Generate qualitative text with the ValidMind library

This notebook shows how to generate qualitative documentation content directly from the ValidMind library using both vm.run_text_generation() and vm.generate_documentation_text(). Instead of switching to the UI to write text manually or trigger generation one section at a time, you can generate content for documentation text blocks programmatically from within a notebook and log it back to the corresponding sections of the model document.

After building an example model and documenting its quantitative results, we’ll show how to generate text for individual content blocks, customize the output with prompts, control the context used for generation, and use a configuration-driven workflow to populate multiple qualitative sections across the document. By the end, you’ll have an end-to-end example of how quantitative test results and AI-generated qualitative content can work together to populate a full model document from Python, giving you a more automated documentation workflow directly in the library.

About ValidMind

ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models.

You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.

Before you begin

This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language.

If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.

New to ValidMind?

If you haven't already seen our documentation on the ValidMind Library, we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.

For access to all features available in this notebook, you'll need access to a ValidMind account.

Register with ValidMind

Key concepts

Validation report: A comprehensive and structured assessment of a model’s development and performance, focusing on verifying its integrity, appropriateness, and alignment with its intended use. It includes analyses of model assumptions, data quality, performance metrics, outcomes of testing procedures, and risk considerations. The validation report supports transparency, regulatory compliance, and informed decision-making by documenting the validator’s independent review and conclusions.

Validation report template: Serves as a standardized framework for conducting and documenting model validation activities. It outlines the required sections, recommended analyses, and expected validation tests, ensuring consistency and completeness across validation reports. The template helps guide validators through a systematic review process while promoting comparability and traceability of validation outcomes.

Tests: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets.

Metrics: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.

Custom metrics: Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.

Inputs: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:

model: A single model that has been initialized in ValidMind with vm.init_model().
dataset: Single dataset that has been initialized in ValidMind with vm.init_dataset().
models: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom metric.
datasets: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom metric. (Learn more: Run tests with multiple datasets)

Parameters: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a metric, customize its behavior, or provide additional context.

Outputs: Custom metrics can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.

Setting up

Install the ValidMind Library

Recommended Python versions

Python 3.8 <= x <= 3.14

To install the library:

%pip install -q validmind

Initialize the ValidMind Library

Register sample model

Let's first register a sample model for use with this notebook:

In a browser, log in to ValidMind.
In the left sidebar, navigate to Inventory and click + Register Model.
Enter the model details and click Next > to continue to assignment of model stakeholders. (Need more help?)
Select your own name under the MODEL OWNER drop-down.
Click Register Model to add the model to your inventory.

Apply documentation template

Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.

In the left sidebar that appears for your model, click Documents and select Development.
Under TEMPLATE, select Binary classification.
Click Use Template to apply the template.

Get your code snippet

Initialize the ValidMind Library with the code snippet unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.

On the left sidebar that appears for your model, select Getting Started and select Development from the DOCUMENT drop-down menu.
Click Copy snippet to clipboard.
Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    api_host="http://localhost:5000/api/v1/tracking",
    api_key="..",
    api_secret="..",
    document="documentation", # requires library    >=2.12.0
    model="..",
)

Initialize the Python environment

Then, let's import the necessary libraries and set up your Python environment for data analysis:

Import Extreme Gradient Boosting (XGBoost) with an alias so that we can reference its functions in later calls. XGBoost is a powerful machine learning library designed for speed and performance, especially in handling structured or tabular data.
Enable matplotlib, a plotting library used for visualizing data. Ensures that any plots you generate will render inline in our notebook output rather than opening in a separate window.

%matplotlib inline

import xgboost as xgb

Getting to know ValidMind

Preview the documentation template

Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate template is selected for your model.

You will upload documentation and test results unique to your model based on this template later on. For now, take a look at the default structure that the template provides with the vm.preview_template() function from the ValidMind library and note the empty sections:

vm.preview_template()

View model documentation in the ValidMind Platform

Next, let's head to the ValidMind Platform to see the template in action:

In a browser, log in to ValidMind.
In the left sidebar, navigate to Inventory and select the model you registered for this notebook.
Click Development under Documents for your model and note how the structure of the documentation matches our preview above.

Build the example model

Import the sample dataset

First, let's import the public Bank Customer Churn Prediction dataset from Kaggle so that we have something to work with.

In our below example, note that:

The target column, Exited has a value of 1 when a customer has churned and 0 otherwise.
The ValidMind Library provides a wrapper to automatically load the dataset as a Pandas DataFrame object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns.

from validmind.datasets.classification import customer_churn

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{customer_churn.target_column}' \n\t• Class labels: {customer_churn.class_labels}"
)

raw_df = customer_churn.load_data()
raw_df.head()

Preprocessing the raw dataset

In this section, we preprocess the raw dataset so it is ready for model training and validation. This includes splitting the data into training, validation, and test subsets to support both model fitting and evaluation on unseen data, and then separating each subset into input features and target labels so the model can learn from customer attributes and predict whether a customer churned.

train_df, validation_df, test_df = customer_churn.preprocess(raw_df)

x_train = train_df.drop(customer_churn.target_column, axis=1)
y_train = train_df[customer_churn.target_column]
x_val = validation_df.drop(customer_churn.target_column, axis=1)
y_val = validation_df[customer_churn.target_column]

Training an XGBoost classifier model

In this section, we train an XGBoost classifier to predict customer churn, using early stopping to halt training if performance does not improve after 10 rounds and reduce unnecessary fitting. We configure the model to evaluate performance with three complementary metrics: error for incorrect predictions, logloss for prediction confidence, and auc for class separation. The model is trained on the training split and evaluated against the validation split during fitting, while verbose=False keeps the training output concise.

model = xgb.XGBClassifier(early_stopping_rounds=10)

model.set_params(
    eval_metric=["error", "logloss", "auc"],
)

model.fit(
    x_train,
    y_train,
    eval_set=[(x_val, y_val)],
    verbose=False,
)

Initialize the ValidMind inputs

We begin by registering the datasets and trained model as ValidMind inputs so they can be referenced consistently throughout the documentation workflow. For the datasets, this means creating ValidMind Dataset objects for the raw, training, and testing data, each with a unique input_id for traceability. Where needed, we also provide supporting metadata such as the target column and class labels so tests can interpret the data correctly.

# Initialize the raw dataset
vm_raw_dataset = vm.init_dataset(
    dataset=raw_df,
    input_id="raw_dataset",
    target_column=customer_churn.target_column,
    class_labels=customer_churn.class_labels,
)

# Initialize the training dataset
vm_train_ds = vm.init_dataset(
    dataset=train_df,
    input_id="train_dataset",
    target_column=customer_churn.target_column,
)

# Initialize the testing dataset
vm_test_ds = vm.init_dataset(
    dataset=test_df,
    input_id="test_dataset",
    target_column=customer_churn.target_column
)

Next, we initialize a ValidMind model object with vm.init_model(). This creates a standardized representation of the trained model that can be passed into ValidMind tests and other library functions, making it possible to evaluate the model and connect its results to the documentation.

# Initialize the model
vm_model = vm.init_model(
    model,
    input_id="model",
)

Finally, we assign predictions from the trained model to the training and testing datasets. The assign_predictions() method links predicted classes and probabilities to each dataset, and can also compute predictions automatically if they are not passed explicitly. This step is what allows ValidMind to run performance and diagnostic tests using the model outputs.

vm_train_ds.assign_predictions(
    model=vm_model,
)
vm_test_ds.assign_predictions(
    model=vm_model,
)

Document test results

In this section, we run the documentation tests defined by the applied template to populate the quantitative parts of the model documentation. The vm.run_documentation_tests() function discovers each test-driven block in the template, executes the corresponding tests, and uploads the resulting artifacts to the ValidMind Platform.

To run the full suite successfully, ValidMind needs to know which model and dataset inputs should be used for each test. This can be done with a shared inputs argument when all tests use the same objects, or with a config dictionary when individual tests require specific inputs or parameters. In this example, we use the default test parameters and provide the input configuration needed for the demo model.

from validmind.utils import preview_test_config

test_config = customer_churn.get_demo_test_config()
preview_test_config(test_config)

Once the configuration is prepared, we pass it to vm.run_documentation_tests() and execute the full suite. The returned full_suite object contains the test results and represents the quantitative documentation that has been generated for the model.

full_suite = vm.run_documentation_tests(config=test_config)

Document qualitative sections

In addition to documenting quantitative results through tests, ValidMind now supports programmatic generation of qualitative content for the text blocks in a model documentation template through vm.run_text_generation(). This function allows you to generate AI-assisted text for a specific content block directly from a notebook and then log it back to the corresponding section of the document. As a result, you can populate qualitative sections without switching to the UI to write text manually or trigger generation one section at a time.

In the next sections, we’ll walk through the main ways to use this functionality. We’ll start by generating text for a single content block with the default behavior, then show how to customize the output with a prompt, how to control the context used for generation by selecting specific sections, and finally how to scale the same pattern across all text blocks in the document.

Generate text for a single content block

First, we’ll use vm.run_text_generation() to generate qualitative text for a single documentation block. By providing a content_id, you can target the exact text placeholder you want to populate and let ValidMind generate content using the current document context. The helper vm.get_content_ids() is useful for inspecting which content blocks are available in the active template, making it easier to identify the IDs you can use when generating and logging text programmatically.

vm.get_content_ids()

vm.run_text_generation(
    content_id="dataset_summary_text",
).log()

Customize the prompt

Next, we’ll customize the generated output by passing a prompt to vm.run_text_generation(). This makes it possible to guide not just the subject of the generated text, but also its structure, tone, level of detail, and presentation format. In practice, this allows you to tailor the output for different documentation needs, such as producing a short narrative summary, a more structured section, or content written for a specific audience, while still relying on the same underlying document context for generation.

prompt = """
Use exactly this structure:

<h3>Dataset Overview</h3>
<p>Explain in 1-2 sentences what the dataset contains and what it is used for.</p>

<h3>Dataset Summary</h3>
<p>Summarize the dataset structure, target outcome, and the main types of input features in 2-3 sentences.</p>

<h3>Key Characteristics</h3>
<ul>
  <li>Include 2-3 concise points about the most important characteristics of the dataset.</li>
</ul>

<h3>Data Quality and Considerations</h3>
<ul>
  <li>Include 2-3 concise points about important quality observations, limitations, or considerations relevant to the dataset.</li>
</ul>

<h3>Overall Assessment</h3>
<p>End with a short balanced conclusion on the dataset's suitability for model development and evaluation.</p>
"""

vm.run_text_generation(
    content_id="dataset_summary_text",
    prompt=prompt,
).log()

Pass section-specific context

Then, we’ll control the context used for generation by passing a selected set of content IDs to vm.run_text_generation(). Rather than relying on the full document, this lets you focus the model on the most relevant parts of the documentation for a given text block. In practice, that means you can generate more targeted qualitative content by choosing which existing test and text blocks should inform the output.

vm.get_content_ids("data_description")

vm.run_text_generation(
    content_id="dataset_summary_text",
    context={"content_ids": vm.get_content_ids("data_description")},
).log()

Append a new text block to a section

Sometimes you may want to generate text for a content_id that is not already defined in the template. In that case, you can still generate the text with vm.run_text_generation() and then use .log(section_id=...) to tell ValidMind where that new text block should be placed in the document.

vm.run_text_generation(
    content_id="intended_use",
    section_id="intended_use",
).log()

Generate text across the document

At this stage, instead of generating one block at a time, we can populate multiple qualitative sections in a single pass.

The vm.generate_documentation_text function reads a configuration dictionary, generates content for each target block, logs the generated text to the ValidMind Platform, and returns a notebook summary grouped by section.

The function uses a config argument to describe which text blocks to generate and how each one should be handled.
The config parameter is a dictionary with the following structure:
```
config = {
    "<content-id>": {
        "section_id": "<section-id>",
        "prompt": "Optional custom prompt",
        "context": {
            "content_ids": ["<content-id-1>", "<content-id-2>"]
        }
    },
    ...
}
```
Each <content-id> represents a documentation text block to populate. Use section_id when the block should be inserted into a specific section, prompt when you want to shape the output more explicitly, and context.content_ids when you want the generation step to focus on selected parts of the document. In this notebook, text_config comes from customer_churn.get_demo_text_config(), which provides the demo setup for the customer churn example.

text_config = customer_churn.get_demo_text_config()
preview_test_config(text_config)

results = vm.generate_documentation_text(config=text_config)

In summary

In this notebook, you learned how to:

Build and document an example customer churn model with ValidMind
Run documentation tests to populate the quantitative sections of a model document
Generate qualitative text for a single documentation content block with vm.run_text_generation()
Customize generated output by passing a prompt
Control generation context by selecting specific sections of the document
Use a configuration-driven workflow to generate qualitative content across the document with vm.generate_documentation_text()

Next steps

You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation.

Work with your model documentation

From the Inventory in the ValidMind Platform, go to the model you registered earlier. (Need more help?)
In the left sidebar that appears for your model, click Development under Documents.

What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. Learn more ...

Discover more learning resources

For a more in-depth introduction to using the ValidMind Library for development, check out our introductory development series and the accompanying interactive training:

ValidMind for model development
Developer Fundamentals

We also offer many interactive notebooks to help you document models:

Or, visit our documentation to learn more about ValidMind.

Upgrade ValidMind

After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.

Retrieve the information for the currently installed version of ValidMind:

%pip show validmind

If the version returned is lower than the version indicated in our production open-source code, restart your notebook and run:

%pip install --upgrade validmind

You may need to restart your kernel after running the upgrade package for changes to be applied.