ValidMind for validation 2 — Start the validation process

Learn how to use ValidMind for your end-to-end validation process with our series of four introductory notebooks. In this second notebook, independently verify the data quality tests performed on the dataset used to train the champion.

You'll learn how to run relevant validation tests with ValidMind, log the results of those tests to the ValidMind Platform, and insert your logged test results as evidence into your validation report. You'll become familiar with the tests available in ValidMind, as well as how to run them. Running tests during validation is crucial to the effective challenge process, as we want to independently evaluate the evidence and assessments provided by the development team.

While running our tests in this notebook, we'll focus on:

• Ensuring that data used for training and testing the champion is of appropriate data quality
• Ensuring that the raw data has been preprocessed appropriately and that the resulting final datasets reflects this

For a full list of out-of-the-box tests and descriptions, use the interactive ValidMind test sandbox.

Learn by doing

Our course tailor-made for validators new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Validator Fundamentals

Prerequisites

In order to independently assess the quality of your datasets with notebook, you'll need to first have:

• Registered a model within the ValidMind Platform and granted yourself access to the model as a validator
• Installed the ValidMind Library in your local environment, allowing you to access all its features

Need help with the above steps?

Refer to the first notebook in this series: 1 — Set up the ValidMind Library for validation

Setting up

Initialize the ValidMind Library

First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Validation from the Document drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="validation-report",
)

Note: you may need to restart the kernel to use updated packages.

2026-09-11 23:51:04,041 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model validation (ID: cmalguc9y02ok199q2db381ib)
📁 Document Type: validation_report

Load the sample dataset

Let's first import the public Bank Customer Churn Prediction dataset from Kaggle, which was used to develop the dummy champion.

We'll use this dataset to review steps that should have been conducted during the initial development and documentation of the champion to ensure that the model was built correctly. By independently performing steps taken by the development team, we can confirm whether the model was built using appropriate and properly processed data.

In our below example, note that:

• The target column, Exited has a value of 1 when a customer has churned and 0 otherwise.
• The ValidMind Library provides a wrapper to automatically load the dataset as a Pandas DataFrame object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns.

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()
raw_df.head()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

Verifying data quality adjustments

Let's say that thanks to the documentation submitted by the development team (Learn more: ValidMind for development), we know that the sample dataset was first modified before being used to train the champion. After performing some data quality assessments on the raw dataset, it was determined that the dataset required rebalancing, and highly correlated features were also removed.

Identify qualitative tests

During validation, we use the same data processing logic and training procedure to confirm that the model's results can be reproduced independently, so let's start by doing some data quality assessments by running a few individual tests just like the development team did.

Use the vm.tests.list_tests() function introduced by the first notebook in this series in combination with vm.tests.list_tags() and vm.tests.list_tasks() to find which prebuilt tests are relevant for data quality assessment:

• tasks represent the kind of modeling task associated with a test. Here we'll focus on classification tasks.
• tags are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the data_quality tag.

# Get the list of available task types
sorted(vm.tests.list_tasks())

['classification',
 'clustering',
 'data_validation',
 'feature_extraction',
 'monitoring',
 'nlp',
 'regression',
 'residual_analysis',
 'text_classification',
 'text_generation',
 'text_qa',
 'text_summarization',
 'time_series_forecasting',
 'visualization']

# Get the list of available tags
sorted(vm.tests.list_tags())

['AUC',
 'analysis',
 'anomaly',
 'anomaly_detection',
 'bias_and_fairness',
 'binary_classification',
 'calibration',
 'categorical_data',
 'classification',
 'classification_metrics',
 'clustering',
 'correlation',
 'credit_risk',
 'data_analysis',
 'data_distribution',
 'data_quality',
 'data_validation',
 'descriptive_statistics',
 'dimensionality_reduction',
 'distribution',
 'embeddings',
 'feature_importance',
 'feature_selection',
 'few_shot',
 'forecasting',
 'frequency_analysis',
 'kmeans',
 'linear_regression',
 'llm',
 'logistic_regression',
 'metadata',
 'model_comparison',
 'model_diagnosis',
 'model_explainability',
 'model_interpretation',
 'model_performance',
 'model_predictions',
 'model_selection',
 'model_training',
 'model_validation',
 'multiclass_classification',
 'nlp',
 'normality',
 'numerical_data',
 'outlier',
 'outliers',
 'qualitative',
 'rag_performance',
 'ragas',
 'regression',
 'retrieval_performance',
 'scorecard',
 'seasonality',
 'senstivity_analysis',
 'sklearn',
 'stationarity',
 'statistical_test',
 'statistics',
 'statsmodels',
 'tabular_data',
 'text_data',
 'threshold_optimization',
 'time_series_data',
 'unit_root_test',
 'visualization',
 'zero_shot']

You can pass tags and tasks as parameters to the vm.tests.list_tests() function to filter the tests based on the tags and task types.

For example, to find tests related to tabular data quality for classification models, you can call list_tests() like this:

vm.tests.list_tests(task="classification", tags=["tabular_data", "data_quality"])

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.data_validation.ClassImbalance	Class Imbalance	Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model....	True	True	['dataset']	{'min_percent_threshold': {'type': 'int', 'default': 10}}	['tabular_data', 'binary_classification', 'multiclass_classification', 'data_quality']	['classification']
validmind.data_validation.DescriptiveStatistics	Descriptive Statistics	Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's...	False	True	['dataset']	{}	['tabular_data', 'time_series_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.Duplicates	Duplicates	Tests dataset for duplicate entries, ensuring model reliability via data quality verification....	False	True	['dataset']	{'min_threshold': {'type': '_empty', 'default': 1}}	['tabular_data', 'data_quality', 'text_data']	['classification', 'regression']
validmind.data_validation.HighCardinality	High Cardinality	Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting....	False	True	['dataset']	{'num_threshold': {'type': 'int', 'default': 100}, 'percent_threshold': {'type': 'float', 'default': 0.1}, 'threshold_type': {'type': 'str', 'default': 'percent'}}	['tabular_data', 'data_quality', 'categorical_data']	['classification', 'regression']
validmind.data_validation.HighPearsonCorrelation	High Pearson Correlation	Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity....	False	True	['dataset']	{'max_threshold': {'type': 'float', 'default': 0.3}, 'top_n_correlations': {'type': 'int', 'default': 10}, 'feature_columns': {'type': 'list', 'default': None}}	['tabular_data', 'data_quality', 'correlation']	['classification', 'regression']
validmind.data_validation.MissingValues	Missing Values	Evaluates dataset quality by ensuring missing value percentage across all features does not exceed a set threshold....	False	True	['dataset']	{'min_percentage_threshold': {'type': 'float', 'default': 1.0}}	['tabular_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.MissingValuesBarPlot	Missing Values Bar Plot	Assesses the percentage and distribution of missing values in the dataset via a bar plot, with emphasis on...	True	False	['dataset']	{'threshold': {'type': 'int', 'default': 80}, 'fig_height': {'type': 'int', 'default': 600}}	['tabular_data', 'data_quality', 'visualization']	['classification', 'regression']
validmind.data_validation.Skewness	Skewness	Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data...	False	True	['dataset']	{'max_threshold': {'type': '_empty', 'default': 1}}	['data_quality', 'tabular_data']	['classification', 'regression']
validmind.plots.BoxPlot	Box Plot	Generates customizable box plots for numerical features in a dataset with optional grouping using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'group_by': {'type': 'Optional', 'default': None}, 'width': {'type': 'int', 'default': 1800}, 'height': {'type': 'int', 'default': 1200}, 'colors': {'type': 'Optional', 'default': None}, 'show_outliers': {'type': 'bool', 'default': True}, 'title_prefix': {'type': 'str', 'default': 'Box Plot of'}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.plots.HistogramPlot	Histogram Plot	Generates customizable histogram plots for numerical features in a dataset using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'bins': {'type': 'Union', 'default': 30}, 'color': {'type': 'str', 'default': 'steelblue'}, 'opacity': {'type': 'float', 'default': 0.7}, 'show_kde': {'type': 'bool', 'default': True}, 'normalize': {'type': 'bool', 'default': False}, 'log_scale': {'type': 'bool', 'default': False}, 'title_prefix': {'type': 'str', 'default': 'Histogram of'}, 'width': {'type': 'int', 'default': 1200}, 'height': {'type': 'int', 'default': 800}, 'n_cols': {'type': 'int', 'default': 2}, 'vertical_spacing': {'type': 'float', 'default': 0.15}, 'horizontal_spacing': {'type': 'float', 'default': 0.1}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.stats.DescriptiveStats	Descriptive Stats	Provides comprehensive descriptive statistics for numerical features in a dataset....	False	True	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'include_advanced': {'type': 'bool', 'default': True}, 'confidence_level': {'type': 'float', 'default': 0.95}}	['tabular_data', 'statistics', 'data_quality']	['classification', 'regression', 'clustering']

Want to learn more about navigating ValidMind tests?

Refer to our notebook outlining the utilities available for viewing and understanding available ValidMind tests: Explore tests

Initialize the ValidMind dataset

With the individual tests we want to run identified, the next step is to connect your data with a ValidMind Dataset object. This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind, but you only need to do it once per dataset.

Initialize a ValidMind dataset object using the init_dataset function from the ValidMind (vm) module. For this example, we'll pass in the following arguments:

• dataset — The raw dataset that you want to provide as input to tests.
• input_id — A unique identifier that allows tracking what inputs are used when running each individual test.
• target_column — A required argument if tests require access to true values. This is the name of the target column in the dataset.

# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test
vm_raw_dataset = vm.init_dataset(
    dataset=raw_df,
    input_id="raw_dataset",
    target_column="Exited",
)

Run data quality tests

Now that we know how to initialize a ValidMind dataset object, we're ready to run some tests!

You run individual tests by calling the run_test function provided by the validmind.tests module. For the examples below, we'll pass in the following arguments:

• test_id — The ID of the test to run, as seen in the ID column when you run list_tests.
• params — A dictionary of parameters for the test. These will override any default_params set in the test definition.

Run tabular data tests

The inputs expected by a test can also be found in the test definition — let's take validmind.data_validation.DescriptiveStatistics as an example.

Note that the output of the describe_test() function below shows that this test expects a dataset as input:

vm.tests.describe_test("validmind.data_validation.DescriptiveStatistics")

▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Descriptive Statistics

Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's dataset.

Purpose

The purpose of the Descriptive Statistics metric is to provide a comprehensive summary of both numerical and categorical data within a dataset. This involves statistics such as count, mean, standard deviation, minimum and maximum values for numerical data. For categorical data, it calculates the count, number of unique values, most common value and its frequency, and the proportion of the most frequent value relative to the total. The goal is to visualize the overall distribution of the variables in the dataset, aiding in understanding the model's behavior and predicting its performance.

Test Mechanism

The testing mechanism utilizes two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function pulls out several summary statistics, while value_counts() accounts for unique values. The resulting data is formatted into two distinct tables, one for numerical and another for categorical variable summaries. These tables provide a clear summary of the main characteristics of the variables, which can be instrumental in assessing the model's performance.

Signs of High Risk

• Skewed data or significant outliers can represent high risk. For numerical data, this may be reflected via a significant difference between the mean and median (50% percentile).
• For categorical data, a lack of diversity (low count of unique values), or overdominance of a single category (high frequency of the top value) can indicate high risk.

Strengths

• Provides a comprehensive summary of the dataset, shedding light on the distribution and characteristics of the variables under consideration.
• It is a versatile and robust method, applicable to both numerical and categorical data.
• Helps highlight crucial anomalies such as outliers, extreme skewness, or lack of diversity, which are vital in understanding model behavior during testing and validation.

Limitations

• While this metric offers a high-level overview of the data, it may fail to detect subtle correlations or complex patterns.
• Does not offer any insights on the relationship between variables.
• Alone, descriptive statistics cannot be used to infer properties about future unseen data.
• Should be used in conjunction with other statistical tests to provide a comprehensive understanding of the model's data.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DescriptiveStatistics", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Now, let's run a few tests to assess the quality of the dataset:

result2 = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_raw_dataset},
    params={"min_percent_threshold": 30},
)

❌ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring the percentage of records in each class against a minimum threshold. For the Exited target, the results table and bar chart show two classes with materially different frequencies. Class 0 represents 79.80% of rows and is marked as Pass, while class 1 represents 20.20% of rows and is marked as Fail under the configured 30% minimum percentage threshold.

Key insights:

• Majority class dominates target distribution: Class 0 accounts for 79.80% of observations, making it the substantially more prevalent outcome in the Exited target.
• Minority class falls below threshold: Class 1 represents 20.20% of rows, which is below the configured min_percent_threshold of 30%, and is therefore marked as Fail.
• Pass/fail outcome differs by class: The test assigns a Pass to class 0 and a Fail to class 1, reflecting that only one of the two target classes meets the configured minimum representation level.

The results show a two-class target distribution with a pronounced concentration in class 0 and lower representation in class 1. Under the 30% threshold applied in this test, the majority class satisfies the minimum representation criterion while the minority class does not. Collectively, the table and chart document an imbalanced class distribution in the Exited target under the configured test setting.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	79.80%	Pass
1	20.20%	Fail

Figures

The output above shows that the validmind.data_validation.ClassImbalance test did not pass according to the value we set for min_percent_threshold — great, this matches what was reported by the development team.

To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.

As this is technically a different dataset, remember to first initialize a new ValidMind Dataset object to pass in as input as required by run_test():

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

# Pass the initialized `balanced_raw_dataset` as input into the test run
result = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_balanced_raw_dataset},
    params={"min_percent_threshold": 30},
)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring the percentage share of each class and comparing those shares against the configured minimum threshold. In this result, the target variable Exited is shown across two classes, 0 and 1, with each class representing 50.00% of rows. The table also reports a pass/fail outcome for each class using the configured minimum percentage threshold of 30. The accompanying bar chart visually reflects the equal class proportions.

Key insights:

• Equal class distribution: The Exited target is evenly split between class 0 and class 1, with each class accounting for 50.00% of the dataset.

• All classes pass threshold: Both target classes are above the configured minimum percentage threshold of 30, and both are marked as Pass.

• No minority class identified: The reported class frequencies are identical across the two target outcomes, with no class appearing at a lower share than the other.

The results show a balanced observed distribution for the Exited target, with both classes present at equal frequency and both exceeding the configured minimum percentage threshold. The table and chart present consistent evidence that no class falls below the test criterion in this dataset.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

Remove highly correlated features

Next, let's also remove highly correlated features from our dataset as outlined by the development team. Removing highly correlated features helps make the model simpler, more stable, and easier to understand.

You can utilize the output from a ValidMind test for further use — in this below example, to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.

First, we'll run validmind.data_validation.HighPearsonCorrelation with the balanced_raw_dataset we initialized previously as input as is for comparison with later runs:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships between features to identify potentially redundant variables or concentrations of multicollinearity. The results table lists the top feature pairs by Pearson correlation coefficient together with a Pass/Fail assessment based on the configured absolute-value threshold of 0.3. In this run, the reported coefficients range from -0.1892 to 0.3314 across the ten strongest feature pairs shown. Only one pair exceeds the threshold and is marked as a failure, while the remaining nine pairs are marked as passing.

Key insights:

• Single threshold breach observed: The pair (Age, Exited) records the highest absolute correlation at 0.3314 and is the only relationship in the reported output that exceeds the 0.3 threshold, resulting in a Fail classification.
• Remaining correlations are modest: All other reported pairwise coefficients have absolute values below 0.19, including (IsActiveMember, Exited) at -0.1892, (Balance, NumOfProducts) at -0.1825, and (Balance, Exited) at 0.1609.
• Top relationships are concentrated around Exited: Among the ten strongest reported pairs, Exited appears in five relationships: with Age (0.3314), IsActiveMember (-0.1892), Balance (0.1609), NumOfProducts (-0.0518), and Tenure (-0.0329).
• Observed signs are mixed but mostly weak: The reported set includes both positive and negative coefficients, with several values close to zero, such as (Balance, HasCrCard) at -0.0387, (Tenure, IsActiveMember) at -0.0361, (Age, Balance) at 0.0330, and (Tenure, Exited) at -0.0329.

The reported correlation structure is characterized by one pair above the configured threshold and a broad remainder of comparatively small-magnitude linear relationships. The most pronounced observed relationship is between Age and Exited, while the other reported pairs remain well below the threshold. Overall, the output indicates a largely low-correlation set among the top reported feature pairs, with a single identified exception under the test criterion.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3314	Fail
(IsActiveMember, Exited)	-0.1892	Pass
(Balance, NumOfProducts)	-0.1825	Pass
(Balance, Exited)	0.1609	Pass
(NumOfProducts, IsActiveMember)	0.0596	Pass
(NumOfProducts, Exited)	-0.0518	Pass
(Balance, HasCrCard)	-0.0387	Pass
(Tenure, IsActiveMember)	-0.0361	Pass
(Age, Balance)	0.0330	Pass
(Tenure, Exited)	-0.0329	Pass

The output above shows that the test did not pass according to the value we set for max_threshold — as reported and expected.

corr_result is an object of type TestResult. We can inspect the result object to see what the test has produced:

print(type(corr_result))
print("Result ID: ", corr_result.result_id)
print("Params: ", corr_result.params)
print("Passed: ", corr_result.passed)
print("Tables: ", corr_result.tables)

<class 'validmind.vm_models.result.result.TestResult'>
Result ID:  validmind.data_validation.HighPearsonCorrelation
Params:  {'max_threshold': 0.3}
Passed:  False
Tables:  [ResultTable]

Let's remove the highly correlated features and create a new VM dataset object.

We'll begin by checking out the table in the result and extracting a list of features that failed the test:

# Extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3314	Fail
1	(IsActiveMember, Exited)	-0.1892	Pass
2	(Balance, NumOfProducts)	-0.1825	Pass
3	(Balance, Exited)	0.1609	Pass
4	(NumOfProducts, IsActiveMember)	0.0596	Pass
5	(NumOfProducts, Exited)	-0.0518	Pass
6	(Balance, HasCrCard)	-0.0387	Pass
7	(Tenure, IsActiveMember)	-0.0361	Pass
8	(Age, Balance)	0.0330	Pass
9	(Tenure, Exited)	-0.0329	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

Next, extract the feature names from the list of strings (example: (Age, Exited) > Age):

high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

Now, it's time to re-initialize the dataset with the highly correlated features removed.

Note the use of a different input_id. This allows tracking the inputs used when running each individual test.

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

Re-running the test with the reduced feature set should pass the test:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant variables or signs of multicollinearity. The results table lists the top feature pairs by absolute Pearson correlation coefficient together with a pass/fail assessment based on the configured threshold of 0.3. In this run, the reported coefficients range from -0.1892 to 0.1609 across the ten strongest observed relationships, and each reported pair is marked as Pass.

Key insights:

• No pair exceeds threshold: All reported absolute correlation coefficients are below the 0.3 threshold. Every listed feature pair therefore received a Pass result in this test run.
• Observed relationships are weak: The largest absolute coefficient is 0.1892 for (IsActiveMember, Exited), followed by 0.1825 for (Balance, NumOfProducts) and 0.1609 for (Balance, Exited). These values indicate limited linear association among the strongest reported pairs.
• Top correlations include both signs: The reported correlations include both negative and positive values. Negative relationships are shown for pairs such as (IsActiveMember, Exited) and (Balance, NumOfProducts), while positive relationships are shown for (Balance, Exited) and (Balance, EstimatedSalary).
• Lower-ranked pairs are near zero: Several of the listed feature pairs have coefficients close to zero, including (HasCrCard, IsActiveMember) at -0.0303 and (Balance, EstimatedSalary) at 0.03. These values indicate minimal linear relationship for those pairs within the reported top ten.

The reported correlation structure does not show any feature pair breaching the configured Pearson threshold of 0.3. The strongest observed associations remain weak in magnitude, and the remaining reported pairs are closer to zero. Collectively, the test output indicates limited linear dependence among the top-ranked feature pairs included in the result set.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1892	Pass
(Balance, NumOfProducts)	-0.1825	Pass
(Balance, Exited)	0.1609	Pass
(NumOfProducts, IsActiveMember)	0.0596	Pass
(NumOfProducts, Exited)	-0.0518	Pass
(Balance, HasCrCard)	-0.0387	Pass
(Tenure, IsActiveMember)	-0.0361	Pass
(Tenure, Exited)	-0.0329	Pass
(HasCrCard, IsActiveMember)	-0.0303	Pass
(Balance, EstimatedSalary)	0.0300	Pass

You can also plot the correlation matrix to visualize the new correlation between features:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.PearsonCorrelationMatrix",
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

Pearson Correlation Matrix

The PearsonCorrelationMatrix test evaluates linear dependency among numerical variables by calculating pairwise Pearson correlation coefficients and displaying them in a heat map. The matrix shown includes CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with coefficients ranging from -0.19 to 0.16 outside the diagonal. The diagonal entries are 1.0 by construction, while off-diagonal values are concentrated close to zero, indicating weak pairwise linear relationships across the variables included in the plot.

Key insights:

• No high pairwise correlations: All off-diagonal correlations remain well below the 0.7 threshold referenced in the test methodology. The largest absolute correlations observed are -0.19, -0.18, and 0.16.
• Exited shows weak associations: Exited has weak negative correlations with IsActiveMember (-0.19), NumOfProducts (-0.05), HasCrCard (-0.03), Tenure (-0.03), CreditScore (-0.02), and EstimatedSalary (-0.02), and a weak positive correlation with Balance (0.16).
• Balance and NumOfProducts are mildly inversely related: The strongest non-target feature relationship is between Balance and NumOfProducts at -0.18, indicating a limited negative linear association.
• EstimatedSalary is largely uncorrelated: EstimatedSalary shows near-zero correlations with the other variables, including -0.02 with CreditScore, 0.02 with Tenure, 0.03 with Balance, 0.01 with NumOfProducts, -0.01 with HasCrCard, approximately 0.00 with IsActiveMember, and -0.02 with Exited.

The correlation matrix indicates a low overall level of linear dependence among the numerical variables included in the test. No variable pair exhibits a strong positive or negative linear relationship based on the reported coefficients, and the observed associations are limited to small-magnitude effects. The most pronounced relationships in the matrix are still weak, with the largest absolute values appearing between Exited and IsActiveMember, Balance and NumOfProducts, and Exited and Balance.

Figures

Documenting test results

Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it. Every test result returned by the run_test() function has a .log() method that can be used to send the test results to the ValidMind Platform.

When logging validation test results to the platform, you'll need to manually add those results to the desired section of the validation report. To demonstrate how to add test results to your validation report, we'll log our data quality tests and insert the results via the ValidMind Platform.

Configure and run comparison tests

Below, we'll perform comparison tests between the original raw dataset (raw_dataset) and the final preprocessed (raw_dataset_preprocessed) dataset, again logging the results to the ValidMind Platform.

We can specify all the tests we'd ike to run in a dictionary called test_config, and we'll pass in the following arguments for each test:

• params: Individual test parameters.
• input_grid: Individual test inputs to compare. In this case, we'll input our two datasets for comparison.

Note here that the input_grid expects the input_id of the dataset as the value rather than the variable name we specified:

# Individual test config with inputs specified
test_config = {
    "validmind.data_validation.ClassImbalance": {
        "input_grid": {"dataset": ["raw_dataset", "raw_dataset_preprocessed"]},
        "params": {"min_percent_threshold": 30}
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "input_grid": {"dataset": ["raw_dataset", "raw_dataset_preprocessed"]},
        "params": {"max_threshold": 0.3}
    },
}

Then batch run and log our tests in test_config:

for t in test_config:
    print(t)
    try:
        # Check if test has input_grid
        if 'input_grid' in test_config[t]:
            # For tests with input_grid, pass the input_grid configuration
            if 'params' in test_config[t]:
                vm.tests.run_test(t, input_grid=test_config[t]['input_grid'], params=test_config[t]['params']).log()
            else:
                vm.tests.run_test(t, input_grid=test_config[t]['input_grid']).log()
        else:
            # Original logic for regular inputs
            if 'params' in test_config[t]:
                vm.tests.run_test(t, inputs=test_config[t]['inputs'], params=test_config[t]['params']).log()
            else:
                vm.tests.run_test(t, inputs=test_config[t]['inputs']).log()
    except Exception as e:
        print(f"Error running test {t}: {str(e)}")

validmind.data_validation.ClassImbalance

❌ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by comparing each class share against the configured minimum percentage threshold of 30%. Results are reported for both the raw dataset and the preprocessed dataset, with class-level percentages and pass/fail outcomes shown for target values Exited = 0 and Exited = 1. In the raw dataset, the two classes account for 79.80% and 20.20% of rows respectively, while in the preprocessed dataset both classes account for 50.00% of rows.

Key insights:

• Raw dataset is skewed: In raw_dataset, Exited = 0 represents 79.80% of observations and passes the 30% threshold, while Exited = 1 represents 20.20% and fails the threshold.

• Preprocessed dataset is balanced: In raw_dataset_preprocessed, both Exited = 0 and Exited = 1 each represent 50.00% of rows, and both classes pass the 30% threshold.

• Class proportions changed materially after preprocessing: The class distribution shifts from a 79.80% / 20.20% split in the raw dataset to an even 50.00% / 50.00% split in the preprocessed dataset.

The test results show different class distributions across the two dataset versions. The raw dataset contains an under-threshold minority class for Exited = 1, whereas the preprocessed dataset shows equal representation of both target classes and full threshold compliance. Collectively, the results document a transition from an imbalanced original class distribution to a balanced preprocessed distribution.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

dataset	Exited	Percentage of Rows (%)	Pass/Fail
raw_dataset	0	79.80%	Pass
raw_dataset	1	20.20%	Fail
raw_dataset_preprocessed	0	50.00%	Pass
raw_dataset_preprocessed	1	50.00%	Pass

Figures

2026-09-11 23:51:50,490 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.ClassImbalance does not exist in model's document

validmind.data_validation.HighPearsonCorrelation

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify highly correlated variable pairs that may indicate redundancy or multicollinearity. The results are reported for both raw_dataset and raw_dataset_preprocessed, listing the strongest observed Pearson correlation coefficients together with their pass/fail status against the configured threshold of 0.3. In raw_dataset, coefficients range from -0.3045 to 0.281 across the reported pairs, while in raw_dataset_preprocessed, coefficients range from -0.1892 to 0.1609. The output shows one failing pair in the raw dataset and no failing pairs in the preprocessed dataset.

Key insights:

• One threshold breach in raw data: The pair (Balance, NumOfProducts) in raw_dataset has a Pearson correlation of -0.3045, which exceeds the absolute threshold of 0.3 and is marked as Fail.
• No threshold breaches after preprocessing: All reported feature pairs in raw_dataset_preprocessed are marked Pass, with the largest absolute coefficient equal to 0.1892 for (IsActiveMember, Exited).
• Strongest raw correlation weakens after preprocessing: (Balance, NumOfProducts) declines in magnitude from -0.3045 in raw_dataset to -0.1825 in raw_dataset_preprocessed, moving from Fail to Pass.
• Observed relationships are generally weak to modest: Aside from the single failing pair in raw_dataset, the remaining reported coefficients are relatively small in magnitude, including 0.281 for (Age, Exited) and -0.1515 for (IsActiveMember, Exited) in the raw dataset.

The reported correlation structure is limited, with only one pair in the raw dataset exceeding the configured threshold and no pairs exceeding it after preprocessing. The most notable relationship is the negative association between Balance and NumOfProducts, which is reduced below threshold in the preprocessed data. Across the remaining reported feature pairs, coefficient magnitudes are comparatively low, indicating that the strongest linear relationships in the output are concentrated in a small number of variables.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

dataset	Columns	Coefficient	Pass/Fail
raw_dataset	(Balance, NumOfProducts)	-0.3045	Fail
raw_dataset	(Age, Exited)	0.2810	Pass
raw_dataset	(IsActiveMember, Exited)	-0.1515	Pass
raw_dataset	(Balance, Exited)	0.1174	Pass
raw_dataset	(Age, IsActiveMember)	0.0873	Pass
raw_dataset	(NumOfProducts, Exited)	-0.0523	Pass
raw_dataset	(Age, NumOfProducts)	-0.0306	Pass
raw_dataset	(CreditScore, IsActiveMember)	0.0306	Pass
raw_dataset	(Tenure, IsActiveMember)	-0.0293	Pass
raw_dataset	(Age, Balance)	0.0290	Pass
raw_dataset_preprocessed	(IsActiveMember, Exited)	-0.1892	Pass
raw_dataset_preprocessed	(Balance, NumOfProducts)	-0.1825	Pass
raw_dataset_preprocessed	(Balance, Exited)	0.1609	Pass
raw_dataset_preprocessed	(NumOfProducts, IsActiveMember)	0.0596	Pass
raw_dataset_preprocessed	(NumOfProducts, Exited)	-0.0518	Pass
raw_dataset_preprocessed	(Balance, HasCrCard)	-0.0387	Pass
raw_dataset_preprocessed	(Tenure, IsActiveMember)	-0.0361	Pass
raw_dataset_preprocessed	(Tenure, Exited)	-0.0329	Pass
raw_dataset_preprocessed	(HasCrCard, IsActiveMember)	-0.0303	Pass
raw_dataset_preprocessed	(Balance, EstimatedSalary)	0.0300	Pass

2026-09-11 23:51:59,013 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.HighPearsonCorrelation does not exist in model's document

Note the output returned indicating that a test-driven block doesn't currently exist in your documentation for some test IDs.

That's expected, as when we run validations tests the results logged need to be manually added to your report as part of your compliance assessment process within the ValidMind Platform.

Log tests with unique identifiers

Next, we'll use the previously initialized vm_balanced_raw_dataset (that still has a highly correlated Age column) as input to run an individual test, then log the result to the ValidMind Platform.

When running individual tests, you can use a custom result_id to tag the individual result with a unique identifier:

• This result_id can be appended to test_id with a : separator.
• The balanced_raw_dataset result identifier will correspond to the balanced_raw_dataset input, the dataset that still has the Age column.

result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)
result.log()

❌ High Pearson Correlation Balanced Raw Dataset

The High Pearson Correlation test evaluates pairwise linear relationships among dataset features to identify feature pairs with elevated correlation relative to the configured threshold. The result table reports the top correlations ranked by coefficient magnitude, along with a Pass/Fail status based on the absolute-value threshold of 0.3. In this run, the listed coefficients range from -0.1892 to 0.3314, and one feature pair is marked as failing the threshold test while the remaining reported pairs pass.

Key insights:

• Single threshold breach observed: The pair (Age, Exited) records the largest coefficient at 0.3314 and is the only result classified as Fail, exceeding the configured threshold of 0.3.
• All other reported correlations are below threshold: The remaining nine reported pairs are classified as Pass, with absolute correlation values ranging from 0.0329 to 0.1892.
• Most reported relationships are weak in magnitude: Aside from (Age, Exited), the strongest absolute coefficients are -0.1892 for (IsActiveMember, Exited), -0.1825 for (Balance, NumOfProducts), and 0.1609 for (Balance, Exited), all remaining below the test limit.
• Both positive and negative associations are present: The reported coefficients include positive values such as 0.3314 and 0.1609, as well as negative values such as -0.1892 and -0.1825, indicating mixed linear relationship directions among the top-ranked pairs.

The reported correlation structure is concentrated in generally low-magnitude pairwise relationships, with one exception. (Age, Exited) is the only listed pair exceeding the 0.3 threshold and therefore represents the sole flagged result in this test output. All other top-ranked correlations remain below the configured limit, with relatively small absolute magnitudes across the remaining reported pairs.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3314	Fail
(IsActiveMember, Exited)	-0.1892	Pass
(Balance, NumOfProducts)	-0.1825	Pass
(Balance, Exited)	0.1609	Pass
(NumOfProducts, IsActiveMember)	0.0596	Pass
(NumOfProducts, Exited)	-0.0518	Pass
(Balance, HasCrCard)	-0.0387	Pass
(Tenure, IsActiveMember)	-0.0361	Pass
(Age, Balance)	0.0330	Pass
(Tenure, Exited)	-0.0329	Pass

2026-09-11 23:52:03,384 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset does not exist in model's document

Add test results to reporting

With some test results logged, let's head to the model we connected to at the beginning of this notebook and learn how to insert a test result into our validation report. (Learn more: Assess compliance)

While the example below focuses on a specific test result, you can follow the same general procedure for your other results:

1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

2. In the left sidebar that appears for your model, click Validation under Documents.

3. Click on 2.2.1. Data Quality to expand that section.

4. Under the Class Imbalance Assessment guideline, click Evidence to expand the evidence panel.

5. Click Link Evidence, then select Validator Evidence.

6. Select the Class Imbalance test results we logged: ValidMind Data Validation Class Imbalance

   
   
   

7. Click Update Linked Evidence to add the test results to the validation report.

8. Confirm that the results for the Class Imbalance test you inserted has been correctly inserted into section 2.2.1. Data Quality of the report.

   • Note that these test results are flagged as Requires Attention — as they include comparative results from our initial raw dataset.
   • Click See evidence details to review the LLM-generated description that summarizes the test results, that confirm that our final preprocessed dataset actually passes our test:

   
   
   

Here in this text editor, you can make qualitative edits to the draft that ValidMind generated to finalize the test results.

Learn more: Work with content blocks

Preparing the preprocessed dataset

Split the preprocessed dataset

With our raw dataset rebalanced with highly correlated features removed, let's now spilt our dataset into train and test in preparation for model evaluation testing.

To start, let's grab the first few rows from the balanced_raw_no_age_df dataset we initialized earlier:

balanced_raw_no_age_df.head()

	CreditScore	Geography	Gender	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
5488	591	France	Female	10	0.00	2	0	0	171099.22	0
5795	434	Spain	Male	3	0.00	1	1	1	2739.71	0
1031	756	Germany	Male	2	124439.49	2	0	1	47093.11	0
2742	722	France	Female	2	88011.40	1	1	1	90655.94	1
6149	723	Germany	Female	10	90450.00	2	0	0	103379.31	1

Before training the model, we need to encode the categorical features in the dataset:

• Use the OneHotEncoder class from the sklearn.preprocessing module to encode the categorical features.
• The categorical features in the dataset are Geography and Gender.

balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
5488	591	10	0.00	2	0	0	171099.22	0	False	False	False
5795	434	3	0.00	1	1	1	2739.71	0	False	True	True
1031	756	2	124439.49	2	0	1	47093.11	0	True	False	True
2742	722	2	88011.40	1	1	1	90655.94	1	False	False	False
6149	723	10	90450.00	2	0	0	103379.31	1	True	False	False

Splitting our dataset into training and testing is essential for proper validation testing, as this helps assess how well the model generalizes to unseen data:

• We start by dividing our balanced_raw_no_age_df dataset into training and test subsets using train_test_split, with 80% of the data allocated to training (train_df) and 20% to testing (test_df).
• From each subset, we separate the features (all columns except "Exited") into X_train and X_test, and the target column ("Exited") into y_train and y_test.

from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

Initialize the split datasets

Next, let's initialize the training and testing datasets so they are available for use:

vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

In summary

In this second notebook, you learned how to:

• Import a sample dataset
• Identify which tests you might want to run with ValidMind
• Initialize ValidMind datasets
• Run individual tests
• Utilize the output from tests you’ve run
• Log test results as evidence to the ValidMind Platform
• Insert test results into your validation report

Next steps

Develop potential challenger models

Now that you're familiar with the basics of using the ValidMind Library, let's use it to develop a challenger model: 3 — Developing a potential challenger

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial