ValidMind for validation 2 — Start the validation process

Learn how to use ValidMind for your end-to-end validation process with our series of four introductory notebooks. In this second notebook, independently verify the data quality tests performed on the dataset used to train the champion.

You'll learn how to run relevant validation tests with ValidMind, log the results of those tests to the ValidMind Platform, and insert your logged test results as evidence into your validation report. You'll become familiar with the tests available in ValidMind, as well as how to run them. Running tests during validation is crucial to the effective challenge process, as we want to independently evaluate the evidence and assessments provided by the development team.

While running our tests in this notebook, we'll focus on:

For a full list of out-of-the-box tests and descriptions, use the interactive ValidMind test sandbox.

Learn by doing

Our course tailor-made for validators new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Validator Fundamentals

Prerequisites

In order to independently assess the quality of your datasets with notebook, you'll need to first have:

Need help with the above steps?

Refer to the first notebook in this series: 1 — Set up the ValidMind Library for validation

Setting up

Initialize the ValidMind Library

First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

  1. On the left sidebar that appears for your model, select Getting Started and select Validation from the Document drop-down menu.

  2. Click Copy snippet to clipboard.

  3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="validation-report",
)
Note: you may need to restart the kernel to use updated packages.
2026-10-30 20:58:01,837 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model validation (ID: cmalguc9y02ok199q2db381ib)
📁 Document Type: validation_report

Load the sample dataset

Let's first import the public Bank Customer Churn Prediction dataset from Kaggle, which was used to develop the dummy champion.

We'll use this dataset to review steps that should have been conducted during the initial development and documentation of the champion to ensure that the model was built correctly. By independently performing steps taken by the development team, we can confirm whether the model was built using appropriate and properly processed data.

In our below example, note that:

  • The target column, Exited has a value of 1 when a customer has churned and 0 otherwise.
  • The ValidMind Library provides a wrapper to automatically load the dataset as a Pandas DataFrame object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns.
from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()
raw_df.head()
Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}
CreditScore Geography Gender Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary Exited
0 619 France Female 42 2 0.00 1 1 1 101348.88 1
1 608 Spain Female 41 1 83807.86 1 0 1 112542.58 0
2 502 France Female 42 8 159660.80 3 1 0 113931.57 1
3 699 France Female 39 1 0.00 2 0 0 93826.63 0
4 850 Spain Female 43 2 125510.82 1 1 1 79084.10 0

Verifying data quality adjustments

Let's say that thanks to the documentation submitted by the development team (Learn more: ValidMind for development), we know that the sample dataset was first modified before being used to train the champion. After performing some data quality assessments on the raw dataset, it was determined that the dataset required rebalancing, and highly correlated features were also removed.

Identify qualitative tests

During validation, we use the same data processing logic and training procedure to confirm that the model's results can be reproduced independently, so let's start by doing some data quality assessments by running a few individual tests just like the development team did.

Use the vm.tests.list_tests() function introduced by the first notebook in this series in combination with vm.tests.list_tags() and vm.tests.list_tasks() to find which prebuilt tests are relevant for data quality assessment:

  • tasks represent the kind of modeling task associated with a test. Here we'll focus on classification tasks.
  • tags are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the data_quality tag.
# Get the list of available task types
sorted(vm.tests.list_tasks())
['classification',
 'clustering',
 'data_validation',
 'feature_extraction',
 'monitoring',
 'nlp',
 'regression',
 'residual_analysis',
 'text_classification',
 'text_generation',
 'text_qa',
 'text_summarization',
 'time_series_forecasting',
 'visualization']
# Get the list of available tags
sorted(vm.tests.list_tags())
['AUC',
 'analysis',
 'anomaly',
 'anomaly_detection',
 'bias_and_fairness',
 'binary_classification',
 'calibration',
 'categorical_data',
 'classification',
 'classification_metrics',
 'clustering',
 'conceptual_soundness',
 'correlation',
 'credit_risk',
 'data_analysis',
 'data_distribution',
 'data_quality',
 'data_validation',
 'descriptive_statistics',
 'dimensionality_reduction',
 'distribution',
 'embeddings',
 'feature_importance',
 'feature_selection',
 'few_shot',
 'forecasting',
 'frequency_analysis',
 'kmeans',
 'linear_regression',
 'llm',
 'logistic_regression',
 'metadata',
 'model_comparison',
 'model_diagnosis',
 'model_explainability',
 'model_interpretation',
 'model_performance',
 'model_predictions',
 'model_selection',
 'model_training',
 'model_validation',
 'multiclass_classification',
 'multicollinearity',
 'nlp',
 'normality',
 'numerical_data',
 'outlier',
 'outliers',
 'qualitative',
 'rag_performance',
 'ragas',
 'regression',
 'retrieval_performance',
 'robustness',
 'scorecard',
 'seasonality',
 'segment_analysis',
 'sensitivity_analysis',
 'sklearn',
 'specification',
 'stability',
 'stationarity',
 'statistical_test',
 'statistics',
 'statsmodels',
 'stress_testing',
 'tabular_data',
 'text_data',
 'threshold_optimization',
 'time_series_data',
 'uncertainty',
 'unit_root_test',
 'visualization',
 'zero_shot']

You can pass tags and tasks as parameters to the vm.tests.list_tests() function to filter the tests based on the tags and task types.

For example, to find tests related to tabular data quality for classification models, you can call list_tests() like this:

vm.tests.list_tests(task="classification", tags=["tabular_data", "data_quality"])
ID Name Description Has Figure Has Table Required Inputs Params Tags Tasks
validmind.data_validation.ClassImbalance Class Imbalance Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model.... True True ['dataset'] {'min_percent_threshold': {'type': 'int', 'default': 10}} ['tabular_data', 'binary_classification', 'multiclass_classification', 'data_quality'] ['classification']
validmind.data_validation.DescriptiveStatistics Descriptive Statistics Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's... False True ['dataset'] {} ['tabular_data', 'time_series_data', 'data_quality'] ['classification', 'regression']
validmind.data_validation.Duplicates Duplicates Tests dataset for duplicate entries, ensuring model reliability via data quality verification.... False True ['dataset'] {'min_threshold': {'type': '_empty', 'default': 1}} ['tabular_data', 'data_quality', 'text_data'] ['classification', 'regression']
validmind.data_validation.HighCardinality High Cardinality Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting.... False True ['dataset'] {'num_threshold': {'type': 'int', 'default': 100}, 'percent_threshold': {'type': 'float', 'default': 0.1}, 'threshold_type': {'type': 'str', 'default': 'percent'}} ['tabular_data', 'data_quality', 'categorical_data'] ['classification', 'regression']
validmind.data_validation.HighPearsonCorrelation High Pearson Correlation Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity.... False True ['dataset'] {'max_threshold': {'type': 'float', 'default': 0.3}, 'top_n_correlations': {'type': 'int', 'default': 10}, 'feature_columns': {'type': 'list', 'default': None}} ['tabular_data', 'data_quality', 'correlation'] ['classification', 'regression']
validmind.data_validation.MissingValues Missing Values Evaluates dataset quality by ensuring missing value percentage across all features does not exceed a set threshold.... False True ['dataset'] {'min_percentage_threshold': {'type': 'float', 'default': 1.0}} ['tabular_data', 'data_quality'] ['classification', 'regression']
validmind.data_validation.MissingValuesBarPlot Missing Values Bar Plot Assesses the percentage and distribution of missing values in the dataset via a bar plot, with emphasis on... True False ['dataset'] {'threshold': {'type': 'int', 'default': 80}, 'fig_height': {'type': 'int', 'default': 600}} ['tabular_data', 'data_quality', 'visualization'] ['classification', 'regression']
validmind.data_validation.Skewness Skewness Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data... False True ['dataset'] {'max_threshold': {'type': '_empty', 'default': 1}} ['data_quality', 'tabular_data'] ['classification', 'regression']
validmind.plots.BoxPlot Box Plot Generates customizable box plots for numerical features in a dataset with optional grouping using Plotly.... True False ['dataset'] {'columns': {'type': 'Optional', 'default': None}, 'group_by': {'type': 'Optional', 'default': None}, 'width': {'type': 'int', 'default': 1800}, 'height': {'type': 'int', 'default': 1200}, 'colors': {'type': 'Optional', 'default': None}, 'show_outliers': {'type': 'bool', 'default': True}, 'title_prefix': {'type': 'str', 'default': 'Box Plot of'}} ['tabular_data', 'visualization', 'data_quality'] ['classification', 'regression', 'clustering']
validmind.plots.HistogramPlot Histogram Plot Generates customizable histogram plots for numerical features in a dataset using Plotly.... True False ['dataset'] {'columns': {'type': 'Optional', 'default': None}, 'bins': {'type': 'Union', 'default': 30}, 'color': {'type': 'str', 'default': 'steelblue'}, 'opacity': {'type': 'float', 'default': 0.7}, 'show_kde': {'type': 'bool', 'default': True}, 'normalize': {'type': 'bool', 'default': False}, 'log_scale': {'type': 'bool', 'default': False}, 'title_prefix': {'type': 'str', 'default': 'Histogram of'}, 'width': {'type': 'int', 'default': 1200}, 'height': {'type': 'int', 'default': 800}, 'n_cols': {'type': 'int', 'default': 2}, 'vertical_spacing': {'type': 'float', 'default': 0.15}, 'horizontal_spacing': {'type': 'float', 'default': 0.1}} ['tabular_data', 'visualization', 'data_quality'] ['classification', 'regression', 'clustering']
validmind.stats.DescriptiveStats Descriptive Stats Provides comprehensive descriptive statistics for numerical features in a dataset.... False True ['dataset'] {'columns': {'type': 'Optional', 'default': None}, 'include_advanced': {'type': 'bool', 'default': True}, 'confidence_level': {'type': 'float', 'default': 0.95}} ['tabular_data', 'statistics', 'data_quality'] ['classification', 'regression', 'clustering']
Want to learn more about navigating ValidMind tests?

Refer to our notebook outlining the utilities available for viewing and understanding available ValidMind tests: Explore tests

Initialize the ValidMind dataset

With the individual tests we want to run identified, the next step is to connect your data with a ValidMind Dataset object. This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind, but you only need to do it once per dataset.

Initialize a ValidMind dataset object using the init_dataset function from the ValidMind (vm) module. For this example, we'll pass in the following arguments:

  • dataset — The raw dataset that you want to provide as input to tests.
  • input_id — A unique identifier that allows tracking what inputs are used when running each individual test.
  • target_column — A required argument if tests require access to true values. This is the name of the target column in the dataset.
# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test
vm_raw_dataset = vm.init_dataset(
    dataset=raw_df,
    input_id="raw_dataset",
    target_column="Exited",
)

Run data quality tests

Now that we know how to initialize a ValidMind dataset object, we're ready to run some tests!

You run individual tests by calling the run_test function provided by the validmind.tests module. For the examples below, we'll pass in the following arguments:

  • test_id — The ID of the test to run, as seen in the ID column when you run list_tests.
  • params — A dictionary of parameters for the test. These will override any default_params set in the test definition.

Run tabular data tests

The inputs expected by a test can also be found in the test definition — let's take validmind.data_validation.DescriptiveStatistics as an example.

Note that the output of the describe_test() function below shows that this test expects a dataset as input:

vm.tests.describe_test("validmind.data_validation.DescriptiveStatistics")
▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Now, let's run a few tests to assess the quality of the dataset:

result2 = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_raw_dataset},
    params={"min_percent_threshold": 30},
)

❌ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring the percentage share of each class against a minimum threshold. In this result, the target variable Exited contains two classes, with class 0 representing 79.80% of rows and class 1 representing 20.20% of rows. The applied minimum percentage threshold is 30%, and the table reports a pass/fail outcome for each class based on that threshold. The accompanying bar chart visually reflects this concentration, with a substantially taller bar for class 0 than for class 1.

Key insights:

  • Class 0 dominates the target: Exited = 0 accounts for 79.80% of observations, making it the clear majority class in the dataset.
  • Class 1 falls below threshold: Exited = 1 represents 20.20% of rows, which is below the configured 30% minimum threshold and is therefore marked as Fail.
  • Asymmetric class distribution observed: The gap between the two classes is 59.60 percentage points, indicating a materially uneven target distribution.

The result shows that the target distribution is concentrated in class 0, while class 1 is under the configured minimum percentage threshold. One class passes and one class fails under the 30% criterion. Collectively, the table and chart indicate that the dataset exhibits class imbalance in the Exited target variable under the test configuration provided.

Parameters:

{
  "min_percent_threshold": 30
}

Tables

Exited Class Imbalance

Exited Percentage of Rows (%) Pass/Fail
0 79.80% Pass
1 20.20% Fail

Figures

ValidMind Figure validmind.data_validation.ClassImbalance:82bc

The output above shows that the validmind.data_validation.ClassImbalance test did not pass according to the value we set for min_percent_threshold — great, this matches what was reported by the development team.

To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.

As this is technically a different dataset, remember to first initialize a new ValidMind Dataset object to pass in as input as required by run_test():

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)
# Pass the initialized `balanced_raw_dataset` as input into the test run
result = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_balanced_raw_dataset},
    params={"min_percent_threshold": 30},
)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring each class’s share of total records against a minimum percentage threshold. In this run, the target variable Exited contains two classes, 0 and 1, and the results table reports each class at 50.00% of rows. The applied minimum percentage threshold is 30%, and the accompanying bar chart shows equal class proportions across both target outcomes.

Key insights:

  • Balanced binary class distribution: Both Exited=0 and Exited=1 represent 50.00% of the dataset, indicating an even split between the two target classes.
  • All classes pass threshold test: Each class exceeds the configured minimum percentage threshold of 30%, and both are marked as Pass.
  • No observed minority class: Neither target class appears under-represented in the reported distribution, as the class shares are identical.

The test results show a fully balanced target distribution for Exited, with both classes represented equally at 50.00%. Relative to the configured 30% threshold, both classes pass without exception. Collectively, the table and chart indicate no observed class concentration in either target outcome.

Parameters:

{
  "min_percent_threshold": 30
}

Tables

Exited Class Imbalance

Exited Percentage of Rows (%) Pass/Fail
0 50.00% Pass
1 50.00% Pass

Figures

ValidMind Figure validmind.data_validation.ClassImbalance:125d

Remove highly correlated features

Next, let's also remove highly correlated features from our dataset as outlined by the development team. Removing highly correlated features helps make the model simpler, more stable, and easier to understand.

You can utilize the output from a ValidMind test for further use — in this below example, to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.

First, we'll run validmind.data_validation.HighPearsonCorrelation with the balanced_raw_dataset we initialized previously as input as is for comparison with later runs:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships between features to identify potentially redundant or highly collinear variable combinations. The result table reports the top feature pairs ranked by Pearson correlation coefficient and applies a pass/fail assessment using the configured absolute correlation threshold of 0.3. In this output, coefficients range from -0.1898 to 0.335 across the ten reported pairs, with one pair exceeding the threshold and the remaining pairs marked as passing.

Key insights:

  • Single threshold breach identified: The pair (Age, Exited) has a Pearson correlation coefficient of 0.335, which exceeds the configured threshold of 0.3 and is the only pair in the reported output marked as Fail.
  • Most reported relationships are weak: The remaining nine reported pairs have coefficients between -0.1898 and 0.1512, all below the threshold in absolute value and therefore classified as Pass.
  • Largest non-failing negative relationship is limited: The strongest negative coefficient in the reported results is -0.1898 for (IsActiveMember, Exited), which remains below the threshold by absolute magnitude.
  • Top reported correlations are concentrated near zero beyond the leading pair: Aside from (Age, Exited), the reported coefficients are relatively small, including values such as -0.1807 for (Balance, NumOfProducts), 0.1512 for (Balance, Exited), and several near-zero values between -0.0502 and 0.0410.

The reported correlation structure shows one feature pair exceeding the configured high-correlation threshold, while all other listed relationships remain below that cutoff. The strongest observed association is between Age and Exited, and the rest of the top reported correlations are modest to minimal in magnitude. Overall, the table indicates a largely low linear correlation profile across the reported feature pairs, with a single flagged relationship in this test output.

Parameters:

{
  "max_threshold": 0.3
}

Tables

Columns Coefficient Pass/Fail
(Age, Exited) 0.3350 Fail
(IsActiveMember, Exited) -0.1898 Pass
(Balance, NumOfProducts) -0.1807 Pass
(Balance, Exited) 0.1512 Pass
(NumOfProducts, Exited) -0.0502 Pass
(Age, NumOfProducts) -0.0418 Pass
(Age, Balance) 0.0410 Pass
(CreditScore, IsActiveMember) 0.0354 Pass
(NumOfProducts, IsActiveMember) 0.0353 Pass
(CreditScore, EstimatedSalary) -0.0321 Pass

The output above shows that the test did not pass according to the value we set for max_threshold — as reported and expected.

corr_result is an object of type TestResult. We can inspect the result object to see what the test has produced:

print(type(corr_result))
print("Result ID: ", corr_result.result_id)
print("Params: ", corr_result.params)
print("Passed: ", corr_result.passed)
print("Tables: ", corr_result.tables)
<class 'validmind.vm_models.result.result.TestResult'>
Result ID:  validmind.data_validation.HighPearsonCorrelation
Params:  {'max_threshold': 0.3}
Passed:  False
Tables:  [ResultTable]

Let's remove the highly correlated features and create a new VM dataset object.

We'll begin by checking out the table in the result and extracting a list of features that failed the test:

# Extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df
Columns Coefficient Pass/Fail
0 (Age, Exited) 0.3350 Fail
1 (IsActiveMember, Exited) -0.1898 Pass
2 (Balance, NumOfProducts) -0.1807 Pass
3 (Balance, Exited) 0.1512 Pass
4 (NumOfProducts, Exited) -0.0502 Pass
5 (Age, NumOfProducts) -0.0418 Pass
6 (Age, Balance) 0.0410 Pass
7 (CreditScore, IsActiveMember) 0.0354 Pass
8 (NumOfProducts, IsActiveMember) 0.0353 Pass
9 (CreditScore, EstimatedSalary) -0.0321 Pass 
# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features
['(Age, Exited)']

Next, extract the feature names from the list of strings (example: (Age, Exited) > Age):

high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features
['Age']

Now, it's time to re-initialize the dataset with the highly correlated features removed.

Note the use of a different input_id. This allows tracking the inputs used when running each individual test.

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

Re-running the test with the reduced feature set should pass the test:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify highly correlated variable pairs that may indicate redundancy or multicollinearity. The result table reports the top 10 strongest feature correlations along with their Pearson coefficients and pass/fail status against the configured threshold of 0.3. In this output, all reported coefficients are below the threshold in absolute value, and all listed feature pairs are marked as Pass. The observed coefficients range from -0.1898 to 0.1512 across the reported pairs.

Key insights:

  • No pair exceeds threshold: All 10 reported feature pairs pass the test, with absolute correlation coefficients remaining below the configured 0.3 threshold.
  • Strongest observed relationship is modest: The largest absolute coefficient is -0.1898 for (IsActiveMember, Exited), which remains materially below the threshold.
  • Top relationships are concentrated below 0.20: The three largest absolute correlations are (IsActiveMember, Exited) at -0.1898, (Balance, NumOfProducts) at -0.1807, and (Balance, Exited) at 0.1512, indicating only modest linear relationships among the strongest reported pairs.
  • Several feature pairs are near zero: Lower-ranked correlations, including (CreditScore, IsActiveMember) at 0.0354, (NumOfProducts, IsActiveMember) at 0.0353, (CreditScore, EstimatedSalary) at -0.0321, and (Tenure, IsActiveMember) at -0.0300, show minimal linear association.

The reported correlation structure does not show evidence of high pairwise linear dependence under the 0.3 threshold used in this test. The strongest observed relationships are modest in magnitude, and several of the listed feature pairs are effectively near zero. Taken together, the result indicates limited pairwise linear redundancy among the top reported correlations.

Parameters:

{
  "max_threshold": 0.3
}

Tables

Columns Coefficient Pass/Fail
(IsActiveMember, Exited) -0.1898 Pass
(Balance, NumOfProducts) -0.1807 Pass
(Balance, Exited) 0.1512 Pass
(NumOfProducts, Exited) -0.0502 Pass
(CreditScore, IsActiveMember) 0.0354 Pass
(NumOfProducts, IsActiveMember) 0.0353 Pass
(CreditScore, EstimatedSalary) -0.0321 Pass
(Tenure, IsActiveMember) -0.0300 Pass
(CreditScore, Exited) -0.0282 Pass
(Tenure, Exited) -0.0247 Pass

You can also plot the correlation matrix to visualize the new correlation between features:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.PearsonCorrelationMatrix",
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

Pearson Correlation Matrix

The PearsonCorrelationMatrix test evaluates linear dependency between numerical variables by computing pairwise Pearson correlation coefficients and presenting them in a heat map. The matrix covers CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with coefficients ranging from -0.19 to 0.15 outside the diagonal. The diagonal entries are 1.0 by construction, while the off-diagonal cells show generally weak positive and negative linear relationships across the variables.

Key insights:

  • No high pairwise correlations observed: All off-diagonal correlation coefficients remain well below the 0.7 absolute-value threshold highlighted by the test. The observed range of -0.19 to 0.15 indicates an absence of strong linear dependence among the numerical variables shown.

  • Exited shows weak associations only: Exited has its largest absolute correlation with IsActiveMember at -0.19, followed by Balance at 0.15. Correlations with CreditScore (-0.03), Tenure (-0.02), NumOfProducts (-0.05), HasCrCard (0.0), and EstimatedSalary (-0.01) are close to zero.

  • Balance-related relationships are modest: Balance shows a negative correlation of -0.18 with NumOfProducts and a positive correlation of 0.15 with Exited. Its remaining correlations are near zero, including 0.02 with CreditScore, -0.01 with Tenure, -0.02 with HasCrCard, 0.01 with IsActiveMember, and 0.0 with EstimatedSalary.

  • Most feature pairs are near zero: Several variable pairs exhibit negligible linear association, typically between -0.03 and 0.04. Examples include CreditScore with NumOfProducts (0.0), HasCrCard with Exited (0.0), EstimatedSalary with Balance (0.0), and EstimatedSalary with NumOfProducts (0.0).

The correlation structure is sparse, with all observed off-diagonal coefficients indicating weak linear relationships rather than concentrated clusters of dependency. The most pronounced associations are limited to IsActiveMember versus Exited (-0.19), Balance versus NumOfProducts (-0.18), and Balance versus Exited (0.15), while the remainder of the matrix is largely centered near zero. Collectively, the result shows limited linear redundancy across the numerical variables included in the test.

Figures

ValidMind Figure validmind.data_validation.PearsonCorrelationMatrix:a8fa

Documenting test results

Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it. Every test result returned by the run_test() function has a .log() method that can be used to send the test results to the ValidMind Platform.

When logging validation test results to the platform, you'll need to manually add those results to the desired section of the validation report. To demonstrate how to add test results to your validation report, we'll log our data quality tests and insert the results via the ValidMind Platform.

Configure and run comparison tests

Below, we'll perform comparison tests between the original raw dataset (raw_dataset) and the final preprocessed (raw_dataset_preprocessed) dataset, again logging the results to the ValidMind Platform.

We can specify all the tests we'd ike to run in a dictionary called test_config, and we'll pass in the following arguments for each test:

  • params: Individual test parameters.
  • input_grid: Individual test inputs to compare. In this case, we'll input our two datasets for comparison.

Note here that the input_grid expects the input_id of the dataset as the value rather than the variable name we specified:

# Individual test config with inputs specified
test_config = {
    "validmind.data_validation.ClassImbalance": {
        "input_grid": {"dataset": ["raw_dataset", "raw_dataset_preprocessed"]},
        "params": {"min_percent_threshold": 30}
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "input_grid": {"dataset": ["raw_dataset", "raw_dataset_preprocessed"]},
        "params": {"max_threshold": 0.3}
    },
}

Then batch run and log our tests in test_config:

for t in test_config:
    print(t)
    try:
        # Check if test has input_grid
        if 'input_grid' in test_config[t]:
            # For tests with input_grid, pass the input_grid configuration
            if 'params' in test_config[t]:
                vm.tests.run_test(t, input_grid=test_config[t]['input_grid'], params=test_config[t]['params']).log()
            else:
                vm.tests.run_test(t, input_grid=test_config[t]['input_grid']).log()
        else:
            # Original logic for regular inputs
            if 'params' in test_config[t]:
                vm.tests.run_test(t, inputs=test_config[t]['inputs'], params=test_config[t]['params']).log()
            else:
                vm.tests.run_test(t, inputs=test_config[t]['inputs']).log()
    except Exception as e:
        print(f"Error running test {t}: {str(e)}")
validmind.data_validation.ClassImbalance

❌ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes by measuring the percentage share of each class and comparing those shares to the configured minimum threshold. In this run, the test was applied to the Exited target for both raw_dataset and raw_dataset_preprocessed using a minimum percentage threshold of 30%. The results table and accompanying bar charts show the class proportions for Exited=0 and Exited=1 in each dataset, along with the corresponding pass/fail outcome for each class.

Key insights:

  • Raw dataset is imbalanced: In raw_dataset, Exited=0 represents 79.80% of rows and passes the 30% threshold, while Exited=1 represents 20.20% and fails the threshold.
  • Preprocessed dataset is balanced: In raw_dataset_preprocessed, both Exited=0 and Exited=1 each account for 50.00% of rows, and both classes pass the test.
  • Minority class share increased materially: The share of Exited=1 increases from 20.20% in raw_dataset to 50.00% in raw_dataset_preprocessed, eliminating the threshold breach observed in the raw data.
  • Pass/fail outcome changes across datasets: The test outcome shifts from one failing class in raw_dataset to no failing classes in raw_dataset_preprocessed.

The results show that class distribution differs materially between the two evaluated datasets. The original raw_dataset contains a skewed target distribution with the Exited=1 class below the 30% threshold, while the preprocessed dataset shows an even 50/50 split across classes and no threshold failures. Taken together, the test output indicates that the class imbalance present in the raw target distribution is no longer present in the preprocessed dataset under the configured threshold.

Parameters:

{
  "min_percent_threshold": 30
}

Tables

dataset Exited Percentage of Rows (%) Pass/Fail
raw_dataset 0 79.80% Pass
raw_dataset 1 20.20% Fail
raw_dataset_preprocessed 0 50.00% Pass
raw_dataset_preprocessed 1 50.00% Pass

Figures

ValidMind Figure validmind.data_validation.ClassImbalance:6174
ValidMind Figure validmind.data_validation.ClassImbalance:8217
2026-10-30 20:58:59,338 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.ClassImbalance does not exist in model's document
validmind.data_validation.HighPearsonCorrelation

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant variables or concentrations of multicollinearity. The results are reported for both raw_dataset and raw_dataset_preprocessed, listing the strongest observed feature pairs together with their Pearson coefficients and pass/fail status against the configured threshold of 0.3. In the raw dataset, one pair exceeds the threshold and is flagged as Fail, while all other listed relationships in both datasets remain within the threshold. The reported coefficients span both positive and negative values, with magnitudes ranging from very weak to modest.

Key insights:

  • One raw-data pair breaches threshold: In raw_dataset, the pair (Balance, NumOfProducts) has a coefficient of -0.3045, which exceeds the absolute threshold of 0.3 and is the only relationship marked as Fail.
  • Preprocessing reduces strongest observed correlation: For (Balance, NumOfProducts), the coefficient changes from -0.3045 in raw_dataset to -0.1807 in raw_dataset_preprocessed, moving from Fail to Pass under the same threshold.
  • Most reported relationships are weak: Aside from the single failed pair, the listed coefficients are all below |0.3|, including 0.2810 for (Age, Exited) in raw_dataset and -0.1898 for (IsActiveMember, Exited) in raw_dataset_preprocessed.
  • Correlation patterns differ across dataset versions: The top reported relationships are not identical between the two datasets; for example, (Age, Exited) appears among the strongest pairs in raw_dataset, while the strongest reported pair in raw_dataset_preprocessed is (IsActiveMember, Exited) at -0.1898.

The results show a limited concentration of higher linear dependence among the reported feature pairs. In the raw dataset, only (Balance, NumOfProducts) crosses the configured threshold, while all listed pairs in the preprocessed dataset remain below it. Overall, the reported correlation structure is dominated by weak to modest relationships, with the preprocessing stage associated with a reduction in the strongest observed absolute correlation.

Parameters:

{
  "max_threshold": 0.3
}

Tables

dataset Columns Coefficient Pass/Fail
raw_dataset (Balance, NumOfProducts) -0.3045 Fail
raw_dataset (Age, Exited) 0.2810 Pass
raw_dataset (IsActiveMember, Exited) -0.1515 Pass
raw_dataset (Balance, Exited) 0.1174 Pass
raw_dataset (Age, IsActiveMember) 0.0873 Pass
raw_dataset (NumOfProducts, Exited) -0.0523 Pass
raw_dataset (Age, NumOfProducts) -0.0306 Pass
raw_dataset (CreditScore, IsActiveMember) 0.0306 Pass
raw_dataset (Tenure, IsActiveMember) -0.0293 Pass
raw_dataset (Age, Balance) 0.0290 Pass
raw_dataset_preprocessed (IsActiveMember, Exited) -0.1898 Pass
raw_dataset_preprocessed (Balance, NumOfProducts) -0.1807 Pass
raw_dataset_preprocessed (Balance, Exited) 0.1512 Pass
raw_dataset_preprocessed (NumOfProducts, Exited) -0.0502 Pass
raw_dataset_preprocessed (CreditScore, IsActiveMember) 0.0354 Pass
raw_dataset_preprocessed (NumOfProducts, IsActiveMember) 0.0353 Pass
raw_dataset_preprocessed (CreditScore, EstimatedSalary) -0.0321 Pass
raw_dataset_preprocessed (Tenure, IsActiveMember) -0.0300 Pass
raw_dataset_preprocessed (CreditScore, Exited) -0.0282 Pass
raw_dataset_preprocessed (Tenure, Exited) -0.0247 Pass 
2026-10-30 20:59:08,843 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.HighPearsonCorrelation does not exist in model's document
Note the output returned indicating that a test-driven block doesn't currently exist in your documentation for some test IDs.

That's expected, as when we run validations tests the results logged need to be manually added to your report as part of your compliance assessment process within the ValidMind Platform.

Log tests with unique identifiers

Next, we'll use the previously initialized vm_balanced_raw_dataset (that still has a highly correlated Age column) as input to run an individual test, then log the result to the ValidMind Platform.

When running individual tests, you can use a custom result_id to tag the individual result with a unique identifier:

  • This result_id can be appended to test_id with a : separator.
  • The balanced_raw_dataset result identifier will correspond to the balanced_raw_dataset input, the dataset that still has the Age column.
result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)
result.log()

❌ High Pearson Correlation Balanced Raw Dataset

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant or highly collinear variables. The results table lists the top feature pairs by Pearson correlation coefficient together with a Pass/Fail assessment against the configured absolute threshold of 0.3. In this run, 10 feature pairs are reported, with coefficients ranging from -0.1898 to 0.335. Only one pair exceeds the threshold and is marked as Fail, while the remaining nine pairs are marked as Pass.

Key insights:

  • Single threshold breach observed: The pair (Age, Exited) has a Pearson correlation coefficient of 0.335, which exceeds the configured threshold of 0.3 and is the only reported Fail.
  • All other reported correlations are below threshold: The remaining nine feature pairs have absolute correlation values below 0.3, with the next largest magnitude at -0.1898 for (IsActiveMember, Exited).
  • Reported relationships are generally weak: Aside from (Age, Exited), all listed coefficients fall within a narrow range from -0.1898 to 0.1512, indicating limited linear association among the other top reported pairs.
  • Both positive and negative associations appear: The reported coefficients include positive values such as (Balance, Exited) = 0.1512 and negative values such as (Balance, NumOfProducts) = -0.1807, showing that the strongest listed relationships are mixed in direction.

Overall, the reported correlation structure is concentrated in low-magnitude pairwise relationships, with only one feature pair crossing the configured high-correlation threshold. The most prominent observed relationship is the positive correlation between Age and Exited at 0.335, while all other top reported pairs remain below the threshold. This indicates that the test output identifies a largely low-correlation set of reported feature relationships with one isolated threshold exceedance.

Parameters:

{
  "max_threshold": 0.3
}

Tables

Columns Coefficient Pass/Fail
(Age, Exited) 0.3350 Fail
(IsActiveMember, Exited) -0.1898 Pass
(Balance, NumOfProducts) -0.1807 Pass
(Balance, Exited) 0.1512 Pass
(NumOfProducts, Exited) -0.0502 Pass
(Age, NumOfProducts) -0.0418 Pass
(Age, Balance) 0.0410 Pass
(CreditScore, IsActiveMember) 0.0354 Pass
(NumOfProducts, IsActiveMember) 0.0353 Pass
(CreditScore, EstimatedSalary) -0.0321 Pass 
2026-10-30 20:59:17,287 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset does not exist in model's document

Add test results to reporting

With some test results logged, let's head to the model we connected to at the beginning of this notebook and learn how to insert a test result into our validation report. (Learn more: Assess compliance)

While the example below focuses on a specific test result, you can follow the same general procedure for your other results:

  1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

  2. In the left sidebar that appears for your model, click Validation under Documents.

  3. Click on 2.2.1. Data Quality to expand that section.

  4. Under the Class Imbalance Assessment guideline, click Evidence to expand the evidence panel.

  5. Click Link Evidence, then select Validator Evidence.

  6. Select the Class Imbalance test results we logged: ValidMind Data Validation Class Imbalance

    Screenshot showing the ClassImbalance test selected

  7. Click Update Linked Evidence to add the test results to the validation report.

  8. Confirm that the results for the Class Imbalance test you inserted has been correctly inserted into section 2.2.1. Data Quality of the report.

    • Note that these test results are flagged as Requires Attention — as they include comparative results from our initial raw dataset.
    • Click See evidence details to review the LLM-generated description that summarizes the test results, that confirm that our final preprocessed dataset actually passes our test:

    Screenshot showing the ClassImbalance test generated description in the text editor

Here in this text editor, you can make qualitative edits to the draft that ValidMind generated to finalize the test results.

Learn more: Work with content blocks

Preparing the preprocessed dataset

Split the preprocessed dataset

With our raw dataset rebalanced with highly correlated features removed, let's now spilt our dataset into train and test in preparation for model evaluation testing.

To start, let's grab the first few rows from the balanced_raw_no_age_df dataset we initialized earlier:

balanced_raw_no_age_df.head()
CreditScore Geography Gender Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary Exited
2367 850 Spain Female 3 99945.93 2 1 0 71179.31 0
3957 776 Spain Male 8 106365.29 1 1 1 148527.56 0
3417 731 Spain Female 1 128932.40 1 1 1 180712.52 0
4163 780 Germany Female 10 95196.26 1 1 0 126310.39 1
7827 825 Germany Female 6 118336.95 1 1 0 26342.33 1

Before training the model, we need to encode the categorical features in the dataset:

  • Use the OneHotEncoder class from the sklearn.preprocessing module to encode the categorical features.
  • The categorical features in the dataset are Geography and Gender.
balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()
CreditScore Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary Exited Geography_Germany Geography_Spain Gender_Male
2367 850 3 99945.93 2 1 0 71179.31 0 False True False
3957 776 8 106365.29 1 1 1 148527.56 0 False True True
3417 731 1 128932.40 1 1 1 180712.52 0 False True False
4163 780 10 95196.26 1 1 0 126310.39 1 True False False
7827 825 6 118336.95 1 1 0 26342.33 1 True False False

Splitting our dataset into training and testing is essential for proper validation testing, as this helps assess how well the model generalizes to unseen data:

  • We start by dividing our balanced_raw_no_age_df dataset into training and test subsets using train_test_split, with 80% of the data allocated to training (train_df) and 20% to testing (test_df).
  • From each subset, we separate the features (all columns except "Exited") into X_train and X_test, and the target column ("Exited") into y_train and y_test.
from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

Initialize the split datasets

Next, let's initialize the training and testing datasets so they are available for use:

vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

In summary

In this second notebook, you learned how to:

Next steps

Develop potential challenger models

Now that you're familiar with the basics of using the ValidMind Library, let's use it to develop a challenger model: 3 — Developing a potential challenger

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial