ValidMind for development 2 — Start the development process

Learn how to use ValidMind for your end-to-end documentation process with our series of four introductory notebooks. In this second notebook, you'll run tests and investigate results, then add the results or evidence to your documentation.

You'll become familiar with the individual tests available in ValidMind, as well as how to run them and change parameters as necessary. Using ValidMind's repository of individual tests as building blocks helps you ensure that a record (model) is being built appropriately.

For a full list of out-of-the-box tests and descriptions, use the interactive ValidMind test sandbox.

Learn by doing

Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Developer Fundamentals

Prerequisites

In order to log test results or evidence to your documentation with this notebook, you'll need to first have:

• Registered a model within the ValidMind Platform with a predefined documentation template
• Installed the ValidMind Library in your local environment, allowing you to access all its features

Need help with the above steps?

Refer to the first notebook in this series: 1 — Set up the ValidMind Library

Setting up

Initialize the ValidMind Library

First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Development from the Document drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="documentation",
)

Note: you may need to restart the kernel to use updated packages.

2026-10-02 20:31:05,134 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model development (ID: cmalgf3qi02ce199qm3rdkl46)
📁 Document Type: model_documentation

Import sample dataset

Then, let's import the public Bank Customer Churn Prediction dataset from Kaggle.

In our below example, note that:

• The target column, Exited has a value of 1 when a customer has churned and 0 otherwise.
• The ValidMind Library provides a wrapper to automatically load the dataset as a Pandas DataFrame object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns.

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()
raw_df.head()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

Identify qualitative tests

Next, let's say we want to do some data quality assessments by running a few individual tests.

Use the vm.tests.list_tests() function introduced by the first notebook in this series in combination with vm.tests.list_tags() and vm.tests.list_tasks() to find which prebuilt tests are relevant for data quality assessment:

• tasks represent the kind of modeling task associated with a test. Here we'll focus on classification tasks.
• tags are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the data_quality tag.

# Get the list of available task types
sorted(vm.tests.list_tasks())

['classification',
 'clustering',
 'data_validation',
 'feature_extraction',
 'monitoring',
 'nlp',
 'regression',
 'residual_analysis',
 'text_classification',
 'text_generation',
 'text_qa',
 'text_summarization',
 'time_series_forecasting',
 'visualization']

# Get the list of available tags
sorted(vm.tests.list_tags())

['AUC',
 'analysis',
 'anomaly',
 'anomaly_detection',
 'bias_and_fairness',
 'binary_classification',
 'calibration',
 'categorical_data',
 'classification',
 'classification_metrics',
 'clustering',
 'correlation',
 'credit_risk',
 'data_analysis',
 'data_distribution',
 'data_quality',
 'data_validation',
 'descriptive_statistics',
 'dimensionality_reduction',
 'distribution',
 'embeddings',
 'feature_importance',
 'feature_selection',
 'few_shot',
 'forecasting',
 'frequency_analysis',
 'kmeans',
 'linear_regression',
 'llm',
 'logistic_regression',
 'metadata',
 'model_comparison',
 'model_diagnosis',
 'model_explainability',
 'model_interpretation',
 'model_performance',
 'model_predictions',
 'model_selection',
 'model_training',
 'model_validation',
 'multiclass_classification',
 'nlp',
 'normality',
 'numerical_data',
 'outlier',
 'outliers',
 'qualitative',
 'rag_performance',
 'ragas',
 'regression',
 'retrieval_performance',
 'scorecard',
 'seasonality',
 'senstivity_analysis',
 'sklearn',
 'stationarity',
 'statistical_test',
 'statistics',
 'statsmodels',
 'tabular_data',
 'text_data',
 'threshold_optimization',
 'time_series_data',
 'unit_root_test',
 'visualization',
 'zero_shot']

You can pass tags and tasks as parameters to the vm.tests.list_tests() function to filter the tests based on the tags and task types.

For example, to find tests related to tabular data quality for classification models, you can call list_tests() like this:

vm.tests.list_tests(task="classification", tags=["tabular_data", "data_quality"])

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.data_validation.ClassImbalance	Class Imbalance	Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model....	True	True	['dataset']	{'min_percent_threshold': {'type': 'int', 'default': 10}}	['tabular_data', 'binary_classification', 'multiclass_classification', 'data_quality']	['classification']
validmind.data_validation.DescriptiveStatistics	Descriptive Statistics	Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's...	False	True	['dataset']	{}	['tabular_data', 'time_series_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.Duplicates	Duplicates	Tests dataset for duplicate entries, ensuring model reliability via data quality verification....	False	True	['dataset']	{'min_threshold': {'type': '_empty', 'default': 1}}	['tabular_data', 'data_quality', 'text_data']	['classification', 'regression']
validmind.data_validation.HighCardinality	High Cardinality	Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting....	False	True	['dataset']	{'num_threshold': {'type': 'int', 'default': 100}, 'percent_threshold': {'type': 'float', 'default': 0.1}, 'threshold_type': {'type': 'str', 'default': 'percent'}}	['tabular_data', 'data_quality', 'categorical_data']	['classification', 'regression']
validmind.data_validation.HighPearsonCorrelation	High Pearson Correlation	Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity....	False	True	['dataset']	{'max_threshold': {'type': 'float', 'default': 0.3}, 'top_n_correlations': {'type': 'int', 'default': 10}, 'feature_columns': {'type': 'list', 'default': None}}	['tabular_data', 'data_quality', 'correlation']	['classification', 'regression']
validmind.data_validation.MissingValues	Missing Values	Evaluates dataset quality by ensuring missing value percentage across all features does not exceed a set threshold....	False	True	['dataset']	{'min_percentage_threshold': {'type': 'float', 'default': 1.0}}	['tabular_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.MissingValuesBarPlot	Missing Values Bar Plot	Assesses the percentage and distribution of missing values in the dataset via a bar plot, with emphasis on...	True	False	['dataset']	{'threshold': {'type': 'int', 'default': 80}, 'fig_height': {'type': 'int', 'default': 600}}	['tabular_data', 'data_quality', 'visualization']	['classification', 'regression']
validmind.data_validation.Skewness	Skewness	Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data...	False	True	['dataset']	{'max_threshold': {'type': '_empty', 'default': 1}}	['data_quality', 'tabular_data']	['classification', 'regression']
validmind.plots.BoxPlot	Box Plot	Generates customizable box plots for numerical features in a dataset with optional grouping using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'group_by': {'type': 'Optional', 'default': None}, 'width': {'type': 'int', 'default': 1800}, 'height': {'type': 'int', 'default': 1200}, 'colors': {'type': 'Optional', 'default': None}, 'show_outliers': {'type': 'bool', 'default': True}, 'title_prefix': {'type': 'str', 'default': 'Box Plot of'}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.plots.HistogramPlot	Histogram Plot	Generates customizable histogram plots for numerical features in a dataset using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'bins': {'type': 'Union', 'default': 30}, 'color': {'type': 'str', 'default': 'steelblue'}, 'opacity': {'type': 'float', 'default': 0.7}, 'show_kde': {'type': 'bool', 'default': True}, 'normalize': {'type': 'bool', 'default': False}, 'log_scale': {'type': 'bool', 'default': False}, 'title_prefix': {'type': 'str', 'default': 'Histogram of'}, 'width': {'type': 'int', 'default': 1200}, 'height': {'type': 'int', 'default': 800}, 'n_cols': {'type': 'int', 'default': 2}, 'vertical_spacing': {'type': 'float', 'default': 0.15}, 'horizontal_spacing': {'type': 'float', 'default': 0.1}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.stats.DescriptiveStats	Descriptive Stats	Provides comprehensive descriptive statistics for numerical features in a dataset....	False	True	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'include_advanced': {'type': 'bool', 'default': True}, 'confidence_level': {'type': 'float', 'default': 0.95}}	['tabular_data', 'statistics', 'data_quality']	['classification', 'regression', 'clustering']

Want to learn more about navigating ValidMind tests?

Refer to our notebook outlining the utilities available for viewing and understanding available ValidMind tests: Explore tests

Initialize the ValidMind dataset

With the individual tests we want to run identified, the next step is to connect your data with a ValidMind Dataset object. This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind, but you only need to do it once per dataset.

Initialize a ValidMind dataset object using the init_dataset function from the ValidMind (vm) module. For this example, we'll pass in the following arguments:

• dataset — The raw dataset that you want to provide as input to tests.
• input_id — A unique identifier that allows tracking what inputs are used when running each individual test.
• target_column — A required argument if tests require access to true values. This is the name of the target column in the dataset.

# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test
vm_raw_dataset = vm.init_dataset(
    dataset=raw_df,
    input_id="raw_dataset",
    target_column="Exited",
)

Running tests on datasets

Now that we know how to initialize a ValidMind dataset object, we're ready to run some tests!

You run individual tests by calling the run_test function provided by the validmind.tests module. For the examples below, we'll pass in the following arguments:

• test_id — The ID of the test to run, as seen in the ID column when you run list_tests.
• params — A dictionary of parameters for the test. These will override any default_params set in the test definition.

Run tabular data tests

The inputs expected by a test can also be found in the test definition — let's take validmind.data_validation.DescriptiveStatistics as an example.

Note that the output of the describe_test() function below shows that this test expects a dataset as input:

vm.tests.describe_test("validmind.data_validation.DescriptiveStatistics")

▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Descriptive Statistics

Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's dataset.

Purpose

The purpose of the Descriptive Statistics metric is to provide a comprehensive summary of both numerical and categorical data within a dataset. This involves statistics such as count, mean, standard deviation, minimum and maximum values for numerical data. For categorical data, it calculates the count, number of unique values, most common value and its frequency, and the proportion of the most frequent value relative to the total. The goal is to visualize the overall distribution of the variables in the dataset, aiding in understanding the model's behavior and predicting its performance.

Test Mechanism

The testing mechanism utilizes two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function pulls out several summary statistics, while value_counts() accounts for unique values. The resulting data is formatted into two distinct tables, one for numerical and another for categorical variable summaries. These tables provide a clear summary of the main characteristics of the variables, which can be instrumental in assessing the model's performance.

Signs of High Risk

• Skewed data or significant outliers can represent high risk. For numerical data, this may be reflected via a significant difference between the mean and median (50% percentile).
• For categorical data, a lack of diversity (low count of unique values), or overdominance of a single category (high frequency of the top value) can indicate high risk.

Strengths

• Provides a comprehensive summary of the dataset, shedding light on the distribution and characteristics of the variables under consideration.
• It is a versatile and robust method, applicable to both numerical and categorical data.
• Helps highlight crucial anomalies such as outliers, extreme skewness, or lack of diversity, which are vital in understanding model behavior during testing and validation.

Limitations

• While this metric offers a high-level overview of the data, it may fail to detect subtle correlations or complex patterns.
• Does not offer any insights on the relationship between variables.
• Alone, descriptive statistics cannot be used to infer properties about future unseen data.
• Should be used in conjunction with other statistical tests to provide a comprehensive understanding of the model's data.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DescriptiveStatistics", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Now, let's run a few tests to assess the quality of the dataset:

result = vm.tests.run_test(
    test_id="validmind.data_validation.DescriptiveStatistics",
    inputs={"dataset": vm_raw_dataset},
)

Descriptive Statistics

The Descriptive Statistics test evaluates the distributional characteristics of numerical and categorical variables in the dataset. The results present summary statistics for eight numerical variables and frequency-based summaries for two categorical variables, with all variables showing a count of 8,000 observations. The numerical table includes central tendency, dispersion, percentile, and range statistics, while the categorical table reports the number of unique values, the most frequent category, and its relative prevalence.

Key insights:

• Balance shows pronounced asymmetry: Balance has a mean of 76,434.10 and a median of 97,264.00, with the 25th percentile at 0.00. This indicates a substantial concentration of zero or low balances alongside a broad upper range extending to 250,898.00.

• Salary is broadly dispersed: EstimatedSalary ranges from 12.00 to 199,992.00, with a mean of 99,790.19, median of 99,505.00, and standard deviation of 57,520.51. The close alignment of mean and median is accompanied by a wide overall spread.

• Credit score distribution is centered near median: CreditScore has a mean of 650.16 and a median of 652.00, with an interquartile range from 583.00 to 717.00 and an observed range from 350.00 to 850.00. The central tendency measures are closely aligned relative to the full score range.

• Age includes a long upper tail: Age has a mean of 38.95 and a median of 37.00, with the 95th percentile at 60.00 and a maximum of 92.00. Most observations are concentrated between 32.00 and 44.00, while the upper end extends materially beyond the upper quartile.

• Product holdings are concentrated at low counts: NumOfProducts has a median of 1.00, a 75th percentile of 2.00, and a maximum of 4.00. This shows that most observations are concentrated in the lower product-count categories.

• Categorical concentration is moderate: Geography contains 3 unique values, with France representing 50.12% of observations, while Gender contains 2 unique values, with Male representing 54.95%. The top categories are the majority classes in both categorical variables, but neither dominates the dataset overwhelmingly.

The results show complete coverage across all reported variables, with each variable containing 8,000 observations. The numerical distributions are mixed: several variables such as CreditScore and EstimatedSalary show closely aligned mean and median values, while Balance displays a stronger departure between central tendency measures and lower-percentile values. The categorical variables exhibit moderate concentration in their most frequent classes, with limited cardinality and majority representation by a single category in each case.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	8000.0	650.1596	96.8462	350.0	583.0	652.0	717.0	778.0	813.0	850.0
Age	8000.0	38.9489	10.4590	18.0	32.0	37.0	44.0	53.0	60.0	92.0
Tenure	8000.0	5.0339	2.8853	0.0	3.0	5.0	8.0	9.0	9.0	10.0
Balance	8000.0	76434.0965	62612.2513	0.0	0.0	97264.0	128045.0	149545.0	162488.0	250898.0
NumOfProducts	8000.0	1.5325	0.5805	1.0	1.0	1.0	2.0	2.0	2.0	4.0
HasCrCard	8000.0	0.7026	0.4571	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	8000.0	0.5199	0.4996	0.0	0.0	1.0	1.0	1.0	1.0	1.0
EstimatedSalary	8000.0	99790.1880	57520.5089	12.0	50857.0	99505.0	149216.0	179486.0	189997.0	199992.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	8000.0	3.0	France	4010.0	50.12
Gender	8000.0	2.0	Male	4396.0	54.95

result2 = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_raw_dataset},
    params={"min_percent_threshold": 30},
)

❌ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring the percentage share of each class and comparing those shares against the configured minimum threshold. In this run, the target variable Exited is shown across two classes, with class 0 representing 79.80% of rows and class 1 representing 20.20% of rows. The applied minimum percentage threshold is 30%, and the result table records a pass/fail outcome for each class relative to that threshold.

Key insights:

• Class 0 is dominant: Exited = 0 accounts for 79.80% of observations, making it the majority class in the target distribution.
• Class 1 falls below threshold: Exited = 1 represents 20.20% of observations, which is below the configured 30% minimum threshold and is therefore marked as Fail.
• Observed class gap is substantial: The difference between the two class shares is 59.60 percentage points, indicating a pronounced skew toward the non-exited class.

The results show a two-class target distribution with a clear concentration in Exited = 0 and a smaller representation of Exited = 1. Relative to the configured 30% threshold, only the majority class passes while the minority class fails. Collectively, the table and chart indicate that the target distribution is imbalanced under the test configuration used in this run.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	79.80%	Pass
1	20.20%	Fail

Figures

The output above shows that the validmind.data_validation.ClassImbalance test did not pass according to the value we set for min_percent_threshold.

To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.

As this is technically a different dataset, remember to first initialize a new ValidMind Dataset object to pass in as input as required by run_test():

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

# Pass the initialized `balanced_raw_dataset` as input into the test run
result = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_balanced_raw_dataset},
    params={"min_percent_threshold": 30},
)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring each class’s share of total records against a minimum percentage threshold. For the target variable Exited, the results show two classes, 0 and 1, each representing 50.00% of rows. The table and corresponding bar chart present an even split across the two classes, and both classes are marked as passing under the configured 30% threshold.

Key insights:

• Perfectly balanced target classes: Both Exited = 0 and Exited = 1 account for 50.00% of observations, indicating an equal class distribution in the evaluated dataset.
• All classes exceed threshold: Each class is above the configured minimum percentage threshold of 30%, and both are recorded as Pass in the test output.
• No minority class detected: The equal 50.00% / 50.00% split shows no under-represented target class in this test result.

The results indicate that the target variable is evenly distributed across its two classes in the evaluated dataset. No class falls below the configured minimum percentage threshold, and the test output shows a pass outcome for both classes. Collectively, these findings show no observed class distribution skew within the target variable under this test configuration.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

Utilize test output

You can utilize the output from a ValidMind test for further use, for example, if you want to remove highly correlated features. Removing highly correlated features helps make the model simpler, more stable, and easier to understand.

Below we demonstrate how to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.

First, we'll run validmind.data_validation.HighPearsonCorrelation with the balanced_raw_dataset we initialized previously as input as is for comparison with later runs:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant or highly collinear variable pairs. The result table reports the top feature pairs ranked by Pearson correlation coefficient, along with a pass/fail assessment against the configured absolute correlation threshold of 0.3. In this run, coefficients range from -0.1807 to 0.3631 across the reported pairs, and only one pair exceeds the threshold. The remaining listed relationships are all below the threshold and are marked as passing.

Key insights:

• One pair exceeds threshold: The pair (Age, Exited) has a Pearson correlation coefficient of 0.3631, which is the only reported relationship with absolute correlation above the 0.3 threshold and is therefore marked Fail.
• All other reported pairs pass: The other nine reported feature pairs have absolute correlation coefficients between 0.0294 and 0.1807, all below the configured threshold and marked Pass.
• Correlations are mostly weak: Aside from (Age, Exited), the largest absolute coefficients are -0.1807 for (IsActiveMember, Exited), -0.1787 for (Balance, NumOfProducts), and 0.1578 for (Balance, Exited), indicating relatively limited linear association among the remaining top-ranked pairs.
• Reported relationships include target associations: Several of the highest-ranked pairs involve Exited, including (Age, Exited), (IsActiveMember, Exited), (Balance, Exited), (NumOfProducts, Exited), and (CreditScore, Exited), with only (Age, Exited) crossing the threshold.

The reported correlation structure is concentrated in a single above-threshold pair, (Age, Exited), while all other listed pairwise relationships remain below the configured cutoff. Across the top reported correlations, most coefficients are small in magnitude, with the next strongest relationships remaining below 0.2 in absolute value. Taken together, the results show limited high linear dependence among the reported feature pairs, with one flagged relationship standing out from an otherwise low-correlation set.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3631	Fail
(IsActiveMember, Exited)	-0.1807	Pass
(Balance, NumOfProducts)	-0.1787	Pass
(Balance, Exited)	0.1578	Pass
(NumOfProducts, Exited)	-0.0583	Pass
(HasCrCard, IsActiveMember)	-0.0504	Pass
(NumOfProducts, IsActiveMember)	0.0463	Pass
(Age, NumOfProducts)	-0.0388	Pass
(Age, HasCrCard)	-0.0322	Pass
(CreditScore, Exited)	-0.0294	Pass

The output above shows that the test did not pass according to the value we set for max_threshold.

corr_result is an object of type TestResult. We can inspect the result object to see what the test has produced:

print(type(corr_result))
print("Result ID: ", corr_result.result_id)
print("Params: ", corr_result.params)
print("Passed: ", corr_result.passed)
print("Tables: ", corr_result.tables)

<class 'validmind.vm_models.result.result.TestResult'>
Result ID:  validmind.data_validation.HighPearsonCorrelation
Params:  {'max_threshold': 0.3}
Passed:  False
Tables:  [ResultTable]

Let's remove the highly correlated features and create a new VM dataset object.

We'll begin by checking out the table in the result and extracting a list of features that failed the test:

# Extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3631	Fail
1	(IsActiveMember, Exited)	-0.1807	Pass
2	(Balance, NumOfProducts)	-0.1787	Pass
3	(Balance, Exited)	0.1578	Pass
4	(NumOfProducts, Exited)	-0.0583	Pass
5	(HasCrCard, IsActiveMember)	-0.0504	Pass
6	(NumOfProducts, IsActiveMember)	0.0463	Pass
7	(Age, NumOfProducts)	-0.0388	Pass
8	(Age, HasCrCard)	-0.0322	Pass
9	(CreditScore, Exited)	-0.0294	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

Next, extract the feature names from the list of strings (example: (Age, Exited) > Age):

high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

Now, it's time to re-initialize the dataset with the highly correlated features removed.

Note the use of a different input_id. This allows tracking the inputs used when running each individual test.

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

Re-running the test with the reduced feature set should pass the test:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant or highly collinear variables. The results table lists the top feature pairs by absolute Pearson correlation coefficient together with Pass/Fail status under the configured threshold of 0.3. In this run, the reported coefficients range from -0.1807 to 0.1578, and all listed pairs are marked as Pass. The strongest observed relationships involve IsActiveMember and Exited, Balance and NumOfProducts, and Balance and Exited.

Key insights:

• All reported pairs passed: Every listed feature pair is marked Pass under the configured maximum threshold of 0.3, with no reported coefficient exceeding the threshold in absolute value.

• Correlation magnitudes are low: The largest absolute coefficient is 0.1807 for (IsActiveMember, Exited), followed by 0.1787 for (Balance, NumOfProducts) and 0.1578 for (Balance, Exited), indicating limited linear association among the top reported pairs.

• Both positive and negative relationships appear: The table includes negative correlations such as (IsActiveMember, Exited) at -0.1807 and (Balance, NumOfProducts) at -0.1787, alongside positive correlations such as (Balance, Exited) at 0.1578, showing mixed relationship direction across feature pairs.

• Several pairs are near zero: Lower-ranked reported pairs, including (CreditScore, Exited) at -0.0294, (CreditScore, IsActiveMember) at 0.0277, (IsActiveMember, EstimatedSalary) at 0.0272, and (HasCrCard, Exited) at -0.0265, show very weak linear relationships.

The reported correlation structure is uniformly below the configured threshold, with all top-ranked pairwise coefficients remaining modest in magnitude. The strongest observed associations are still below 0.2 in absolute value, while several reported pairs are close to zero. Taken together, the results indicate that the top reported feature pairs do not exhibit high linear correlation under this test configuration.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1807	Pass
(Balance, NumOfProducts)	-0.1787	Pass
(Balance, Exited)	0.1578	Pass
(NumOfProducts, Exited)	-0.0583	Pass
(HasCrCard, IsActiveMember)	-0.0504	Pass
(NumOfProducts, IsActiveMember)	0.0463	Pass
(CreditScore, Exited)	-0.0294	Pass
(CreditScore, IsActiveMember)	0.0277	Pass
(IsActiveMember, EstimatedSalary)	0.0272	Pass
(HasCrCard, Exited)	-0.0265	Pass

You can also plot the correlation matrix to visualize the new correlation between features:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.PearsonCorrelationMatrix",
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

Pearson Correlation Matrix

The Pearson Correlation Matrix test evaluates linear dependency between numerical variables in the dataset using pairwise Pearson correlation coefficients. The result is presented as a symmetric heat map across CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with diagonal values of 1.0 and off-diagonal values indicating the direction and magnitude of linear association. The observed off-diagonal correlations are generally small in magnitude, with the largest absolute values visible at -0.18 and 0.16. No pairwise correlation shown in the matrix approaches the high-correlation threshold of 0.7 referenced in the test description.

Key insights:

• Correlations are uniformly low: Off-diagonal coefficients are concentrated near zero, indicating weak linear relationships across the numerical variables included in the test.
• Largest negative relationship is limited: The most negative correlations shown are -0.18 between Balance and NumOfProducts and -0.18 between IsActiveMember and Exited, both remaining weak in magnitude.
• Largest positive relationship is modest: The highest positive correlation displayed is 0.16 between Balance and Exited, which is materially below the high-correlation threshold defined for this test.
• Exited has weak pairwise associations: Exited shows correlations of -0.03 with CreditScore, -0.02 with Tenure, 0.16 with Balance, -0.06 with NumOfProducts, -0.03 with HasCrCard, -0.18 with IsActiveMember, and 0.0 with EstimatedSalary, indicating no strong linear dependence with any single numerical variable.
• EstimatedSalary is largely independent of other variables: EstimatedSalary has correlations ranging from -0.02 to 0.03 with the other variables shown, making it one of the least linearly associated features in the matrix.

The correlation structure observed in this test is sparse and low in magnitude across the variables shown. The largest pairwise relationships are limited to weak associations, and no values indicate substantial linear redundancy under the threshold described by the test. Overall, the result shows that the numerical variables in this dataset maintain a high degree of pairwise independence on a linear correlation basis.

Figures

Documenting test results

Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it.

Every test result returned by the run_test() function has a .log() method that can be used to send the test results to the ValidMind Platform:

• When using run_documentation_tests(), documentation sections will be automatically populated with the results of all tests registered in the documentation template.
• When logging individual test results to the platform, you'll need to manually add those results to the desired section of the documentation.

To demonstrate how to add test results to your documentation, we'll populate the entire Data Preparation section of the documentation using the clean vm_raw_dataset_preprocessed dataset as input, and then document an additional individual result for the highly correlated dataset vm_balanced_raw_dataset.

Run and log multiple tests

run_documentation_tests() allows you to run multiple tests at once and automatically log the results to your documentation. Below, we'll run the tests using the previously initialized vm_raw_dataset_preprocessed as input — this will populate the entire Data Preparation section for every test that is part of the documentation template.

For this example, we'll pass in the following arguments:

• inputs: Any inputs to be passed to the tests.
• config: A dictionary <test_id>:<test_config> that allows configuring each test individually. Each test config requires the following:
  • params: Individual test parameters.
  • inputs: Individual test inputs. This overrides any inputs passed from the run_documentation_tests() function.

When including explicit configuration for individual tests, you'll need to specify the inputs even if they mirror what is included in your global configuration.

# Individual test config with inputs specified
test_config = {
    "validmind.data_validation.ClassImbalance": {
        "params": {"min_percent_threshold": 30},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "params": {"max_threshold": 0.3},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
}

# Global test config
tests_suite = vm.run_documentation_tests(
    inputs={
        "dataset": vm_raw_dataset_preprocessed,
    },
    config=test_config,
    section=["data_preparation"],
)

Test suite complete!

26/26 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Data Preparation

Data Preparation

▶ Test Result: Dataset Description (validmind.data_validation.DatasetDescription)

Dataset Description

The Dataset Description test evaluates the structure, completeness, and distinct-value profile of each dataset column. The results summarize 10 columns across numeric and categorical types, reporting count, missingness, and distinct-value counts for each field. All listed columns have 3,232 observed records, with missing-value statistics and distinct-value proportions shown for each variable.

Key insights:

• No missing values observed: Every column reports a count of 3,232 and a missing count of 0, corresponding to 0.0% missingness across all 10 variables.

• EstimatedSalary is fully unique: EstimatedSalary has 3,232 distinct values, equal to 100.0% of observations, making it the only column with a distinct count matching the full dataset size.

• Balance shows high granularity: Balance contains 2,184 distinct values, representing 67.57% of observations, which is the second-highest distinct-value ratio among the variables.

• Categorical fields have low cardinality: Geography has 3 distinct values, while Gender, HasCrCard, IsActiveMember, and Exited each have 2 distinct values. This indicates a compact categorical structure across these fields.

• Numeric fields vary in cardinality: Among numeric variables, CreditScore has 444 distinct values, Tenure has 11, and NumOfProducts has 4, showing substantial variation in value granularity across numeric inputs.

The dataset description indicates a fully populated dataset with no observed missingness in any reported field. Variable cardinality differs materially across columns, ranging from binary categorical variables to fully unique numeric values in EstimatedSalary, with Balance also showing relatively high distinctness. Overall, the results describe a dataset composed of complete records with a mix of low-cardinality categorical variables and numeric variables with both discrete and highly granular value distributions.

Tables

Dataset Description

Name	Type	Count	Missing	Missing %	Distinct	Distinct %
CreditScore	Numeric	3232.0	0	0.0	444	0.1374
Geography	Categorical	3232.0	0	0.0	3	0.0009
Gender	Categorical	3232.0	0	0.0	2	0.0006
Tenure	Numeric	3232.0	0	0.0	11	0.0034
Balance	Numeric	3232.0	0	0.0	2184	0.6757
NumOfProducts	Numeric	3232.0	0	0.0	4	0.0012
HasCrCard	Categorical	3232.0	0	0.0	2	0.0006
IsActiveMember	Categorical	3232.0	0	0.0	2	0.0006
EstimatedSalary	Numeric	3232.0	0	0.0	3232	1.0000
Exited	Categorical	3232.0	0	0.0	2	0.0006

▶ Test Result: Class Imbalance (validmind.data_validation.ClassImbalance)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring the percentage of records in each class and comparing those percentages to the configured minimum threshold. In this result, the target variable Exited contains two classes, 0 and 1, each representing 50.00% of rows. The table and accompanying bar chart show identical class proportions, and both classes are marked as Pass against the 30% minimum threshold.

Key insights:

• Perfectly even class split: The Exited target is distributed evenly, with class 0 at 50.00% and class 1 at 50.00% of the dataset.
• All classes exceed threshold: Both observed classes pass the configured minimum percentage threshold of 30%, with each class exceeding the threshold by 20 percentage points.
• No underrepresented target class: The test identifies no class below the threshold, and the results show no imbalance between the two target categories.

The results show a fully balanced binary target distribution for Exited under the configured test criterion. Both classes have equal representation and pass the threshold check, indicating that this test did not detect class underrepresentation in the dataset.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

▶ Test Result: Duplicates (validmind.data_validation.Duplicates)

✅ Duplicates

The Duplicates test evaluates whether the dataset contains exact duplicate rows that could affect data quality and the representation of training observations. The results table reports both the absolute count of duplicate rows and their share of total rows for the dataset assessed. In this run, the table shows 0 duplicate rows and a duplicate-row percentage of 0.0%.

Key insights:

• No duplicate rows detected: The dataset contains 0 duplicate rows based on the executed duplicate check.
• Duplicate share is zero: The percentage of rows identified as duplicates is 0.0%, indicating no duplicated observations in the assessed dataset.

The test result shows no exact duplicate rows in the dataset under evaluation. Both the count-based and percentage-based outputs are zero, indicating that duplication was not observed in this test run.

Tables

Duplicate Rows Results for Dataset

Number of Duplicates	Percentage of Rows (%)
0	0.0

▶ Test Result: High Cardinality (validmind.data_validation.HighCardinality)

✅ High Cardinality

The High Cardinality test evaluates the number of unique values in categorical columns to identify features with large numbers of distinct categories. The results table reports each categorical column alongside its number of distinct values, percentage of distinct values, and pass/fail outcome against the configured threshold. In this run, two categorical columns were evaluated: Geography and Gender. Both columns are recorded as passing, with 3 and 2 distinct values respectively and corresponding distinct-value percentages of 0.0928% and 0.0619%.

Key insights:

• All evaluated columns passed: Both categorical features assessed in this test, Geography and Gender, have pass statuses in the result table, indicating that neither exceeded the test threshold for distinct values.

• Low distinct counts across categories: Geography contains 3 distinct values and Gender contains 2 distinct values. The observed cardinality is limited for both evaluated categorical fields.

• Distinct-value percentages are minimal: The percentage of distinct values is 0.0928% for Geography and 0.0619% for Gender, showing very low distinct-value concentration relative to the dataset size.

The test results show that the categorical columns evaluated in this run exhibit low cardinality based on both absolute distinct counts and distinct-value percentages. No evaluated categorical feature failed the threshold-based assessment. Across the two reported columns, the observed category structure is limited and uniform in passing status.

Tables

Column	Number of Distinct Values	Percentage of Distinct Values (%)	Pass/Fail
Geography	3	0.0928	Pass
Gender	2	0.0619	Pass

▶ Test Result: Missing Values (validmind.data_validation.MissingValues)

✅ Missing Values

The Missing Values test evaluates dataset completeness by measuring the proportion of missing entries in each feature against the configured threshold. The results are presented at the column level for 10 variables, showing the count of missing values, the corresponding percentage of missing values, and the pass/fail outcome for each feature. In this test run, CreditScore, Geography, Gender, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited each report 0 missing values and 0.0% missingness.

Key insights:

• No missing values detected: All 10 evaluated columns show 0 missing values, corresponding to 0.0% missingness in every feature.
• Uniform pass status across features: Every column is marked as Pass, indicating that each feature remains within the test threshold for allowable missingness.
• Completeness spans predictors and target: The zero-missingness result applies across all listed variables, including both input features and the Exited target field.

The test results show complete observed data coverage across the evaluated dataset fields, with no missing entries recorded in any column. The consistent 0.0% missingness and universal pass status indicate that missing-value incidence was not present in this test run for either predictors or the target variable.

Tables

Column	Number of Missing Values	Percentage of Missing Values (%)	Pass/Fail
CreditScore	0	0.0	Pass
Geography	0	0.0	Pass
Gender	0	0.0	Pass
Tenure	0	0.0	Pass
Balance	0	0.0	Pass
NumOfProducts	0	0.0	Pass
HasCrCard	0	0.0	Pass
IsActiveMember	0	0.0	Pass
EstimatedSalary	0	0.0	Pass
Exited	0	0.0	Pass

▶ Test Result: Skewness (validmind.data_validation.Skewness)

❌ Skewness

The Skewness test evaluates the asymmetry of numerical feature distributions by comparing each column’s skewness value against the defined maximum threshold of 1. The results table reports skewness values and pass/fail outcomes for eight numerical columns in the dataset. Observed skewness values range from -0.9067 to 1.2233, with seven columns passing the threshold test and one column failing. The reported results cover CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited.

Key insights:

• Single threshold breach observed: NumOfProducts is the only column that fails the test, with skewness of 1.2233, exceeding the maximum threshold of 1.
• Most variables show low skewness: Seven of eight numerical columns pass the test, with skewness values clustered between -0.9067 and 0.1627 for the passing variables.
• Near-symmetric continuous variables: CreditScore (-0.0708), Tenure (0.0414), and EstimatedSalary (-0.0142) are all close to zero, indicating minimal distributional asymmetry in these columns.
• Largest negative skew among passing variables: HasCrCard records skewness of -0.9067, representing the most pronounced left-skewed distribution among columns that remain within the threshold.
• Target variable is symmetric by this measure: Exited has a reported skewness of 0.0 and passes the test.

The results indicate that skewness is limited across most numerical columns, with only one feature exceeding the configured threshold. The most material concentration of asymmetry is in NumOfProducts, while the remaining variables remain within the defined tolerance range. Overall, the dataset exhibits predominantly low to moderate skewness under this test configuration.

Tables

Skewness Results for Dataset

Column	Skewness	Pass/Fail
CreditScore	-0.0708	Pass
Tenure	0.0414	Pass
Balance	-0.2568	Pass
NumOfProducts	1.2233	Fail
HasCrCard	-0.9067	Pass
IsActiveMember	0.1627	Pass
EstimatedSalary	-0.0142	Pass
Exited	0.0000	Pass

▶ Test Result: Unique Rows (validmind.data_validation.UniqueRows)

❌ Unique Rows

The UniqueRows test evaluates dataset diversity by measuring the number and percentage of unique values present in each column relative to the total row count. The results are reported at the column level, showing the count of unique values, the corresponding percentage of unique values, and a pass/fail outcome for each field. In this run, three columns passed the test and seven columns failed, with observed uniqueness percentages ranging from 0.0619% to 100.0%.

Key insights:

• Uniqueness is concentrated in three columns: EstimatedSalary shows 3,232 unique values and 100.0% uniqueness, Balance shows 2,184 unique values and 67.5743% uniqueness, and CreditScore shows 444 unique values and 13.7376% uniqueness. These are the only columns that passed the test.

• Most columns have very low uniqueness: Seven columns failed the test, with uniqueness percentages between 0.0619% and 0.3403%. Geography has 3 unique values (0.0928%), Gender 2 (0.0619%), Tenure 11 (0.3403%), NumOfProducts 4 (0.1238%), HasCrCard 2 (0.0619%), IsActiveMember 2 (0.0619%), and Exited 2 (0.0619%).

• EstimatedSalary is fully distinct: EstimatedSalary has 3,232 unique values out of 3,232 rows, corresponding to 100.0% uniqueness. This is the highest observed uniqueness level in the dataset.

• Binary and low-cardinality fields dominate failures: The failed columns are primarily variables with 2 to 11 unique values. Four columns—Gender, HasCrCard, IsActiveMember, and Exited—each contain 2 unique values and share the same uniqueness percentage of 0.0619%.

The results show a clear split between a small set of high-uniqueness columns and a larger set of low-cardinality columns. EstimatedSalary, Balance, and CreditScore meet the test threshold, while the remaining seven fields do not. Overall, the observed uniqueness profile is dominated by repeated values across categorical and discrete columns, with full uniqueness present only in EstimatedSalary.

Tables

Column	Number of Unique Values	Percentage of Unique Values (%)	Pass/Fail
CreditScore	444	13.7376	Pass
Geography	3	0.0928	Fail
Gender	2	0.0619	Fail
Tenure	11	0.3403	Fail
Balance	2184	67.5743	Pass
NumOfProducts	4	0.1238	Fail
HasCrCard	2	0.0619	Fail
IsActiveMember	2	0.0619	Fail
EstimatedSalary	3232	100.0000	Pass
Exited	2	0.0619	Fail

▶ Test Result: Too Many Zero Values (validmind.data_validation.TooManyZeroValues)

❌ Too Many Zero Values

The TooManyZeroValues test evaluates numerical columns for zero-value prevalence relative to a defined threshold. The result table reports the row count, zero-value count, zero-value percentage, and pass/fail outcome for each assessed numerical variable. Four variables are listed in the output—Tenure, Balance, HasCrCard, and IsActiveMember—and all four are marked as failing the test. Reported zero-value percentages range from 3.9604% to 54.0532% across 3,232 rows for each variable.

Key insights:

• All assessed variables failed: Each of the four reported numerical variables exceeds the configured zero-value threshold and is marked as Fail in the output.

• IsActiveMember has the highest zero concentration: IsActiveMember contains 1,747 zero values out of 3,232 rows, corresponding to 54.0532%, the highest proportion among the variables tested.

• Balance also shows substantial zero prevalence: Balance contains 1,049 zero values, representing 32.4567% of observations, making it the second-highest zero-rate in the result set.

• Binary-style fields show large zero shares: HasCrCard and IsActiveMember record zero-value percentages of 29.3626% and 54.0532%, respectively, indicating that zeros account for a material share of observations in both variables.

• Tenure has the lowest reported zero rate: Tenure records 128 zero values, or 3.9604% of 3,232 rows, which is the smallest zero-value proportion among the reported variables, though it still fails the test.

The test results show that zero values are present above the defined threshold in every numerical variable included in the output. The most concentrated zero presence appears in IsActiveMember and Balance, while HasCrCard also shows a substantial share of zeros. Tenure has a comparatively lower zero rate, but it remains above the test threshold and is therefore also classified as failing.

Tables

Variable	Row Count	Number of Zero Values	Percentage of Zero Values (%)	Pass/Fail
Tenure	3232	128	3.9604	Fail
Balance	3232	1049	32.4567	Fail
HasCrCard	3232	949	29.3626	Fail
IsActiveMember	3232	1747	54.0532	Fail

▶ Test Result: IQR Outliers Table (validmind.data_validation.IQROutliersTable)

IQR Outliers Table

The IQROutliersTable test evaluates numerical features for observations outside the interquartile range-based bounds and summarizes the frequency and values of those outliers. The results table reports outlier counts and distributional summaries for the features in which outliers were detected. In this run, outliers are reported for CreditScore and NumOfProducts, including the total number of outlying observations and the minimum, quartile, median, and maximum values among those outliers.

Key insights:

• Outliers are concentrated in two features: Outliers were identified only for CreditScore and NumOfProducts in the reported results, with counts of 7 and 44 respectively.
• NumOfProducts has the highest outlier count: NumOfProducts contains 44 outliers, materially more than CreditScore with 7. All reported outlier summary statistics for NumOfProducts are equal to 4, indicating all detected outliers take the same value.
• CreditScore outliers are low-valued and tightly clustered: CreditScore outliers range from 350 to 365, with the 25th percentile and median both at 350 and the 75th percentile at 354.5. Relative to the reported mean of 648.1952, the outlier values are concentrated well below the feature average.

The results show that IQR-based outliers are present in two numerical features, with the larger concentration occurring in NumOfProducts. The NumOfProducts outliers are uniform at a value of 4, while CreditScore outliers are confined to a narrow low-end range between 350 and 365. Collectively, the table indicates that the detected outlier behavior is limited in scope and feature-specific within the reported output.

Tables

Summary of Outliers Detected by IQR Method

Variable	Total Count of Outliers	Mean Value of Variable	Minimum Outlier Value	Outlier Value at 25th Percentile	Outlier Value at 50th Percentile	Outlier Value at 75th Percentile	Maximum Outlier Value
CreditScore	7	648.1952	350	350.0	350.0	354.5	365
NumOfProducts	44	1.5111	4	4.0	4.0	4.0	4

▶ Test Result: IQR Outliers Bar Plot (validmind.data_validation.IQROutliersBarPlot)

IQR Outliers Bar Plot

The IQROutliersBarPlot test evaluates the distribution of IQR-based outliers across percentile bands for numeric features. The result is presented as bar plots for CreditScore and NumOfProducts, with outlier counts shown separately for the 0–25, 25–50, 50–75, and 75–100 percentile ranges. In these plots, CreditScore shows outlier counts in the upper-half percentile bands, while NumOfProducts shows outliers only in the highest percentile band.

Key insights:

• Outliers are concentrated in upper percentiles: Both plotted features show no outliers in the 0–25 and 25–50 percentile bands. All detected outliers occur in the 50–75 or 75–100 ranges.

• NumOfProducts outliers are exclusively highest-band: NumOfProducts has all observed outliers in the 75–100 percentile range, with an outlier count of approximately 44 and no outliers in the lower three percentile bands.

• CreditScore shows fewer, split upper-tail outliers: CreditScore has a smaller number of outliers, with about 5 in the 50–75 percentile band and about 2 in the 75–100 percentile band, and none in the lower half of the distribution.

• Outlier magnitude differs substantially by feature: The observed outlier count for NumOfProducts is materially higher than for CreditScore, indicating that detected outliers are much more concentrated in NumOfProducts within the plotted results.

The results show that detected IQR-based outliers are confined to the upper portion of the distributions for both displayed features. NumOfProducts exhibits a pronounced concentration of outliers in the highest percentile band, whereas CreditScore shows a smaller number of outliers distributed across the two upper percentile ranges. Collectively, the plots indicate that outlier behavior is feature-specific and concentrated in upper-tail values within the tested variables.

Figures

▶ Test Result: Descriptive Statistics (validmind.data_validation.DescriptiveStatistics)

Descriptive Statistics

The Descriptive Statistics test evaluates the distributional characteristics of numerical and categorical variables in the dataset. The results present summary statistics for seven numerical variables and frequency-based summaries for two categorical variables across 3,232 records. For numerical fields, the output includes counts, central tendency, dispersion, and percentile values, while the categorical summary reports the number of unique values, the most frequent category, and its share of observations.

Key insights:

• No missingness in reported fields: All listed numerical and categorical variables have a count of 3,232, indicating complete population coverage for the reported fields in this dataset.

• Balance shows pronounced lower-tail concentration: Balance has a minimum and 25th percentile of 0.0, while the median is 102,606 and the mean is 81,323.4531. This indicates that at least one quarter of observations are at zero, with the distribution shifting sharply upward above the lower quartile.

• EstimatedSalary is broadly dispersed but centered: EstimatedSalary ranges from 12 to 199,863, with mean 99,900.9304 and median 99,579. The close alignment of mean and median, alongside a standard deviation of 57,041.5497, indicates wide spread without a large shift between average and midpoint.

• CreditScore is relatively centered around the median: CreditScore has mean 648.1952 and median 650, with an interquartile range from 578 to 719 and observed values spanning 350 to 850. The proximity of mean and median indicates limited asymmetry in the central portion of the distribution.

• Product holding is concentrated at lower counts: NumOfProducts has a median of 1, a 75th percentile of 2, and a maximum of 4, with mean 1.5111. Most observations are concentrated in the lower product-count range.

• Categorical concentration is moderate: Geography contains 3 unique values, with France as the most frequent category at 1,477 observations (45.7%). Gender contains 2 unique values, with Male as the most frequent category at 1,629 observations (50.4%), indicating no single category dominates either variable.

The descriptive statistics show complete coverage for all reported variables and a mix of distribution shapes across the dataset. Numerical variables such as CreditScore and EstimatedSalary exhibit mean and median values that are closely aligned, while Balance displays a distinct concentration at zero with substantially higher values above the lower quartile. The categorical variables show limited cardinality with moderate concentration in the top category, but neither reported field is dominated by a single category.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	3232.0	648.1952	100.4718	350.0	578.0	650.0	719.0	782.0	818.0	850.0
Tenure	3232.0	5.0090	2.8936	0.0	3.0	5.0	8.0	9.0	10.0	10.0
Balance	3232.0	81323.4531	61631.1885	0.0	0.0	102606.0	129313.0	150483.0	163804.0	250898.0
NumOfProducts	3232.0	1.5111	0.6694	1.0	1.0	1.0	2.0	2.0	3.0	4.0
HasCrCard	3232.0	0.7064	0.4555	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	3232.0	0.4595	0.4984	0.0	0.0	0.0	1.0	1.0	1.0	1.0
EstimatedSalary	3232.0	99900.9304	57041.5497	12.0	51796.0	99579.0	148498.0	178833.0	188430.0	199863.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	3232.0	3.0	France	1477.0	45.7
Gender	3232.0	2.0	Male	1629.0	50.4

▶ Test Result: Pearson Correlation Matrix (validmind.data_validation.PearsonCorrelationMatrix)

Pearson Correlation Matrix

The Pearson Correlation Matrix test evaluates linear dependency among numerical variables by measuring pairwise Pearson correlation coefficients and presenting them in a heat map. The result shows correlations across CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with coefficients ranging from -0.18 to 0.16 outside the diagonal. The diagonal contains 1.0 self-correlations, and the off-diagonal cells are concentrated near zero, with the largest observed magnitudes appearing between Balance and NumOfProducts (-0.18), IsActiveMember and Exited (-0.18), and Balance and Exited (0.16).

Key insights:

• Correlations are uniformly low: All off-diagonal Pearson coefficients shown in the matrix fall within a narrow range of approximately -0.18 to 0.16, indicating weak linear relationships across the numerical variables included in the test.
• No high-correlation pairs observed: The heat map does not display any pairwise correlations approaching the 0.7 absolute-value threshold described in the test methodology. No variable pair is highlighted as strongly linearly dependent.
• Largest negative relationships are limited in magnitude: The most negative coefficients shown are -0.18 for Balance versus NumOfProducts and -0.18 for IsActiveMember versus Exited. These are the strongest inverse relationships in the matrix, but both remain weak in absolute terms.
• Exited shows weak association with other variables: Correlations between Exited and the other numerical variables range from -0.18 to 0.16. The largest positive value is with Balance (0.16), while the largest negative value is with IsActiveMember (-0.18).
• EstimatedSalary is nearly uncorrelated with all variables: EstimatedSalary shows coefficients between -0.02 and 0.03 with the other variables, including 0.0 with Exited and NumOfProducts, indicating minimal linear association in this matrix.

Overall, the correlation structure is sparse and low in magnitude, with no evidence of strong pairwise linear dependency among the numerical variables shown. The most pronounced relationships are limited to weak correlations around ±0.18, and the target variable Exited also exhibits only weak linear associations with the other features. Collectively, the result indicates limited redundancy based on linear correlation in the tested dataset.

Figures

▶ Test Result: High Pearson Correlation (validmind.data_validation.HighPearsonCorrelation)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant variables or multicollinearity. The results table reports the top feature pairs ranked by Pearson correlation coefficient, along with Pass/Fail status using a maximum threshold of 0.3. In this run, the reported coefficients range from -0.1807 to 0.1578, and all listed pairs are marked as Pass.

Key insights:

• No pair exceeds threshold: All reported absolute correlation coefficients are below the 0.3 threshold. Every listed feature pair therefore receives a Pass result in the test output.
• Largest correlation is modest: The strongest reported relationship is between IsActiveMember and Exited with a coefficient of -0.1807. The next largest absolute values are -0.1787 for Balance and NumOfProducts, and 0.1578 for Balance and Exited.
• Reported relationships are weak: The top-ranked correlations all remain close to zero in magnitude, with the full reported range spanning from -0.1807 to 0.1578. Several pairs, including CreditScore with Exited (-0.0294) and HasCrCard with Exited (-0.0265), show especially small linear association.
• Both positive and negative associations appear: The output includes negative coefficients such as IsActiveMember with Exited (-0.1807) and Balance with NumOfProducts (-0.1787), as well as positive coefficients such as Balance with Exited (0.1578) and NumOfProducts with IsActiveMember (0.0463). This indicates mixed directional relationships across the reported feature pairs.

The reported correlation structure shows no high pairwise linear dependence under the specified 0.3 threshold. The strongest observed associations are modest in magnitude, and all listed feature pairs pass the test. Overall, the results indicate that the top reported pairwise Pearson correlations in this dataset are weak and distributed across both positive and negative directions.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1807	Pass
(Balance, NumOfProducts)	-0.1787	Pass
(Balance, Exited)	0.1578	Pass
(NumOfProducts, Exited)	-0.0583	Pass
(HasCrCard, IsActiveMember)	-0.0504	Pass
(NumOfProducts, IsActiveMember)	0.0463	Pass
(CreditScore, Exited)	-0.0294	Pass
(CreditScore, IsActiveMember)	0.0277	Pass
(IsActiveMember, EstimatedSalary)	0.0272	Pass
(HasCrCard, Exited)	-0.0265	Pass

Run and log an individual test

Next, we'll use the previously initialized vm_balanced_raw_dataset (that still has a highly correlated Age column) as input to run an individual test, then log the result to the ValidMind Platform.

When running individual tests, you can use a custom result_id to tag the individual result with a unique identifier:

• This result_id can be appended to test_id with a : separator.
• The balanced_raw_dataset result identifier will correspond to the balanced_raw_dataset input, the dataset that still has the Age column.

result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)
result.log()

❌ High Pearson Correlation Balanced Raw Dataset

The High Pearson Correlation test evaluates pairwise linear relationships between features to identify highly correlated variable pairs that may indicate redundancy or multicollinearity. The results table reports the top feature pairs by Pearson correlation coefficient together with a Pass/Fail assessment against the configured absolute threshold of 0.3. In this run, 10 feature pairs are listed, with coefficients spanning from -0.1807 to 0.3631. Only one pair exceeds the threshold and is marked Fail, while the remaining nine pairs are marked Pass.

Key insights:

• Single threshold breach observed: The pair (Age, Exited) has the highest reported correlation at 0.3631, exceeding the 0.3 threshold and receiving a Fail designation.
• Remaining correlations are low: All other reported feature pairs have absolute correlation coefficients below 0.2, including (IsActiveMember, Exited) at -0.1807, (Balance, NumOfProducts) at -0.1787, and (Balance, Exited) at 0.1578.
• Correlations are concentrated near zero: Several listed relationships are very weak in magnitude, including (NumOfProducts, Exited) at -0.0583, (HasCrCard, IsActiveMember) at -0.0504, (NumOfProducts, IsActiveMember) at 0.0463, (Age, NumOfProducts) at -0.0388, (Age, HasCrCard) at -0.0322, and (CreditScore, Exited) at -0.0294.

The reported correlation structure is limited to a single feature pair above the configured threshold, namely (Age, Exited) at 0.3631. All other displayed pairwise relationships remain below the threshold and are comparatively small in magnitude, with the strongest of these not exceeding an absolute value of 0.1807. Overall, the table shows one flagged linear relationship alongside predominantly weak pairwise correlations among the top reported pairs.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3631	Fail
(IsActiveMember, Exited)	-0.1807	Pass
(Balance, NumOfProducts)	-0.1787	Pass
(Balance, Exited)	0.1578	Pass
(NumOfProducts, Exited)	-0.0583	Pass
(HasCrCard, IsActiveMember)	-0.0504	Pass
(NumOfProducts, IsActiveMember)	0.0463	Pass
(Age, NumOfProducts)	-0.0388	Pass
(Age, HasCrCard)	-0.0322	Pass
(CreditScore, Exited)	-0.0294	Pass

2026-10-02 20:32:39,857 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset does not exist in model's document

Note the output returned indicating that a test-driven block doesn't currently exist in your documentation for this particular test ID.

That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.

Add individual test results to documentation

With the test results logged, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation (Learn more: Work with test results):

1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

2. In the left sidebar that appears for your model, click Development under Documents.

3. Locate the Data Preparation section and click on 2.3. Correlations and Interactions to expand that section.

4. Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a + button appears, indicating that you can insert a new block.

   
   
   

5. Click + and then select Test-Driven Block under FROM LIBRARY:

   • Click on VM Library under TEST-DRIVEN in the left sidebar.
   • In the search bar, type in HighPearsonCorrelation.
   • Select HighPearsonCorrelation:balanced_raw_dataset as the test.

   A preview of the test gets shown:

   
   
   

6. Finally, click Insert 1 Test Result to Document to add the test result to the documentation.

   Confirm that the individual results for the high correlation test has been correctly inserted into section 2.3. Correlations and Interactions of the documentation.

7. Finalize the documentation by editing the test result's description block to explain the changes you made to the raw data and the reasons behind them as shown in the screenshot below:

   

Running model evaluation tests

So far, we've focused on the data assessment and pre-processing that usually occurs prior to any models being built. Now, let's instead assume we have already built a model and we want to incorporate some model results into our documentation.

Train simple logistic regression model

Using ValidMind tests, we'll train a simple logistic regression model on our dataset and evaluate its performance by using the LogisticRegression class from the sklearn.linear_model.

To start, let's grab the first few rows from the balanced_raw_no_age_df dataset with the highly correlated features removed we initialized earlier:

balanced_raw_no_age_df.head()

	CreditScore	Geography	Gender	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
6985	655	France	Female	2	0.00	1	1	1	71047.51	0
223	706	Spain	Female	7	0.00	2	1	1	110899.30	0
6244	613	Spain	Male	0	0.00	2	0	1	133897.32	0
1793	572	France	Male	2	140431.98	1	1	0	26450.57	1
6352	542	France	Female	1	0.00	1	1	1	178256.58	1

Before training the model, we need to encode the categorical features in the dataset:

• Use the OneHotEncoder class from the sklearn.preprocessing module to encode the categorical features.
• The categorical features in the dataset are Geography and Gender.

balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
6985	655	2	0.00	1	1	1	71047.51	0	False	False	False
223	706	7	0.00	2	1	1	110899.30	0	False	True	False
6244	613	0	0.00	2	0	1	133897.32	0	False	True	True
1793	572	2	140431.98	1	1	0	26450.57	1	False	False	True
6352	542	1	0.00	1	1	1	178256.58	1	False	False	False

We'll split our preprocessed dataset into training and testing, to help assess how well the model generalizes to unseen data:

• We start by dividing our balanced_raw_no_age_df dataset into training and test subsets using train_test_split, with 80% of the data allocated to training (train_df) and 20% to testing (test_df).
• From each subset, we separate the features (all columns except "Exited") into X_train and X_test, and the target column ("Exited") into y_train and y_test.

from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

Then using GridSearchCV, we'll find the best-performing hyperparameters or settings and save them:

from sklearn.linear_model import LogisticRegression

# Logistic Regression grid params
log_reg_params = {
    "penalty": ["l1", "l2"],
    "C": [0.001, 0.01, 0.1, 1, 10, 100, 1000],
    "solver": ["liblinear"],
}

# Grid search for Logistic Regression
from sklearn.model_selection import GridSearchCV

grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)
grid_log_reg.fit(X_train, y_train)

# Logistic Regression best estimator
log_reg = grid_log_reg.best_estimator_

Initialize ValidMind datasets

The last step for evaluating the model's performance is to initialize the ValidMind Dataset and Model objects in preparation for assigning model predictions to each dataset.

# Initialize the datasets into their own dataset objects
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

Initialize a ValidMind model

You'll also need to initialize a ValidMind model object (vm_model) that can be passed to other functions for analysis and tests on the data for each of our three models.

• Despite the naming convention, ValidMind model objects can be any type of record you want to test, document, validate, or monitor with the ValidMind Library.
• From classical statistical and machine learning models, to generative and agentic AI systems and more, the ValidMind model object provides a consistent wrapper around your record so it can be passed as a unified input to any ValidMind test or test suite, with results sent directly to the ValidMind Platform.

Initialize your model object with vm.init_model():

# Register the model
vm_model = vm.init_model(log_reg, input_id="log_reg_model_v1")

Assign predictions

Once the model has been registered you can assign predictions to the training and testing datasets.

• The assign_predictions() method from the Dataset object can link existing predictions to any number of models.
• This method links the model's class prediction values and probabilities to our vm_train_ds and vm_test_ds datasets.

If no prediction values are passed, the method will compute predictions automatically:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

2026-10-02 20:32:40,836 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-10-02 20:32:40,837 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-10-02 20:32:40,837 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-10-02 20:32:40,839 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-10-02 20:32:40,840 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-10-02 20:32:40,841 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-10-02 20:32:40,841 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-10-02 20:32:40,842 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Run the model evaluation tests

In this next example, we'll focus on running the tests within the Model Development section of the documentation. Only tests associated with this section will be executed, and the corresponding results will be updated in the documentation.

• Note the additional config that is passed to run_documentation_tests() — this allows you to override inputs or params in certain tests.
• In our case, we want to explicitly use the vm_train_ds for the validmind.model_validation.sklearn.ClassifierPerformance:in_sample test, since it's supposed to run on the training dataset and not the test dataset.

test_config = {
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "dataset": vm_train_ds,
            "model": vm_model,
        },
    }
}
results = vm.run_documentation_tests(
    section=["model_development"],
    inputs={
        "dataset": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds
        "model": vm_model,
        "datasets": (
            vm_train_ds,
            vm_test_ds,
        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds
    },
    config=test_config,
)

Test suite complete!

34/34 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Development

Model Development

▶ Test Result: Model Metadata (validmind.model_validation.ModelMetadata)

Model Metadata

The ModelMetadata test compares core metadata fields across models to summarize their implementation characteristics. The result is presented as a metadata table covering modeling technique, modeling framework, framework version, and programming language. In this test output, one model entry is shown with values populated for each of these fields.

Key insights:

• Single model entry documented: The result contains one metadata record, indicating that the comparison output in this test run consists of a single model entry rather than multiple models.
• Framework identified as sklearn: The modeling framework is listed as sklearn, providing the implementation framework captured in the metadata.
• Framework version explicitly recorded: The framework version is reported as 1.7.2, showing that version information is available in the test output.
• Programming language captured: The programming language is listed as Python, indicating that the language field is present and populated.
• Modeling technique specified: The modeling technique is recorded as SKlearnModel, showing that the model type field is available in the metadata summary.

The test result shows that the metadata summary contains populated values for all reported fields in the output: modeling technique, framework, framework version, and programming language. The result is limited to a single model record, so the output documents the model’s implementation attributes but does not display cross-model differences within this test run.

Tables

Modeling Technique	Modeling Framework	Framework Version	Programming Language
SKlearnModel	sklearn	1.7.2	Python

▶ Test Result: Dataset Split (validmind.data_validation.DatasetSplit)

Dataset Split

The DatasetSplit test evaluates the distribution of observations across the available model datasets. The result table reports the absolute size and share of the total sample for each split used in model development. In this run, the table includes training and testing datasets, along with the total combined sample size.

Key insights:

• Training data is the majority split: train_dataset_final contains 2,585 observations, representing 79.98% of the total dataset.
• Testing data accounts for one-fifth: test_dataset_final contains 647 observations, representing 20.02% of the total dataset.
• Two-way split is documented: The reported datasets consist of training and testing only, with a total sample size of 3,232 observations.

The dataset split result shows a total of 3,232 observations allocated between a training sample of 2,585 records and a testing sample of 647 records. The distribution is concentrated in the training dataset at 79.98%, with the remaining 20.02% assigned to testing. No validation dataset is reported in the provided result.

Tables

Dataset	Size	Proportion
train_dataset_final	2585	79.98%
test_dataset_final	647	20.02%
Total	3232	100%

▶ Test Result: Population Stability Index (validmind.model_validation.sklearn.PopulationStabilityIndex)

Population Stability Index

The Population Stability Index (PSI) test evaluates the stability of distributions between the reference and comparison datasets. The results compare the bin-level population shares for train_dataset_final and test_dataset_final across 10 bins, with counts, percentages, and PSI contributions reported for each bin. The table and plot show how observations are distributed across bins in the initial and new datasets and how each bin contributes to the total PSI. The total PSI reported for the comparison is 0.0178.

Key insights:

• Low overall PSI: The total PSI is 0.0178, indicating a small aggregate difference between the distributions of train_dataset_final and test_dataset_final in this test output.

• Drift is concentrated in the final bin: Bin 9 has the largest individual PSI contribution at 0.0070. Its population share increases from 4.7969% in the initial dataset to 6.8006% in the new dataset.

• Mid-distribution bins are closely aligned: Bins 3, 4, and 5 show very small PSI contributions of 0.0004, 0.0000, and 0.0000, respectively. Their population percentages are nearly identical between datasets, including 14.7776% versus 14.6832% in bin 4 and 13.5783% versus 13.6012% in bin 5.

• Early-bin differences are modest: Bins 0 through 2 contribute PSI values of 0.0025, 0.0022, and 0.0018. The largest percentage differences in this range are in bin 0, where the share declines from 6.4603% to 5.2550%, and in bin 1, where it rises from 9.0522% to 10.5100%.

• Several upper-middle bins shift downward: Bins 6 and 8 have lower shares in the new dataset than in the initial dataset, moving from 12.3017% to 10.9737% and from 6.0735% to 5.1005%, respectively. Their PSI contributions are 0.0015 and 0.0017.

The PSI results show that the compared distributions are closely aligned overall, with a total PSI of 0.0178 and minimal contribution from most bins. The largest observed deviation is concentrated in bin 9, while the center of the distribution remains nearly unchanged across datasets. Taken together, the results indicate that distributional differences in this comparison are limited and localized rather than broad-based across the full bin structure.

Tables

Population Stability Index for train_dataset_final and test_dataset_final Datasets

Bin	Count Initial	Percent Initial (%)	Count New	Percent New (%)	PSI
0	167	6.4603	34	5.2550	0.0025
1	234	9.0522	68	10.5100	0.0022
2	308	11.9149	68	10.5100	0.0018
3	328	12.6886	87	13.4467	0.0004
4	382	14.7776	95	14.6832	0.0000
5	351	13.5783	88	13.6012	0.0000
6	318	12.3017	71	10.9737	0.0015
7	216	8.3559	59	9.1190	0.0007
8	157	6.0735	33	5.1005	0.0017
9	124	4.7969	44	6.8006	0.0070
Total	2585	100.0000	647	100.0000	0.0178

Figures

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

The Confusion Matrix test evaluates classification performance by comparing predicted labels with observed labels and displaying the counts of true negatives, false positives, false negatives, and true positives. The heatmap shows a binary classification result with 203 true negatives, 110 false positives, 129 false negatives, and 205 true positives. Correct classifications are concentrated in the true positive and true negative cells, while misclassifications are split between false negatives and false positives.

Key insights:

• Correct classifications slightly exceed errors: The model records 408 correct predictions in total, consisting of 205 true positives and 203 true negatives, compared with 239 total misclassifications.
• Positive and negative classes are similarly captured: True positives (205) and true negatives (203) are nearly equal, indicating similar counts of correct predictions across both classes.
• False negatives exceed false positives: The matrix shows 129 false negatives versus 110 false positives, indicating more missed positive cases than incorrect positive assignments.
• Error counts remain material in both directions: Both error cells are substantial, with more than 100 observations in each of the false negative and false positive categories.

The confusion matrix indicates that the model produces more correct classifications than incorrect ones, with balanced volumes of true positives and true negatives. At the same time, misclassification is present in both directions, and false negatives are modestly higher than false positives. Overall, the result reflects a mixed classification outcome with meaningful correct detection in both classes alongside material residual error.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

The Classifier Performance test evaluates classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The reported in-sample results include class-level precision, recall, and F1 for classes 0 and 1, along with weighted and macro averages across classes. Accuracy is reported as 0.624, and ROC AUC is reported as 0.6765. The class-specific metrics are closely aligned, with precision, recall, and F1 values for both classes concentrated near 0.62 to 0.63.

Key insights:

• Balanced class-level performance: Class 0 and class 1 show similar results across all three classification metrics. Precision is 0.6264 for class 0 and 0.6215 for class 1, recall is 0.6293 versus 0.6186, and F1 is 0.6279 versus 0.6200.
• Aggregate averages are consistent: Weighted average and macro average metrics are nearly identical, with precision at 0.6240 for both, recall at 0.6240 and 0.6239, and F1 at 0.6240 and 0.6239 respectively. This indicates limited divergence between class-level and aggregate performance summaries.
• Moderate overall classification effectiveness: Accuracy is 0.624, matching the aggregate precision, recall, and F1 values at approximately 0.624. ROC AUC is higher at 0.6765, indicating stronger rank-order discrimination than the threshold-based classification metrics alone.

The in-sample performance results show a closely balanced classification profile across both classes, with minimal spread between class-specific and averaged metrics. Overall threshold-based performance is concentrated at 0.624, while ROC AUC of 0.6765 is comparatively higher. Taken together, the results reflect uniform model behavior across classes with moderate in-sample discriminatory performance.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6264	0.6293	0.6279
1	0.6215	0.6186	0.6200
Weighted Average	0.6240	0.6240	0.6240
Macro Average	0.6240	0.6239	0.6239

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6240
ROC AUC	0.6765

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

The Classifier Performance test evaluates classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The out-of-sample results are presented for both classes individually and as macro and weighted averages, alongside overall accuracy and ROC AUC. Class-level precision ranges from 0.6114 to 0.6508, recall ranges from 0.6138 to 0.6486, and F1-scores are 0.6295 for class 0 and 0.6317 for class 1. Aggregate performance shows weighted and macro average F1 of 0.6306, with overall accuracy of 0.6306 and ROC AUC of 0.6876.

Key insights:

• Balanced class-level F1 performance: F1-scores are closely aligned across classes at 0.6295 for class 0 and 0.6317 for class 1, indicating similar overall classification effectiveness for both outcomes.
• Precision-recall tradeoff differs by class: Class 0 shows higher recall than precision (0.6486 vs. 0.6114), while class 1 shows higher precision than recall (0.6508 vs. 0.6138), indicating different error profiles across the two classes.
• Macro and weighted averages are nearly identical: Macro average precision, recall, and F1 are 0.6311, 0.6312, and 0.6306, while weighted averages are 0.6318, 0.6306, and 0.6306, showing minimal divergence between the two aggregate views.
• Overall discrimination exceeds accuracy: ROC AUC is 0.6876 compared with accuracy of 0.6306, indicating stronger rank-order separation than is reflected by the single-threshold classification accuracy.

The out-of-sample performance results show a broadly even pattern across the two classes, with nearly identical class-level F1-scores and very similar macro and weighted averages. The main distinction in class behavior appears in the precision-recall balance, where class 0 emphasizes recall and class 1 emphasizes precision. Overall, the reported metrics place aggregate threshold-based performance at 0.6306 accuracy and 0.6306 weighted F1, with ROC AUC of 0.6876 indicating comparatively stronger discrimination across score thresholds.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6114	0.6486	0.6295
1	0.6508	0.6138	0.6317
Weighted Average	0.6318	0.6306	0.6306
Macro Average	0.6311	0.6312	0.6306

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6306
ROC AUC	0.6876

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

The PrecisionRecallCurve test evaluates the trade-off between precision and recall across classification thresholds for this binary model. The result is presented as a precision-recall curve with recall on the x-axis and precision on the y-axis, showing how precision changes as recall increases. In this run, the curve begins with high precision at very low recall, remains mostly in the upper-middle portion of the plot through much of the recall range, and trends downward as recall approaches 1.0.

Key insights:

• High initial precision at low recall: At very low recall levels, precision reaches approximately 1.0 before quickly settling lower, indicating the highest-confidence predictions are associated with very high precision over a narrow recall range.

• Mid-range precision is relatively stable: Across a substantial middle portion of the curve, roughly from recall 0.1 to 0.8, precision remains broadly around 0.60 to 0.75 with gradual variation rather than abrupt deterioration.

• Precision declines at high recall: As recall increases toward the right end of the curve, precision decreases steadily and finishes near approximately 0.52 to 0.55 at recall close to 1.0, reflecting a weaker precision level when attempting to capture nearly all positive cases.

• Early-threshold region is more variable: The leftmost portion of the curve shows noticeable fluctuation in precision at very low recall, after which the curve becomes smoother across the remaining threshold range.

Overall, the curve shows a model that maintains moderate precision across most recall levels, with the strongest precision concentrated at very low recall and a gradual reduction as recall expands. The observed shape indicates a relatively stable middle operating region rather than a sharp trade-off through most of the threshold range. The main change in behavior occurs near full recall, where precision compresses to the low-0.5 range.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

The ROCCurve test evaluates classification performance by plotting the Receiver Operating Characteristic curve and calculating the Area Under the Curve (AUC) to measure class discrimination across threshold levels. The result for log_reg_model_v1 on test_dataset_final shows a single ROC curve for a binary classification setting, plotted against the random-classifier reference line. The chart reports an AUC of 0.69, with the ROC curve remaining above the diagonal baseline across the displayed false positive rate range.

Key insights:

• AUC exceeds random baseline: The reported AUC is 0.69, compared with the reference random-performance level of 0.50 shown in the plot legend.
• ROC curve stays above diagonal: The model ROC curve is consistently positioned above the dashed random-classifier line, indicating positive discrimination across thresholds shown in the figure.
• Discrimination is present but not strong: The curve rises gradually rather than sharply toward the upper-left corner, and the reported AUC remains materially below 1.0.

The ROC result indicates that the model demonstrates measurable ability to distinguish between the two classes on test_dataset_final. Performance is above the random benchmark, as reflected by both the curve position and the AUC of 0.69. At the same time, the curve shape and AUC magnitude show moderate rather than near-perfect separation.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

The TrainingTestDegradation test evaluates whether model performance degradation between the training and test datasets remains within a predefined threshold across classification metrics. The results table reports precision, recall, and F1-score for classes 0 and 1 on both datasets, along with the calculated degradation percentage and pass/fail status. Observed degradation values range from -4.7179% to 2.3924%, where negative values reflect higher test-set performance than training-set performance. All six metric evaluations are marked as Pass.

Key insights:

• All metrics passed threshold: Every evaluated metric for both classes received a Pass result. The reported degradation percentages all remain below the stated 10% maximum threshold.

• Degradation magnitudes are small: Positive degradation appears in only two cases: class 0 precision at 2.3924% and class 1 recall at 0.7747%. These are the only instances where test performance is lower than training performance.

• Several test metrics improved: Four of the six evaluated metrics show negative degradation, indicating higher scores on the test dataset than on the training dataset. The largest improvement is for class 1 precision, with degradation of -4.7179%, corresponding to an increase from 0.6215 to 0.6508.

• F1-scores are stable across datasets: F1-score changes are limited for both classes, with class 0 moving from 0.6279 to 0.6295 and class 1 from 0.6200 to 0.6317. The associated degradation values of -0.2526% and -1.8912% indicate limited change between training and test performance on this combined metric.

The test results show limited differences between training and test performance across all evaluated classification metrics. All degradation values remain within the configured threshold, and most metrics are slightly higher on the test dataset than on the training dataset. Overall, the observed train-test performance relationship is stable across both classes and across precision, recall, and F1-score.

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
0	Precision	0.6264	0.6114	2.3924	Pass
0	Recall	0.6293	0.6486	-3.0581	Pass
0	F1-Score	0.6279	0.6295	-0.2526	Pass
1	Precision	0.6215	0.6508	-4.7179	Pass
1	Recall	0.6186	0.6138	0.7747	Pass
1	F1-Score	0.6200	0.6317	-1.8912	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

The Minimum Accuracy test evaluates whether the model’s classification accuracy on the evaluated dataset meets or exceeds a predefined minimum threshold. The result table reports an accuracy score of 0.6306 against a threshold of 0.7000, along with an overall test outcome of Fail. This result presents the direct comparison between observed predictive accuracy and the configured minimum acceptance level.

Key insights:

• Accuracy is below threshold: The observed accuracy score is 0.6306, which is lower than the specified minimum threshold of 0.7000 by 0.0694.
• Test outcome is fail: The recorded Pass/Fail status is Fail, reflecting that the measured accuracy did not meet the minimum criterion defined for this test.

The result indicates that the model’s observed classification accuracy on the tested dataset did not reach the minimum threshold established in the Minimum Accuracy test. The shortfall between the measured score and the configured threshold is 0.0694, and the test is therefore recorded as failed.

Tables

Score	Threshold	Pass/Fail
0.6306	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

The MinimumF1Score test evaluates whether the model’s validation-set F1 score meets a predefined minimum threshold. The result table reports a validation F1 score of 0.6317, a threshold of 0.5, and an overall test outcome of Pass. These values provide the basis for assessing whether the observed balance between precision and recall satisfies the configured minimum standard.

Key insights:

• F1 score exceeds threshold: The observed validation F1 score is 0.6317 compared with a minimum threshold of 0.5, placing the result 0.1317 above the configured cutoff.
• Test outcome is Pass: The reported pass/fail status is Pass, indicating that the validation-set F1 score satisfies the criterion defined by this test.

The result shows that the model’s validation F1 score exceeded the specified minimum threshold and therefore passed the MinimumF1Score test. Taken together, the reported score, threshold, and pass status indicate that the model met the configured minimum performance level for this metric on the validation dataset.

Tables

Score	Threshold	Pass/Fail
0.6317	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

The Minimum ROC AUC Score test evaluates whether the model’s ROC AUC on the validation dataset meets or exceeds a predefined minimum threshold. The reported result includes the observed ROC AUC score, the threshold applied in the test, and the resulting pass/fail status. In this execution, the model produced a ROC AUC score of 0.6876 against a threshold of 0.5, and the test outcome was recorded as Pass.

Key insights:

• Threshold exceeded: The observed ROC AUC score of 0.6876 is above the configured minimum threshold of 0.5.
• Test outcome is pass: The test result is recorded as Pass based on the comparison between the observed score and the threshold.
• Positive margin to threshold: The score exceeds the threshold by 0.1876, indicating separation from the minimum acceptance level used in this test.

The test result shows that the model satisfied the minimum ROC AUC criterion defined for this validation check. The observed score of 0.6876 was higher than the 0.5 threshold, and the test therefore passed. Collectively, the result and supporting values indicate that the model met the specific acceptance condition evaluated by this test.

Tables

Score	Threshold	Pass/Fail
0.6876	0.5	Pass

▶ Test Result: Permutation Feature Importance (validmind.model_validation.sklearn.PermutationFeatureImportance)

Permutation Feature Importance

The Permutation Feature Importance test evaluates the relative contribution of each input feature by measuring the change in model performance after randomly permuting that feature’s values. The results are presented as a horizontal bar chart of permutation importances, with positive values indicating a larger reduction in performance when the feature is shuffled and negative values indicating no beneficial contribution under this measure. The chart shows a clear ranking across the modeled inputs, with the largest positive importances concentrated in a small subset of features and several features near zero or below zero.

Key insights:

• Geography_Germany is most influential: Geography_Germany has the largest permutation importance in the chart, at approximately 0.05, making it the most impactful feature under this test.
• IsActiveMember shows strong importance: IsActiveMember has the second-highest importance, at roughly 0.033, indicating a substantial contribution to model performance.
• Gender_Male has moderate contribution: Gender_Male ranks third, with permutation importance around 0.018, below the top two features but clearly above the remaining variables.
• Only one additional feature is marginally positive: NumOfProducts is slightly positive at approximately 0.004, while all other remaining features are at or below zero.
• Several features are non-contributory under permutation: Tenure, EstimatedSalary, Balance, CreditScore, Geography_Spain, and HasCrCard display near-zero to negative permutation importance, with HasCrCard appearing most negative at roughly -0.011.

The permutation importance profile is concentrated in three features, led by Geography_Germany and IsActiveMember, with Gender_Male contributing at a lower but still positive level. NumOfProducts adds only a small positive signal, while the remaining features do not improve performance under this permutation-based measure and several register negative values. Overall, the result indicates an uneven importance distribution across inputs, with model performance primarily associated with a limited subset of features in this test.

Figures

▶ Test Result: SHAP Global Importance (validmind.model_validation.sklearn.SHAPGlobalImportance)

SHAP Global Importance

The SHAPGlobalImportance test evaluates global feature importance using SHAP values to show which input variables contribute most to the model output and how their values affect prediction direction. The results include a normalized feature importance chart and a SHAP summary plot. The importance chart ranks features by mean absolute SHAP value, while the summary plot shows the distribution and direction of feature-level impacts across observations. Together, these figures display both the relative importance of each feature and the sign of their contributions to the model output.

Key insights:

• Importance is concentrated in a few features: IsActiveMember and Geography_Germany are the two highest-ranked features, each with normalized importance close to 100, followed by Gender_Male at roughly 64 and Balance at roughly 41. The remaining features decline materially after these top four.

• IsActiveMember shows strong directional separation: In the summary plot, higher feature values for IsActiveMember are concentrated on the negative SHAP side, while lower values are concentrated on the positive SHAP side. This indicates a clear directional split in model impact for this feature.

• Geography_Germany has a large positive impact when high: Higher values for Geography_Germany appear on the positive SHAP side, reaching the largest positive values among all features, while lower values are concentrated on the negative side. This pattern is visually strong and consistent across the plotted observations.

• Balance contributes mainly in the positive direction at higher values: Balance has the fourth-highest global importance and shows higher feature values clustered on the positive SHAP side. Lower feature values are positioned closer to zero or slightly negative, indicating an asymmetric contribution pattern.

• Lower-ranked features have limited marginal impact: Tenure and EstimatedSalary have the smallest normalized SHAP importance, and their summary plot distributions are tightly centered around zero. Geography_Spain, NumOfProducts, and CreditScore also show comparatively smaller spreads than the top-ranked variables.

The SHAP results show that model behavior is driven primarily by a small set of variables, with IsActiveMember and Geography_Germany contributing the largest overall effects and clear directional patterns in the summary plot. Gender_Male and Balance also have material influence, though at lower magnitude than the top two features. The remaining features contribute more modestly, with several displaying SHAP values concentrated near zero, indicating comparatively limited influence on the model output.

Figures

▶ Test Result: Weakspots Diagnosis (validmind.model_validation.sklearn.WeakspotsDiagnosis)

❌ Weakspots Diagnosis

The WeakspotsDiagnosis test evaluates model performance across feature-space slices by comparing accuracy, precision, recall, and F1 between training and test datasets within each bin. The results are presented for multiple features, including continuous and binary variables, with metric-specific thresholds shown in the plots. Across the reported slices, the tables and figures show substantial variation in metric levels by feature bin, with several bins falling below the plotted thresholds and some slices based on very small record counts. The outputs also show side-by-side training and test values, allowing direct comparison of slice-level behavior across datasets.

Key insights:

• Low-sample slices are highly volatile: Several bins contain very few test observations and show extreme metric values. Examples include Balance (21314.62, 42629.24] with 2 test records and all metrics at 0.0, Balance (42629.24, 63943.86] with 4 test records and precision 1.0, and NumOfProducts (3.7, 4.0] with 3 test records and F1 0.8.

• CreditScore weak spot at lowest bin: The lowest CreditScore slice (349.5, 400.0] shows the weakest test performance, with accuracy 0.25, recall 0.25, and F1 0.4 on 4 test records, compared with training values of 0.7 accuracy, 0.7 recall, and 0.8235 F1 on 10 records.

• Balance performance varies sharply by bin: Test F1 ranges from 0.0 in Balance (21314.62, 42629.24] to 0.7429 in (106573.1, 127887.72]. Mid-range balance bins generally show stronger test F1 values, including 0.6916, 0.7429, 0.7231, and 0.7170 between 85258.48 and 170516.96, while the lowest and upper-end bins are weaker.

• EstimatedSalary shows mixed slice behavior: Test accuracy ranges from 0.5224 to 0.7377 across salary bins, and test F1 ranges from 0.4667 in (119922.282, 139907.399] to 0.7778 in (-188.271, 19996.697]. The test recall in (119922.282, 139907.399] is 0.4242, while several lower and middle salary slices remain above 0.63 on F1.

• Binary features reveal asymmetric recall patterns: For Gender_Male, the test slice (-0.001, 0.1] has recall 0.7796 and F1 0.7178, while (0.9, 1.0] has recall 0.4054 and F1 0.4898. A similar pattern appears for IsActiveMember, where (-0.001, 0.1] has recall 0.7593 and F1 0.7009, versus (0.9, 1.0] with recall 0.3475 and F1 0.4530.

• Geography slices differ materially by category: For Geography_Germany, the positive slice (0.9, 1.0] has test recall 0.9118 and F1 0.7898, while the complement slice (-0.001, 0.1] has recall 0.4091 and F1 0.4836. For Geography_Spain, the pattern reverses, with (-0.001, 0.1] at test F1 0.6667 versus (0.9, 1.0] at 0.4746.

• NumOfProducts separates into strong and weak bins: Test F1 is 0.6883 for (0.997, 1.3], 0.4 for (1.9, 2.2], 0.6667 for (2.8, 3.1], and 0.8 for (3.7, 4.0]. The (1.9, 2.2] bin is the weakest among the larger product-count slices, with low precision of 0.3086 despite recall of 0.5682.

• Tenure has localized strong and weak regions: Test accuracy peaks at 0.7727 for Tenure (4.0, 5.0], with F1 0.7761, while weaker test F1 values appear at (6.0, 7.0] with 0.5490 and (7.0, 8.0] with 0.5517. Test precision is highest at (5.0, 6.0] with 0.84, while recall is highest at (4.0, 5.0] with 0.7647.

The weak spot analysis shows that performance is not uniform across the feature space and varies materially by feature slice. The most pronounced weakness appears in low-sample bins, where metrics are unstable and in some cases collapse to zero, while several larger slices also show materially lower recall or F1 than adjacent regions. Binary segmentation results indicate persistent differences between complementary groups for features such as Gender_Male, IsActiveMember, Geography_Germany, and Geography_Spain. Continuous features such as Balance, CreditScore, EstimatedSalary, and Tenure show localized regions of stronger and weaker test performance rather than a uniform pattern across their ranges.

Tables

Slice	Number of Records	Feature	Accuracy	Precision	Recall	F1	Dataset
(-213.146, 21314.62]	216	Balance	0.6343	0.5600	0.3294	0.4148	test_dataset_final
(21314.62, 42629.24]	2	Balance	0.0000	0.0000	0.0000	0.0000	test_dataset_final
(42629.24, 63943.86]	4	Balance	0.5000	1.0000	0.5000	0.6667	test_dataset_final
(63943.86, 85258.48]	42	Balance	0.5000	0.4545	0.5263	0.4878	test_dataset_final
(85258.48, 106573.1]	90	Balance	0.6333	0.6852	0.6981	0.6916	test_dataset_final
(106573.1, 127887.72]	113	Balance	0.6814	0.7536	0.7324	0.7429	test_dataset_final
(127887.72, 149202.34]	108	Balance	0.6667	0.6620	0.7966	0.7231	test_dataset_final
(149202.34, 170516.96]	46	Balance	0.6739	0.6786	0.7600	0.7170	test_dataset_final
(170516.96, 191831.58]	17	Balance	0.3529	0.3846	0.6250	0.4762	test_dataset_final
(191831.58, 213146.2]	5	Balance	0.4000	0.6667	0.5000	0.5714	test_dataset_final
(-213.146, 21314.62]	835	Balance	0.6563	0.5814	0.3175	0.4107	train_dataset_final
(21314.62, 42629.24]	12	Balance	0.5833	0.6667	0.5714	0.6154	train_dataset_final
(42629.24, 63943.86]	52	Balance	0.5192	0.5294	0.3462	0.4186	train_dataset_final
(63943.86, 85258.48]	142	Balance	0.5563	0.5667	0.4789	0.5191	train_dataset_final
(85258.48, 106573.1]	336	Balance	0.6161	0.6019	0.7384	0.6632	train_dataset_final
(106573.1, 127887.72]	525	Balance	0.6648	0.7079	0.7778	0.7412	train_dataset_final
(127887.72, 149202.34]	417	Balance	0.5899	0.6022	0.7269	0.6587	train_dataset_final
(149202.34, 170516.96]	182	Balance	0.5879	0.5520	0.7841	0.6479	train_dataset_final
(170516.96, 191831.58]	65	Balance	0.5385	0.5778	0.7027	0.6341	train_dataset_final
(191831.58, 213146.2]	19	Balance	0.4211	0.7000	0.4667	0.5600	train_dataset_final
(349.5, 400.0]	4	CreditScore	0.2500	1.0000	0.2500	0.4000	test_dataset_final
(400.0, 450.0]	16	CreditScore	0.5625	0.6250	0.5556	0.5882	test_dataset_final
(450.0, 500.0]	29	CreditScore	0.6207	0.5556	0.7692	0.6452	test_dataset_final
(500.0, 550.0]	65	CreditScore	0.5385	0.5455	0.5455	0.5455	test_dataset_final
(550.0, 600.0]	97	CreditScore	0.6289	0.6122	0.6383	0.6250	test_dataset_final
(600.0, 650.0]	119	CreditScore	0.6471	0.6786	0.6129	0.6441	test_dataset_final
(650.0, 700.0]	128	CreditScore	0.6484	0.6615	0.6515	0.6565	test_dataset_final
(700.0, 750.0]	96	CreditScore	0.6458	0.7292	0.6250	0.6731	test_dataset_final
(750.0, 800.0]	56	CreditScore	0.6607	0.6923	0.6207	0.6545	test_dataset_final
(800.0, 850.0]	37	CreditScore	0.6757	0.6364	0.4667	0.5385	test_dataset_final
(349.5, 400.0]	10	CreditScore	0.7000	1.0000	0.7000	0.8235	train_dataset_final
(400.0, 450.0]	48	CreditScore	0.6250	0.6207	0.7200	0.6667	train_dataset_final
(450.0, 500.0]	131	CreditScore	0.6107	0.5658	0.7049	0.6277	train_dataset_final
(500.0, 550.0]	275	CreditScore	0.6436	0.6490	0.6853	0.6667	train_dataset_final
(550.0, 600.0]	377	CreditScore	0.6154	0.6058	0.6667	0.6348	train_dataset_final
(600.0, 650.0]	454	CreditScore	0.6035	0.6216	0.5897	0.6053	train_dataset_final
(650.0, 700.0]	476	CreditScore	0.6155	0.6053	0.5974	0.6013	train_dataset_final
(700.0, 750.0]	384	CreditScore	0.6094	0.6108	0.5455	0.5763	train_dataset_final
(750.0, 800.0]	237	CreditScore	0.6962	0.7212	0.6356	0.6757	train_dataset_final
(800.0, 850.0]	193	CreditScore	0.6269	0.5714	0.5714	0.5714	train_dataset_final
(-188.271, 19996.697]	61	EstimatedSalary	0.7377	0.7179	0.8485	0.7778	test_dataset_final
(19996.697, 39981.814]	59	EstimatedSalary	0.6441	0.5758	0.7308	0.6441	test_dataset_final
(39981.814, 59966.931]	65	EstimatedSalary	0.6462	0.6765	0.6571	0.6667	test_dataset_final
(59966.931, 79952.048]	80	EstimatedSalary	0.6375	0.7353	0.5556	0.6329	test_dataset_final
(79952.048, 99937.165]	72	EstimatedSalary	0.6250	0.5897	0.6765	0.6301	test_dataset_final
(99937.165, 119922.282]	61	EstimatedSalary	0.6721	0.6875	0.6875	0.6875	test_dataset_final
(119922.282, 139907.399]	67	EstimatedSalary	0.5224	0.5185	0.4242	0.4667	test_dataset_final
(139907.399, 159892.516]	62	EstimatedSalary	0.6613	0.6667	0.6000	0.6316	test_dataset_final
(159892.516, 179877.633]	58	EstimatedSalary	0.6207	0.7826	0.5143	0.6207	test_dataset_final
(179877.633, 199862.75]	62	EstimatedSalary	0.5484	0.5556	0.4839	0.5172	test_dataset_final
(-188.271, 19996.697]	265	EstimatedSalary	0.6000	0.6045	0.6045	0.6045	train_dataset_final
(19996.697, 39981.814]	244	EstimatedSalary	0.6311	0.6640	0.6336	0.6484	train_dataset_final
(39981.814, 59966.931]	233	EstimatedSalary	0.6695	0.6696	0.6638	0.6667	train_dataset_final
(59966.931, 79952.048]	270	EstimatedSalary	0.6259	0.6124	0.6077	0.6100	train_dataset_final
(79952.048, 99937.165]	272	EstimatedSalary	0.6213	0.5625	0.6050	0.5830	train_dataset_final
(99937.165, 119922.282]	252	EstimatedSalary	0.5595	0.5736	0.5692	0.5714	train_dataset_final
(119922.282, 139907.399]	274	EstimatedSalary	0.6241	0.5915	0.6512	0.6199	train_dataset_final
(139907.399, 159892.516]	267	EstimatedSalary	0.6442	0.6129	0.6179	0.6154	train_dataset_final
(159892.516, 179877.633]	269	EstimatedSalary	0.6394	0.6822	0.6111	0.6447	train_dataset_final
(179877.633, 199862.75]	239	EstimatedSalary	0.6276	0.6529	0.6270	0.6397	train_dataset_final
(-0.001, 0.1]	321	Gender_Male	0.6449	0.6651	0.7796	0.7178	test_dataset_final
(0.9, 1.0]	326	Gender_Male	0.6166	0.6186	0.4054	0.4898	test_dataset_final
(-0.001, 0.1]	1282	Gender_Male	0.6357	0.6441	0.7648	0.6993	train_dataset_final
(0.9, 1.0]	1303	Gender_Male	0.6124	0.5774	0.4371	0.4975	train_dataset_final
(-0.001, 0.1]	449	Geography_Germany	0.6147	0.5912	0.4091	0.4836	test_dataset_final
(0.9, 1.0]	198	Geography_Germany	0.6667	0.6966	0.9118	0.7898	test_dataset_final
(-0.001, 0.1]	1791	Geography_Germany	0.6147	0.5661	0.4149	0.4789	train_dataset_final
(0.9, 1.0]	794	Geography_Germany	0.6448	0.6648	0.9189	0.7715	train_dataset_final
(-0.001, 0.1]	494	Geography_Spain	0.6417	0.6808	0.6531	0.6667	test_dataset_final
(0.9, 1.0]	153	Geography_Spain	0.5948	0.5091	0.4444	0.4746	test_dataset_final
(-0.001, 0.1]	1975	Geography_Spain	0.6238	0.6273	0.6601	0.6433	train_dataset_final
(0.9, 1.0]	610	Geography_Spain	0.6246	0.5913	0.4607	0.5179	train_dataset_final
(-0.001, 0.1]	209	HasCrCard	0.6220	0.6441	0.6726	0.6580	test_dataset_final
(0.9, 1.0]	438	HasCrCard	0.6347	0.6548	0.5837	0.6172	test_dataset_final
(-0.001, 0.1]	740	HasCrCard	0.6108	0.6143	0.6562	0.6345	train_dataset_final
(0.9, 1.0]	1845	HasCrCard	0.6293	0.6249	0.6027	0.6136	train_dataset_final
(-0.001, 0.1]	360	IsActiveMember	0.6111	0.6508	0.7593	0.7009	test_dataset_final
(0.9, 1.0]	287	IsActiveMember	0.6551	0.6508	0.3475	0.4530	test_dataset_final
(-0.001, 0.1]	1387	IsActiveMember	0.6164	0.6362	0.7883	0.7041	train_dataset_final
(0.9, 1.0]	1198	IsActiveMember	0.6327	0.5694	0.3340	0.4211	train_dataset_final
(0.997, 1.3]	388	NumOfProducts	0.6289	0.7465	0.6386	0.6883	test_dataset_final
(1.9, 2.2]	216	NumOfProducts	0.6528	0.3086	0.5682	0.4000	test_dataset_final
(2.8, 3.1]	40	NumOfProducts	0.5250	1.0000	0.5000	0.6667	test_dataset_final
(3.7, 4.0]	3	NumOfProducts	0.6667	1.0000	0.6667	0.8000	test_dataset_final
(0.997, 1.3]	1468	NumOfProducts	0.6144	0.6857	0.6479	0.6663	train_dataset_final
(1.9, 2.2]	928	NumOfProducts	0.6595	0.3746	0.5859	0.4570	train_dataset_final
(2.8, 3.1]	148	NumOfProducts	0.5473	0.9747	0.5423	0.6968	train_dataset_final
(3.7, 4.0]	41	NumOfProducts	0.4390	1.0000	0.4390	0.6102	train_dataset_final
(-0.01, 1.0]	89	Tenure	0.6629	0.7143	0.6863	0.7000	test_dataset_final
(1.0, 2.0]	62	Tenure	0.5968	0.5312	0.6296	0.5763	test_dataset_final
(2.0, 3.0]	63	Tenure	0.5238	0.5750	0.6389	0.6053	test_dataset_final
(3.0, 4.0]	62	Tenure	0.5645	0.6207	0.5294	0.5714	test_dataset_final
(4.0, 5.0]	66	Tenure	0.7727	0.7879	0.7647	0.7761	test_dataset_final
(5.0, 6.0]	74	Tenure	0.7297	0.8400	0.5676	0.6774	test_dataset_final
(6.0, 7.0]	53	Tenure	0.5660	0.5833	0.5185	0.5490	test_dataset_final
(7.0, 8.0]	71	Tenure	0.6338	0.5161	0.5926	0.5517	test_dataset_final
(8.0, 9.0]	66	Tenure	0.5909	0.6129	0.5588	0.5846	test_dataset_final
(9.0, 10.0]	41	Tenure	0.6098	0.7619	0.5926	0.6667	test_dataset_final
(-0.01, 1.0]	372	Tenure	0.6102	0.6359	0.6517	0.6437	train_dataset_final
(1.0, 2.0]	272	Tenure	0.5809	0.5639	0.5725	0.5682	train_dataset_final
(2.0, 3.0]	273	Tenure	0.5971	0.6000	0.6087	0.6043	train_dataset_final
(3.0, 4.0]	285	Tenure	0.5614	0.5390	0.6058	0.5704	train_dataset_final
(4.0, 5.0]	258	Tenure	0.6202	0.6045	0.6429	0.6231	train_dataset_final
(5.0, 6.0]	242	Tenure	0.6074	0.5948	0.5897	0.5923	train_dataset_final
(6.0, 7.0]	245	Tenure	0.6980	0.6800	0.6182	0.6476	train_dataset_final
(7.0, 8.0]	254	Tenure	0.6496	0.6833	0.6165	0.6482	train_dataset_final
(8.0, 9.0]	254	Tenure	0.6969	0.7190	0.6692	0.6932	train_dataset_final
(9.0, 10.0]	130	Tenure	0.6538	0.6346	0.5593	0.5946	train_dataset_final

Figures

▶ Test Result: Overfit Diagnosis (validmind.model_validation.sklearn.OverfitDiagnosis)

Overfit Diagnosis

The Overfit Diagnosis test evaluates differences between training and test performance across feature-level slices to identify regions where the AUC gap exceeds the 0.04 threshold. The reported results list flagged slices for CreditScore, Tenure, Balance, NumOfProducts, EstimatedSalary, and Geography_Germany, with training and test record counts and corresponding AUC values for each slice. Observed gaps range from 0.0416 to 0.5714 in the tabulated flagged regions, and the accompanying plots show that the magnitude and direction of the train-test gap vary across bins within each feature.

Key insights:

• Balance shows the largest gaps: Balance contains the highest flagged train-test AUC differences, including gaps of 0.5714 for (21314.62, 42629.24], 0.5118 for (42629.24, 63943.86], 0.3500 for (191831.58, 213146.2], and 0.2109 for (170516.96, 191831.58]. These slices also have relatively small sample sizes, with test counts of 2, 4, 5, and 17 respectively.

• Tenure has the most flagged regions: Four Tenure slices exceed the threshold: (2.0, 3.0] with a gap of 0.1198, (6.0, 7.0] with 0.1026, (8.0, 9.0] with 0.0952, and (7.0, 8.0] with 0.0558. Training AUCs in these bins range from 0.6404 to 0.7515, while test AUCs range from 0.5206 to 0.6633.

• EstimatedSalary contains multiple elevated segments: EstimatedSalary has two flagged slices, with the largest gap at (119922.282, 139907.399] equal to 0.0960 and another at (179877.633, 199862.75] equal to 0.0448. These correspond to training/test AUC pairs of 0.6816/0.5856 and 0.6754/0.6306.

• CreditScore gaps are narrower but above threshold: Two CreditScore slices are flagged: (500.0, 550.0] with a gap of 0.0784 and (800.0, 850.0] with a gap of 0.0416. The associated AUC pairs are 0.6816 versus 0.6032 and 0.6992 versus 0.6576.

• NumOfProducts affects a large segment: The flagged NumOfProducts slice (1.9, 2.2] has a gap of 0.0565 with 928 training records and 216 test records, making it one of the largest segments among the flagged regions. Its training AUC is 0.7011 and test AUC is 0.6447.

• Geography_Germany is marginally above threshold: The slice (0.9, 1.0] for Geography_Germany shows a gap of 0.0494, with 794 training records and 198 test records. The corresponding AUC values are 0.6115 on training and 0.5620 on test.

The results identify overfit-sensitive regions across several features rather than a single isolated segment. The most pronounced gaps are concentrated in Balance, while Tenure contributes the highest number of flagged slices. CreditScore, EstimatedSalary, NumOfProducts, and Geography_Germany also contain segments with train-test AUC differences above the 0.04 cutoff, indicating that the observed divergence is distributed across both continuous and encoded categorical inputs.

Tables

Overfit Diagnosis

Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
CreditScore	(500.0, 550.0]	275	65	0.6816	0.6032	0.0784
CreditScore	(800.0, 850.0]	193	37	0.6992	0.6576	0.0416
Tenure	(2.0, 3.0]	273	63	0.6404	0.5206	0.1198
Tenure	(6.0, 7.0]	245	53	0.7407	0.6382	0.1026
Tenure	(7.0, 8.0]	254	71	0.7191	0.6633	0.0558
Tenure	(8.0, 9.0]	254	66	0.7515	0.6562	0.0952
Balance	(21314.62, 42629.24]	12	2	0.5714	0.0000	0.5714
Balance	(42629.24, 63943.86]	52	4	0.5118	0.0000	0.5118
Balance	(170516.96, 191831.58]	65	17	0.5164	0.3056	0.2109
Balance	(191831.58, 213146.2]	19	5	0.3500	0.0000	0.3500
NumOfProducts	(1.9, 2.2]	928	216	0.7011	0.6447	0.0565
EstimatedSalary	(119922.282, 139907.399]	274	67	0.6816	0.5856	0.0960
EstimatedSalary	(179877.633, 199862.75]	239	62	0.6754	0.6306	0.0448
Geography_Germany	(0.9, 1.0]	794	198	0.6115	0.5620	0.0494

Figures

▶ Test Result: Robustness Diagnosis (validmind.model_validation.sklearn.RobustnessDiagnosis)

✅ Robustness Diagnosis

The Robustness Diagnosis test evaluates model performance under increasing Gaussian noise applied to numeric input features. The results report AUC and performance decay for both the training dataset and the test dataset across perturbation sizes from 0.0 to 0.5 standard deviations. Baseline AUC is 0.6765 on train_dataset_final and 0.6876 on test_dataset_final, and the table and plot show how these values change as perturbation size increases. All evaluated perturbation levels are marked as passed for both datasets.

Key insights:

• AUC declines as noise increases: Relative to baseline, AUC generally decreases as perturbation size rises. On train_dataset_final, AUC moves from 0.6765 at baseline to 0.6638 at perturbation sizes 0.4 and 0.5, while on test_dataset_final it moves from 0.6876 to 0.6643 at perturbation size 0.5.

• Test performance shows larger decay: Performance decay is consistently higher on test_dataset_final than on train_dataset_final from perturbation size 0.3 onward. At perturbation size 0.5, decay is 0.0233 on the test dataset versus 0.0127 on the training dataset.

• Minor non-monotonic variation appears at low-to-mid noise levels: The degradation pattern is not strictly monotonic at every step. On train_dataset_final, AUC improves from 0.6709 at 0.2 perturbation to 0.6736 at 0.3, and on test_dataset_final, AUC at 0.1 is 0.6881 versus a baseline of 0.6876, corresponding to a performance decay of -0.0004.

• All evaluated scenarios passed: Every baseline and perturbed run is marked Passed = true in both datasets across all tested perturbation sizes from 0.0 through 0.5.

The results show that model discrimination, as measured by AUC, is retained across all tested perturbation levels, with gradual deterioration as Gaussian noise increases. The largest observed degradation occurs in the test dataset at higher perturbation sizes, while small local fluctuations appear at lower and intermediate noise levels. Across the full evaluated range, all runs satisfy the test’s pass criterion.

Tables

Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
Baseline (0.0)	train_dataset_final	2585	0.6765	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.6876	0.0000	True
0.1	train_dataset_final	2585	0.6747	0.0018	True
0.1	test_dataset_final	647	0.6881	-0.0004	True
0.2	train_dataset_final	2585	0.6709	0.0056	True
0.2	test_dataset_final	647	0.6833	0.0044	True
0.3	train_dataset_final	2585	0.6736	0.0030	True
0.3	test_dataset_final	647	0.6761	0.0115	True
0.4	train_dataset_final	2585	0.6638	0.0127	True
0.4	test_dataset_final	647	0.6690	0.0187	True
0.5	train_dataset_final	2585	0.6638	0.0127	True
0.5	test_dataset_final	647	0.6643	0.0233	True

Figures

In summary

In this second notebook, you learned how to:

• Import a sample dataset
• Identify which tests you might want to run with ValidMind
• Initialize ValidMind datasets and model objects
• Run individual tests
• Utilize the output from tests you've run
• Log test results from sets of or individual tests as evidence to the ValidMind Platform
• Add supplementary individual test results to your documentation
• Assign predictions to your ValidMind model objects

Next steps

Integrate custom tests

Now that you're familiar with the basics of using the ValidMind Library to run and log tests to provide evidence for your documentation, let's learn how to incorporate your own custom tests into ValidMind: 3 — Integrate custom tests

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial