ValidMind for development 2 — Start the development process

Learn how to use ValidMind for your end-to-end documentation process with our series of four introductory notebooks. In this second notebook, you'll run tests and investigate results, then add the results or evidence to your documentation.

You'll become familiar with the individual tests available in ValidMind, as well as how to run them and change parameters as necessary. Using ValidMind's repository of individual tests as building blocks helps you ensure that a record (model) is being built appropriately.

For a full list of out-of-the-box tests and descriptions, use the interactive ValidMind test sandbox.

Learn by doing

Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Developer Fundamentals

Prerequisites

In order to log test results or evidence to your documentation with this notebook, you'll need to first have:

• Registered a model within the ValidMind Platform with a predefined documentation template
• Installed the ValidMind Library in your local environment, allowing you to access all its features

Need help with the above steps?

Refer to the first notebook in this series: 1 — Set up the ValidMind Library

Setting up

Initialize the ValidMind Library

First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Development from the Document drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="documentation",
)

Note: you may need to restart the kernel to use updated packages.

2026-09-11 23:45:50,859 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model development (ID: cmalgf3qi02ce199qm3rdkl46)
📁 Document Type: model_documentation

Import sample dataset

Then, let's import the public Bank Customer Churn Prediction dataset from Kaggle.

In our below example, note that:

• The target column, Exited has a value of 1 when a customer has churned and 0 otherwise.
• The ValidMind Library provides a wrapper to automatically load the dataset as a Pandas DataFrame object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns.

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()
raw_df.head()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

Identify qualitative tests

Next, let's say we want to do some data quality assessments by running a few individual tests.

Use the vm.tests.list_tests() function introduced by the first notebook in this series in combination with vm.tests.list_tags() and vm.tests.list_tasks() to find which prebuilt tests are relevant for data quality assessment:

• tasks represent the kind of modeling task associated with a test. Here we'll focus on classification tasks.
• tags are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the data_quality tag.

# Get the list of available task types
sorted(vm.tests.list_tasks())

['classification',
 'clustering',
 'data_validation',
 'feature_extraction',
 'monitoring',
 'nlp',
 'regression',
 'residual_analysis',
 'text_classification',
 'text_generation',
 'text_qa',
 'text_summarization',
 'time_series_forecasting',
 'visualization']

# Get the list of available tags
sorted(vm.tests.list_tags())

['AUC',
 'analysis',
 'anomaly',
 'anomaly_detection',
 'bias_and_fairness',
 'binary_classification',
 'calibration',
 'categorical_data',
 'classification',
 'classification_metrics',
 'clustering',
 'correlation',
 'credit_risk',
 'data_analysis',
 'data_distribution',
 'data_quality',
 'data_validation',
 'descriptive_statistics',
 'dimensionality_reduction',
 'distribution',
 'embeddings',
 'feature_importance',
 'feature_selection',
 'few_shot',
 'forecasting',
 'frequency_analysis',
 'kmeans',
 'linear_regression',
 'llm',
 'logistic_regression',
 'metadata',
 'model_comparison',
 'model_diagnosis',
 'model_explainability',
 'model_interpretation',
 'model_performance',
 'model_predictions',
 'model_selection',
 'model_training',
 'model_validation',
 'multiclass_classification',
 'nlp',
 'normality',
 'numerical_data',
 'outlier',
 'outliers',
 'qualitative',
 'rag_performance',
 'ragas',
 'regression',
 'retrieval_performance',
 'scorecard',
 'seasonality',
 'senstivity_analysis',
 'sklearn',
 'stationarity',
 'statistical_test',
 'statistics',
 'statsmodels',
 'tabular_data',
 'text_data',
 'threshold_optimization',
 'time_series_data',
 'unit_root_test',
 'visualization',
 'zero_shot']

You can pass tags and tasks as parameters to the vm.tests.list_tests() function to filter the tests based on the tags and task types.

For example, to find tests related to tabular data quality for classification models, you can call list_tests() like this:

vm.tests.list_tests(task="classification", tags=["tabular_data", "data_quality"])

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.data_validation.ClassImbalance	Class Imbalance	Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model....	True	True	['dataset']	{'min_percent_threshold': {'type': 'int', 'default': 10}}	['tabular_data', 'binary_classification', 'multiclass_classification', 'data_quality']	['classification']
validmind.data_validation.DescriptiveStatistics	Descriptive Statistics	Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's...	False	True	['dataset']	{}	['tabular_data', 'time_series_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.Duplicates	Duplicates	Tests dataset for duplicate entries, ensuring model reliability via data quality verification....	False	True	['dataset']	{'min_threshold': {'type': '_empty', 'default': 1}}	['tabular_data', 'data_quality', 'text_data']	['classification', 'regression']
validmind.data_validation.HighCardinality	High Cardinality	Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting....	False	True	['dataset']	{'num_threshold': {'type': 'int', 'default': 100}, 'percent_threshold': {'type': 'float', 'default': 0.1}, 'threshold_type': {'type': 'str', 'default': 'percent'}}	['tabular_data', 'data_quality', 'categorical_data']	['classification', 'regression']
validmind.data_validation.HighPearsonCorrelation	High Pearson Correlation	Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity....	False	True	['dataset']	{'max_threshold': {'type': 'float', 'default': 0.3}, 'top_n_correlations': {'type': 'int', 'default': 10}, 'feature_columns': {'type': 'list', 'default': None}}	['tabular_data', 'data_quality', 'correlation']	['classification', 'regression']
validmind.data_validation.MissingValues	Missing Values	Evaluates dataset quality by ensuring missing value percentage across all features does not exceed a set threshold....	False	True	['dataset']	{'min_percentage_threshold': {'type': 'float', 'default': 1.0}}	['tabular_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.MissingValuesBarPlot	Missing Values Bar Plot	Assesses the percentage and distribution of missing values in the dataset via a bar plot, with emphasis on...	True	False	['dataset']	{'threshold': {'type': 'int', 'default': 80}, 'fig_height': {'type': 'int', 'default': 600}}	['tabular_data', 'data_quality', 'visualization']	['classification', 'regression']
validmind.data_validation.Skewness	Skewness	Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data...	False	True	['dataset']	{'max_threshold': {'type': '_empty', 'default': 1}}	['data_quality', 'tabular_data']	['classification', 'regression']
validmind.plots.BoxPlot	Box Plot	Generates customizable box plots for numerical features in a dataset with optional grouping using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'group_by': {'type': 'Optional', 'default': None}, 'width': {'type': 'int', 'default': 1800}, 'height': {'type': 'int', 'default': 1200}, 'colors': {'type': 'Optional', 'default': None}, 'show_outliers': {'type': 'bool', 'default': True}, 'title_prefix': {'type': 'str', 'default': 'Box Plot of'}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.plots.HistogramPlot	Histogram Plot	Generates customizable histogram plots for numerical features in a dataset using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'bins': {'type': 'Union', 'default': 30}, 'color': {'type': 'str', 'default': 'steelblue'}, 'opacity': {'type': 'float', 'default': 0.7}, 'show_kde': {'type': 'bool', 'default': True}, 'normalize': {'type': 'bool', 'default': False}, 'log_scale': {'type': 'bool', 'default': False}, 'title_prefix': {'type': 'str', 'default': 'Histogram of'}, 'width': {'type': 'int', 'default': 1200}, 'height': {'type': 'int', 'default': 800}, 'n_cols': {'type': 'int', 'default': 2}, 'vertical_spacing': {'type': 'float', 'default': 0.15}, 'horizontal_spacing': {'type': 'float', 'default': 0.1}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.stats.DescriptiveStats	Descriptive Stats	Provides comprehensive descriptive statistics for numerical features in a dataset....	False	True	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'include_advanced': {'type': 'bool', 'default': True}, 'confidence_level': {'type': 'float', 'default': 0.95}}	['tabular_data', 'statistics', 'data_quality']	['classification', 'regression', 'clustering']

Want to learn more about navigating ValidMind tests?

Refer to our notebook outlining the utilities available for viewing and understanding available ValidMind tests: Explore tests

Initialize the ValidMind dataset

With the individual tests we want to run identified, the next step is to connect your data with a ValidMind Dataset object. This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind, but you only need to do it once per dataset.

Initialize a ValidMind dataset object using the init_dataset function from the ValidMind (vm) module. For this example, we'll pass in the following arguments:

• dataset — The raw dataset that you want to provide as input to tests.
• input_id — A unique identifier that allows tracking what inputs are used when running each individual test.
• target_column — A required argument if tests require access to true values. This is the name of the target column in the dataset.

# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test
vm_raw_dataset = vm.init_dataset(
    dataset=raw_df,
    input_id="raw_dataset",
    target_column="Exited",
)

Running tests on datasets

Now that we know how to initialize a ValidMind dataset object, we're ready to run some tests!

You run individual tests by calling the run_test function provided by the validmind.tests module. For the examples below, we'll pass in the following arguments:

• test_id — The ID of the test to run, as seen in the ID column when you run list_tests.
• params — A dictionary of parameters for the test. These will override any default_params set in the test definition.

Run tabular data tests

The inputs expected by a test can also be found in the test definition — let's take validmind.data_validation.DescriptiveStatistics as an example.

Note that the output of the describe_test() function below shows that this test expects a dataset as input:

vm.tests.describe_test("validmind.data_validation.DescriptiveStatistics")

▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Descriptive Statistics

Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's dataset.

Purpose

The purpose of the Descriptive Statistics metric is to provide a comprehensive summary of both numerical and categorical data within a dataset. This involves statistics such as count, mean, standard deviation, minimum and maximum values for numerical data. For categorical data, it calculates the count, number of unique values, most common value and its frequency, and the proportion of the most frequent value relative to the total. The goal is to visualize the overall distribution of the variables in the dataset, aiding in understanding the model's behavior and predicting its performance.

Test Mechanism

The testing mechanism utilizes two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function pulls out several summary statistics, while value_counts() accounts for unique values. The resulting data is formatted into two distinct tables, one for numerical and another for categorical variable summaries. These tables provide a clear summary of the main characteristics of the variables, which can be instrumental in assessing the model's performance.

Signs of High Risk

• Skewed data or significant outliers can represent high risk. For numerical data, this may be reflected via a significant difference between the mean and median (50% percentile).
• For categorical data, a lack of diversity (low count of unique values), or overdominance of a single category (high frequency of the top value) can indicate high risk.

Strengths

• Provides a comprehensive summary of the dataset, shedding light on the distribution and characteristics of the variables under consideration.
• It is a versatile and robust method, applicable to both numerical and categorical data.
• Helps highlight crucial anomalies such as outliers, extreme skewness, or lack of diversity, which are vital in understanding model behavior during testing and validation.

Limitations

• While this metric offers a high-level overview of the data, it may fail to detect subtle correlations or complex patterns.
• Does not offer any insights on the relationship between variables.
• Alone, descriptive statistics cannot be used to infer properties about future unseen data.
• Should be used in conjunction with other statistical tests to provide a comprehensive understanding of the model's data.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DescriptiveStatistics", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Now, let's run a few tests to assess the quality of the dataset:

result = vm.tests.run_test(
    test_id="validmind.data_validation.DescriptiveStatistics",
    inputs={"dataset": vm_raw_dataset},
)

Descriptive Statistics

The Descriptive Statistics test evaluates the distributional characteristics of numerical and categorical variables in the dataset. The results are presented in separate summary tables for eight numerical variables and two categorical variables, each based on 8,000 observations. For numerical fields, the tables report counts, central tendency, dispersion, and percentile values from the minimum through the 95th percentile and maximum. For categorical fields, the tables summarize the number of unique values, the most frequent category, and the frequency and proportion of that category.

Key insights:

• Complete coverage across all variables: All reported numerical and categorical variables have a count of 8,000, indicating that the summary statistics were computed on a fully populated sample for the variables shown.

• Balance shows pronounced asymmetry: Balance has a mean of 76,434.10 and a median of 97,264.00, with the 25th percentile at 0.00 and the minimum also at 0.00. This pattern indicates a substantial concentration at zero alongside a wide spread up to a maximum of 250,898.00.

• CreditScore and Age are broadly distributed: CreditScore spans from 350 to 850 with a mean of 650.16 and median of 652.00, while Age ranges from 18 to 92 with a mean of 38.95 and median of 37.00. In both cases, mean and median are relatively close compared with the full observed range.

• NumOfProducts is concentrated at low values: NumOfProducts has a mean of 1.53 and median of 1.00, with the 75th, 90th, and 95th percentiles at 2.00 and a maximum of 4.00. The distribution is concentrated primarily between 1 and 2 products.

• Binary indicators differ in prevalence: HasCrCard has a mean of 0.7026, while IsActiveMember has a mean of 0.5199. These values show that credit card ownership is more prevalent than active membership status in the sample.

• Categorical concentration is moderate: Geography contains 3 unique values, with France as the top category at 4,010 observations (50.12%), and Gender contains 2 unique values, with Male as the top category at 4,396 observations (54.95%). Neither categorical variable is dominated by an extreme single-category share based on the reported proportions.

The descriptive statistics show a complete dataset across the variables reported, with different distributional patterns across features. CreditScore, Age, and EstimatedSalary exhibit broad numeric ranges, while Balance stands out for its zero-valued lower quartile and wide dispersion. NumOfProducts is concentrated at lower values, and the binary and categorical variables show moderate class concentration rather than extreme dominance by a single value.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	8000.0	650.1596	96.8462	350.0	583.0	652.0	717.0	778.0	813.0	850.0
Age	8000.0	38.9489	10.4590	18.0	32.0	37.0	44.0	53.0	60.0	92.0
Tenure	8000.0	5.0339	2.8853	0.0	3.0	5.0	8.0	9.0	9.0	10.0
Balance	8000.0	76434.0965	62612.2513	0.0	0.0	97264.0	128045.0	149545.0	162488.0	250898.0
NumOfProducts	8000.0	1.5325	0.5805	1.0	1.0	1.0	2.0	2.0	2.0	4.0
HasCrCard	8000.0	0.7026	0.4571	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	8000.0	0.5199	0.4996	0.0	0.0	1.0	1.0	1.0	1.0	1.0
EstimatedSalary	8000.0	99790.1880	57520.5089	12.0	50857.0	99505.0	149216.0	179486.0	189997.0	199992.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	8000.0	3.0	France	4010.0	50.12
Gender	8000.0	2.0	Male	4396.0	54.95

result2 = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_raw_dataset},
    params={"min_percent_threshold": 30},
)

❌ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring each class’s share of total records against a specified minimum percentage threshold. In this run, the threshold was set to 30%, and the results are reported for the Exited target classes. The table and accompanying bar chart show that class Exited = 0 represents 79.80% of rows, while class Exited = 1 represents 20.20% of rows, with pass/fail status assigned for each class relative to the threshold.

Key insights:

• Majority class dominates distribution: Exited = 0 accounts for 79.80% of observations, making it the substantially larger class in the target distribution.
• Minority class falls below threshold: Exited = 1 represents 20.20% of observations, which is below the configured 30% minimum threshold and is therefore marked as Fail.
• Pass/fail outcome is split by class: The class-level test result passes for Exited = 0 and fails for Exited = 1, indicating that the threshold criterion is not met uniformly across the target classes.

The results show a two-class target distribution with a pronounced concentration in Exited = 0 and a smaller share in Exited = 1. Under the configured 30% threshold, only the majority class satisfies the minimum class representation criterion, while the minority class does not. Collectively, these findings document an imbalanced class distribution under the parameters used for this test.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	79.80%	Pass
1	20.20%	Fail

Figures

The output above shows that the validmind.data_validation.ClassImbalance test did not pass according to the value we set for min_percent_threshold.

To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.

As this is technically a different dataset, remember to first initialize a new ValidMind Dataset object to pass in as input as required by run_test():

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

# Pass the initialized `balanced_raw_dataset` as input into the test run
result = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_balanced_raw_dataset},
    params={"min_percent_threshold": 30},
)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring the percentage share of each class and comparing those shares against a minimum threshold. For the Exited target, the result table and chart show two classes, 0 and 1, each representing 50.00% of rows. The configured minimum percentage threshold for this run is 30%, and the result records a pass/fail status for each class against that threshold.

Key insights:

• Even class distribution observed: Both Exited = 0 and Exited = 1 account for 50.00% of the dataset, indicating an equal split between the two target classes.
• All classes exceed threshold: Each class is above the configured 30% minimum percentage threshold, and both classes are marked as Pass.
• No underrepresented target class: The results do not show any class with a share below the test threshold in this binary target distribution.

The test result shows a balanced binary target distribution for Exited, with both classes occurring at identical frequencies. Since each class represents 50.00% of observations and exceeds the 30% threshold, the test identifies no class-level imbalance under the configured criterion.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

Utilize test output

You can utilize the output from a ValidMind test for further use, for example, if you want to remove highly correlated features. Removing highly correlated features helps make the model simpler, more stable, and easier to understand.

Below we demonstrate how to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.

First, we'll run validmind.data_validation.HighPearsonCorrelation with the balanced_raw_dataset we initialized previously as input as is for comparison with later runs:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among dataset features to identify highly correlated pairs that may indicate redundancy or multicollinearity. The result table reports the top correlation pairs, their Pearson coefficients, and a Pass/Fail assessment against the configured absolute-value threshold of 0.3. Across the reported pairs, coefficients range from -0.1905 to 0.3462, with one pair exceeding the threshold and the remaining pairs classified as passing.

Key insights:

• One pair exceeds threshold: The pair (Age, Exited) has a Pearson correlation coefficient of 0.3462, which is the only reported result marked Fail against the 0.3 threshold.
• Most reported correlations are weak: The remaining reported coefficients fall between -0.1905 and 0.1492 in magnitude, and all are classified as Pass under the test threshold.
• Largest non-failing negative relationship is modest: (IsActiveMember, Exited) shows the strongest negative coefficient among the reported passing pairs at -0.1905, remaining below the threshold in absolute value.
• Other feature relationships are limited in magnitude: Pairs including (Balance, NumOfProducts) at -0.1789 and (Balance, Exited) at 0.1492 are among the larger remaining correlations, while the rest are closer to zero.

The reported correlation structure is characterized by a single pair above the configured threshold and a broader set of pairwise relationships with relatively low absolute correlation values. The most pronounced observed relationship is between Age and Exited, while all other reported feature pairs remain below the test cutoff. Overall, the table indicates limited high linear dependence among the reported top correlations, with the exception of the identified failing pair.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3462	Fail
(IsActiveMember, Exited)	-0.1905	Pass
(Balance, NumOfProducts)	-0.1789	Pass
(Balance, Exited)	0.1492	Pass
(Age, Balance)	0.0620	Pass
(NumOfProducts, Exited)	-0.0583	Pass
(Age, NumOfProducts)	-0.0549	Pass
(CreditScore, Exited)	-0.0450	Pass
(CreditScore, EstimatedSalary)	-0.0399	Pass
(Tenure, EstimatedSalary)	0.0376	Pass

The output above shows that the test did not pass according to the value we set for max_threshold.

corr_result is an object of type TestResult. We can inspect the result object to see what the test has produced:

print(type(corr_result))
print("Result ID: ", corr_result.result_id)
print("Params: ", corr_result.params)
print("Passed: ", corr_result.passed)
print("Tables: ", corr_result.tables)

<class 'validmind.vm_models.result.result.TestResult'>
Result ID:  validmind.data_validation.HighPearsonCorrelation
Params:  {'max_threshold': 0.3}
Passed:  False
Tables:  [ResultTable]

Let's remove the highly correlated features and create a new VM dataset object.

We'll begin by checking out the table in the result and extracting a list of features that failed the test:

# Extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3462	Fail
1	(IsActiveMember, Exited)	-0.1905	Pass
2	(Balance, NumOfProducts)	-0.1789	Pass
3	(Balance, Exited)	0.1492	Pass
4	(Age, Balance)	0.0620	Pass
5	(NumOfProducts, Exited)	-0.0583	Pass
6	(Age, NumOfProducts)	-0.0549	Pass
7	(CreditScore, Exited)	-0.0450	Pass
8	(CreditScore, EstimatedSalary)	-0.0399	Pass
9	(Tenure, EstimatedSalary)	0.0376	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

Next, extract the feature names from the list of strings (example: (Age, Exited) > Age):

high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

Now, it's time to re-initialize the dataset with the highly correlated features removed.

Note the use of a different input_id. This allows tracking the inputs used when running each individual test.

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

Re-running the test with the reduced feature set should pass the test:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant or highly collinear variables. The results table reports the top feature pairs ranked by Pearson correlation coefficient, along with Pass/Fail status based on the configured absolute-value threshold of 0.3. In this run, the reported coefficients range from -0.1905 to 0.1492, and all listed feature pairs are marked as Pass.

Key insights:

• No pair exceeds threshold: All reported absolute correlation coefficients are below the 0.3 threshold used in the test. Every listed feature pair therefore receives a Pass result.
• Strongest observed relationship is modest: The largest absolute coefficient is -0.1905 for (IsActiveMember, Exited), which remains below the configured threshold and indicates only a limited linear association within the reported pairs.
• Top correlations are concentrated below 0.20: The next-largest absolute coefficients are -0.1789 for (Balance, NumOfProducts) and 0.1492 for (Balance, Exited). The remaining reported coefficients are all below 0.06 in absolute value.
• Reported relationships include both directions: The table contains both negative and positive coefficients, with negative associations such as (IsActiveMember, Exited) and (Balance, NumOfProducts) and positive associations such as (Balance, Exited) and (Tenure, EstimatedSalary).

Overall, the reported Pearson correlation results do not show any feature pair with an absolute linear relationship above the configured 0.3 threshold. The strongest observed associations are modest in magnitude, and the majority of the listed top correlations are small. Based on the reported output, the tested feature pairs do not exhibit high pairwise linear correlation under this test configuration.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1905	Pass
(Balance, NumOfProducts)	-0.1789	Pass
(Balance, Exited)	0.1492	Pass
(NumOfProducts, Exited)	-0.0583	Pass
(CreditScore, Exited)	-0.0450	Pass
(CreditScore, EstimatedSalary)	-0.0399	Pass
(Tenure, EstimatedSalary)	0.0376	Pass
(CreditScore, IsActiveMember)	0.0327	Pass
(NumOfProducts, IsActiveMember)	0.0322	Pass
(Tenure, Exited)	-0.0318	Pass

You can also plot the correlation matrix to visualize the new correlation between features:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.PearsonCorrelationMatrix",
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

Pearson Correlation Matrix

The PearsonCorrelationMatrix test evaluates linear dependency among numerical variables by displaying pairwise Pearson correlation coefficients in a heat map. The result covers CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with coefficients ranging from -1 to 1 and 1.0 values along the diagonal. Off-diagonal correlations in the matrix are generally small in magnitude, with the largest observed pairwise relationships visible between IsActiveMember and Exited, Balance and NumOfProducts, and Balance and Exited.

Key insights:

• No high pairwise correlations observed: All off-diagonal correlations shown in the heat map are well below the 0.7 absolute threshold referenced in the test description. The matrix therefore does not show strongly collinear variable pairs among the numerical fields included.

• Exited has weak relationships with inputs: The target variable Exited shows its largest correlations with IsActiveMember (-0.19), Balance (0.15), and NumOfProducts (-0.06). Correlations with CreditScore (-0.04), Tenure (-0.03), HasCrCard (-0.01), and EstimatedSalary (-0.01) are near zero.

• Balance is the most connected feature: Balance has the largest observed feature-to-feature correlation with NumOfProducts at -0.18 and also shows a 0.15 correlation with Exited. Its remaining relationships are small, including 0.03 with EstimatedSalary and values near zero with the other variables.

• Most feature pairs are near zero: Several pairs, including CreditScore with Tenure (0.01), CreditScore with Balance (0.01), HasCrCard with NumOfProducts (-0.01), and EstimatedSalary with Exited (-0.01), exhibit negligible linear association. This pattern is consistent across most of the matrix.

The correlation matrix indicates a low overall level of linear dependency across the numerical variables included in the test. The most pronounced relationships are modest in magnitude, with the largest absolute correlations at -0.19, -0.18, and 0.15. Collectively, the results show limited redundancy among these variables on a Pearson linear-correlation basis.

Figures

Documenting test results

Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it.

Every test result returned by the run_test() function has a .log() method that can be used to send the test results to the ValidMind Platform:

• When using run_documentation_tests(), documentation sections will be automatically populated with the results of all tests registered in the documentation template.
• When logging individual test results to the platform, you'll need to manually add those results to the desired section of the documentation.

To demonstrate how to add test results to your documentation, we'll populate the entire Data Preparation section of the documentation using the clean vm_raw_dataset_preprocessed dataset as input, and then document an additional individual result for the highly correlated dataset vm_balanced_raw_dataset.

Run and log multiple tests

run_documentation_tests() allows you to run multiple tests at once and automatically log the results to your documentation. Below, we'll run the tests using the previously initialized vm_raw_dataset_preprocessed as input — this will populate the entire Data Preparation section for every test that is part of the documentation template.

For this example, we'll pass in the following arguments:

• inputs: Any inputs to be passed to the tests.
• config: A dictionary <test_id>:<test_config> that allows configuring each test individually. Each test config requires the following:
  • params: Individual test parameters.
  • inputs: Individual test inputs. This overrides any inputs passed from the run_documentation_tests() function.

When including explicit configuration for individual tests, you'll need to specify the inputs even if they mirror what is included in your global configuration.

# Individual test config with inputs specified
test_config = {
    "validmind.data_validation.ClassImbalance": {
        "params": {"min_percent_threshold": 30},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "params": {"max_threshold": 0.3},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
}

# Global test config
tests_suite = vm.run_documentation_tests(
    inputs={
        "dataset": vm_raw_dataset_preprocessed,
    },
    config=test_config,
    section=["data_preparation"],
)

Test suite complete!

26/26 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Data Preparation

Data Preparation

▶ Test Result: Dataset Description (validmind.data_validation.DatasetDescription)

Dataset Description

The Dataset Description test evaluates the structure, completeness, and cardinality of each dataset column. The results summarize 10 variables across numeric and categorical types, reporting count, missingness, and distinct-value levels for each field. All listed columns contain 3,232 observed values with no missing entries, and the output shows variation in feature cardinality from binary categorical fields to fully unique numeric values.

Key insights:

• No missing values observed: All 10 columns report 3,232 non-missing observations and 0.0% missingness, indicating complete population coverage within the analyzed dataset.

• EstimatedSalary is fully unique: EstimatedSalary has 3,232 distinct values, corresponding to a distinct ratio of 1.0, making it the highest-cardinality field in the dataset.

• Balance also shows high cardinality: Balance contains 2,188 distinct values, representing 67.7% of observations, which is materially higher than the distinctness levels of the other numeric features aside from EstimatedSalary.

• Categorical fields are low-cardinality: Geography has 3 distinct values, while Gender, HasCrCard, IsActiveMember, and Exited each have 2 distinct values, indicating compact category sets across these fields.

• Numeric variables vary substantially in granularity: Numeric columns range from 4 distinct values for NumOfProducts and 11 for Tenure to 434 for CreditScore, 2,188 for Balance, and 3,232 for EstimatedSalary, showing a wide spread in observed value granularity.

The dataset description indicates a fully populated input table with no missing values across all reported variables. The most prominent structural characteristic is the contrast between low-cardinality categorical and discrete numeric fields and the much higher-cardinality continuous numeric fields, particularly EstimatedSalary and Balance. Overall, the results document a complete dataset with heterogeneous feature granularity across the 10 analyzed columns.

Tables

Dataset Description

Name	Type	Count	Missing	Missing %	Distinct	Distinct %
CreditScore	Numeric	3232.0	0	0.0	434	0.1343
Geography	Categorical	3232.0	0	0.0	3	0.0009
Gender	Categorical	3232.0	0	0.0	2	0.0006
Tenure	Numeric	3232.0	0	0.0	11	0.0034
Balance	Numeric	3232.0	0	0.0	2188	0.6770
NumOfProducts	Numeric	3232.0	0	0.0	4	0.0012
HasCrCard	Categorical	3232.0	0	0.0	2	0.0006
IsActiveMember	Categorical	3232.0	0	0.0	2	0.0006
EstimatedSalary	Numeric	3232.0	0	0.0	3232	1.0000
Exited	Categorical	3232.0	0	0.0	2	0.0006

▶ Test Result: Class Imbalance (validmind.data_validation.ClassImbalance)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes by measuring the percentage share of each class and comparing those shares to the configured minimum threshold. In this run, the target variable Exited contains two classes, 0 and 1, and the results table reports each class at 50.00% of rows. The applied threshold is 30%, and the accompanying plot shows equal bar heights for both classes, consistent with the tabulated percentages. Both classes are marked as Pass under the test criterion.

Key insights:

• Even class distribution observed: The Exited target is split evenly, with class 0 at 50.00% and class 1 at 50.00% of total rows.
• All classes exceed threshold: Both observed classes are above the configured minimum percentage threshold of 30%, and each receives a Pass result.
• No class-level disparity in frequency: The chart and table show identical representation across the two target classes, with no difference in reported class proportions.

The test result shows a balanced binary target distribution for Exited under the configured class imbalance criterion. Both classes are equally represented and both pass the 30% minimum threshold check. Based on the reported percentages and pass/fail outcomes, the dataset does not exhibit class imbalance within this test definition.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

▶ Test Result: Duplicates (validmind.data_validation.Duplicates)

✅ Duplicates

The Duplicates test evaluates the dataset for exact duplicate rows to quantify the extent of repeated records in the data used for modeling. The result table reports both the absolute number of duplicate rows and their share of the dataset. In this run, the table shows 0 duplicate rows and a corresponding duplicate rate of 0.0%.

Key insights:

• No duplicate rows detected: The dataset contains 0 duplicate rows based on the test output, indicating that no exact row-level repetitions were identified.
• Duplicate rate is zero: The reported percentage of duplicate rows is 0.0%, showing that duplicates do not contribute to the observed dataset composition.

The test result indicates that no exact duplicate records were identified in the evaluated dataset. Both the count and percentage measures are zero, showing no observable row-level duplication in this test run.

Tables

Duplicate Rows Results for Dataset

Number of Duplicates	Percentage of Rows (%)
0	0.0

▶ Test Result: High Cardinality (validmind.data_validation.HighCardinality)

✅ High Cardinality

The High Cardinality test evaluates the number of unique values in categorical columns to identify features with large numbers of distinct categories. The result table reports, for each categorical column assessed, the number of distinct values, the percentage of distinct values, and the corresponding pass/fail outcome against the test threshold. In this run, two categorical columns were evaluated: Geography and Gender. Both columns are reported as passing, with 3 distinct values for Geography and 2 distinct values for Gender.

Key insights:

• All assessed categorical columns passed: Both categorical features included in the result table, Geography and Gender, received a Pass outcome under the applied high-cardinality threshold.
• Low distinct counts across features: Geography contains 3 distinct values and Gender contains 2 distinct values, indicating a small number of unique categories in each assessed column.
• Distinct-value percentages are minimal: The reported percentage of distinct values is 0.0928% for Geography and 0.0619% for Gender, reflecting limited uniqueness relative to the dataset size used in the test.

The test results show that the categorical columns assessed in this run do not exhibit high cardinality under the applied threshold. Geography and Gender both have low distinct counts and low distinct-value percentages, and neither column triggered a failure in the test output.

Tables

Column	Number of Distinct Values	Percentage of Distinct Values (%)	Pass/Fail
Geography	3	0.0928	Pass
Gender	2	0.0619	Pass

▶ Test Result: Missing Values (validmind.data_validation.MissingValues)

✅ Missing Values

The Missing Values test evaluates dataset completeness by measuring the proportion of missing values in each feature against the configured threshold. The results are presented at the column level, showing the number and percentage of missing values together with a pass/fail outcome for each feature. In this test run, all listed variables report 0 missing values and 0.0% missingness, and each column is marked as Pass.

Key insights:

• No missing values detected: All 10 evaluated columns show 0 missing values, corresponding to 0.0% missingness in every case.

• All features passed the test: Every variable in the results table is marked as Pass, indicating no column exceeded the missing-value threshold used in the test.

• Completeness is consistent across fields: Missingness results are uniform across numeric, categorical, and target fields included in the table, with no variation in observed completeness.

The test results show complete observed data coverage across all evaluated features in this dataset snapshot. No column-level missingness is present in the reported variables, and the pass/fail outcomes are uniformly favorable. Taken together, the results indicate that missing values were not identified as a data quality issue in this test run.

Tables

Column	Number of Missing Values	Percentage of Missing Values (%)	Pass/Fail
CreditScore	0	0.0	Pass
Geography	0	0.0	Pass
Gender	0	0.0	Pass
Tenure	0	0.0	Pass
Balance	0	0.0	Pass
NumOfProducts	0	0.0	Pass
HasCrCard	0	0.0	Pass
IsActiveMember	0	0.0	Pass
EstimatedSalary	0	0.0	Pass
Exited	0	0.0	Pass

▶ Test Result: Skewness (validmind.data_validation.Skewness)

❌ Skewness

The Skewness test evaluates the asymmetry of numerical feature distributions by calculating skewness for each numeric column and comparing the result to the defined maximum threshold of 1. The results table reports skewness values and pass/fail outcomes for eight numeric variables in the dataset. Seven columns are marked as passing the test, while one column exceeds the threshold and is marked as failing. Observed skewness values range from -0.8775 to 1.2233 across the evaluated variables.

Key insights:

• Single threshold breach identified: NumOfProducts records skewness of 1.2233 and is the only column with a fail result, indicating asymmetry above the maximum threshold of 1.
• Most variables show limited skewness: Seven of eight evaluated columns pass the test, with skewness values for these variables ranging from -0.8775 to 0.1428.
• Near-symmetric distributions in several fields: Exited has skewness of 0.0, while Tenure (-0.004) and EstimatedSalary (-0.0095) are also very close to zero.
• Negative skewness is present but within threshold: HasCrCard shows the strongest negative skewness at -0.8775, followed by Balance at -0.2707 and CreditScore at -0.0737, and all remain within the passing threshold.

The skewness assessment shows that the dataset is largely within the defined asymmetry tolerance, with seven of eight numeric variables passing the threshold-based evaluation. The only exception is NumOfProducts, which exceeds the threshold at 1.2233 and stands apart from the remaining variables. Several fields are effectively symmetric based on values near zero, while the observed negative skewness in other passing variables remains below the test limit.

Tables

Skewness Results for Dataset

Column	Skewness	Pass/Fail
CreditScore	-0.0737	Pass
Tenure	-0.0040	Pass
Balance	-0.2707	Pass
NumOfProducts	1.2233	Fail
HasCrCard	-0.8775	Pass
IsActiveMember	0.1428	Pass
EstimatedSalary	-0.0095	Pass
Exited	0.0000	Pass

▶ Test Result: Unique Rows (validmind.data_validation.UniqueRows)

❌ Unique Rows

The UniqueRows test evaluates dataset diversity by comparing the proportion of unique values in each column against a prescribed minimum threshold. The results are reported at the column level, showing the number of unique values, the percentage of unique values, and the corresponding pass/fail outcome for each field. In this run, three columns passed the test and seven failed. Percentages of unique values range from 0.0619% to 100.0%, with EstimatedSalary showing full uniqueness and several binary or low-cardinality fields showing very limited uniqueness.

Key insights:

• Only three columns passed: CreditScore, Balance, and EstimatedSalary met the uniqueness threshold, with 13.4282%, 67.698%, and 100.0% unique values respectively.
• EstimatedSalary is fully unique: EstimatedSalary contains 3,232 unique values and a uniqueness rate of 100.0%, the highest observed in the test output.
• Balance shows high uniqueness: Balance has 2,188 unique values, corresponding to 67.698% unique values, making it the second most diverse column in the result set.
• Most failed columns are low-cardinality: Geography (3 unique values), Gender (2), Tenure (11), NumOfProducts (4), HasCrCard (2), IsActiveMember (2), and Exited (2) all failed, with uniqueness percentages between 0.0619% and 0.3403%.
• Several variables are binary: Gender, HasCrCard, IsActiveMember, and Exited each contain 2 unique values and share the same uniqueness percentage of 0.0619%.

The test results show a mixed uniqueness profile across the dataset, with high uniqueness concentrated in EstimatedSalary and Balance, and moderate uniqueness in CreditScore. The remaining columns have low observed cardinality and did not meet the configured threshold. Overall, the result is characterized by a small set of high-diversity numerical fields alongside a larger group of low-diversity categorical or discrete fields.

Tables

Column	Number of Unique Values	Percentage of Unique Values (%)	Pass/Fail
CreditScore	434	13.4282	Pass
Geography	3	0.0928	Fail
Gender	2	0.0619	Fail
Tenure	11	0.3403	Fail
Balance	2188	67.6980	Pass
NumOfProducts	4	0.1238	Fail
HasCrCard	2	0.0619	Fail
IsActiveMember	2	0.0619	Fail
EstimatedSalary	3232	100.0000	Pass
Exited	2	0.0619	Fail

▶ Test Result: Too Many Zero Values (validmind.data_validation.TooManyZeroValues)

❌ Too Many Zero Values

The TooManyZeroValues test evaluates numerical columns for excessive concentrations of zero values relative to the dataset row count. The results table reports, for each assessed variable, the total number of rows, the number of zero values, the percentage of zero values, and the corresponding pass/fail status against the configured threshold. Four numerical variables are listed in the output—Tenure, Balance, HasCrCard, and IsActiveMember—and each is marked as failing the test. Reported zero-value percentages range from 4.5173% to 53.5582%.

Key insights:

• All assessed variables failed: Each of the four reported numerical variables exceeded the test threshold and received a fail status, indicating that zero values were present above the configured limit in every assessed column.

• IsActiveMember has the highest zero share: IsActiveMember contains 1,731 zero values out of 3,232 rows, corresponding to 53.5582%, which is the largest zero-value concentration among the reported variables.

• Balance shows substantial zero concentration: Balance contains 1,044 zero values out of 3,232 rows, representing 32.302% of observations, making it the second-highest zero-percentage variable in the result set.

• Binary-style indicators also contain many zeros: HasCrCard records 967 zero values, or 29.9196%, and IsActiveMember records 53.5582%, showing that both listed indicator-like variables have material zero concentrations.

• Tenure has the lowest but still failing rate: Tenure records 146 zero values out of 3,232 rows, equal to 4.5173%, which is the smallest zero-value percentage reported but still above the test threshold.

The test results show that all four reported numerical variables exceeded the zero-value threshold, with material variation in magnitude across columns. The highest concentrations are observed in IsActiveMember and Balance, followed by HasCrCard, while Tenure has a comparatively lower but still failing zero share. Collectively, the output indicates that zero values are a widespread characteristic across the numerical variables included in this test result.

Tables

Variable	Row Count	Number of Zero Values	Percentage of Zero Values (%)	Pass/Fail
Tenure	3232	146	4.5173	Fail
Balance	3232	1044	32.3020	Fail
HasCrCard	3232	967	29.9196	Fail
IsActiveMember	3232	1731	53.5582	Fail

▶ Test Result: IQR Outliers Table (validmind.data_validation.IQROutliersTable)

IQR Outliers Table

The Interquartile Range Outliers Table test identifies numerical-feature outliers using the IQR method and summarizes their frequency and distribution. The results table reports outliers for two variables: CreditScore and NumOfProducts. For each variable, the table shows the total count of detected outliers, the mean value of the full variable, and the minimum, quartile, median, and maximum values observed among the outlier subset.

Key insights:

• Outliers are concentrated in two variables: The test detected outliers only for CreditScore and NumOfProducts, with 11 and 44 outliers respectively.
• NumOfProducts outliers are uniform: All detected outliers for NumOfProducts take the value 4, as indicated by identical minimum, 25th percentile, median, 75th percentile, and maximum outlier values.
• CreditScore outliers are low-end values: CreditScore outliers range from 350 to 376, with outlier quartiles at 350.0, 359.0, and 370.0, while the reported mean of the variable is 649.6761.
• NumOfProducts shows higher outlier frequency: NumOfProducts has 44 detected outliers compared with 11 for CreditScore, indicating a larger concentration of observations outside the IQR bounds for that feature.

The IQR-based outlier summary shows that detected outliers are limited to two numerical variables and exhibit distinct patterns across them. NumOfProducts outliers are concentrated entirely at a single value, while CreditScore outliers span a narrow low-value range relative to the reported variable mean. Collectively, the result indicates that the observed outlier behavior is localized rather than broadly distributed across multiple reported features.

Tables

Summary of Outliers Detected by IQR Method

Variable	Total Count of Outliers	Mean Value of Variable	Minimum Outlier Value	Outlier Value at 25th Percentile	Outlier Value at 50th Percentile	Outlier Value at 75th Percentile	Maximum Outlier Value
CreditScore	11	649.6761	350	350.0	359.0	370.0	376
NumOfProducts	44	1.5111	4	4.0	4.0	4.0	4

▶ Test Result: IQR Outliers Bar Plot (validmind.data_validation.IQROutliersBarPlot)

IQR Outliers Bar Plot

The IQROutliersBarPlot test evaluates the distribution of outliers across percentile ranges for numeric features using the interquartile range method. The results are shown as bar plots by feature, with outlier counts reported across the 0–25, 25–50, 50–75, and 75–100 percentile bands. In the displayed results, CreditScore has outliers in the upper three percentile bands, while NumOfProducts shows outliers only in the 75–100 percentile band.

Key insights:

• CreditScore outliers are distributed above the first quartile: CreditScore shows 5 outliers in the 25–50 percentile band, 3 in the 50–75 band, and 3 in the 75–100 band, with no outliers in the 0–25 band.
• NumOfProducts outliers are concentrated in the top percentile band: NumOfProducts shows 44 outliers in the 75–100 percentile band and no outliers in the 0–25, 25–50, or 50–75 bands.
• Outlier volume differs materially by feature: The observed outlier count for NumOfProducts is substantially higher than for CreditScore, with 44 outliers versus 11 total outliers across the displayed percentile bands for CreditScore.

The results show two distinct outlier patterns across the displayed features. CreditScore has a moderate number of outliers spread across the 25–100 percentile range, whereas NumOfProducts exhibits a much larger concentration of outliers entirely in the highest percentile band. This indicates that the detected outlier behavior is feature-specific and more concentrated in the upper tail for NumOfProducts than for CreditScore.

Figures

▶ Test Result: Descriptive Statistics (validmind.data_validation.DescriptiveStatistics)

Descriptive Statistics

The Descriptive Statistics test evaluates the distributional characteristics of numerical and categorical variables in the dataset. The results are presented in separate summary tables for seven numerical variables and two categorical variables, showing counts, central tendency, dispersion, percentile ranges, and category concentration. All reported variables have a count of 3,232 observations, and the categorical summary identifies the number of unique values, the most frequent category, and its observed frequency share.

Key insights:

• Complete coverage across reported variables: Each numerical and categorical variable has a reported count of 3,232, indicating consistent observation counts across the summarized fields.

• Balance shows pronounced dispersion: Balance has a mean of 81,839.3185, a median of 103,413, a standard deviation of 61,729.1455, and a minimum of 0. The 25th percentile is also 0, while the 75th percentile reaches 129,514, indicating a wide spread with substantial mass at zero.

• Salary is widely distributed: EstimatedSalary ranges from 12 to 199,992, with a mean of 100,524.5484 and a median of 101,892. Its standard deviation of 57,668.6825 indicates broad dispersion across the observed salary values.

• Credit score is centered near its median: CreditScore has a mean of 649.6761 and a median of 649, with quartiles at 582 and 718 and an observed range from 350 to 850. The close alignment between mean and median indicates limited difference in central tendency measures.

• Product holdings are concentrated at lower values: NumOfProducts has a median of 1, a 75th percentile of 2, a mean of 1.5111, and a maximum of 4. This indicates that most observed records are concentrated in the lower product-count range.

• Binary indicators are unevenly distributed: HasCrCard has a mean of 0.7008, while IsActiveMember has a mean of 0.4644. These values indicate different class balances across the two binary variables, with credit card ownership more prevalent than active membership.

• Categorical concentration is moderate: Geography contains 3 unique values, with France as the top category at 1,475 observations or 45.64%. Gender contains 2 unique values, with Female as the top category at 1,624 observations or 50.25%, indicating near-even representation by frequency.

The descriptive statistics show a dataset with uniform record counts across the reported fields and mixed distributional characteristics across variables. The most pronounced dispersion appears in Balance and EstimatedSalary, while CreditScore is centered closely around its median and NumOfProducts is concentrated at lower values. The categorical variables show limited cardinality with moderate concentration in the top category, particularly a near-even split for Gender and a less-than-majority leading category for Geography.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	3232.0	649.6761	98.5005	350.0	582.0	649.0	718.0	780.0	818.0	850.0
Tenure	3232.0	5.0436	2.9095	0.0	3.0	5.0	8.0	9.0	10.0	10.0
Balance	3232.0	81839.3185	61729.1455	0.0	0.0	103413.0	129514.0	150924.0	164265.0	250898.0
NumOfProducts	3232.0	1.5111	0.6694	1.0	1.0	1.0	2.0	2.0	3.0	4.0
HasCrCard	3232.0	0.7008	0.4580	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	3232.0	0.4644	0.4988	0.0	0.0	0.0	1.0	1.0	1.0	1.0
EstimatedSalary	3232.0	100524.5484	57668.6825	12.0	50688.0	101892.0	149653.0	179831.0	190385.0	199992.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	3232.0	3.0	France	1475.0	45.64
Gender	3232.0	2.0	Female	1624.0	50.25

▶ Test Result: Pearson Correlation Matrix (validmind.data_validation.PearsonCorrelationMatrix)

Pearson Correlation Matrix

The PearsonCorrelationMatrix test evaluates linear dependency between numerical variables by displaying pairwise Pearson correlation coefficients across the dataset. The result is presented as a symmetric heat map covering CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with coefficients ranging from -1 to 1 and 1.0 on the diagonal. Most off-diagonal values are close to zero, with the largest observed non-diagonal magnitudes appearing in the relationships between IsActiveMember and Exited (-0.19), Balance and NumOfProducts (-0.18), and Balance and Exited (0.15).

Key insights:

• Correlations are uniformly low: All displayed off-diagonal correlations are small in magnitude, and none approach the 0.7 absolute-value threshold highlighted by the test methodology.
• Exited has limited linear association: Exited shows weak correlations with the other variables, with the largest magnitudes observed for IsActiveMember (-0.19), Balance (0.15), NumOfProducts (-0.06), CreditScore (-0.04), and Tenure (-0.03).
• Balance shows the strongest feature linkage: Among predictor-to-predictor relationships, Balance and NumOfProducts exhibit the largest absolute correlation at -0.18, while all other pairwise feature correlations remain closer to zero.
• Most feature pairs are near independent linearly: Variables such as CreditScore, Tenure, HasCrCard, and EstimatedSalary show pairwise coefficients generally between approximately -0.04 and 0.04 across most relationships.

The correlation structure is sparse and characterized by weak linear relationships across the numerical variables included in the matrix. No pairwise relationship in the displayed result indicates high linear dependency under the test’s stated threshold. The strongest observed associations are limited to small-magnitude relationships involving Exited and the Balance–NumOfProducts pair, while the remaining variables exhibit correlations clustered near zero.

Figures

▶ Test Result: High Pearson Correlation (validmind.data_validation.HighPearsonCorrelation)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant variables or multicollinearity. The results table reports the top feature pairs ranked by absolute Pearson correlation coefficient, along with pass/fail status based on the configured threshold of 0.3. In this run, the reported coefficients range from -0.1905 to 0.1492, and all listed feature pairs are marked as Pass.

Key insights:

• No correlations exceed threshold: All reported absolute correlation coefficients are below the 0.3 threshold. Every listed feature pair is therefore classified as Pass in the test output.
• Strongest relationship is modest: The largest absolute correlation is -0.1905 between IsActiveMember and Exited, which remains materially below the configured threshold.
• Top relationships are concentrated below 0.20: The next largest absolute coefficients are -0.1789 for Balance and NumOfProducts, and 0.1492 for Balance and Exited. These values indicate that the strongest observed linear relationships in the reported output remain limited in magnitude.
• Remaining associations are weak: The other listed pairs have coefficients between -0.0583 and 0.0376 in magnitude, including NumOfProducts with Exited, CreditScore with Exited, and Tenure with EstimatedSalary.

The reported correlation structure shows that none of the top-ranked feature pairs crosses the configured high-correlation threshold. The strongest observed pairwise relationships are modest, with the highest absolute coefficient remaining below 0.2. Taken together, the test output indicates limited linear dependence among the reported feature pairs under this threshold setting.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1905	Pass
(Balance, NumOfProducts)	-0.1789	Pass
(Balance, Exited)	0.1492	Pass
(NumOfProducts, Exited)	-0.0583	Pass
(CreditScore, Exited)	-0.0450	Pass
(CreditScore, EstimatedSalary)	-0.0399	Pass
(Tenure, EstimatedSalary)	0.0376	Pass
(CreditScore, IsActiveMember)	0.0327	Pass
(NumOfProducts, IsActiveMember)	0.0322	Pass
(Tenure, Exited)	-0.0318	Pass

Run and log an individual test

Next, we'll use the previously initialized vm_balanced_raw_dataset (that still has a highly correlated Age column) as input to run an individual test, then log the result to the ValidMind Platform.

When running individual tests, you can use a custom result_id to tag the individual result with a unique identifier:

• This result_id can be appended to test_id with a : separator.
• The balanced_raw_dataset result identifier will correspond to the balanced_raw_dataset input, the dataset that still has the Age column.

result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)
result.log()

❌ High Pearson Correlation Balanced Raw Dataset

The High Pearson Correlation test evaluates pairwise linear relationships between features to identify highly correlated variable pairs that may indicate redundancy or multicollinearity. The result table reports the top 10 feature pairs ranked by Pearson correlation coefficient, together with a Pass/Fail assessment based on the configured absolute-value threshold of 0.3. Observed coefficients range from -0.1905 to 0.3462, with one pair exceeding the threshold and the remaining pairs falling below it.

Key insights:

• Single threshold breach identified: The pair (Age, Exited) records the highest correlation at 0.3462 and is the only entry marked Fail, exceeding the configured threshold of 0.3.
• Remaining correlations are low in magnitude: The other nine reported pairs have absolute correlation values between 0.0376 and 0.1905, and all are marked Pass under the test threshold.
• Top relationships are mixed in direction: Among the strongest reported pairs, both positive and negative coefficients appear, including (IsActiveMember, Exited) = -0.1905 and (Balance, NumOfProducts) = -0.1789, indicating that the reported linear relationships are not uniformly directional.
• Largest non-failing coefficients remain well below the threshold: After the failing (Age, Exited) pair, the next strongest absolute correlations are 0.1905, 0.1789, and 0.1492, leaving a gap relative to the 0.3 threshold.

The reported correlation structure is limited in concentration, with only one pair crossing the configured threshold and all other top-ranked relationships remaining below it. The strongest observed relationship is between Age and Exited, while the rest of the listed pairs show comparatively smaller linear associations. Overall, the test output indicates that high Pearson correlation is isolated within the reported top pair rather than broadly present across the displayed feature pairs.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3462	Fail
(IsActiveMember, Exited)	-0.1905	Pass
(Balance, NumOfProducts)	-0.1789	Pass
(Balance, Exited)	0.1492	Pass
(Age, Balance)	0.0620	Pass
(NumOfProducts, Exited)	-0.0583	Pass
(Age, NumOfProducts)	-0.0549	Pass
(CreditScore, Exited)	-0.0450	Pass
(CreditScore, EstimatedSalary)	-0.0399	Pass
(Tenure, EstimatedSalary)	0.0376	Pass

2026-09-11 23:47:03,999 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset does not exist in model's document

Note the output returned indicating that a test-driven block doesn't currently exist in your documentation for this particular test ID.

That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.

Add individual test results to documentation

With the test results logged, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation (Learn more: Work with test results):

1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

2. In the left sidebar that appears for your model, click Development under Documents.

3. Locate the Data Preparation section and click on 2.3. Correlations and Interactions to expand that section.

4. Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a + button appears, indicating that you can insert a new block.

   
   
   

5. Click + and then select Test-Driven Block under FROM LIBRARY:

   • Click on VM Library under TEST-DRIVEN in the left sidebar.
   • In the search bar, type in HighPearsonCorrelation.
   • Select HighPearsonCorrelation:balanced_raw_dataset as the test.

   A preview of the test gets shown:

   
   
   

6. Finally, click Insert 1 Test Result to Document to add the test result to the documentation.

   Confirm that the individual results for the high correlation test has been correctly inserted into section 2.3. Correlations and Interactions of the documentation.

7. Finalize the documentation by editing the test result's description block to explain the changes you made to the raw data and the reasons behind them as shown in the screenshot below:

   

Running model evaluation tests

So far, we've focused on the data assessment and pre-processing that usually occurs prior to any models being built. Now, let's instead assume we have already built a model and we want to incorporate some model results into our documentation.

Train simple logistic regression model

Using ValidMind tests, we'll train a simple logistic regression model on our dataset and evaluate its performance by using the LogisticRegression class from the sklearn.linear_model.

To start, let's grab the first few rows from the balanced_raw_no_age_df dataset with the highly correlated features removed we initialized earlier:

balanced_raw_no_age_df.head()

	CreditScore	Geography	Gender	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
5862	789	France	Male	6	0.00	2	1	0	103603.65	0
2855	778	Germany	Male	9	182275.23	2	1	0	190631.23	0
5605	464	Germany	Male	4	155994.15	1	0	0	143665.44	0
3687	610	Spain	Female	3	0.00	1	1	0	38276.84	1
247	614	France	Male	4	137104.47	1	1	0	127166.49	1

Before training the model, we need to encode the categorical features in the dataset:

• Use the OneHotEncoder class from the sklearn.preprocessing module to encode the categorical features.
• The categorical features in the dataset are Geography and Gender.

balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
5862	789	6	0.00	2	1	0	103603.65	0	False	False	True
2855	778	9	182275.23	2	1	0	190631.23	0	True	False	True
5605	464	4	155994.15	1	0	0	143665.44	0	True	False	True
3687	610	3	0.00	1	1	0	38276.84	1	False	True	False
247	614	4	137104.47	1	1	0	127166.49	1	False	False	True

We'll split our preprocessed dataset into training and testing, to help assess how well the model generalizes to unseen data:

• We start by dividing our balanced_raw_no_age_df dataset into training and test subsets using train_test_split, with 80% of the data allocated to training (train_df) and 20% to testing (test_df).
• From each subset, we separate the features (all columns except "Exited") into X_train and X_test, and the target column ("Exited") into y_train and y_test.

from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

Then using GridSearchCV, we'll find the best-performing hyperparameters or settings and save them:

from sklearn.linear_model import LogisticRegression

# Logistic Regression grid params
log_reg_params = {
    "penalty": ["l1", "l2"],
    "C": [0.001, 0.01, 0.1, 1, 10, 100, 1000],
    "solver": ["liblinear"],
}

# Grid search for Logistic Regression
from sklearn.model_selection import GridSearchCV

grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)
grid_log_reg.fit(X_train, y_train)

# Logistic Regression best estimator
log_reg = grid_log_reg.best_estimator_

Initialize ValidMind datasets

The last step for evaluating the model's performance is to initialize the ValidMind Dataset and Model objects in preparation for assigning model predictions to each dataset.

# Initialize the datasets into their own dataset objects
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

Initialize a ValidMind model

You'll also need to initialize a ValidMind model object (vm_model) that can be passed to other functions for analysis and tests on the data for each of our three models.

• Despite the naming convention, ValidMind model objects can be any type of record you want to test, document, validate, or monitor with the ValidMind Library.
• From classical statistical and machine learning models, to generative and agentic AI systems and more, the ValidMind model object provides a consistent wrapper around your record so it can be passed as a unified input to any ValidMind test or test suite, with results sent directly to the ValidMind Platform.

Initialize your model object with vm.init_model():

# Register the model
vm_model = vm.init_model(log_reg, input_id="log_reg_model_v1")

Assign predictions

Once the model has been registered you can assign predictions to the training and testing datasets.

• The assign_predictions() method from the Dataset object can link existing predictions to any number of models.
• This method links the model's class prediction values and probabilities to our vm_train_ds and vm_test_ds datasets.

If no prediction values are passed, the method will compute predictions automatically:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

2026-09-11 23:47:05,125 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-09-11 23:47:05,128 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-09-11 23:47:05,128 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-09-11 23:47:05,131 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-09-11 23:47:05,133 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-09-11 23:47:05,135 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-09-11 23:47:05,136 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-09-11 23:47:05,137 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Run the model evaluation tests

In this next example, we'll focus on running the tests within the Model Development section of the documentation. Only tests associated with this section will be executed, and the corresponding results will be updated in the documentation.

• Note the additional config that is passed to run_documentation_tests() — this allows you to override inputs or params in certain tests.
• In our case, we want to explicitly use the vm_train_ds for the validmind.model_validation.sklearn.ClassifierPerformance:in_sample test, since it's supposed to run on the training dataset and not the test dataset.

test_config = {
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "dataset": vm_train_ds,
            "model": vm_model,
        },
    }
}
results = vm.run_documentation_tests(
    section=["model_development"],
    inputs={
        "dataset": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds
        "model": vm_model,
        "datasets": (
            vm_train_ds,
            vm_test_ds,
        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds
    },
    config=test_config,
)

Test suite complete!

34/34 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Development

Model Development

▶ Test Result: Model Metadata (validmind.model_validation.ModelMetadata)

Model Metadata

The ModelMetadata test compares core metadata fields across models to provide a standardized view of implementation characteristics. The result is presented as a summary table covering modeling technique, modeling framework, framework version, and programming language. In this test output, one model entry is reported with values populated for each of these metadata fields.

Key insights:

• Single model metadata record: The summary table contains one model entry, indicating that the comparison output in this result is limited to a single reported model.
• Framework and technique identified: The model is documented with modeling technique SKlearnModel and modeling framework sklearn, providing a consistent identification of the implementation stack in the reported metadata.
• Version and language are specified: The framework version is listed as 1.7.2 and the programming language is listed as Python, with no missing values shown in the reported fields.

The reported metadata provides a complete high-level implementation profile for the model entry included in this test result. The table documents the modeling technique, framework, framework version, and programming language without missing values in the displayed fields. As presented, the result serves as a concise reference for the model’s recorded technical metadata.

Tables

Modeling Technique	Modeling Framework	Framework Version	Programming Language
SKlearnModel	sklearn	1.7.2	Python

▶ Test Result: Dataset Split (validmind.data_validation.DatasetSplit)

Dataset Split

The DatasetSplit test evaluates the allocation of records across the available model datasets. The results are presented as a table showing the absolute size and share of total observations for each split. In this run, the table includes training and testing datasets along with the overall total, allowing the distribution of records across model development splits to be directly observed.

Key insights:

• Training data is the dominant split: train_dataset_final contains 2,585 records, representing 79.98% of the total dataset.
• Test data comprises one-fifth of records: test_dataset_final contains 647 records, representing 20.02% of the total dataset.
• Two-way split is reflected in results: The reported datasets consist of training and testing only, with a combined total of 3,232 records shown in the summary row.

The observed dataset allocation is concentrated in the training split, with the remaining records assigned to testing. The reported results show a total of 3,232 observations distributed across two documented datasets: 2,585 in training and 647 in testing. No separate validation dataset is listed in the test output.

Tables

Dataset	Size	Proportion
train_dataset_final	2585	79.98%
test_dataset_final	647	20.02%
Total	3232	100%

▶ Test Result: Population Stability Index (validmind.model_validation.sklearn.PopulationStabilityIndex)

Population Stability Index

The Population Stability Index (PSI) test evaluates the stability of model output distributions by comparing the bin-level distribution of observations between the initial and new datasets. The results are presented across 10 bins with counts, population shares, and bin-level PSI contributions for train_dataset_final and test_dataset_final. The total PSI reported for the comparison is 0.0039, with bin-level PSI contributions ranging from 0.0 to 0.0015. The accompanying plot shows that the initial and new population ratios are closely aligned across bins, with only small differences in selected upper and lower bins.

Key insights:

• Overall PSI is very low: The total PSI for train_dataset_final versus test_dataset_final is 0.0039, indicating that the aggregate distribution difference captured by this test is small.
• Bin distributions are closely aligned: Population shares are similar across all bins, including Bin 1 at 8.4720% versus 8.5008% and Bin 6 at 11.1412% versus 11.4374%, with only modest variation elsewhere.
• Largest contribution occurs in Bin 7: Bin 7 has the highest individual PSI contribution at 0.0015, corresponding to a decrease from 9.0135% in the initial dataset to 7.8825% in the new dataset.
• Secondary differences are concentrated in edge bins: Additional non-zero PSI contributions are relatively small and appear most visibly in Bin 0 (0.0010) and Bin 9 (0.0006), where the new dataset shows 5.5641% versus 4.8356% in Bin 0 and 4.3277% versus 4.8356% in Bin 9.
• Mid-range bins remain particularly stable: Bins 2 through 6 show very small PSI contributions between 0.0001 and 0.0003, alongside closely matching population shares between the two datasets.

The PSI result indicates a highly similar distributional profile between the initial and new datasets in this comparison. Observed differences are minor in magnitude and are concentrated primarily in Bin 7, with smaller contributions from the lowest and highest bins. Across the middle of the distribution, the initial and new population shares remain closely matched, and these bins contribute minimally to the total PSI.

Tables

Population Stability Index for train_dataset_final and test_dataset_final Datasets

Bin	Count Initial	Percent Initial (%)	Count New	Percent New (%)	PSI
0	125	4.8356	36	5.5641	0.0010
1	219	8.4720	55	8.5008	0.0000
2	309	11.9536	80	12.3648	0.0001
3	355	13.7331	93	14.3740	0.0003
4	364	14.0812	93	14.3740	0.0001
5	402	15.5513	97	14.9923	0.0002
6	288	11.1412	74	11.4374	0.0001
7	233	9.0135	51	7.8825	0.0015
8	165	6.3830	40	6.1824	0.0001
9	125	4.8356	28	4.3277	0.0006
Total	2585	100.0000	647	100.0000	0.0039

Figures

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

The Confusion Matrix test evaluates classification performance by comparing predicted class labels with observed class labels. The heatmap presents counts for true positives, true negatives, false positives, and false negatives across the binary target classes. In this result, the matrix shows 201 true positives, 207 true negatives, 116 false positives, and 123 false negatives. These values provide a direct view of the model’s correct classifications and misclassifications by class.

Key insights:

• Correct classifications exceed errors: The model records 201 true positives and 207 true negatives, for a total of 408 correct classifications, compared with 116 false positives and 123 false negatives, for 239 misclassifications.

• Balanced performance across predicted classes: True positive and true negative counts are similar in magnitude, at 201 and 207 respectively. The two error types are also close, with 123 false negatives and 116 false positives.

• Slightly higher false negatives: False negatives exceed false positives by 7 observations, indicating marginally more missed positive cases than incorrect positive assignments.

The confusion matrix shows that correct classifications are more frequent than misclassifications, with similar counts for true positives and true negatives. Error counts are also closely aligned, although false negatives are modestly higher than false positives. Overall, the result reflects relatively even classification behavior across the two classes, with both types of prediction error present at comparable levels.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

The Classifier Performance test evaluates classification model performance using precision, recall, F1-score, accuracy, and ROC AUC. The in-sample results are presented by class for precision, recall, and F1, alongside weighted and macro averages, with separate summary values for overall accuracy and ROC AUC. Class-level metrics are reported for classes 0 and 1, and the aggregate metrics provide a consolidated view of overall classification behavior within the development sample.

Key insights:

• Class-level performance is highly symmetric: Precision, recall, and F1 are closely aligned across both classes. Class 0 records precision 0.6390, recall 0.6381, and F1 0.6385, while class 1 records precision 0.6383, recall 0.6393, and F1 0.6388.
• Aggregate averages match class performance: Weighted average and macro average are both 0.6387 for precision, recall, and F1. This indicates that the aggregate summary metrics are nearly identical to the class-specific results shown in the report.
• Accuracy aligns with average classification metrics: Overall accuracy is 0.6387, matching the weighted and macro averages reported for precision, recall, and F1. The reported overall classification performance is therefore consistent across these summary measures.
• ROC AUC exceeds threshold-based metrics: ROC AUC is 0.6807, which is higher than the reported accuracy, precision, recall, and F1 values of approximately 0.639. This shows stronger ranking performance than the threshold-based classification metrics.

The in-sample results show a balanced performance profile across classes, with nearly identical precision, recall, and F1 values for classes 0 and 1 and no material divergence between macro and weighted averages. Overall accuracy is consistent with these aggregate classification metrics at 0.6387. ROC AUC of 0.6807 is higher than the threshold-based metrics, indicating that the model’s ranking performance is stronger than its reported point-classification results on the development sample.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6390	0.6381	0.6385
1	0.6383	0.6393	0.6388
Weighted Average	0.6387	0.6387	0.6387
Macro Average	0.6387	0.6387	0.6387

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6387
ROC AUC	0.6807

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

The Classifier Performance test evaluates out-of-sample classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The results are presented for both classes individually, along with macro and weighted averages across classes. Class-level precision ranges from 0.6273 to 0.6341, recall ranges from 0.6204 to 0.6409, and F1-scores range from 0.6271 to 0.6340. Aggregate performance is summarized by accuracy of 0.6306 and ROC AUC of 0.6751.

Key insights:

• Balanced class-level performance: Precision, recall, and F1-scores are closely aligned across Class 0 and Class 1. F1 is 0.6340 for Class 0 and 0.6271 for Class 1, indicating limited divergence in performance between the two classes.
• Macro and weighted averages are identical: Macro-average and weighted-average precision, recall, and F1 are all 0.6306 to 0.6307. This shows that the aggregate metrics are numerically consistent across the two averaging approaches in this test output.
• Recall differs modestly by class: Recall is 0.6409 for Class 0 and 0.6204 for Class 1, a difference of 0.0205. This is the largest class-level separation among the reported precision, recall, and F1 metrics.
• ROC AUC exceeds accuracy: ROC AUC is 0.6751 compared with accuracy of 0.6306. The reported ranking-based discrimination measure is therefore higher than the threshold-dependent accuracy metric.

Overall, the out-of-sample results show closely grouped class-specific precision, recall, and F1-scores, with aggregate averages matching across macro and weighted views. The largest class-specific difference appears in recall, while overall accuracy is 0.6306. The ROC AUC of 0.6751 indicates stronger performance on the reported discrimination measure than on the reported accuracy measure.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6273	0.6409	0.6340
1	0.6341	0.6204	0.6271
Weighted Average	0.6307	0.6306	0.6306
Macro Average	0.6307	0.6306	0.6306

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6306
ROC AUC	0.6751

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

The Precision-Recall Curve test evaluates the trade-off between precision and recall across classification thresholds for a binary classification model. The result is presented as a precision-recall curve with recall on the x-axis and precision on the y-axis, showing how precision changes as recall increases. In this output, precision is highest at very low recall levels, remains around the low- to mid-0.70 range through much of the mid-recall region, and declines progressively as recall approaches 1.0.

Key insights:

• Highest precision at low recall: Precision reaches approximately 1.0 at near-zero recall and remains above 0.9 only in the earliest part of the curve, indicating the strongest positive predictive concentration at the most selective thresholds.
• Mid-range recall shows moderate stability: Across roughly the 0.1 to 0.6 recall range, precision fluctuates but remains broadly within about 0.65 to 0.79, with local variation rather than abrupt deterioration.
• Precision declines at high recall: After approximately 0.6 recall, the curve trends downward more steadily, with precision moving from the mid-0.60 range toward about 0.51 as recall approaches 1.0.
• Curve remains above 0.5 throughout: Across the displayed threshold range, precision stays above approximately 0.5, including at the highest recall levels shown.

The curve indicates a clear precision-recall trade-off across thresholds. The strongest precision is concentrated at very low recall, while broader positive capture is associated with a gradual reduction in precision. Performance through the middle recall range is comparatively stable, followed by a more consistent decline in precision in the higher-recall region.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

The ROCCurve test evaluates classification performance by plotting the Receiver Operating Characteristic curve and quantifying discrimination with the Area Under the Curve (AUC). The result shows a single ROC curve for log_reg_model_v1 on test_dataset_final, indicating a binary classification assessment on the test dataset. The chart includes both the model ROC curve and the random-classification reference line, with the reported model AUC equal to 0.68.

Key insights:

• AUC exceeds random baseline: The reported AUC is 0.68, which is above the random-reference AUC of 0.50 shown in the plot. This indicates the model curve maintains separation from the diagonal benchmark across much of the threshold range.

• ROC curve remains above diagonal: The plotted ROC curve lies predominantly above the randomness line from low to high false positive rates. This reflects positive discrimination between classes across evaluated thresholds.

• Discrimination is moderate rather than extreme: The curve rises steadily rather than sharply toward the upper-left corner. This shape is consistent with measured discrimination that is present but not near-perfect, as reflected by the AUC value of 0.68.

The ROC result shows that log_reg_model_v1 demonstrates measurable classification discrimination on test_dataset_final, with an AUC of 0.68 and a curve that remains above the random baseline. The observed curve shape indicates performance stronger than chance across thresholds, while the gradual ascent and distance from the upper-left corner indicate that discrimination is moderate rather than highly pronounced.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

The TrainingTestDegradation test evaluates whether performance degradation between the training and test datasets remains below the predefined threshold for key classification metrics. The results table reports precision, recall, and F1-score for classes 0 and 1 on both datasets, together with the calculated degradation percentages and pass/fail outcomes. Observed degradation values range from -0.4413% to 2.9639%, and all reported metrics are marked as passing.

Key insights:

• All metrics passed threshold: Every evaluated metric for both classes is labeled "Pass," indicating that all measured degradation values remained below the configured limit used by the test.
• Largest degradation in class 1 recall: Class 1 recall declines from 0.6393 on the training dataset to 0.6204 on the test dataset, corresponding to the highest reported degradation of 2.9639%.
• Class 0 recall improved on test data: Class 0 recall increases from 0.6381 in training to 0.6409 in testing, producing a negative degradation value of -0.4413%.
• Train-test performance gaps are small: Across all reported metrics, training and test scores remain close, with absolute score differences ranging from 0.0028 to 0.0189.

The test results show limited differences between training and test performance across precision, recall, and F1-score for both classes. The largest observed reduction occurs in class 1 recall, while class 0 recall is slightly higher on the test dataset than on the training dataset. Taken together, the results indicate that the measured classification metrics remain stable across the two datasets within the threshold applied by this test.

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
1	Precision	0.6383	0.6341	0.6676	Pass
1	Recall	0.6393	0.6204	2.9639	Pass
1	F1-Score	0.6388	0.6271	1.8283	Pass
0	Precision	0.6390	0.6273	1.8413	Pass
0	Recall	0.6381	0.6409	-0.4413	Pass
0	F1-Score	0.6385	0.6340	0.7122	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

The Minimum Accuracy test evaluates whether the model’s prediction accuracy on the evaluated dataset meets or exceeds a specified minimum threshold. The result table reports an accuracy score of 0.6306 against a threshold of 0.7, along with the corresponding pass/fail outcome. In this run, the reported outcome is Fail because the observed score is below the configured threshold.

Key insights:

• Accuracy below threshold: The observed accuracy score is 0.6306, which is lower than the minimum threshold of 0.7 used in this test.
• Test outcome is fail: The result is recorded as Fail, reflecting that the measured accuracy did not meet the threshold condition defined by the test.
• Shortfall is measurable: The difference between the reported score and the threshold is 0.0694, indicating the magnitude by which the result falls below the minimum requirement in this test.

The test result shows that the model’s measured accuracy on the evaluated dataset did not satisfy the minimum threshold applied in this assessment. The observed score of 0.6306 is below the threshold of 0.7, and the test therefore returns a Fail outcome. Collectively, the result and supporting values indicate that the model did not meet the accuracy level specified by this test configuration.

Tables

Score	Threshold	Pass/Fail
0.6306	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

The MinimumF1Score test evaluates whether the model’s F1 score on the validation dataset meets a predefined minimum threshold. The result table reports a validation F1 score of 0.6271 against a threshold of 0.5, along with an overall test outcome of Pass. These values provide the direct basis for assessing whether the observed precision-recall balance satisfied the configured minimum criterion.

Key insights:

• Threshold exceeded on validation set: The reported F1 score is 0.6271, which is above the minimum threshold of 0.5 by 0.1271.
• Test outcome recorded as Pass: The result table explicitly classifies the test as Pass, consistent with the observed score-threshold comparison.
• Positive precision-recall summary captured: The test records a single F1 statistic for the validation sample, providing a combined measure of precision and recall performance at 0.6271.

The test result shows that the validation F1 score exceeded the configured minimum threshold and was therefore recorded as a pass. The observed margin above threshold is 0.1271, indicating that the model satisfied the criterion defined for this check based on the reported validation-set F1 score.

Tables

Score	Threshold	Pass/Fail
0.6271	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

The Minimum ROC AUC Score test evaluates whether the model’s ROC AUC score meets or exceeds a predefined minimum threshold. The result table reports a ROC AUC score of 0.6751, a threshold of 0.5, and a test outcome of Pass. These values summarize the model’s observed discrimination performance under the threshold-based validation defined by the test.

Key insights:

• Threshold exceeded: The reported ROC AUC score of 0.6751 is above the specified threshold of 0.5 by 0.1751.
• Test outcome is pass: The test result is recorded as Pass, consistent with the score exceeding the minimum threshold.
• Documented discrimination level: The reported ROC AUC value provides the quantitative basis used in this test to assess class discrimination performance.

The test result shows that the model’s ROC AUC score exceeded the predefined minimum threshold and therefore satisfied the pass condition for this validation. The documented outcome is based on a score of 0.6751 relative to a threshold of 0.5, as reflected directly in the test result table.

Tables

Score	Threshold	Pass/Fail
0.6751	0.5	Pass

▶ Test Result: Permutation Feature Importance (validmind.model_validation.sklearn.PermutationFeatureImportance)

Permutation Feature Importance

The Permutation Feature Importance test evaluates the relative significance of each input feature by measuring the change in model performance after randomly permuting feature values. The result is presented as a horizontal importance chart, with larger positive values indicating a larger decrease in performance when a feature is shuffled. The chart ranks features from highest to lowest impact, showing a small set of variables with clearly larger importance values and several variables with near-zero or negative values.

Key insights:

• IsActiveMember is the most influential feature: IsActiveMember has the largest permutation importance in the chart, at approximately 0.045, indicating the greatest observed reduction in model performance when this feature is permuted.

• Geography_Germany has similarly high importance: Geography_Germany is the second-highest feature, with an importance value close to IsActiveMember at roughly 0.044, placing it among the dominant drivers in the permutation ranking.

• Importance drops after the top two features: Gender_Male and NumOfProducts follow at materially lower levels, approximately 0.024 and 0.014 respectively, indicating a noticeable step-down in marginal contribution relative to the top two features.

• Most remaining features contribute little under permutation: Balance shows a modest positive importance near 0.004, while CreditScore, Geography_Spain, and HasCrCard are all close to zero, reflecting limited measured performance sensitivity to their permutation.

• Two features show negative importance: EstimatedSalary and Tenure have negative permutation importance values, with Tenure appearing more negative than EstimatedSalary. In this test output, permuting these features did not reduce model performance and was associated with a slight performance improvement.

The permutation importance profile is concentrated in a small number of features, with IsActiveMember and Geography_Germany contributing the largest observed performance impact by a wide margin. A second tier is formed by Gender_Male and NumOfProducts, while most other variables show minimal effect when permuted. The presence of negative importance values for EstimatedSalary and Tenure indicates that these features did not provide positive measured contribution in this test result.

Figures

▶ Test Result: SHAP Global Importance (validmind.model_validation.sklearn.SHAPGlobalImportance)

SHAP Global Importance

The SHAP Global Importance test evaluates global feature importance by measuring the magnitude and direction of feature contributions to model output using SHAP values. The results include a normalized mean importance bar chart and a SHAP summary plot for the ten displayed features. The bar chart ranks features by average absolute SHAP value, while the summary plot shows the distribution of instance-level SHAP impacts and the relationship between feature values and contribution direction for each feature.

Key insights:

• Importance is concentrated in two features: IsActiveMember and Geography_Germany are the two highest-ranked features in the normalized importance chart, with values near 100 and 90 respectively, materially above the remaining features.

• Secondary importance drops after the top tier: After the two leading features, normalized importance declines to Gender_Male at roughly the mid-50s and Balance at roughly the low-40s, followed by a further drop to the remaining variables, all of which are below 20.

• Lowest-ranked features contribute minimally: Geography_Spain and EstimatedSalary have the smallest normalized SHAP importance, both appearing close to 5 or below in the bar chart.

• Directional separation is visible for several binary features: In the summary plot, IsActiveMember, Geography_Germany, and Gender_Male show clear separation between low and high feature values, with points concentrated on opposite sides of zero SHAP value for the two value states.

• Balance shows predominantly positive SHAP contributions: The Balance feature exhibits SHAP values distributed largely on the positive side of the axis, extending to approximately 0.4, with higher feature values concentrated toward larger positive impacts.

• Several lower-ranked features remain centered near zero: Tenure, CreditScore, Geography_Spain, and EstimatedSalary have comparatively tight SHAP distributions clustered around zero, indicating smaller instance-level effects relative to the leading features.

The SHAP results show that model explanation is dominated by a small set of features, led by IsActiveMember and Geography_Germany, with a noticeable reduction in contribution magnitude across the rest of the feature set. The summary plot further indicates that the highest-ranked variables have clearer directional effects on model output, while lower-ranked variables are more concentrated around zero and contribute less strongly at the instance level. Overall, the observed feature attribution pattern is uneven, with global importance and local impact concentrated in the top-ranked predictors.

Figures

▶ Test Result: Weakspots Diagnosis (validmind.model_validation.sklearn.WeakspotsDiagnosis)

❌ Weakspots Diagnosis

The WeakspotsDiagnosis test evaluates model performance across bins of the feature space to identify localized regions where accuracy, precision, recall, or F1 fall below the configured thresholds. The results are presented by feature and slice for both the training and test datasets, with thresholds shown at 0.75 for accuracy, 0.5 for precision and recall, and 0.7 for F1. The output includes continuous-feature bins for Balance, CreditScore, EstimatedSalary, and Tenure, and binary or categorical slices for Gender_Male, Geography_Germany, Geography_Spain, HasCrCard, IsActiveMember, and NumOfProducts. Across these slices, the tables and charts show both the absolute metric levels and the train-test differences within each segment.

Key insights:

• Balance contains the weakest localized regions: Several Balance test slices show very low performance, especially in sparse high-balance bins. Test F1 is 0.0 for (25089.809, 50179.618] with 2 records and (200718.472, 225808.281] with 3 records, 0.25 for (175628.663, 200718.472] with 8 records, and 0.4091 in the largest low-balance slice (-250.898, 25089.809] with 212 records.

• Sparse Balance bins show large train-test gaps: The largest disparities appear in high-balance ranges with limited test observations. For example, in (200718.472, 225808.281], train F1 is 0.8696 versus test F1 of 0.0, and in (175628.663, 200718.472], train F1 is 0.6154 versus test F1 of 0.25.

• CreditScore performance is uneven at the low end: The lowest CreditScore slice (349.5, 400.0] shows materially weaker test performance, with accuracy 0.25, recall 0.25, and F1 0.4 on 4 records, versus train F1 of 0.8235 on 10 records. Other CreditScore bins are mostly above 0.5 on precision and recall but remain below the 0.75 accuracy threshold in nearly all slices.

• EstimatedSalary shows broad mid-range variability: Test accuracy for EstimatedSalary ranges from 0.5161 to 0.7286, and test F1 ranges from 0.4815 to 0.7532. The weakest test F1 occurs in (100042.115, 120032.188] at 0.4815, driven by precision 0.52 and recall 0.4483, while the strongest test segment is (-108.151, 20081.823] with F1 0.7532.

• IsActiveMember splits performance sharply by segment: For IsActiveMember = 0, test recall is 0.804 and F1 is 0.7323, while for IsActiveMember = 1, test recall drops to 0.328 and F1 to 0.402. This contrast is also present in training, where F1 is 0.712 for the 0 slice and 0.4593 for the 1 slice.

• Geography segments differ materially: Geography_Germany = 1 has stronger test performance than Geography_Germany = 0, with test F1 of 0.774 versus 0.478 and recall 0.9328 versus 0.4. Geography_Spain shows the reverse pattern, with test F1 of 0.6456 for the 0 slice and 0.5294 for the 1 slice.

• NumOfProducts weakens at higher product counts: Test accuracy declines from 0.6273 for 1 product and 0.6695 for 2 products to 0.4138 for 3 products and 0.4444 for 4 products. Recall is below 0.5 in the 3-product slice at 0.3929 and in the 4-product slice at 0.4444, while those slices also have small test sample sizes of 29 and 9.

• Gender and HasCrCard are comparatively stable: Gender_Male and HasCrCard show modest train-test differences across their two slices. For Gender_Male, test F1 is 0.6866 for the 0 slice and 0.5474 for the 1 slice; for HasCrCard, test F1 is 0.5987 for the 0 slice and 0.6364 for the 1 slice, with corresponding training values remaining close.

The weak spot analysis shows that localized underperformance is concentrated most clearly in specific Balance, CreditScore, IsActiveMember, Geography, and NumOfProducts segments. The most severe weaknesses appear in sparse Balance and high NumOfProducts slices, where test metrics are low and train-test differences are large. More stable behavior is observed for Gender_Male and HasCrCard, while EstimatedSalary and Tenure show moderate variation across bins without the same degree of segment-level breakdown.

Tables

Slice	Number of Records	Feature	Accuracy	Precision	Recall	F1	Dataset
(-250.898, 25089.809]	212	Balance	0.6321	0.6279	0.3034	0.4091	test_dataset_final
(25089.809, 50179.618]	2	Balance	0.5000	0.0000	0.0000	0.0000	test_dataset_final
(50179.618, 75269.427]	25	Balance	0.4400	0.3846	0.4545	0.4167	test_dataset_final
(75269.427, 100359.236]	75	Balance	0.5867	0.5745	0.7105	0.6353	test_dataset_final
(100359.236, 125449.045]	145	Balance	0.7241	0.7701	0.7701	0.7701	test_dataset_final
(125449.045, 150538.854]	125	Balance	0.6720	0.6437	0.8485	0.7320	test_dataset_final
(150538.854, 175628.663]	52	Balance	0.5192	0.4865	0.7500	0.5902	test_dataset_final
(175628.663, 200718.472]	8	Balance	0.2500	0.3333	0.2000	0.2500	test_dataset_final
(200718.472, 225808.281]	3	Balance	0.0000	0.0000	0.0000	0.0000	test_dataset_final
(225808.281, 250898.09]	0	Balance	0.0000	0.0000	0.0000	0.0000	test_dataset_final
(-250.898, 25089.809]	835	Balance	0.6479	0.5562	0.3003	0.3900	train_dataset_final
(25089.809, 50179.618]	22	Balance	0.4545	0.5556	0.3846	0.4545	train_dataset_final
(50179.618, 75269.427]	92	Balance	0.5435	0.6667	0.4151	0.5116	train_dataset_final
(75269.427, 100359.236]	286	Balance	0.6119	0.6026	0.6573	0.6288	train_dataset_final
(100359.236, 125449.045]	594	Balance	0.7020	0.7207	0.8164	0.7656	train_dataset_final
(125449.045, 150538.854]	486	Balance	0.6255	0.6380	0.7818	0.7026	train_dataset_final
(150538.854, 175628.663]	201	Balance	0.5672	0.5429	0.7677	0.6360	train_dataset_final
(175628.663, 200718.472]	54	Balance	0.5370	0.5263	0.7407	0.6154	train_dataset_final
(200718.472, 225808.281]	13	Balance	0.7692	1.0000	0.7692	0.8696	train_dataset_final
(225808.281, 250898.09]	2	Balance	0.5000	1.0000	0.5000	0.6667	train_dataset_final
(349.5, 400.0]	4	CreditScore	0.2500	1.0000	0.2500	0.4000	test_dataset_final
(400.0, 450.0]	12	CreditScore	0.7500	0.7778	0.8750	0.8235	test_dataset_final
(450.0, 500.0]	28	CreditScore	0.7857	0.8333	0.8333	0.8333	test_dataset_final
(500.0, 550.0]	62	CreditScore	0.5968	0.6207	0.5625	0.5902	test_dataset_final
(550.0, 600.0]	93	CreditScore	0.6022	0.6364	0.6731	0.6542	test_dataset_final
(600.0, 650.0]	124	CreditScore	0.5968	0.5714	0.6102	0.5902	test_dataset_final
(650.0, 700.0]	129	CreditScore	0.6589	0.6909	0.5846	0.6333	test_dataset_final
(700.0, 750.0]	100	CreditScore	0.5800	0.5333	0.5333	0.5333	test_dataset_final
(750.0, 800.0]	56	CreditScore	0.6786	0.6333	0.7308	0.6786	test_dataset_final
(800.0, 850.0]	39	CreditScore	0.7179	0.6667	0.5333	0.5926	test_dataset_final
(349.5, 400.0]	10	CreditScore	0.7000	1.0000	0.7000	0.8235	train_dataset_final
(400.0, 450.0]	45	CreditScore	0.5778	0.6522	0.5769	0.6122	train_dataset_final
(450.0, 500.0]	113	CreditScore	0.6018	0.5797	0.7143	0.6400	train_dataset_final
(500.0, 550.0]	274	CreditScore	0.6533	0.6601	0.7014	0.6801	train_dataset_final
(550.0, 600.0]	369	CreditScore	0.6585	0.6543	0.6685	0.6613	train_dataset_final
(600.0, 650.0]	493	CreditScore	0.6166	0.5992	0.6118	0.6054	train_dataset_final
(650.0, 700.0]	474	CreditScore	0.6561	0.6456	0.6595	0.6525	train_dataset_final
(700.0, 750.0]	389	CreditScore	0.6247	0.6413	0.5960	0.6178	train_dataset_final
(750.0, 800.0]	239	CreditScore	0.6904	0.7196	0.6364	0.6754	train_dataset_final
(800.0, 850.0]	179	CreditScore	0.5866	0.5595	0.5595	0.5595	train_dataset_final
(-108.151, 20081.823]	70	EstimatedSalary	0.7286	0.7436	0.7632	0.7532	test_dataset_final
(20081.823, 40071.896]	69	EstimatedSalary	0.7101	0.7179	0.7568	0.7368	test_dataset_final
(40071.896, 60061.969]	62	EstimatedSalary	0.6613	0.6129	0.6786	0.6441	test_dataset_final
(60061.969, 80052.042]	62	EstimatedSalary	0.5161	0.5588	0.5588	0.5588	test_dataset_final
(80052.042, 100042.115]	66	EstimatedSalary	0.5758	0.5294	0.6000	0.5625	test_dataset_final
(100042.115, 120032.188]	75	EstimatedSalary	0.6267	0.5200	0.4483	0.4815	test_dataset_final
(120032.188, 140022.261]	68	EstimatedSalary	0.5735	0.6410	0.6250	0.6329	test_dataset_final
(140022.261, 160012.334]	59	EstimatedSalary	0.6102	0.6667	0.4667	0.5490	test_dataset_final
(160012.334, 180002.407]	63	EstimatedSalary	0.6349	0.7059	0.6486	0.6761	test_dataset_final
(180002.407, 199992.48]	53	EstimatedSalary	0.6604	0.5714	0.5714	0.5714	test_dataset_final
(-108.151, 20081.823]	243	EstimatedSalary	0.6420	0.6667	0.6615	0.6641	train_dataset_final
(20081.823, 40071.896]	251	EstimatedSalary	0.6653	0.6496	0.6387	0.6441	train_dataset_final
(40071.896, 60061.969]	258	EstimatedSalary	0.6512	0.6434	0.6535	0.6484	train_dataset_final
(60061.969, 80052.042]	268	EstimatedSalary	0.6604	0.6742	0.6496	0.6617	train_dataset_final
(80052.042, 100042.115]	231	EstimatedSalary	0.6104	0.6371	0.6371	0.6371	train_dataset_final
(100042.115, 120032.188]	280	EstimatedSalary	0.6321	0.6028	0.6439	0.6227	train_dataset_final
(120032.188, 140022.261]	276	EstimatedSalary	0.6123	0.5643	0.6320	0.5962	train_dataset_final
(140022.261, 160012.334]	231	EstimatedSalary	0.6753	0.6991	0.6583	0.6781	train_dataset_final
(160012.334, 180002.407]	281	EstimatedSalary	0.6014	0.6099	0.6014	0.6056	train_dataset_final
(180002.407, 199992.48]	266	EstimatedSalary	0.6429	0.6562	0.6222	0.6388	train_dataset_final
(-0.001, 0.1]	317	Gender_Male	0.6372	0.6562	0.7200	0.6866	test_dataset_final
(0.9, 1.0]	330	Gender_Male	0.6242	0.6000	0.5034	0.5474	test_dataset_final
(-0.001, 0.1]	1307	Gender_Male	0.6496	0.6553	0.7698	0.7079	train_dataset_final
(0.9, 1.0]	1278	Gender_Male	0.6275	0.6063	0.4746	0.5324	train_dataset_final
(-0.001, 0.1]	434	Geography_Germany	0.6175	0.5938	0.4000	0.4780	test_dataset_final
(0.9, 1.0]	213	Geography_Germany	0.6573	0.6614	0.9328	0.7740	test_dataset_final
(-0.001, 0.1]	1793	Geography_Germany	0.6252	0.5865	0.4391	0.5022	train_dataset_final
(0.9, 1.0]	792	Geography_Germany	0.6692	0.6802	0.9365	0.7880	train_dataset_final
(-0.001, 0.1]	516	Geography_Spain	0.6298	0.6350	0.6566	0.6456	test_dataset_final
(0.9, 1.0]	131	Geography_Spain	0.6336	0.6279	0.4576	0.5294	test_dataset_final
(-0.001, 0.1]	1964	Geography_Spain	0.6446	0.6483	0.6915	0.6692	train_dataset_final
(0.9, 1.0]	621	Geography_Spain	0.6200	0.5854	0.4428	0.5042	train_dataset_final
(-0.001, 0.1]	159	HasCrCard	0.6038	0.5663	0.6351	0.5987	test_dataset_final
(0.9, 1.0]	488	HasCrCard	0.6393	0.6581	0.6160	0.6364	test_dataset_final
(-0.001, 0.1]	808	HasCrCard	0.6238	0.6336	0.6548	0.6440	train_dataset_final
(0.9, 1.0]	1777	HasCrCard	0.6455	0.6407	0.6319	0.6363	train_dataset_final
(-0.001, 0.1]	329	IsActiveMember	0.6444	0.6723	0.8040	0.7323	test_dataset_final
(0.9, 1.0]	318	IsActiveMember	0.6164	0.5190	0.3280	0.4020	test_dataset_final
(-0.001, 0.1]	1402	IsActiveMember	0.6227	0.6431	0.7976	0.7120	train_dataset_final
(0.9, 1.0]	1183	IsActiveMember	0.6577	0.6209	0.3644	0.4593	train_dataset_final
(0.997, 1.3]	373	NumOfProducts	0.6273	0.6934	0.6652	0.6790	test_dataset_final
(1.9, 2.2]	236	NumOfProducts	0.6695	0.4333	0.5909	0.5000	test_dataset_final
(2.8, 3.1]	29	NumOfProducts	0.4138	1.0000	0.3929	0.5641	test_dataset_final
(3.7, 4.0]	9	NumOfProducts	0.4444	1.0000	0.4444	0.6154	test_dataset_final
(0.997, 1.3]	1483	NumOfProducts	0.6365	0.7092	0.6800	0.6943	train_dataset_final
(1.9, 2.2]	908	NumOfProducts	0.6652	0.3514	0.5707	0.4349	train_dataset_final
(2.8, 3.1]	159	NumOfProducts	0.5597	0.9881	0.5461	0.7034	train_dataset_final
(3.7, 4.0]	35	NumOfProducts	0.4000	1.0000	0.4000	0.5714	train_dataset_final
(-0.01, 1.0]	90	Tenure	0.6111	0.5962	0.6889	0.6392	test_dataset_final
(1.0, 2.0]	61	Tenure	0.5902	0.6765	0.6216	0.6479	test_dataset_final
(2.0, 3.0]	67	Tenure	0.5672	0.6552	0.5000	0.5672	test_dataset_final
(3.0, 4.0]	70	Tenure	0.6429	0.5714	0.6667	0.6154	test_dataset_final
(4.0, 5.0]	66	Tenure	0.5455	0.6000	0.5676	0.5833	test_dataset_final
(5.0, 6.0]	61	Tenure	0.6230	0.6207	0.6000	0.6102	test_dataset_final
(6.0, 7.0]	53	Tenure	0.7547	0.6667	0.6316	0.6486	test_dataset_final
(7.0, 8.0]	68	Tenure	0.6765	0.7500	0.5833	0.6562	test_dataset_final
(8.0, 9.0]	70	Tenure	0.7143	0.6750	0.7941	0.7297	test_dataset_final
(9.0, 10.0]	41	Tenure	0.5854	0.5294	0.5000	0.5143	test_dataset_final
(-0.01, 1.0]	379	Tenure	0.6412	0.6651	0.6908	0.6777	train_dataset_final
(1.0, 2.0]	251	Tenure	0.6414	0.6240	0.6446	0.6341	train_dataset_final
(2.0, 3.0]	282	Tenure	0.6348	0.6078	0.6838	0.6436	train_dataset_final
(3.0, 4.0]	243	Tenure	0.6173	0.6875	0.6241	0.6543	train_dataset_final
(4.0, 5.0]	253	Tenure	0.6798	0.6641	0.6911	0.6773	train_dataset_final
(5.0, 6.0]	257	Tenure	0.6342	0.6230	0.6129	0.6179	train_dataset_final
(6.0, 7.0]	272	Tenure	0.5993	0.5360	0.5678	0.5514	train_dataset_final
(7.0, 8.0]	265	Tenure	0.6679	0.6525	0.6210	0.6364	train_dataset_final
(8.0, 9.0]	257	Tenure	0.6420	0.6557	0.6154	0.6349	train_dataset_final
(9.0, 10.0]	126	Tenure	0.6190	0.6724	0.5735	0.6190	train_dataset_final

Figures

▶ Test Result: Overfit Diagnosis (validmind.model_validation.sklearn.OverfitDiagnosis)

Overfit Diagnosis

The Overfit Diagnosis test evaluates differences in predictive performance between the training and test samples across feature-based slices to identify localized overfitting regions. Results are reported as AUC by segment, with the table listing training AUC, test AUC, and the resulting gap for slices that exceed the default threshold of 0.04. The identified regions span four features in the tabular output: CreditScore, Tenure, Balance, and EstimatedSalary. Observed gaps range from 0.0464 to 0.4444, with segment record counts varying substantially across slices.

Key insights:

• Largest gap occurs in Balance: The Balance slice (25089.809, 50179.618] shows the largest reported gap at 0.4444, with training AUC of 0.4444 and test AUC of 0.0000. This slice also has the smallest sample size in the table, with 22 training records and 2 test records.

• Tenure contains the largest well-populated gap: Within Tenure, the slice (4.0, 5.0] has a gap of 0.1611, with training AUC of 0.7380 and test AUC of 0.5769 across 253 training and 66 test records. Other Tenure slices above threshold are (2.0, 3.0] at 0.0529 and (9.0, 10.0] at 0.0620.

• CreditScore shows repeated moderate gaps: Three CreditScore ranges exceed the threshold: (500.0, 550.0] with a gap of 0.0464, (550.0, 600.0] with 0.0751, and (700.0, 750.0] with 0.0684. These slices have comparatively larger segment sizes, ranging from 274 to 389 training records and 62 to 100 test records.

• EstimatedSalary gaps are concentrated in mid-to-upper ranges: The EstimatedSalary slices (60061.969, 80052.042], (80052.042, 100042.115], and (100042.115, 120032.188] exceed the threshold with gaps of 0.1146, 0.0581, and 0.0651 respectively. Their training AUC values range from 0.6656 to 0.6808, while test AUC values range from 0.5662 to 0.6185.

• Balance shows multiple elevated segments: In addition to the largest gap, Balance has three other slices above threshold: (50179.618, 75269.427] at 0.1552, (75269.427, 100359.236] at 0.0572, and (175628.663, 200718.472] at 0.2815. These results indicate that elevated train-test AUC separation appears in more than one Balance range.

The test identifies multiple localized regions where training AUC exceeds test AUC by more than the 0.04 threshold. The most pronounced separations occur in selected Balance and Tenure slices, while CreditScore and EstimatedSalary exhibit smaller but repeated above-threshold gaps across several ranges. Overall, the results indicate that the train-test performance difference is not uniform across the feature space and is concentrated in specific segments with varying sample sizes.

Tables

Overfit Diagnosis

Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
CreditScore	(500.0, 550.0]	274	62	0.6943	0.6479	0.0464
CreditScore	(550.0, 600.0]	369	93	0.6975	0.6224	0.0751
CreditScore	(700.0, 750.0]	389	100	0.6744	0.6061	0.0684
Tenure	(2.0, 3.0]	282	67	0.6691	0.6162	0.0529
Tenure	(4.0, 5.0]	253	66	0.7380	0.5769	0.1611
Tenure	(9.0, 10.0]	126	41	0.6876	0.6256	0.0620
Balance	(25089.809, 50179.618]	22	2	0.4444	0.0000	0.4444
Balance	(50179.618, 75269.427]	92	25	0.6033	0.4481	0.1552
Balance	(75269.427, 100359.236]	286	75	0.6568	0.5996	0.0572
Balance	(175628.663, 200718.472]	54	8	0.4815	0.2000	0.2815
EstimatedSalary	(60061.969, 80052.042]	268	62	0.6808	0.5662	0.1146
EstimatedSalary	(80052.042, 100042.115]	231	66	0.6767	0.6185	0.0581
EstimatedSalary	(100042.115, 120032.188]	280	75	0.6656	0.6004	0.0651

Figures

▶ Test Result: Robustness Diagnosis (validmind.model_validation.sklearn.RobustnessDiagnosis)

✅ Robustness Diagnosis

The Robustness Diagnosis test evaluates model performance under increasing Gaussian noise applied to numerical input features. Results are reported as AUC and performance decay across perturbation sizes from 0.0 to 0.5 for both the training dataset (train_dataset_final) and test dataset (test_dataset_final). Baseline AUC values are 0.6807 on the training data and 0.6751 on the test data, and all evaluated perturbation levels are marked as passed. The accompanying plot shows the AUC trajectories for both datasets across the perturbation range and the corresponding threshold lines.

Key insights:

• Baseline train and test AUC are similar: Baseline AUC is 0.6807 for train_dataset_final and 0.6751 for test_dataset_final, indicating close starting performance before perturbation.

• Performance remains within pass criteria: Every evaluated perturbation setting, including the maximum perturbation size of 0.5, is marked as passed for both datasets.

• Small perturbations show minimal change: At perturbation size 0.1, AUC increases slightly to 0.6815 on the training data and 0.6758 on the test data, producing small negative decay values of -0.0008 and -0.0007 respectively.

• Largest deterioration occurs at perturbation 0.5: At the highest perturbation size, AUC declines to 0.6612 on the training data and 0.6560 on the test data, corresponding to performance decay values of 0.0196 and 0.0191.

• Test AUC peaks at perturbation 0.4: The test dataset reaches its highest observed AUC of 0.6901 at perturbation size 0.4, with a performance decay of -0.0150, while the training dataset at the same perturbation level records AUC of 0.6736.

• Training performance declines more steadily: Training AUC moves from 0.6807 at baseline to 0.6782, 0.6705, 0.6736, and 0.6612 as perturbation increases from 0.2 to 0.5, showing an overall downward pattern despite a small rebound at 0.4.

The results show that AUC remains in a relatively narrow range across the tested noise levels, with all perturbation settings passing on both datasets. The most pronounced reduction in performance is observed at perturbation size 0.5, where both training and test AUC decline by approximately 0.02 relative to baseline. At intermediate perturbation levels, the response is not strictly monotonic, particularly on the test dataset, which records its highest AUC at perturbation size 0.4 before declining at 0.5.

Tables

Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
Baseline (0.0)	train_dataset_final	2585	0.6807	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.6751	0.0000	True
0.1	train_dataset_final	2585	0.6815	-0.0008	True
0.1	test_dataset_final	647	0.6758	-0.0007	True
0.2	train_dataset_final	2585	0.6782	0.0025	True
0.2	test_dataset_final	647	0.6729	0.0022	True
0.3	train_dataset_final	2585	0.6705	0.0102	True
0.3	test_dataset_final	647	0.6728	0.0023	True
0.4	train_dataset_final	2585	0.6736	0.0072	True
0.4	test_dataset_final	647	0.6901	-0.0150	True
0.5	train_dataset_final	2585	0.6612	0.0196	True
0.5	test_dataset_final	647	0.6560	0.0191	True

Figures

In summary

In this second notebook, you learned how to:

• Import a sample dataset
• Identify which tests you might want to run with ValidMind
• Initialize ValidMind datasets and model objects
• Run individual tests
• Utilize the output from tests you've run
• Log test results from sets of or individual tests as evidence to the ValidMind Platform
• Add supplementary individual test results to your documentation
• Assign predictions to your ValidMind model objects

Next steps

Integrate custom tests

Now that you're familiar with the basics of using the ValidMind Library to run and log tests to provide evidence for your documentation, let's learn how to incorporate your own custom tests into ValidMind: 3 — Integrate custom tests

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial