ValidMind for development 2 — Start the development process

Learn how to use ValidMind for your end-to-end documentation process with our series of four introductory notebooks. In this second notebook, you'll run tests and investigate results, then add the results or evidence to your documentation.

You'll become familiar with the individual tests available in ValidMind, as well as how to run them and change parameters as necessary. Using ValidMind's repository of individual tests as building blocks helps you ensure that a record (model) is being built appropriately.

For a full list of out-of-the-box tests and descriptions, use the interactive ValidMind test sandbox.

Learn by doing

Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Developer Fundamentals

Prerequisites

In order to log test results or evidence to your documentation with this notebook, you'll need to first have:

• Registered a model within the ValidMind Platform with a predefined documentation template
• Installed the ValidMind Library in your local environment, allowing you to access all its features

Need help with the above steps?

Refer to the first notebook in this series: 1 — Set up the ValidMind Library

Setting up

Initialize the ValidMind Library

First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Development from the Document drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="documentation",
)

Note: you may need to restart the kernel to use updated packages.

2026-10-30 20:51:48,569 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model development (ID: cmalgf3qi02ce199qm3rdkl46)
📁 Document Type: model_documentation

Import sample dataset

Then, let's import the public Bank Customer Churn Prediction dataset from Kaggle.

In our below example, note that:

• The target column, Exited has a value of 1 when a customer has churned and 0 otherwise.
• The ValidMind Library provides a wrapper to automatically load the dataset as a Pandas DataFrame object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns.

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()
raw_df.head()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

Identify qualitative tests

Next, let's say we want to do some data quality assessments by running a few individual tests.

Use the vm.tests.list_tests() function introduced by the first notebook in this series in combination with vm.tests.list_tags() and vm.tests.list_tasks() to find which prebuilt tests are relevant for data quality assessment:

• tasks represent the kind of modeling task associated with a test. Here we'll focus on classification tasks.
• tags are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the data_quality tag.

# Get the list of available task types
sorted(vm.tests.list_tasks())

['classification',
 'clustering',
 'data_validation',
 'feature_extraction',
 'monitoring',
 'nlp',
 'regression',
 'residual_analysis',
 'text_classification',
 'text_generation',
 'text_qa',
 'text_summarization',
 'time_series_forecasting',
 'visualization']

# Get the list of available tags
sorted(vm.tests.list_tags())

['AUC',
 'analysis',
 'anomaly',
 'anomaly_detection',
 'bias_and_fairness',
 'binary_classification',
 'calibration',
 'categorical_data',
 'classification',
 'classification_metrics',
 'clustering',
 'conceptual_soundness',
 'correlation',
 'credit_risk',
 'data_analysis',
 'data_distribution',
 'data_quality',
 'data_validation',
 'descriptive_statistics',
 'dimensionality_reduction',
 'distribution',
 'embeddings',
 'feature_importance',
 'feature_selection',
 'few_shot',
 'forecasting',
 'frequency_analysis',
 'kmeans',
 'linear_regression',
 'llm',
 'logistic_regression',
 'metadata',
 'model_comparison',
 'model_diagnosis',
 'model_explainability',
 'model_interpretation',
 'model_performance',
 'model_predictions',
 'model_selection',
 'model_training',
 'model_validation',
 'multiclass_classification',
 'multicollinearity',
 'nlp',
 'normality',
 'numerical_data',
 'outlier',
 'outliers',
 'qualitative',
 'rag_performance',
 'ragas',
 'regression',
 'retrieval_performance',
 'robustness',
 'scorecard',
 'seasonality',
 'segment_analysis',
 'sensitivity_analysis',
 'sklearn',
 'specification',
 'stability',
 'stationarity',
 'statistical_test',
 'statistics',
 'statsmodels',
 'stress_testing',
 'tabular_data',
 'text_data',
 'threshold_optimization',
 'time_series_data',
 'uncertainty',
 'unit_root_test',
 'visualization',
 'zero_shot']

You can pass tags and tasks as parameters to the vm.tests.list_tests() function to filter the tests based on the tags and task types.

For example, to find tests related to tabular data quality for classification models, you can call list_tests() like this:

vm.tests.list_tests(task="classification", tags=["tabular_data", "data_quality"])

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.data_validation.ClassImbalance	Class Imbalance	Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model....	True	True	['dataset']	{'min_percent_threshold': {'type': 'int', 'default': 10}}	['tabular_data', 'binary_classification', 'multiclass_classification', 'data_quality']	['classification']
validmind.data_validation.DescriptiveStatistics	Descriptive Statistics	Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's...	False	True	['dataset']	{}	['tabular_data', 'time_series_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.Duplicates	Duplicates	Tests dataset for duplicate entries, ensuring model reliability via data quality verification....	False	True	['dataset']	{'min_threshold': {'type': '_empty', 'default': 1}}	['tabular_data', 'data_quality', 'text_data']	['classification', 'regression']
validmind.data_validation.HighCardinality	High Cardinality	Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting....	False	True	['dataset']	{'num_threshold': {'type': 'int', 'default': 100}, 'percent_threshold': {'type': 'float', 'default': 0.1}, 'threshold_type': {'type': 'str', 'default': 'percent'}}	['tabular_data', 'data_quality', 'categorical_data']	['classification', 'regression']
validmind.data_validation.HighPearsonCorrelation	High Pearson Correlation	Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity....	False	True	['dataset']	{'max_threshold': {'type': 'float', 'default': 0.3}, 'top_n_correlations': {'type': 'int', 'default': 10}, 'feature_columns': {'type': 'list', 'default': None}}	['tabular_data', 'data_quality', 'correlation']	['classification', 'regression']
validmind.data_validation.MissingValues	Missing Values	Evaluates dataset quality by ensuring missing value percentage across all features does not exceed a set threshold....	False	True	['dataset']	{'min_percentage_threshold': {'type': 'float', 'default': 1.0}}	['tabular_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.MissingValuesBarPlot	Missing Values Bar Plot	Assesses the percentage and distribution of missing values in the dataset via a bar plot, with emphasis on...	True	False	['dataset']	{'threshold': {'type': 'int', 'default': 80}, 'fig_height': {'type': 'int', 'default': 600}}	['tabular_data', 'data_quality', 'visualization']	['classification', 'regression']
validmind.data_validation.Skewness	Skewness	Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data...	False	True	['dataset']	{'max_threshold': {'type': '_empty', 'default': 1}}	['data_quality', 'tabular_data']	['classification', 'regression']
validmind.plots.BoxPlot	Box Plot	Generates customizable box plots for numerical features in a dataset with optional grouping using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'group_by': {'type': 'Optional', 'default': None}, 'width': {'type': 'int', 'default': 1800}, 'height': {'type': 'int', 'default': 1200}, 'colors': {'type': 'Optional', 'default': None}, 'show_outliers': {'type': 'bool', 'default': True}, 'title_prefix': {'type': 'str', 'default': 'Box Plot of'}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.plots.HistogramPlot	Histogram Plot	Generates customizable histogram plots for numerical features in a dataset using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'bins': {'type': 'Union', 'default': 30}, 'color': {'type': 'str', 'default': 'steelblue'}, 'opacity': {'type': 'float', 'default': 0.7}, 'show_kde': {'type': 'bool', 'default': True}, 'normalize': {'type': 'bool', 'default': False}, 'log_scale': {'type': 'bool', 'default': False}, 'title_prefix': {'type': 'str', 'default': 'Histogram of'}, 'width': {'type': 'int', 'default': 1200}, 'height': {'type': 'int', 'default': 800}, 'n_cols': {'type': 'int', 'default': 2}, 'vertical_spacing': {'type': 'float', 'default': 0.15}, 'horizontal_spacing': {'type': 'float', 'default': 0.1}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.stats.DescriptiveStats	Descriptive Stats	Provides comprehensive descriptive statistics for numerical features in a dataset....	False	True	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'include_advanced': {'type': 'bool', 'default': True}, 'confidence_level': {'type': 'float', 'default': 0.95}}	['tabular_data', 'statistics', 'data_quality']	['classification', 'regression', 'clustering']

Want to learn more about navigating ValidMind tests?

Refer to our notebook outlining the utilities available for viewing and understanding available ValidMind tests: Explore tests

Initialize the ValidMind dataset

With the individual tests we want to run identified, the next step is to connect your data with a ValidMind Dataset object. This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind, but you only need to do it once per dataset.

Initialize a ValidMind dataset object using the init_dataset function from the ValidMind (vm) module. For this example, we'll pass in the following arguments:

• dataset — The raw dataset that you want to provide as input to tests.
• input_id — A unique identifier that allows tracking what inputs are used when running each individual test.
• target_column — A required argument if tests require access to true values. This is the name of the target column in the dataset.

# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test
vm_raw_dataset = vm.init_dataset(
    dataset=raw_df,
    input_id="raw_dataset",
    target_column="Exited",
)

Running tests on datasets

Now that we know how to initialize a ValidMind dataset object, we're ready to run some tests!

You run individual tests by calling the run_test function provided by the validmind.tests module. For the examples below, we'll pass in the following arguments:

• test_id — The ID of the test to run, as seen in the ID column when you run list_tests.
• params — A dictionary of parameters for the test. These will override any default_params set in the test definition.

Run tabular data tests

The inputs expected by a test can also be found in the test definition — let's take validmind.data_validation.DescriptiveStatistics as an example.

Note that the output of the describe_test() function below shows that this test expects a dataset as input:

vm.tests.describe_test("validmind.data_validation.DescriptiveStatistics")

▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Descriptive Statistics

Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's dataset.

Purpose

The purpose of the Descriptive Statistics metric is to provide a comprehensive summary of both numerical and categorical data within a dataset. This involves statistics such as count, mean, standard deviation, minimum and maximum values for numerical data. For categorical data, it calculates the count, number of unique values, most common value and its frequency, and the proportion of the most frequent value relative to the total. The goal is to visualize the overall distribution of the variables in the dataset, aiding in understanding the model's behavior and predicting its performance.

Test Mechanism

The testing mechanism utilizes two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function pulls out several summary statistics, while value_counts() accounts for unique values. The resulting data is formatted into two distinct tables, one for numerical and another for categorical variable summaries. These tables provide a clear summary of the main characteristics of the variables, which can be instrumental in assessing the model's performance.

Signs of High Risk

• Skewed data or significant outliers can represent high risk. For numerical data, this may be reflected via a significant difference between the mean and median (50% percentile).
• For categorical data, a lack of diversity (low count of unique values), or overdominance of a single category (high frequency of the top value) can indicate high risk.

Strengths

• Provides a comprehensive summary of the dataset, shedding light on the distribution and characteristics of the variables under consideration.
• It is a versatile and robust method, applicable to both numerical and categorical data.
• Helps highlight crucial anomalies such as outliers, extreme skewness, or lack of diversity, which are vital in understanding model behavior during testing and validation.

Limitations

• While this metric offers a high-level overview of the data, it may fail to detect subtle correlations or complex patterns.
• Does not offer any insights on the relationship between variables.
• Alone, descriptive statistics cannot be used to infer properties about future unseen data.
• Should be used in conjunction with other statistical tests to provide a comprehensive understanding of the model's data.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DescriptiveStatistics", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Now, let's run a few tests to assess the quality of the dataset:

result = vm.tests.run_test(
    test_id="validmind.data_validation.DescriptiveStatistics",
    inputs={"dataset": vm_raw_dataset},
)

Descriptive Statistics

The Descriptive Statistics test evaluates the distributional characteristics of numerical and categorical variables in the dataset. The results present summary statistics for eight numerical variables and frequency-based summaries for two categorical variables, each with 8,000 observed records. The numerical table reports central tendency, dispersion, and percentile values from the minimum through the 95th percentile and maximum, while the categorical table reports category counts, number of unique values, top category, and top-category concentration.

Key insights:

• Complete coverage across variables: All reported numerical and categorical variables have a count of 8,000, indicating full population coverage within the summarized dataset.

• Balance shows a zero-heavy lower tail: Balance has a minimum of 0, a 25th percentile of 0, and a median of 97,264, indicating that at least one quarter of observations are at zero while the distribution above that point extends materially higher.

• Balance exhibits high dispersion: Balance has a mean of 76,434.10 and a standard deviation of 62,612.25, with values ranging from 0 to 250,898, showing a wide spread relative to its mean.

• CreditScore and Age are broadly distributed: CreditScore ranges from 350 to 850 with a median of 652 and standard deviation of 96.85, while Age ranges from 18 to 92 with a median of 37 and standard deviation of 10.46, indicating substantial dispersion in both variables.

• NumOfProducts is concentrated at low counts: NumOfProducts has a median of 1, a 75th percentile of 2, a 95th percentile of 2, and a maximum of 4, showing that most observations are concentrated in the lower product-count values.

• Categorical concentration is moderate: Geography contains 3 unique values, with France representing 50.12% of observations, and Gender contains 2 unique values, with Male representing 54.95% of observations; neither categorical variable is dominated by an extreme single-category share.

The descriptive statistics show a dataset with complete recorded observations across all summarized fields and a mix of continuous, discrete, and binary features. The most pronounced distributional feature is Balance, where the lower quartile is zero and variability is high across the observed range. Other numerical variables such as CreditScore, Age, and EstimatedSalary span broad ranges, while NumOfProducts is more tightly concentrated at lower values. The categorical variables show limited cardinality with moderate concentration in the most frequent category.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	8000.0	650.1596	96.8462	350.0	583.0	652.0	717.0	778.0	813.0	850.0
Age	8000.0	38.9489	10.4590	18.0	32.0	37.0	44.0	53.0	60.0	92.0
Tenure	8000.0	5.0339	2.8853	0.0	3.0	5.0	8.0	9.0	9.0	10.0
Balance	8000.0	76434.0965	62612.2513	0.0	0.0	97264.0	128045.0	149545.0	162488.0	250898.0
NumOfProducts	8000.0	1.5325	0.5805	1.0	1.0	1.0	2.0	2.0	2.0	4.0
HasCrCard	8000.0	0.7026	0.4571	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	8000.0	0.5199	0.4996	0.0	0.0	1.0	1.0	1.0	1.0	1.0
EstimatedSalary	8000.0	99790.1880	57520.5089	12.0	50857.0	99505.0	149216.0	179486.0	189997.0	199992.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	8000.0	3.0	France	4010.0	50.12
Gender	8000.0	2.0	Male	4396.0	54.95

result2 = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_raw_dataset},
    params={"min_percent_threshold": 30},
)

❌ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring each class share against a specified minimum percentage threshold. For the Exited target, the results table and bar chart show two classes with markedly different representation levels. Class 0 accounts for 79.80% of rows and is marked as passing, while class 1 accounts for 20.20% of rows and is marked as failing under the configured 30% threshold.

Key insights:

• Majority class dominates distribution: Class 0 represents 79.80% of the dataset, making it the clearly predominant target class in the observed distribution.
• Minority class falls below threshold: Class 1 represents 20.20% of rows, which is below the configured minimum percentage threshold of 30%, resulting in a fail outcome for that class.
• Pass/fail split reflects asymmetric class representation: Only one of the two target classes satisfies the threshold criterion, with class 0 passing and class 1 failing.

The test result shows that the target distribution is uneven across the two Exited classes under the applied 30% threshold. Most observations are concentrated in class 0, while class 1 remains materially less represented at 20.20%. The pass/fail outcomes align directly with this distribution, with the minority class identified as below the configured minimum share.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	79.80%	Pass
1	20.20%	Fail

Figures

The output above shows that the validmind.data_validation.ClassImbalance test did not pass according to the value we set for min_percent_threshold.

To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.

As this is technically a different dataset, remember to first initialize a new ValidMind Dataset object to pass in as input as required by run_test():

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

# Pass the initialized `balanced_raw_dataset` as input into the test run
result = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_balanced_raw_dataset},
    params={"min_percent_threshold": 30},
)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring the proportion of observations in each class against a minimum percentage threshold. In this run, the target variable Exited contains two classes, 0 and 1, and the results table reports the percentage of rows and pass/fail status for each class. The configured minimum percentage threshold is 30%, and both classes are shown at 50.00%. The accompanying bar chart reflects this even split, with both class proportions plotted at the same height.

Key insights:

• Balanced binary target distribution: The Exited target is evenly split between class 0 and class 1, with each representing 50.00% of the dataset.
• All classes exceed threshold: Both target classes are above the configured 30% minimum percentage threshold and are marked as Pass.
• No observed minority class: Neither class appears under-represented in the reported class distribution, as the difference in class share is 0 percentage points.

The test results show a fully balanced binary target distribution for Exited, with both classes present at equal frequency. Relative to the configured 30% threshold, no class is flagged for imbalance, and the result indicates no class-specific under-representation in the evaluated dataset.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

Utilize test output

You can utilize the output from a ValidMind test for further use, for example, if you want to remove highly correlated features. Removing highly correlated features helps make the model simpler, more stable, and easier to understand.

Below we demonstrate how to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.

First, we'll run validmind.data_validation.HighPearsonCorrelation with the balanced_raw_dataset we initialized previously as input as is for comparison with later runs:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify highly correlated variable pairs that may indicate redundancy or multicollinearity. The result table lists the strongest observed correlations, including each feature pair, its Pearson correlation coefficient, and a Pass/Fail status based on the configured absolute threshold of 0.3. The reported coefficients span from -0.1837 to 0.3569 across the top 10 pairs shown. Only one pair exceeds the threshold and is marked as failing, while the remaining listed pairs are marked as passing.

Key insights:

• Single threshold breach identified: The pair (Age, Exited) records the highest absolute correlation at 0.3569 and is the only result marked Fail, indicating it is the only listed pair above the 0.3 threshold.
• Most correlations are weak: The other nine reported coefficients range from -0.1837 to 0.1456 in absolute terms, and all are marked Pass, indicating limited linear association among those listed pairs under the configured threshold.
• Negative relationships are modest: The strongest negative coefficient is -0.1837 for (IsActiveMember, Exited), followed by -0.1595 for (Balance, NumOfProducts); both remain below the threshold.
• Top reported pairs cluster near zero: Several listed correlations are small in magnitude, including (HasCrCard, IsActiveMember) at -0.0492, (Age, NumOfProducts) at -0.0452, (NumOfProducts, IsActiveMember) at 0.0438, (Age, Balance) at 0.0436, and (Tenure, IsActiveMember) at -0.0316.

The observed correlation structure is limited, with only one reported pair exceeding the configured threshold and all remaining listed relationships falling well below it. The strongest observed relationship is between Age and Exited, while the rest of the top reported coefficients are modest to minimal in magnitude. Collectively, the results show that high linear correlation is concentrated in a single reported pair rather than broadly distributed across the listed feature relationships.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3569	Fail
(IsActiveMember, Exited)	-0.1837	Pass
(Balance, NumOfProducts)	-0.1595	Pass
(Balance, Exited)	0.1456	Pass
(NumOfProducts, Exited)	-0.0581	Pass
(HasCrCard, IsActiveMember)	-0.0492	Pass
(Age, NumOfProducts)	-0.0452	Pass
(NumOfProducts, IsActiveMember)	0.0438	Pass
(Age, Balance)	0.0436	Pass
(Tenure, IsActiveMember)	-0.0316	Pass

The output above shows that the test did not pass according to the value we set for max_threshold.

corr_result is an object of type TestResult. We can inspect the result object to see what the test has produced:

print(type(corr_result))
print("Result ID: ", corr_result.result_id)
print("Params: ", corr_result.params)
print("Passed: ", corr_result.passed)
print("Tables: ", corr_result.tables)

<class 'validmind.vm_models.result.result.TestResult'>
Result ID:  validmind.data_validation.HighPearsonCorrelation
Params:  {'max_threshold': 0.3}
Passed:  False
Tables:  [ResultTable]

Let's remove the highly correlated features and create a new VM dataset object.

We'll begin by checking out the table in the result and extracting a list of features that failed the test:

# Extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3569	Fail
1	(IsActiveMember, Exited)	-0.1837	Pass
2	(Balance, NumOfProducts)	-0.1595	Pass
3	(Balance, Exited)	0.1456	Pass
4	(NumOfProducts, Exited)	-0.0581	Pass
5	(HasCrCard, IsActiveMember)	-0.0492	Pass
6	(Age, NumOfProducts)	-0.0452	Pass
7	(NumOfProducts, IsActiveMember)	0.0438	Pass
8	(Age, Balance)	0.0436	Pass
9	(Tenure, IsActiveMember)	-0.0316	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

Next, extract the feature names from the list of strings (example: (Age, Exited) > Age):

high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

Now, it's time to re-initialize the dataset with the highly correlated features removed.

Note the use of a different input_id. This allows tracking the inputs used when running each individual test.

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

Re-running the test with the reduced feature set should pass the test:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify highly correlated pairs that may indicate redundancy or multicollinearity. The result table lists the top 10 feature pairs by Pearson correlation coefficient after removing duplicate and self-correlations, using an absolute-value threshold of 0.3 for pass/fail classification. In this run, all reported feature pairs are marked Pass, and the observed coefficients range from -0.1837 to 0.1456. The strongest reported relationships are between IsActiveMember and Exited (-0.1837), Balance and NumOfProducts (-0.1595), and Balance and Exited (0.1456).

Key insights:

• No pair exceeds threshold: All 10 reported feature pairs are classified as Pass under the 0.3 absolute correlation threshold, indicating that none of the listed linear relationships crosses the test limit.

• Largest correlations are modest: The highest absolute coefficient is 0.1837 for IsActiveMember and Exited, followed by 0.1595 for Balance and NumOfProducts and 0.1456 for Balance and Exited. These values remain materially below the configured threshold.

• Observed relationships are weak overall: The reported coefficients are concentrated near zero, with several of the lower-ranked pairs falling below 0.06 in absolute value, including NumOfProducts and Exited (-0.0581), HasCrCard and IsActiveMember (-0.0492), and NumOfProducts and IsActiveMember (0.0438).

• Both positive and negative associations appear: The top correlations include both negative and positive coefficients, with negative relationships such as IsActiveMember and Exited (-0.1837) and positive relationships such as Balance and Exited (0.1456), indicating no single directional pattern across the strongest reported pairs.

The test result shows a correlation structure in which all reported pairwise linear relationships remain below the configured high-correlation threshold. The largest absolute correlations are modest, and most reported coefficients are close to zero. Collectively, the output indicates limited pairwise linear dependency among the listed features within the scope of this test.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1837	Pass
(Balance, NumOfProducts)	-0.1595	Pass
(Balance, Exited)	0.1456	Pass
(NumOfProducts, Exited)	-0.0581	Pass
(HasCrCard, IsActiveMember)	-0.0492	Pass
(NumOfProducts, IsActiveMember)	0.0438	Pass
(Tenure, IsActiveMember)	-0.0316	Pass
(Tenure, EstimatedSalary)	0.0289	Pass
(Tenure, HasCrCard)	0.0286	Pass
(CreditScore, Exited)	-0.0271	Pass

You can also plot the correlation matrix to visualize the new correlation between features:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.PearsonCorrelationMatrix",
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

Pearson Correlation Matrix

The PearsonCorrelationMatrix test evaluates linear dependency among numerical variables using pairwise Pearson correlation coefficients. The result is presented as a symmetric heat map covering CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with coefficients ranging from -1 to 1 and 1.0 values on the diagonal. Off-diagonal correlations in this matrix are generally close to zero, with the largest observed absolute correlations appearing between IsActiveMember and Exited (-0.18), Balance and NumOfProducts (-0.16), and Balance and Exited (0.15).

Key insights:

• No high pairwise correlations observed: All off-diagonal correlation coefficients remain well below the 0.7 absolute threshold referenced by the test. The matrix does not show any strongly correlated variable pairs among the numerical features included.

• Exited has weak linear relationships: Correlations between Exited and the other variables are small in magnitude, with the largest values at -0.18 for IsActiveMember, 0.15 for Balance, and -0.06 for NumOfProducts. The remaining relationships with Exited are near zero, including 0.00 for EstimatedSalary and approximately 0.00 for HasCrCard.

• Balance shows the most notable feature relationships: Among predictor-to-predictor pairs, Balance has the largest observed association with NumOfProducts at -0.16. Its correlations with the remaining variables are minimal, including 0.01 with CreditScore, -0.01 with Tenure, -0.02 with HasCrCard, -0.01 with IsActiveMember, and 0.02 with EstimatedSalary.

• Most variables are effectively uncorrelated: Several pairs display coefficients at or very near zero, such as Tenure with NumOfProducts (0.00), EstimatedSalary with NumOfProducts (-0.00), and CreditScore with multiple variables between -0.03 and 0.02. This indicates a sparse linear dependence structure across the variables in the matrix.

The correlation matrix shows a generally weak linear dependence structure across the numerical variables assessed. No pairwise relationship approaches the high-correlation threshold defined in the test, and the largest observed associations remain limited in magnitude. Overall, the result indicates low redundancy on the basis of Pearson linear correlation within the variables shown.

Figures

Documenting test results

Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it.

Every test result returned by the run_test() function has a .log() method that can be used to send the test results to the ValidMind Platform:

• When using run_documentation_tests(), documentation sections will be automatically populated with the results of all tests registered in the documentation template.
• When logging individual test results to the platform, you'll need to manually add those results to the desired section of the documentation.

To demonstrate how to add test results to your documentation, we'll populate the entire Data Preparation section of the documentation using the clean vm_raw_dataset_preprocessed dataset as input, and then document an additional individual result for the highly correlated dataset vm_balanced_raw_dataset.

Run and log multiple tests

run_documentation_tests() allows you to run multiple tests at once and automatically log the results to your documentation. Below, we'll run the tests using the previously initialized vm_raw_dataset_preprocessed as input — this will populate the entire Data Preparation section for every test that is part of the documentation template.

For this example, we'll pass in the following arguments:

• inputs: Any inputs to be passed to the tests.
• config: A dictionary <test_id>:<test_config> that allows configuring each test individually. Each test config requires the following:
  • params: Individual test parameters.
  • inputs: Individual test inputs. This overrides any inputs passed from the run_documentation_tests() function.

When including explicit configuration for individual tests, you'll need to specify the inputs even if they mirror what is included in your global configuration.

# Individual test config with inputs specified
test_config = {
    "validmind.data_validation.ClassImbalance": {
        "params": {"min_percent_threshold": 30},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "params": {"max_threshold": 0.3},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
}

# Global test config
tests_suite = vm.run_documentation_tests(
    inputs={
        "dataset": vm_raw_dataset_preprocessed,
    },
    config=test_config,
    section=["data_preparation"],
)

Test suite complete!

26/26 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Data Preparation

Data Preparation

▶ Test Result: Dataset Description (validmind.data_validation.DatasetDescription)

Dataset Description

The Dataset Description test evaluates the structure, completeness, and distinct-value characteristics of each column in the dataset. The results summarize 10 variables across numeric and categorical types, reporting counts, missingness, and distinct-value levels for each field. All columns contain 3,232 observed records, and the table shows the distribution of variable cardinality ranging from binary categorical fields to fully unique numeric values.

Key insights:

• No missing values observed: All 10 columns have a count of 3,232 with 0 missing values and 0.0% missingness, indicating complete population of the dataset across all listed variables.

• EstimatedSalary is fully unique: EstimatedSalary contains 3,232 distinct values, corresponding to a distinct ratio of 1.0, making it the highest-cardinality field in the dataset.

• Balance also shows high cardinality: Balance has 2,191 distinct values, representing 67.79% of all records, which is materially higher than the remaining fields other than EstimatedSalary.

• Categorical fields are low-cardinality: Geography has 3 distinct values, while Gender, HasCrCard, IsActiveMember, and Exited each have 2 distinct values, indicating compact categorical domains.

• Numeric fields differ substantially in granularity: Among numeric variables, CreditScore has 436 distinct values, Tenure has 11, NumOfProducts has 4, Balance has 2,191, and EstimatedSalary has 3,232, showing substantial variation in numeric value resolution across features.

The dataset description indicates a fully complete dataset with no observed missingness in any reported column. The primary structural variation arises from differences in feature cardinality: several categorical and discrete numeric fields have very limited value ranges, while Balance and especially EstimatedSalary exhibit high distinctness. Overall, the results describe a dataset with consistent row-level completeness and mixed feature granularity across predictors and the target variable.

Tables

Dataset Description

Name	Type	Count	Missing	Missing %	Distinct	Distinct %
CreditScore	Numeric	3232.0	0	0.0	436	0.1349
Geography	Categorical	3232.0	0	0.0	3	0.0009
Gender	Categorical	3232.0	0	0.0	2	0.0006
Tenure	Numeric	3232.0	0	0.0	11	0.0034
Balance	Numeric	3232.0	0	0.0	2191	0.6779
NumOfProducts	Numeric	3232.0	0	0.0	4	0.0012
HasCrCard	Categorical	3232.0	0	0.0	2	0.0006
IsActiveMember	Categorical	3232.0	0	0.0	2	0.0006
EstimatedSalary	Numeric	3232.0	0	0.0	3232	1.0000
Exited	Categorical	3232.0	0	0.0	2	0.0006

▶ Test Result: Class Imbalance (validmind.data_validation.ClassImbalance)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring each class’s share of total records against a minimum percentage threshold. In this run, the target variable Exited contains two classes, 0 and 1, and the results table reports the percentage of rows for each class alongside a pass/fail outcome. The configured minimum percentage threshold is 30%, and both classes are shown at 50.00%. The accompanying bar chart reflects the same equal split between the two target classes.

Key insights:

• Perfectly even class distribution: The Exited target is split evenly between class 0 and class 1, with each representing 50.00% of the dataset.
• All classes pass threshold: Both target classes exceed the configured 30% minimum percentage threshold and receive a Pass result.
• No detected underrepresented class: The test output shows no class below the threshold, indicating no minority class under the applied criterion.

The results show a fully balanced binary target distribution for Exited, with both classes represented equally in the dataset. Relative to the 30% threshold used in this test, no class falls below the minimum share, and the test identifies no class-level imbalance under the reported criterion.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

▶ Test Result: Duplicates (validmind.data_validation.Duplicates)

✅ Duplicates

The Duplicates test evaluates whether the dataset contains exact duplicate rows. The result table reports both the absolute count of duplicate rows and their share of the dataset. In this run, the table shows 0 duplicate rows and a corresponding duplicate rate of 0.0%.

Key insights:

• No duplicate rows detected: The dataset contains 0 duplicate rows based on the executed duplicate check.
• Duplicate rate is zero: Duplicate rows account for 0.0% of the dataset, indicating no repeated records were identified in the evaluated data.

The duplicate check identified no exact row-level duplication in the dataset. The reported count and percentage are both zero, indicating that duplicate records were not present in the tested data under this method.

Tables

Duplicate Rows Results for Dataset

Number of Duplicates	Percentage of Rows (%)
0	0.0

▶ Test Result: High Cardinality (validmind.data_validation.HighCardinality)

✅ High Cardinality

The High Cardinality test evaluates the number of unique values in categorical columns to identify features with a large count of distinct categories. The results table reports the categorical columns assessed, along with each column’s number of distinct values, percentage of distinct values, and pass/fail status against the test threshold. In this run, two categorical variables were evaluated: Geography and Gender. Geography has 3 distinct values with 0.0928% distinctness, and Gender has 2 distinct values with 0.0619% distinctness; both columns passed the test.

Key insights:

• No categorical failures observed: Both evaluated categorical columns, Geography and Gender, received a Pass result in the test output.
• Low distinct counts across categories: Geography contains 3 distinct values and Gender contains 2 distinct values, indicating limited category counts among the assessed categorical features.
• Geography has the higher cardinality: Among the evaluated columns, Geography has the larger distinct count and distinct-value percentage at 3 values and 0.0928%, compared with Gender at 2 values and 0.0619%.

The results show that all categorical columns assessed by this test remained below the applied high-cardinality threshold. Distinct counts are low for both evaluated features, with only a small difference between Geography and Gender. Within the scope of the tested categorical variables, the output indicates no high-cardinality findings.

Tables

Column	Number of Distinct Values	Percentage of Distinct Values (%)	Pass/Fail
Geography	3	0.0928	Pass
Gender	2	0.0619	Pass

▶ Test Result: Missing Values (validmind.data_validation.MissingValues)

✅ Missing Values

The Missing Values test evaluates dataset completeness by measuring the proportion of missing entries in each feature against the configured threshold. The results are presented at the column level, showing the number of missing values, the corresponding missing-value percentage, and the Pass/Fail outcome for each variable. In this run, all 10 evaluated columns display 0 missing values and 0.0% missingness, and each column is marked as Pass.

Key insights:

• No missing values detected: Every evaluated feature, including CreditScore, Geography, Gender, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, records 0 missing values.
• Uniform 0.0% missingness: The percentage of missing values is 0.0% for all columns, indicating no variation in completeness across the dataset fields included in the test.
• All columns passed threshold check: Each of the 10 evaluated columns is marked Pass, indicating that no feature exceeded the test threshold for missing-value percentage.

The results show complete observed data coverage across all tested fields, with no missing entries identified in any evaluated column. Missingness is uniformly absent rather than concentrated in specific variables, and the threshold assessment produces a pass outcome for the full set of tested features.

Tables

Column	Number of Missing Values	Percentage of Missing Values (%)	Pass/Fail
CreditScore	0	0.0	Pass
Geography	0	0.0	Pass
Gender	0	0.0	Pass
Tenure	0	0.0	Pass
Balance	0	0.0	Pass
NumOfProducts	0	0.0	Pass
HasCrCard	0	0.0	Pass
IsActiveMember	0	0.0	Pass
EstimatedSalary	0	0.0	Pass
Exited	0	0.0	Pass

▶ Test Result: Skewness (validmind.data_validation.Skewness)

❌ Skewness

The Skewness test evaluates asymmetry in the distribution of numeric variables by comparing each column’s skewness value against the configured threshold of 1. The results table reports skewness values and pass/fail outcomes for eight numerical columns in the dataset. Observed skewness values range from -0.8439 to 1.2258, with seven columns passing the test and one column failing.

Key insights:

• Single threshold breach observed: NumOfProducts is the only column exceeding the threshold, with skewness of 1.2258 and a fail result.
• Most variables show limited skewness: Seven of eight numerical columns pass the test, with skewness values between -0.8439 and 0.1565 for all passing variables.
• Several columns are near symmetric: Exited has skewness of 0.0, while EstimatedSalary (-0.0093), Tenure (0.0387), and CreditScore (-0.0921) are also close to zero.
• Negative skew dominates among passing variables: HasCrCard (-0.8439) shows the largest-magnitude skewness among passing columns, followed by Balance (-0.268), while positive skew among passing columns remains modest.

The results indicate that skewness is generally low across the numeric dataset, with passing outcomes for the large majority of evaluated variables. Distributional asymmetry is concentrated in NumOfProducts, which is the only feature above the threshold. Outside this exception, the remaining variables exhibit either near-symmetric behavior or moderate negative skew while remaining within the defined limit.

Tables

Skewness Results for Dataset

Column	Skewness	Pass/Fail
CreditScore	-0.0921	Pass
Tenure	0.0387	Pass
Balance	-0.2680	Pass
NumOfProducts	1.2258	Fail
HasCrCard	-0.8439	Pass
IsActiveMember	0.1565	Pass
EstimatedSalary	-0.0093	Pass
Exited	0.0000	Pass

▶ Test Result: Unique Rows (validmind.data_validation.UniqueRows)

❌ Unique Rows

The UniqueRows test evaluates dataset diversity by measuring the number and percentage of unique values in each column against a prescribed threshold. The results are reported at the column level, showing each field’s unique value count, percentage of unique values, and pass/fail outcome. Across the 10 evaluated columns, 3 columns passed and 7 failed. The highest uniqueness was observed in EstimatedSalary at 100.0%, followed by Balance at 67.7908% and CreditScore at 13.4901%, while the remaining columns showed uniqueness below 1%.

Key insights:

• Pass results are concentrated in numeric fields: EstimatedSalary passed with 3,232 unique values and 100.0% uniqueness, Balance passed with 2,191 unique values and 67.7908%, and CreditScore passed with 436 unique values and 13.4901%.
• Most columns failed the uniqueness threshold: 7 of 10 columns failed, including Geography, Gender, Tenure, NumOfProducts, HasCrCard, IsActiveMember, and Exited.
• Several variables have very limited distinct values: Gender, HasCrCard, IsActiveMember, and Exited each contain 2 unique values, with 0.0619% uniqueness. Geography contains 3 unique values at 0.0928%, and NumOfProducts contains 4 unique values at 0.1238%.
• Uniqueness varies substantially across columns: Reported uniqueness percentages range from 0.0619% to 100.0%, indicating a wide spread in the level of distinct observed values across the dataset.
• Tenure remains low-uniqueness despite more categories: Tenure contains 11 unique values but still failed the test, with a uniqueness rate of 0.3403%.

The test results show that column-level uniqueness is uneven across the dataset, with only EstimatedSalary, Balance, and CreditScore exceeding the prescribed threshold. Most other columns have a small number of distinct observed values and are marked as failing under this test. Overall, the result reflects a dataset containing a mix of highly variable continuous fields and low-cardinality fields with limited uniqueness.

Tables

Column	Number of Unique Values	Percentage of Unique Values (%)	Pass/Fail
CreditScore	436	13.4901	Pass
Geography	3	0.0928	Fail
Gender	2	0.0619	Fail
Tenure	11	0.3403	Fail
Balance	2191	67.7908	Pass
NumOfProducts	4	0.1238	Fail
HasCrCard	2	0.0619	Fail
IsActiveMember	2	0.0619	Fail
EstimatedSalary	3232	100.0000	Pass
Exited	2	0.0619	Fail

▶ Test Result: Too Many Zero Values (validmind.data_validation.TooManyZeroValues)

❌ Too Many Zero Values

The TooManyZeroValues test evaluates numerical columns for excessive concentrations of zero values relative to a defined threshold. The results table reports row count, zero-value count, zero-value percentage, and pass/fail status for each evaluated numerical variable. Four variables are listed in the output—Tenure, Balance, HasCrCard, and IsActiveMember—and each is marked as failing the test. Reported zero-value percentages range from 4.703% to 53.8985% across these variables.

Key insights:

• All evaluated variables failed: Tenure, Balance, HasCrCard, and IsActiveMember each exceeded the configured zero-value threshold and are recorded with a fail status in the results.

• IsActiveMember has the highest zero share: IsActiveMember contains 1,742 zero values out of 3,232 rows, corresponding to 53.8985%, which is the largest zero-value proportion among the evaluated variables.

• Balance and HasCrCard show substantial zero concentration: Balance records 1,042 zero values (32.2401%) and HasCrCard records 988 zero values (30.5693%), indicating that approximately one-third of observations in both variables are zero.

• Tenure has the lowest but still failing zero rate: Tenure contains 152 zero values out of 3,232 rows, or 4.703%, which is the smallest zero-value proportion reported, but it remains above the test threshold and is therefore classified as a failure.

The test results show that each numerical variable evaluated contains zero values at levels above the configured threshold. The degree of zero concentration varies materially across variables, with IsActiveMember exhibiting the highest proportion, Balance and HasCrCard showing similarly elevated levels, and Tenure presenting the lowest observed zero rate. Collectively, the output indicates that elevated zero frequency is present across all four reported variables rather than being isolated to a single field.

Tables

Variable	Row Count	Number of Zero Values	Percentage of Zero Values (%)	Pass/Fail
Tenure	3232	152	4.7030	Fail
Balance	3232	1042	32.2401	Fail
HasCrCard	3232	988	30.5693	Fail
IsActiveMember	3232	1742	53.8985	Fail

▶ Test Result: IQR Outliers Table (validmind.data_validation.IQROutliersTable)

IQR Outliers Table

The IQROutliersTable test evaluates numerical features for observations falling outside the interquartile range thresholds and summarizes the count and distribution of those outlier values. The results table reports outlier statistics for two variables: CreditScore and NumOfProducts. For each variable, the table includes the total count of outliers, the mean value of the full variable, and the minimum, quartile, median, and maximum values among the detected outliers. In this run, 11 outliers were identified for CreditScore and 44 for NumOfProducts.

Key insights:

• NumOfProducts has the most outliers: NumOfProducts records 44 outliers, compared with 11 for CreditScore, making it the more concentrated source of IQR-flagged observations in this result set.

• NumOfProducts outliers are uniform: All detected outliers for NumOfProducts take the value 4, as shown by identical minimum, 25th percentile, median, 75th percentile, and maximum outlier values.

• CreditScore outliers are concentrated at low values: CreditScore outliers range from 350 to 376, with quartiles at 350, 359, and 370, while the reported mean value of the variable is 647.8914.

• CreditScore outlier spread is limited: The CreditScore outlier range spans 26 points from 350 to 376, indicating a relatively narrow band of flagged values.

The IQR outlier summary identifies outliers in two numerical variables, with NumOfProducts contributing the larger count and exhibiting a single repeated outlier value. CreditScore shows fewer outliers, all concentrated within a narrow low-value interval relative to the reported variable mean. Collectively, the results indicate that the detected outliers are localized rather than broadly dispersed across multiple numeric features in the reported table.

Tables

Summary of Outliers Detected by IQR Method

Variable	Total Count of Outliers	Mean Value of Variable	Minimum Outlier Value	Outlier Value at 25th Percentile	Outlier Value at 50th Percentile	Outlier Value at 75th Percentile	Maximum Outlier Value
CreditScore	11	647.8914	350	350.0	359.0	370.0	376
NumOfProducts	44	1.5111	4	4.0	4.0	4.0	4

▶ Test Result: IQR Outliers Bar Plot (validmind.data_validation.IQROutliersBarPlot)

IQR Outliers Bar Plot

The IQROutliersBarPlot test evaluates the distribution of IQR-based outliers across percentile bands for numeric features. The results shown cover CreditScore and NumOfProducts, with bar heights indicating the count of observations classified as outliers within each percentile range. For CreditScore, outliers appear in the 25–50, 50–75, and 75–100 bands, while the 0–25 band shows none. For NumOfProducts, all observed outliers are concentrated in the 75–100 percentile band.

Key insights:

• CreditScore outliers are distributed across upper bands: CreditScore shows 5 outliers in the 25–50 percentile band, 3 in the 50–75 band, and 3 in the 75–100 band, with no outliers in the 0–25 band. This indicates that outlier observations are not isolated to only the most extreme percentile range.

• NumOfProducts outliers are highly concentrated: NumOfProducts shows no outliers in the 0–25, 25–50, or 50–75 percentile bands, and approximately 44 outliers in the 75–100 band. The outlier pattern is therefore entirely concentrated in the upper percentile range.

• Outlier magnitude differs materially by feature: The total visible outlier count for NumOfProducts is substantially higher than for CreditScore. In the plotted results, NumOfProducts has a single large concentration of outliers, whereas CreditScore has a smaller number spread across multiple percentile bands.

These results show two distinct outlier patterns across the plotted numeric features. CreditScore exhibits a moderate number of outliers distributed across the middle-to-upper percentile bands, while NumOfProducts exhibits a much larger concentration confined to the highest percentile band. Collectively, the plots indicate that outlier presence varies materially by feature both in count and in where it appears along the percentile distribution.

Figures

▶ Test Result: Descriptive Statistics (validmind.data_validation.DescriptiveStatistics)

Descriptive Statistics

The Descriptive Statistics test evaluates the distributional characteristics of numerical and categorical variables in the dataset. The results summarize 3,232 observations for each reported field and present central tendency, dispersion, percentile, and range information for seven numerical variables, alongside category counts and concentration measures for two categorical variables. The numerical table shows substantial variation across variables in both scale and spread, while the categorical table shows the number of distinct values and the frequency share of the most common category for Geography and Gender.

Key insights:

• No missingness in reported fields: All listed numerical and categorical variables have a count of 3,232, indicating complete coverage for the variables included in this result.

• Balance shows the widest dispersion: Balance has a mean of 82,049.64 and a standard deviation of 61,818.13, with values ranging from 0 to 250,898. Its 25th percentile is 0 while the median is 103,343, indicating a pronounced concentration at zero combined with a broad upper range.

• EstimatedSalary is broadly distributed: EstimatedSalary spans from 12 to 199,909, with a mean of 100,124.97 and median of 100,262. The 25th and 75th percentiles of 49,905 and 149,864 indicate a wide distribution around a central value near 100,000.

• CreditScore is centered near its median: CreditScore has a mean of 647.89 and a median of 650, with an interquartile range from 580 to 714 and an overall range from 350 to 850. The close alignment of mean and median indicates limited central-tendency asymmetry in the reported summary.

• NumOfProducts is concentrated at low counts: NumOfProducts has a median of 1, a 75th percentile of 2, and a maximum of 4. The mean of 1.51 indicates the distribution is concentrated around one to two products.

• Binary indicators are unevenly distributed: HasCrCard has a mean of 0.6943, indicating the value 1 accounts for about 69.43% of observations, while IsActiveMember has a mean of 0.4610, indicating the value 1 accounts for about 46.10% of observations.

• Categorical concentration is moderate: Geography contains 3 unique values, with France as the most frequent category at 1,492 observations or 46.16%. Gender contains 2 unique values, with Male as the top category at 1,650 observations or 51.05%, indicating near-balanced representation across the two categories.

The descriptive statistics show complete observed coverage across the reported fields and heterogeneous distributional behavior across variables. The most pronounced dispersion appears in Balance, which combines a zero-valued lower quartile with a substantially higher median and broad range, while CreditScore and EstimatedSalary appear more centrally distributed based on the reported mean and median values. The categorical variables exhibit moderate concentration in their top categories, with neither Geography nor Gender dominated by a single category to an extreme degree in the reported sample.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	3232.0	647.8914	97.5874	350.0	580.0	650.0	714.0	776.0	812.0	850.0
Tenure	3232.0	4.9805	2.9023	0.0	3.0	5.0	7.0	9.0	10.0	10.0
Balance	3232.0	82049.6353	61818.1276	0.0	0.0	103343.0	129764.0	150834.0	165304.0	250898.0
NumOfProducts	3232.0	1.5111	0.6717	1.0	1.0	1.0	2.0	2.0	3.0	4.0
HasCrCard	3232.0	0.6943	0.4608	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	3232.0	0.4610	0.4986	0.0	0.0	0.0	1.0	1.0	1.0	1.0
EstimatedSalary	3232.0	100124.9724	57812.6530	12.0	49905.0	100262.0	149864.0	179289.0	189603.0	199909.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	3232.0	3.0	France	1492.0	46.16
Gender	3232.0	2.0	Male	1650.0	51.05

▶ Test Result: Pearson Correlation Matrix (validmind.data_validation.PearsonCorrelationMatrix)

Pearson Correlation Matrix

The PearsonCorrelationMatrix test evaluates linear dependency between numerical variables in the dataset using pairwise Pearson correlation coefficients. The result is presented as a symmetric heat map covering CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with coefficients ranging from -1 to 1 and diagonal values of 1.0. Off-diagonal correlations in this matrix are generally close to zero, with the largest absolute values observed between IsActiveMember and Exited (-0.18), Balance and NumOfProducts (-0.16), and Balance and Exited (0.15).

Key insights:

• No high pairwise correlations observed: All off-diagonal correlations are well below the 0.70 threshold referenced in the test description. The matrix does not show any strongly linear-dependent variable pairs.
• Exited has weak linear relationships: The target variable Exited shows weak correlations with the numerical inputs, including -0.18 with IsActiveMember, 0.15 with Balance, -0.06 with NumOfProducts, and values near zero for the remaining variables.
• Balance shows the most visible feature relationships: Among predictor pairs, Balance has the largest observed associations, including -0.16 with NumOfProducts and 0.15 with Exited, while its correlations with other variables remain close to zero.
• Most variables are effectively uncorrelated linearly: CreditScore, Tenure, HasCrCard, and EstimatedSalary exhibit pairwise coefficients concentrated around zero, indicating minimal linear association across the numerical feature set.

The correlation structure is sparse, with all non-diagonal coefficients remaining weak in magnitude. The most notable observed relationships are limited to small negative correlations between IsActiveMember and Exited and between Balance and NumOfProducts, along with a small positive correlation between Balance and Exited. Overall, the result indicates limited linear redundancy among the numerical variables included in this test.

Figures

▶ Test Result: High Pearson Correlation (validmind.data_validation.HighPearsonCorrelation)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant or highly collinear variables. The results table lists the top 10 feature pairs by Pearson correlation coefficient after removing duplicate and self-correlations, together with a Pass/Fail flag based on the configured absolute correlation threshold of 0.3. In this run, all reported coefficients fall within a narrow range from -0.1837 to 0.1456, and every listed pair is marked as Pass.

Key insights:

• No pair exceeds threshold: All 10 reported feature pairs pass the test under the 0.3 threshold. The largest absolute coefficient is 0.1837 for (IsActiveMember, Exited), which remains below the configured limit.
• Observed relationships are weak: The reported coefficients range from -0.1837 to 0.1456, indicating low-magnitude linear associations across the strongest pairs returned by the test.
• Strongest negative association involves activity status: (IsActiveMember, Exited) shows the largest absolute correlation at -0.1837. Among the listed pairs, this is the most pronounced inverse linear relationship.
• Strongest positive association involves balance and churn: (Balance, Exited) has the largest positive coefficient at 0.1456. Other positive coefficients in the table are smaller, including (NumOfProducts, IsActiveMember) at 0.0438 and (Tenure, EstimatedSalary) at 0.0289.

The reported correlation structure does not show any feature pair with linear dependence above the configured threshold. The strongest associations identified by the test are limited in magnitude, with the highest absolute value remaining below 0.2. Collectively, the results indicate that the top-ranked pairwise linear relationships in this dataset are weak based on the Pearson correlation measure used in this test.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1837	Pass
(Balance, NumOfProducts)	-0.1595	Pass
(Balance, Exited)	0.1456	Pass
(NumOfProducts, Exited)	-0.0581	Pass
(HasCrCard, IsActiveMember)	-0.0492	Pass
(NumOfProducts, IsActiveMember)	0.0438	Pass
(Tenure, IsActiveMember)	-0.0316	Pass
(Tenure, EstimatedSalary)	0.0289	Pass
(Tenure, HasCrCard)	0.0286	Pass
(CreditScore, Exited)	-0.0271	Pass

Run and log an individual test

Next, we'll use the previously initialized vm_balanced_raw_dataset (that still has a highly correlated Age column) as input to run an individual test, then log the result to the ValidMind Platform.

When running individual tests, you can use a custom result_id to tag the individual result with a unique identifier:

• This result_id can be appended to test_id with a : separator.
• The balanced_raw_dataset result identifier will correspond to the balanced_raw_dataset input, the dataset that still has the Age column.

result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)
result.log()

❌ High Pearson Correlation Balanced Raw Dataset

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant or highly collinear variable pairs. The result table reports the top 10 feature pairs ranked by Pearson correlation coefficient, along with Pass/Fail status based on the configured absolute correlation threshold of 0.3. Reported coefficients range from -0.1837 to 0.3569, and only one pair exceeds the threshold. The highest observed correlation is between Age and Exited, while the remaining listed relationships are below the test limit.

Key insights:

• Only one pair breaches threshold: The pair (Age, Exited) has a Pearson correlation of 0.3569, which exceeds the 0.3 threshold and is the only result marked Fail in the reported output.

• All other listed pairs pass: The remaining nine reported pairs have absolute correlations below 0.3, with values ranging from 0.1837 down to 0.0316, and are all marked Pass.

• Most observed relationships are weak: Aside from (Age, Exited), the largest absolute correlation in the table is -0.1837 for (IsActiveMember, Exited), indicating that the reported top correlations are otherwise limited in magnitude.

• Both positive and negative associations appear: The reported coefficients include positive values such as 0.1456 for (Balance, Exited) and negative values such as -0.1595 for (Balance, NumOfProducts), showing that the strongest listed relationships are not concentrated in a single direction.

The reported correlation structure is concentrated around one threshold breach and a set of otherwise low-magnitude pairwise relationships. Age and Exited form the only feature pair with an absolute Pearson correlation above the configured limit, while all other listed pairs remain below that level. Overall, the table indicates limited high linear dependence among the reported top correlations, with the single exception of the Age–Exited relationship.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3569	Fail
(IsActiveMember, Exited)	-0.1837	Pass
(Balance, NumOfProducts)	-0.1595	Pass
(Balance, Exited)	0.1456	Pass
(NumOfProducts, Exited)	-0.0581	Pass
(HasCrCard, IsActiveMember)	-0.0492	Pass
(Age, NumOfProducts)	-0.0452	Pass
(NumOfProducts, IsActiveMember)	0.0438	Pass
(Age, Balance)	0.0436	Pass
(Tenure, IsActiveMember)	-0.0316	Pass

2026-10-30 20:53:39,020 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset does not exist in model's document

Note the output returned indicating that a test-driven block doesn't currently exist in your documentation for this particular test ID.

That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.

Add individual test results to documentation

With the test results logged, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation (Learn more: Work with test results):

1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

2. In the left sidebar that appears for your model, click Development under Documents.

3. Locate the Data Preparation section and click on 2.3. Correlations and Interactions to expand that section.

4. Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a + button appears, indicating that you can insert a new block.

   
   
   

5. Click + and then select Test-Driven Block under FROM LIBRARY:

   • Click on VM Library under TEST-DRIVEN in the left sidebar.
   • In the search bar, type in HighPearsonCorrelation.
   • Select HighPearsonCorrelation:balanced_raw_dataset as the test.

   A preview of the test gets shown:

   
   
   

6. Finally, click Insert 1 Test Result to Document to add the test result to the documentation.

   Confirm that the individual results for the high correlation test has been correctly inserted into section 2.3. Correlations and Interactions of the documentation.

7. Finalize the documentation by editing the test result's description block to explain the changes you made to the raw data and the reasons behind them as shown in the screenshot below:

   

Running model evaluation tests

So far, we've focused on the data assessment and pre-processing that usually occurs prior to any models being built. Now, let's instead assume we have already built a model and we want to incorporate some model results into our documentation.

Train simple logistic regression model

Using ValidMind tests, we'll train a simple logistic regression model on our dataset and evaluate its performance by using the LogisticRegression class from the sklearn.linear_model.

To start, let's grab the first few rows from the balanced_raw_no_age_df dataset with the highly correlated features removed we initialized earlier:

balanced_raw_no_age_df.head()

	CreditScore	Geography	Gender	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
1741	600	France	Male	4	143744.77	2	1	0	104076.51	0
6223	585	France	Male	4	138227.19	2	1	1	55287.84	0
3875	763	France	Male	3	0.00	2	1	0	134281.11	0
5695	554	Germany	Female	0	108477.27	1	0	1	140003.00	1
3780	661	Germany	Female	8	128751.65	2	1	0	189398.18	1

Before training the model, we need to encode the categorical features in the dataset:

• Use the OneHotEncoder class from the sklearn.preprocessing module to encode the categorical features.
• The categorical features in the dataset are Geography and Gender.

balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
1741	600	4	143744.77	2	1	0	104076.51	0	False	False	True
6223	585	4	138227.19	2	1	1	55287.84	0	False	False	True
3875	763	3	0.00	2	1	0	134281.11	0	False	False	True
5695	554	0	108477.27	1	0	1	140003.00	1	True	False	False
3780	661	8	128751.65	2	1	0	189398.18	1	True	False	False

We'll split our preprocessed dataset into training and testing, to help assess how well the model generalizes to unseen data:

• We start by dividing our balanced_raw_no_age_df dataset into training and test subsets using train_test_split, with 80% of the data allocated to training (train_df) and 20% to testing (test_df).
• From each subset, we separate the features (all columns except "Exited") into X_train and X_test, and the target column ("Exited") into y_train and y_test.

from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

Then using GridSearchCV, we'll find the best-performing hyperparameters or settings and save them:

from sklearn.linear_model import LogisticRegression

# Logistic Regression grid params
log_reg_params = {
    "penalty": ["l1", "l2"],
    "C": [0.001, 0.01, 0.1, 1, 10, 100, 1000],
    "solver": ["liblinear"],
}

# Grid search for Logistic Regression
from sklearn.model_selection import GridSearchCV

grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)
grid_log_reg.fit(X_train, y_train)

# Logistic Regression best estimator
log_reg = grid_log_reg.best_estimator_

Initialize ValidMind datasets

The last step for evaluating the model's performance is to initialize the ValidMind Dataset and Model objects in preparation for assigning model predictions to each dataset.

# Initialize the datasets into their own dataset objects
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

Initialize a ValidMind model

You'll also need to initialize a ValidMind model object (vm_model) that can be passed to other functions for analysis and tests on the data for each of our three models.

• Despite the naming convention, ValidMind model objects can be any type of record you want to test, document, validate, or monitor with the ValidMind Library.
• From classical statistical and machine learning models, to generative and agentic AI systems and more, the ValidMind model object provides a consistent wrapper around your record so it can be passed as a unified input to any ValidMind test or test suite, with results sent directly to the ValidMind Platform.

Initialize your model object with vm.init_model():

# Register the model
vm_model = vm.init_model(log_reg, input_id="log_reg_model_v1")

Assign predictions

Once the model has been registered you can assign predictions to the training and testing datasets.

• The assign_predictions() method from the Dataset object can link existing predictions to any number of models.
• This method links the model's class prediction values and probabilities to our vm_train_ds and vm_test_ds datasets.

If no prediction values are passed, the method will compute predictions automatically:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

2026-10-30 20:53:40,039 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-10-30 20:53:40,040 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-10-30 20:53:40,041 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-10-30 20:53:40,043 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-10-30 20:53:40,045 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-10-30 20:53:40,046 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-10-30 20:53:40,047 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-10-30 20:53:40,048 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Run the model evaluation tests

In this next example, we'll focus on running the tests within the Model Development section of the documentation. Only tests associated with this section will be executed, and the corresponding results will be updated in the documentation.

• Note the additional config that is passed to run_documentation_tests() — this allows you to override inputs or params in certain tests.
• In our case, we want to explicitly use the vm_train_ds for the validmind.model_validation.sklearn.ClassifierPerformance:in_sample test, since it's supposed to run on the training dataset and not the test dataset.

test_config = {
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "dataset": vm_train_ds,
            "model": vm_model,
        },
    }
}
results = vm.run_documentation_tests(
    section=["model_development"],
    inputs={
        "dataset": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds
        "model": vm_model,
        "datasets": (
            vm_train_ds,
            vm_test_ds,
        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds
    },
    config=test_config,
)

Test suite complete!

34/34 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Development

Model Development

▶ Test Result: Model Metadata (validmind.model_validation.ModelMetadata)

Model Metadata

The ModelMetadata test compares core metadata fields across models to summarize their implementation characteristics. The result is presented as a metadata table covering modeling technique, modeling framework, framework version, and programming language. For the reported model entry, the table identifies a single configuration consisting of the SKlearnModel technique implemented in the sklearn framework, using framework version 1.7.2 and Python.

Key insights:

• Single model entry reported: The result table contains one model metadata record, so the output documents the model configuration rather than showing cross-model variation.
• SKlearn-based implementation: The modeling technique is listed as SKlearnModel, and the modeling framework is listed as sklearn, indicating alignment between the model wrapper and underlying framework.
• Framework version explicitly specified: The framework version is reported as 1.7.2, providing a precise version reference for the recorded implementation.
• Python language recorded: The programming language field is populated as Python, completing the set of reported high-level implementation metadata.

The reported metadata provides a complete high-level implementation record for the model entry included in the test output. The result identifies the model as a Python-based sklearn implementation using framework version 1.7.2, with no additional model records shown for comparison.

Tables

Modeling Technique	Modeling Framework	Framework Version	Programming Language
SKlearnModel	sklearn	1.7.2	Python

▶ Test Result: Dataset Split (validmind.data_validation.DatasetSplit)

Dataset Split

The DatasetSplit test evaluates the distribution of observations across the model’s available data splits. The result table reports the absolute size and share of the total sample for each dataset included in the split. In this execution, the table contains training and test datasets, along with the combined total. The reported values show 2,585 observations in train_dataset_final, 647 observations in test_dataset_final, and 3,232 observations overall.

Key insights:

• Training data comprises most records: train_dataset_final contains 2,585 observations, representing 79.98% of the total dataset.
• Test data represents one-fifth of sample: test_dataset_final contains 647 observations, accounting for 20.02% of the total dataset.
• Two-way split is documented: The reported split includes training and test datasets only; no validation dataset is listed in the result table.

The results document a two-part dataset split across 3,232 total observations, with the majority allocated to training and the remainder allocated to testing. The distribution is explicitly quantified in both absolute counts and proportions, providing a clear view of how model development and evaluation data were partitioned in this test run.

Tables

Dataset	Size	Proportion
train_dataset_final	2585	79.98%
test_dataset_final	647	20.02%
Total	3232	100%

▶ Test Result: Population Stability Index (validmind.model_validation.sklearn.PopulationStabilityIndex)

Population Stability Index

The Population Stability Index (PSI) test evaluates the stability of the model output distribution by comparing the bin-level distribution of observations between the initial and new datasets. The results are presented across 10 bins, showing count, population share, and bin-level PSI contributions for train_dataset_final versus test_dataset_final. The initial dataset contains 2,585 observations and the new dataset contains 647 observations, with total PSI reported as 0.0367. Bin-level population shares and PSI contributions are also shown in the accompanying chart, which highlights where the relative distribution differences occur.

Key insights:

• Low overall PSI: The total PSI is 0.0367 based on the comparison between train_dataset_final and test_dataset_final, indicating that the aggregate distribution difference observed across bins is limited in magnitude.

• Drift is dispersed across several bins: PSI contributions are spread across multiple bins rather than concentrated in a single segment. The largest individual contributions occur in bin 3 (0.0080), bin 9 (0.0078), bin 4 (0.0075), and bin 6 (0.0070).

• Largest share increases occur in mid and upper bins: The new dataset has higher population shares than the initial dataset in bin 2 (14.0649% vs. 12.5725%), bin 4 (15.3014% vs. 12.1083%), bin 7 (12.0556% vs. 10.7930%), bin 8 (4.4822% vs. 4.2166%), and bin 9 (9.1190% vs. 6.6538%).

• Largest share decreases occur in several central bins: The new dataset has lower population shares in bin 3 (10.3555% vs. 13.4236%), bin 5 (14.2195% vs. 15.9381%), and bin 6 (8.9645% vs. 11.6441%), which account for several of the larger PSI contributions.

• Early-bin differences are minimal: Bins 0 and 1 show only small changes in population share, with PSI contributions of 0.0007 and 0.0006 respectively, and bin 8 is similarly small at 0.0002.

The PSI results show a modest overall distribution shift between the initial and new datasets, with total PSI equal to 0.0367. Differences are distributed across several bins, with the most material changes occurring in bins 3, 4, 6, and 9 rather than being isolated to a single region of the distribution. The observed pattern reflects offsetting increases and decreases across mid-range and upper bins, while the lowest bins remain largely unchanged.

Tables

Population Stability Index for train_dataset_final and test_dataset_final Datasets

Bin	Count Initial	Percent Initial (%)	Count New	Percent New (%)	PSI
0	118	4.5648	26	4.0185	0.0007
1	209	8.0851	48	7.4189	0.0006
2	325	12.5725	91	14.0649	0.0017
3	347	13.4236	67	10.3555	0.0080
4	313	12.1083	99	15.3014	0.0075
5	412	15.9381	92	14.2195	0.0020
6	301	11.6441	58	8.9645	0.0070
7	279	10.7930	78	12.0556	0.0014
8	109	4.2166	29	4.4822	0.0002
9	172	6.6538	59	9.1190	0.0078
Total	2585	100.0000	647	100.0000	0.0367

Figures

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

The ConfusionMatrix test evaluates classification performance by comparing predicted labels against observed labels and summarizing outcomes as true positives, true negatives, false positives, and false negatives. The heatmap shows four outcome counts for the binary classification task: 201 true negatives, 199 true positives, 133 false positives, and 114 false negatives. The diagonal cells represent correct classifications, while the off-diagonal cells represent misclassifications across the two classes.

Key insights:

• Correct classifications exceed errors: The matrix contains 201 true negatives and 199 true positives, compared with 133 false positives and 114 false negatives. This indicates that correct predictions are more frequent than misclassifications in both classes combined.

• Negative class is classified slightly better: True negatives total 201 versus 133 false positives for true class 0, while true positives total 199 versus 114 false negatives for true class 1. The negative class therefore has a slightly higher count of correct classifications than the positive class.

• False positives are more common than false negatives: The model produces 133 false positives compared with 114 false negatives. Misclassification is therefore somewhat more concentrated in predicting class 1 when the true label is 0.

• Class outcome counts are relatively balanced: The observed class 0 outcomes sum to 334 (201 TN + 133 FP) and the observed class 1 outcomes sum to 313 (199 TP + 114 FN). Predicted class 0 outcomes sum to 315 (201 TN + 114 FN) and predicted class 1 outcomes sum to 332 (199 TP + 133 FP), showing similar volumes across classes.

The confusion matrix shows that the model produces more correct than incorrect classifications overall, with 400 observations on the diagonal and 247 off the diagonal. Performance is relatively similar across the two classes, with slightly stronger correct classification for class 0 and a somewhat higher number of false positives than false negatives. The class totals in both the observed and predicted labels are broadly balanced, indicating that the result is not dominated by a single class outcome.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

The Classifier Performance test evaluates classification model performance using precision, recall, F1-score, accuracy, and ROC AUC. The reported results are for a binary classifier with class-level metrics shown for classes 0 and 1, together with weighted and macro averages. Class-specific precision ranges from 0.6357 to 0.6363, recall ranges from 0.6209 to 0.6508, and F1-score ranges from 0.6285 to 0.6432. Overall performance is summarized by accuracy of 0.636 and ROC AUC of 0.6884.

Key insights:

• Balanced class precision: Precision is nearly identical across the two classes, at 0.6363 for class 0 and 0.6357 for class 1, indicating similar positive predictive performance by class.
• Recall differs modestly by class: Recall is 0.6209 for class 0 and 0.6508 for class 1, a gap of 0.0299, showing somewhat stronger class identification for class 1 than for class 0.
• F1-scores are closely aligned: F1-scores are 0.6285 for class 0 and 0.6432 for class 1, with weighted and macro average F1 both near 0.636, indicating limited dispersion in class-level combined precision-recall performance.
• Aggregate metrics are internally consistent: Accuracy is 0.636, while weighted average precision, recall, and F1 are 0.6360, 0.6360, and 0.6359 respectively, showing close alignment between overall classification rate and averaged class performance.
• ROC AUC exceeds accuracy level: ROC AUC is 0.6884, which is higher than the reported accuracy of 0.636, indicating stronger ranking performance than the single-threshold classification metrics alone.

The in-sample results show broadly even performance across the two classes, with nearly identical precision and only modest separation in recall and F1-score. Aggregate metrics are tightly clustered around 0.636, indicating a consistent overall performance profile across summary measures. ROC AUC of 0.6884 is higher than the threshold-based metrics, showing that the model’s discrimination measured across thresholds is stronger than its reported point classification performance.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6363	0.6209	0.6285
1	0.6357	0.6508	0.6432
Weighted Average	0.6360	0.6360	0.6359
Macro Average	0.6360	0.6359	0.6358

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6360
ROC AUC	0.6884

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

The Classifier Performance test evaluates out-of-sample classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The reported results present class-level precision, recall, and F1 for classes 0 and 1, together with macro and weighted averages across classes. Class 0 shows precision 0.6381, recall 0.6018, and F1 0.6194, while class 1 shows precision 0.5994, recall 0.6358, and F1 0.6171. Aggregate metrics report weighted-average F1 of 0.6183, accuracy of 0.6182, and ROC AUC of 0.6664.

Key insights:

• Balanced aggregate performance: Macro-average and weighted-average metrics are nearly identical, with precision at 0.6187 vs. 0.6194, recall at 0.6188 vs. 0.6182, and F1 at 0.6182 vs. 0.6183. This indicates limited divergence between class-averaged and support-weighted performance in the reported sample.

• Class-level tradeoff between precision and recall: Class 0 has higher precision than class 1 (0.6381 vs. 0.5994), while class 1 has higher recall than class 0 (0.6358 vs. 0.6018). The model therefore captures class 1 cases somewhat more completely, while predictions for class 0 are somewhat more precise.

• Similar F1 across both classes: F1-scores are closely aligned at 0.6194 for class 0 and 0.6171 for class 1. This shows that overall harmonic-mean performance is comparable across the two classes despite the precision-recall tradeoff.

• Discrimination exceeds accuracy level: ROC AUC is 0.6664, compared with accuracy of 0.6182. The reported ranking-based discrimination measure is therefore higher than the threshold-based classification accuracy.

The out-of-sample results show closely aligned performance across the two classes, with class-specific differences appearing mainly in the balance between precision and recall rather than in overall F1. Aggregate macro and weighted averages are nearly the same, indicating consistent summarized performance across averaging methods in the reported sample. Overall, the model records accuracy and F1 near 0.62, with ROC AUC of 0.6664 indicating somewhat stronger discrimination than the point classification metrics alone.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6381	0.6018	0.6194
1	0.5994	0.6358	0.6171
Weighted Average	0.6194	0.6182	0.6183
Macro Average	0.6187	0.6188	0.6182

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6182
ROC AUC	0.6664

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

The PrecisionRecallCurve test evaluates the trade-off between precision and recall across classification thresholds for a binary model. The plotted curve shows precision on the vertical axis and recall on the horizontal axis, with the trace beginning at very high precision at near-zero recall and then moving through a generally downward-sloping path as recall increases. Across most of the curve, precision remains between approximately 0.50 and 0.75, with localized fluctuations at low recall and a more gradual decline through the mid- to high-recall range.

Key insights:

• Precision declines as recall rises: The curve shows a clear inverse relationship between the two metrics, with precision moving from the upper range near low recall toward approximately 0.49 by the highest recall levels shown.
• Strongest precision at very low recall: The left edge of the curve reaches precision near 1.0 at near-zero recall, indicating the highest precision is concentrated at the most restrictive thresholds.
• Mid-range performance is moderate and stable: Through roughly the middle of the recall range, precision stays close to about 0.60 to 0.64, with only modest oscillation relative to the sharper variation observed at very low recall.
• High recall corresponds to lower precision: At recall above approximately 0.8, the curve remains in the mid-0.50 range before declining further toward the rightmost end of the plot.

The result shows a typical precision-recall trade-off, with higher precision concentrated at low recall and progressively lower precision as recall expands. Most threshold positions appear to produce moderate precision levels rather than extreme values, and the curve’s shape indicates that performance is comparatively steadier in the middle of the recall range than at the leftmost edge. Overall, the plot reflects a gradual deterioration in precision as the model captures a larger share of positive cases.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

The ROCCurve test evaluates classification performance by plotting the receiver operating characteristic curve and calculating the area under the curve (AUC) to measure class discrimination across thresholds. The result shown is for log_reg_model_v1 on test_dataset_final, with a single ROC curve plotted against the random-classification reference line. The figure reports an AUC of 0.67, and the curve remains above the diagonal benchmark across most of the false positive rate range.

Key insights:

• AUC indicates positive discrimination: The reported AUC is 0.67, which is above the random baseline of 0.50 shown in the figure. This indicates the model distinguishes between the two classes better than random classification on the test dataset.
• ROC curve stays above baseline: The ROC curve lies above the diagonal reference line for most of the plotted range, showing higher true positive rates than the random benchmark at comparable false positive rates.
• Performance is moderate rather than strong: The curve shows gradual improvement in true positive rate as false positive rate increases, rather than a sharp rise near the y-axis. This shape is consistent with moderate separation between classes in the test results.

Overall, the ROC result shows that log_reg_model_v1 achieves measurable discriminatory ability on test_dataset_final, with an AUC of 0.67 and a curve that consistently outperforms the random baseline. The observed curve shape indicates classification performance that is above chance but not highly concentrated at low false positive rates.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

The TrainingTestDegradation test evaluates whether model performance degradation between the training and test datasets remains within a predefined threshold across key classification metrics. The results table reports train and test scores for precision, recall, and F1-score separately for classes 0 and 1, along with the calculated degradation percentage and pass/fail outcome. Observed degradation values range from -0.2836% to 5.7080%, and all evaluated metrics are marked as Pass against the configured threshold.

Key insights:

• All metrics passed threshold: Every evaluated train-to-test comparison for both classes and all three metrics is marked Pass. No degradation value exceeds the 10% limit described for this test.
• Largest degradation is class 1 precision: Class 1 precision declines from 0.6357 on the training dataset to 0.5994 on the test dataset, corresponding to the highest reported degradation at 5.7080%.
• Class 0 precision improved slightly: Class 0 precision increases from 0.6363 in training to 0.6381 in testing, producing a negative degradation value of -0.2836%.
• Recall degradation remains limited across classes: Recall degradation is 3.0775% for class 0 and 2.3084% for class 1. Test recall values remain close to training values in both cases.
• F1-score reductions are moderate: F1-score decreases from 0.6285 to 0.6194 for class 0 and from 0.6432 to 0.6171 for class 1, corresponding to degradations of 1.4462% and 4.0582%, respectively.

Overall, the results show limited performance change between the training and test datasets across all evaluated classification metrics. The largest observed degradation occurs in class 1 precision, while class 0 precision shows a slight improvement on the test dataset. Taken together, the reported values indicate that train-to-test performance differences remained within the defined acceptance threshold for both classes and all assessed metrics.

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
0	Precision	0.6363	0.6381	-0.2836	Pass
0	Recall	0.6209	0.6018	3.0775	Pass
0	F1-Score	0.6285	0.6194	1.4462	Pass
1	Precision	0.6357	0.5994	5.7080	Pass
1	Recall	0.6508	0.6358	2.3084	Pass
1	F1-Score	0.6432	0.6171	4.0582	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

The MinimumAccuracy test evaluates whether the model’s prediction accuracy meets or exceeds a specified minimum threshold. The result table reports an accuracy score of 0.6182 against a threshold of 0.7, along with the associated pass/fail outcome. This test output presents the direct comparison between the observed classification accuracy and the configured minimum benchmark.

Key insights:

• Accuracy is below threshold: The observed accuracy score is 0.6182, which is lower than the configured threshold of 0.7.
• Test result is a failure: The test outcome is recorded as Fail, reflecting that the measured accuracy did not meet the minimum required level.
• Gap to threshold is measurable: The observed score is 0.0818 below the threshold, quantifying the difference between current performance and the benchmark used in the test.

The result indicates that the model’s observed classification accuracy on the evaluated dataset did not satisfy the minimum threshold defined by the test. Both the reported score of 0.6182 and the recorded Fail outcome are aligned with the threshold comparison. Collectively, the result shows that overall prediction correctness, as measured by accuracy in this test, remained below the specified benchmark.

Tables

Score	Threshold	Pass/Fail
0.6182	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

The MinimumF1Score test evaluates whether the model’s F1 score on the validation set meets a predefined minimum threshold. The result table reports a validation F1 score of 0.6171 alongside a threshold of 0.5, with the test outcome recorded as Pass. These values provide the direct comparison used to determine whether the model satisfies the minimum performance criterion defined for this test.

Key insights:

• F1 score exceeds threshold: The validation F1 score is 0.6171 versus a minimum threshold of 0.5, placing the observed score 0.1171 above the defined cutoff.
• Test result is passing: The recorded Pass/Fail outcome is Pass, indicating the model met the minimum F1 score criterion under this validation assessment.
• Threshold comparison is straightforward: The result is based on a single quantitative comparison between the observed F1 score and the predefined threshold, with the score remaining above the cutoff.

The test result shows that the model achieved an F1 score of 0.6171 on the validation set and met the minimum threshold of 0.5. The margin above the threshold is 0.1171, and the reported outcome is Pass. Collectively, these results indicate that the model satisfied the specific minimum F1 score condition evaluated by this test.

Tables

Score	Threshold	Pass/Fail
0.6171	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

The Minimum ROC AUC Score test evaluates whether the model’s ROC AUC performance on the validation dataset meets or exceeds a predefined minimum threshold. The reported result includes the observed ROC AUC score, the threshold used for evaluation, and the resulting pass/fail outcome. In this test run, the model achieved a ROC AUC score of 0.6664 against a threshold of 0.5, and the test result is recorded as Pass.

Key insights:

• Score exceeds threshold: The observed ROC AUC score of 0.6664 is above the minimum threshold of 0.5 used in the test.
• Test result is pass: The test outcome is explicitly recorded as Pass based on the comparison between the observed score and the threshold.
• Positive margin above minimum: The reported score exceeds the threshold by 0.1664, indicating performance above the minimum criterion defined for this test.

The result shows that the model satisfied the minimum ROC AUC requirement on the validation dataset under the configured test setting. The recorded pass outcome is directly supported by the observed score of 0.6664 relative to the threshold of 0.5. This test therefore documents that the model met the specified minimum discrimination criterion in this evaluation.

Tables

Score	Threshold	Pass/Fail
0.6664	0.5	Pass

▶ Test Result: Permutation Feature Importance (validmind.model_validation.sklearn.PermutationFeatureImportance)

Permutation Feature Importance

The Permutation Feature Importance test evaluates the relative contribution of each input feature by measuring the change in model performance after randomly permuting feature values. The result is presented as a horizontal bar chart of permutation importance scores across the model inputs. The chart shows the largest importance values for Geography_Germany and IsActiveMember, followed by Gender_Male, with smaller positive values for CreditScore, Tenure, and Balance. EstimatedSalary and HasCrCard are effectively at zero, while NumOfProducts and Geography_Spain are slightly negative.

Key insights:

• Geography_Germany is the dominant feature: Geography_Germany has the highest permutation importance in the chart, at slightly above 0.04, making it the single most influential feature in this test result.

• IsActiveMember is also highly influential: IsActiveMember shows the second-largest importance, just below 0.04, with a magnitude close to Geography_Germany and notably above all remaining features.

• Importance drops materially after the top three: After Geography_Germany, IsActiveMember, and Gender_Male (approximately 0.02), the remaining positive importances are much smaller, with CreditScore, Tenure, and Balance clustered around low single-digit hundredths.

• Several features contribute little or not at all: EstimatedSalary and HasCrCard are approximately zero in permutation importance, indicating little measurable change in model performance when these features are permuted in this test.

• Two features show negative importance: NumOfProducts and Geography_Spain have slightly negative permutation importance values, meaning performance did not decline after permutation and was marginally higher in the plotted result.

The permutation importance profile indicates that model performance is concentrated in a small subset of features, led by Geography_Germany and IsActiveMember, with Gender_Male providing a secondary contribution. Most other variables have limited incremental impact, and a small number show near-zero or slightly negative importance. Overall, the result reflects a feature importance structure with clear concentration rather than broadly distributed influence across inputs.

Figures

▶ Test Result: SHAP Global Importance (validmind.model_validation.sklearn.SHAPGlobalImportance)

SHAP Global Importance

The SHAPGlobalImportance test evaluates global feature importance by measuring the average magnitude and direction of feature contributions to model output using SHAP values. The results are presented through a normalized feature importance bar chart and a SHAP summary plot. The importance chart ranks features by relative contribution, while the summary plot shows the distribution of SHAP values for each feature together with feature-value coloring to indicate how low and high values align with positive or negative model impact.

Key insights:

• Importance is concentrated in three features: IsActiveMember, Geography_Germany, and Gender_Male are the three most important features in the normalized importance plot, with IsActiveMember at 100%, Geography_Germany just below it, and Gender_Male materially lower but still clearly above the remaining variables.

• Secondary importance drops sharply: After the top three features, normalized importance declines notably to Balance at roughly one-third of the top feature, followed by NumOfProducts and then smaller contributions from CreditScore, Tenure, HasCrCard, Geography_Spain, and EstimatedSalary.

• IsActiveMember shows strong directional separation: In the summary plot, low values for IsActiveMember are associated with negative SHAP values around -0.4, while high values are associated with positive SHAP values around +0.4, indicating a wide and clearly separated contribution range.

• Geography_Germany has large positive and negative effects: Geography_Germany displays one of the widest SHAP spreads, with low values concentrated around negative SHAP values and high values extending to strongly positive SHAP values near +0.6.

• Balance contribution is mostly positive at higher values: For Balance, higher feature values are concentrated on the positive side of the SHAP axis, with many observations between approximately 0.0 and +0.25 and an isolated point further to the right, while lower values cluster closer to zero or slightly negative values.

• EstimatedSalary has minimal influence: EstimatedSalary appears at the bottom of the importance ranking and its SHAP values are tightly concentrated around zero, indicating very limited contribution to model output relative to the other features shown.

The SHAP results indicate that model behavior is driven primarily by a small set of features, led by IsActiveMember and Geography_Germany, with Gender_Male also contributing materially. Several lower-ranked features contribute over narrower SHAP ranges, and EstimatedSalary contributes negligibly. The summary plot also shows that the direction of impact differs by feature, with some variables exhibiting clear separation between low and high feature values and others remaining concentrated near zero.

Figures

▶ Test Result: Weakspots Diagnosis (validmind.model_validation.sklearn.WeakspotsDiagnosis)

❌ Weakspots Diagnosis

The WeakspotsDiagnosis test evaluates model performance across slices of the feature space by comparing accuracy, precision, recall, and F1 between the training and test datasets within each bin. The reported results cover multiple numerical and binary features, including CreditScore, Tenure, Balance, NumOfProducts, EstimatedSalary, Gender_Male, Geography_Germany, Geography_Spain, HasCrCard, and IsActiveMember. For each feature, the tables and plots show per-slice metric values alongside fixed thresholds, allowing direct comparison of test performance levels and train-test differences by segment.

Key insights:

• NumOfProducts shows the sharpest deterioration: In the test dataset, performance declines materially as NumOfProducts increases. Accuracy drops from 0.6480 for (0.997, 1.3] to 0.1667 for (3.7, 4.0], recall falls from 0.7078 to 0.1667, and F1 falls from 0.7014 to 0.2857. The weakest segment also has only 6 test records.

• Balance performance is uneven across slices: Test F1 values for Balance range from 0.3636 in (47677.512, 71516.268] to 0.7273 in (95355.024, 119193.78] among non-empty test bins. Test recall is particularly low in the lowest balance slice (-238.388, 23838.756] at 0.3289, while mid-to-upper balance ranges generally show higher recall, including 0.8000 in (95355.024, 119193.78] and 0.7568 in (143032.536, 166871.292].

• Binary geography variables split performance by segment: For Geography_Germany, the test dataset shows a large difference between the two slices: when the indicator is in (-0.001, 0.1], recall is 0.3966 and F1 is 0.4522, versus recall 0.9552 and F1 0.7734 in (0.9, 1.0]. Geography_Spain shows the opposite directional pattern in recall, with 0.7012 for (-0.001, 0.1] and 0.3710 for (0.9, 1.0], while F1 declines from 0.6447 to 0.4646.

• IsActiveMember segments differ mainly in recall: In the test dataset, (-0.001, 0.1] has recall 0.7789 and F1 0.6813, while (0.9, 1.0] has recall 0.3860 and F1 0.4632. Accuracy moves in the opposite direction, from 0.5892 to 0.6531, indicating materially different error balance across the two slices.

• CreditScore remains broadly stable with localized weaker bins: Across the more populated CreditScore test bins, accuracy ranges from 0.5769 to 0.6750, precision from 0.5000 to 0.7308, recall from 0.5000 to 0.7692, and F1 from 0.5000 to 0.6667. The weakest populated test bin is (400.0, 450.0], with accuracy 0.4286 and F1 0.5000, based on 14 records.

• EstimatedSalary contains both stronger and weaker regions: Test accuracy ranges from 0.4464 in (20001.354, 39991.128] to 0.7612 in (139939.998, 159929.772]. Test F1 ranges from 0.4746 to 0.7143, and recall ranges from 0.5128 to 0.7727. This indicates noticeable variation in performance across salary bands rather than a monotonic pattern.

• Several extreme results occur in very small slices: Multiple bins have very low record counts and perfect or zero metrics, including Balance (23838.756, 47677.512] with 2 test records and all metrics equal to 1.0, CreditScore (349.5, 400.0] with 2 test records and all metrics equal to 1.0, and Balance (214548.804, 238387.56] with 0 test records and all metrics equal to 0.0. These slices are present in the reported output but are sparsely populated.

Overall, the weak spot analysis shows that model performance varies materially across feature-space slices rather than remaining uniform across all segments. The largest concentration of weaker test results appears in specific Balance ranges, higher NumOfProducts categories, and certain binary-segment splits such as Geography_Germany, Geography_Spain, and IsActiveMember, where recall and F1 differ substantially by slice. By contrast, CreditScore and several Tenure and EstimatedSalary bands display more moderate variation, though still with localized weaker bins and some train-test differences.

Tables

Slice	Number of Records	Feature	Accuracy	Precision	Recall	F1	Dataset
(-238.388, 23838.756]	189	Balance	0.6349	0.5814	0.3289	0.4202	test_dataset_final
(23838.756, 47677.512]	2	Balance	1.0000	1.0000	1.0000	1.0000	test_dataset_final
(47677.512, 71516.268]	15	Balance	0.5333	0.3333	0.4000	0.3636	test_dataset_final
(71516.268, 95355.024]	56	Balance	0.6607	0.7000	0.6774	0.6885	test_dataset_final
(95355.024, 119193.78]	135	Balance	0.6667	0.6667	0.8000	0.7273	test_dataset_final
(119193.78, 143032.536]	140	Balance	0.6000	0.5714	0.7536	0.6500	test_dataset_final
(143032.536, 166871.292]	71	Balance	0.5634	0.5600	0.7568	0.6437	test_dataset_final
(166871.292, 190710.048]	31	Balance	0.4839	0.4706	0.5333	0.5000	test_dataset_final
(190710.048, 214548.804]	7	Balance	0.5714	0.5000	0.6667	0.5714	test_dataset_final
(214548.804, 238387.56]	0	Balance	0.0000	0.0000	0.0000	0.0000	test_dataset_final
(-238.388, 23838.756]	855	Balance	0.6433	0.5495	0.3415	0.4213	train_dataset_final
(23838.756, 47677.512]	18	Balance	0.4444	1.0000	0.2857	0.4444	train_dataset_final
(47677.512, 71516.268]	77	Balance	0.5065	0.5938	0.4318	0.5000	train_dataset_final
(71516.268, 95355.024]	230	Balance	0.5652	0.5583	0.5877	0.5726	train_dataset_final
(95355.024, 119193.78]	503	Balance	0.7078	0.7333	0.8214	0.7749	train_dataset_final
(119193.78, 143032.536]	542	Balance	0.6642	0.6639	0.8007	0.7259	train_dataset_final
(143032.536, 166871.292]	252	Balance	0.5635	0.5608	0.7970	0.6584	train_dataset_final
(166871.292, 190710.048]	84	Balance	0.5595	0.5556	0.7955	0.6542	train_dataset_final
(190710.048, 214548.804]	21	Balance	0.4286	0.6923	0.5294	0.6000	train_dataset_final
(214548.804, 238387.56]	3	Balance	1.0000	1.0000	1.0000	1.0000	train_dataset_final
(349.5, 400.0]	2	CreditScore	1.0000	1.0000	1.0000	1.0000	test_dataset_final
(400.0, 450.0]	14	CreditScore	0.4286	0.5000	0.5000	0.5000	test_dataset_final
(450.0, 500.0]	26	CreditScore	0.5769	0.5556	0.7692	0.6452	test_dataset_final
(500.0, 550.0]	70	CreditScore	0.5857	0.6250	0.6410	0.6329	test_dataset_final
(550.0, 600.0]	89	CreditScore	0.6067	0.5778	0.6190	0.5977	test_dataset_final
(600.0, 650.0]	122	CreditScore	0.6557	0.6364	0.7000	0.6667	test_dataset_final
(650.0, 700.0]	130	CreditScore	0.6154	0.5075	0.6667	0.5763	test_dataset_final
(700.0, 750.0]	87	CreditScore	0.6092	0.6136	0.6136	0.6136	test_dataset_final
(750.0, 800.0]	67	CreditScore	0.6269	0.7308	0.5135	0.6032	test_dataset_final
(800.0, 850.0]	40	CreditScore	0.6750	0.6250	0.5882	0.6061	test_dataset_final
(349.5, 400.0]	12	CreditScore	0.5833	1.0000	0.5833	0.7368	train_dataset_final
(400.0, 450.0]	48	CreditScore	0.6042	0.6296	0.6538	0.6415	train_dataset_final
(450.0, 500.0]	114	CreditScore	0.6316	0.6377	0.7213	0.6769	train_dataset_final
(500.0, 550.0]	270	CreditScore	0.6889	0.6906	0.7007	0.6957	train_dataset_final
(550.0, 600.0]	378	CreditScore	0.6455	0.6456	0.6856	0.6650	train_dataset_final
(600.0, 650.0]	481	CreditScore	0.5884	0.5772	0.6017	0.5892	train_dataset_final
(650.0, 700.0]	502	CreditScore	0.6175	0.6023	0.6463	0.6235	train_dataset_final
(700.0, 750.0]	390	CreditScore	0.6256	0.6373	0.6181	0.6276	train_dataset_final
(750.0, 800.0]	234	CreditScore	0.7051	0.6694	0.7364	0.7013	train_dataset_final
(800.0, 850.0]	156	CreditScore	0.6667	0.7419	0.5610	0.6389	train_dataset_final
(-188.318, 20001.354]	62	EstimatedSalary	0.7258	0.6333	0.7600	0.6909	test_dataset_final
(20001.354, 39991.128]	56	EstimatedSalary	0.4464	0.4000	0.5833	0.4746	test_dataset_final
(39991.128, 59980.902]	59	EstimatedSalary	0.5932	0.6471	0.6471	0.6471	test_dataset_final
(59980.902, 79970.676]	69	EstimatedSalary	0.7101	0.7353	0.6944	0.7143	test_dataset_final
(79970.676, 99960.45]	52	EstimatedSalary	0.5577	0.6552	0.5938	0.6230	test_dataset_final
(99960.45, 119950.224]	59	EstimatedSalary	0.6441	0.5862	0.6538	0.6182	test_dataset_final
(119950.224, 139939.998]	71	EstimatedSalary	0.6338	0.7059	0.6000	0.6486	test_dataset_final
(139939.998, 159929.772]	67	EstimatedSalary	0.7612	0.6071	0.7727	0.6800	test_dataset_final
(159929.772, 179919.546]	81	EstimatedSalary	0.5309	0.5128	0.5128	0.5128	test_dataset_final
(179919.546, 199909.32]	71	EstimatedSalary	0.5634	0.5500	0.6286	0.5867	test_dataset_final
(-188.318, 20001.354]	268	EstimatedSalary	0.6604	0.6759	0.6901	0.6829	train_dataset_final
(20001.354, 39991.128]	255	EstimatedSalary	0.6784	0.6977	0.6767	0.6870	train_dataset_final
(39991.128, 59980.902]	255	EstimatedSalary	0.6314	0.5865	0.6667	0.6240	train_dataset_final
(59980.902, 79970.676]	271	EstimatedSalary	0.6421	0.6615	0.6187	0.6394	train_dataset_final
(79970.676, 99960.45]	263	EstimatedSalary	0.6008	0.5556	0.6612	0.6038	train_dataset_final
(99960.45, 119950.224]	250	EstimatedSalary	0.6040	0.6529	0.5809	0.6148	train_dataset_final
(119950.224, 139939.998]	257	EstimatedSalary	0.6770	0.6560	0.6721	0.6640	train_dataset_final
(139939.998, 159929.772]	262	EstimatedSalary	0.6069	0.6045	0.6183	0.6113	train_dataset_final
(159929.772, 179919.546]	266	EstimatedSalary	0.6015	0.6149	0.6500	0.6319	train_dataset_final
(179919.546, 199909.32]	238	EstimatedSalary	0.6597	0.6640	0.6803	0.6721	train_dataset_final
(-0.001, 0.1]	314	Gender_Male	0.6051	0.6109	0.7803	0.6853	test_dataset_final
(0.9, 1.0]	333	Gender_Male	0.6306	0.5766	0.4571	0.5100	test_dataset_final
(-0.001, 0.1]	1268	Gender_Male	0.6317	0.6471	0.7787	0.7068	train_dataset_final
(0.9, 1.0]	1317	Gender_Male	0.6401	0.6142	0.4914	0.5460	train_dataset_final
(-0.001, 0.1]	435	Geography_Germany	0.6046	0.5259	0.3966	0.4522	test_dataset_final
(0.9, 1.0]	212	Geography_Germany	0.6462	0.6497	0.9552	0.7734	test_dataset_final
(-0.001, 0.1]	1807	Geography_Germany	0.6231	0.5825	0.4598	0.5139	train_dataset_final
(0.9, 1.0]	778	Geography_Germany	0.6658	0.6816	0.9385	0.7896	train_dataset_final
(-0.001, 0.1]	503	Geography_Spain	0.6143	0.5966	0.7012	0.6447	test_dataset_final
(0.9, 1.0]	144	Geography_Spain	0.6319	0.6216	0.3710	0.4646	test_dataset_final
(-0.001, 0.1]	1979	Geography_Spain	0.6362	0.6408	0.6928	0.6657	train_dataset_final
(0.9, 1.0]	606	Geography_Spain	0.6353	0.6093	0.4888	0.5424	train_dataset_final
(-0.001, 0.1]	196	HasCrCard	0.5867	0.5490	0.6154	0.5803	test_dataset_final
(0.9, 1.0]	451	HasCrCard	0.6319	0.6217	0.6441	0.6327	test_dataset_final
(-0.001, 0.1]	792	HasCrCard	0.6212	0.6241	0.6427	0.6333	train_dataset_final
(0.9, 1.0]	1793	HasCrCard	0.6425	0.6409	0.6544	0.6476	train_dataset_final
(-0.001, 0.1]	353	IsActiveMember	0.5892	0.6055	0.7789	0.6813	test_dataset_final
(0.9, 1.0]	294	IsActiveMember	0.6531	0.5789	0.3860	0.4632	test_dataset_final
(-0.001, 0.1]	1389	IsActiveMember	0.6371	0.6534	0.8207	0.7276	train_dataset_final
(0.9, 1.0]	1196	IsActiveMember	0.6346	0.5757	0.3623	0.4447	train_dataset_final
(0.997, 1.3]	375	NumOfProducts	0.6480	0.6951	0.7078	0.7014	test_dataset_final
(1.9, 2.2]	227	NumOfProducts	0.5947	0.2727	0.4615	0.3429	test_dataset_final
(2.8, 3.1]	39	NumOfProducts	0.5385	0.9500	0.5278	0.6786	test_dataset_final
(3.7, 4.0]	6	NumOfProducts	0.1667	1.0000	0.1667	0.2857	test_dataset_final
(0.997, 1.3]	1486	NumOfProducts	0.6406	0.7081	0.6940	0.7010	train_dataset_final
(1.9, 2.2]	907	NumOfProducts	0.6560	0.3697	0.6027	0.4583	train_dataset_final
(2.8, 3.1]	154	NumOfProducts	0.5195	0.9605	0.5069	0.6636	train_dataset_final
(3.7, 4.0]	38	NumOfProducts	0.4474	1.0000	0.4474	0.6182	train_dataset_final
(-0.01, 1.0]	96	Tenure	0.6458	0.6029	0.8542	0.7069	test_dataset_final
(1.0, 2.0]	68	Tenure	0.5441	0.4722	0.5862	0.5231	test_dataset_final
(2.0, 3.0]	68	Tenure	0.5000	0.4865	0.5455	0.5143	test_dataset_final
(3.0, 4.0]	76	Tenure	0.6842	0.6944	0.6579	0.6757	test_dataset_final
(4.0, 5.0]	51	Tenure	0.5882	0.5714	0.6400	0.6038	test_dataset_final
(5.0, 6.0]	51	Tenure	0.6471	0.6000	0.5455	0.5714	test_dataset_final
(6.0, 7.0]	82	Tenure	0.6220	0.5758	0.5278	0.5507	test_dataset_final
(7.0, 8.0]	53	Tenure	0.5849	0.6071	0.6071	0.6071	test_dataset_final
(8.0, 9.0]	59	Tenure	0.6949	0.7586	0.6667	0.7097	test_dataset_final
(9.0, 10.0]	43	Tenure	0.6744	0.7059	0.5714	0.6316	test_dataset_final
(-0.01, 1.0]	363	Tenure	0.6033	0.6456	0.6520	0.6488	train_dataset_final
(1.0, 2.0]	281	Tenure	0.6014	0.5556	0.6589	0.6028	train_dataset_final
(2.0, 3.0]	272	Tenure	0.6066	0.6118	0.6596	0.6348	train_dataset_final
(3.0, 4.0]	269	Tenure	0.6283	0.6241	0.6241	0.6241	train_dataset_final
(4.0, 5.0]	249	Tenure	0.6667	0.6912	0.6963	0.6937	train_dataset_final
(5.0, 6.0]	278	Tenure	0.6475	0.6250	0.6439	0.6343	train_dataset_final
(6.0, 7.0]	234	Tenure	0.6538	0.5909	0.6436	0.6161	train_dataset_final
(7.0, 8.0]	258	Tenure	0.6822	0.7083	0.6439	0.6746	train_dataset_final
(8.0, 9.0]	253	Tenure	0.6482	0.6615	0.6565	0.6590	train_dataset_final
(9.0, 10.0]	128	Tenure	0.6484	0.6724	0.6000	0.6341	train_dataset_final

Figures

▶ Test Result: Overfit Diagnosis (validmind.model_validation.sklearn.OverfitDiagnosis)

Overfit Diagnosis

The Overfit Diagnosis test evaluates potential overfitting by comparing training and test AUC across feature-based slices and highlighting regions where the gap exceeds the 0.04 threshold. The reported results identify flagged slices across CreditScore, Tenure, Balance, NumOfProducts, IsActiveMember, and EstimatedSalary, with the largest positive gaps concentrated in selected CreditScore, Tenure, and EstimatedSalary intervals. The output also shows that some plotted features, including HasCrCard, Geography_Germany, Geography_Spain, and Gender_Male, remain below the threshold in all displayed slices.

Key insights:

• Largest gap in low CreditScore slice: The widest reported train-test separation appears for CreditScore in (400.0, 450.0], with Training AUC 0.7115, Test AUC 0.4583, and Gap 0.2532 across 48 training and 14 test records.
• Multiple CreditScore regions are flagged: Five CreditScore slices exceed the 0.04 threshold: (400.0, 450.0] (0.2532), (450.0, 500.0] (0.0981), (500.0, 550.0] (0.1962), (550.0, 600.0] (0.0539), and (750.0, 800.0] (0.1021).
• Tenure gaps peak at 2 to 3 years: The largest Tenure gap is in (2.0, 3.0], where Training AUC is 0.6768 and Test AUC is 0.4874, producing a gap of 0.1893. Additional flagged Tenure slices are (1.0, 2.0] at 0.0595, (4.0, 5.0] at 0.1086, and (7.0, 8.0] at 0.0731.
• EstimatedSalary shows several elevated gaps: Six EstimatedSalary slices exceed the threshold, led by (20001.354, 39991.128] with Training AUC 0.7469, Test AUC 0.5234, and Gap 0.2235. Other flagged intervals include (39991.128, 59980.902] at 0.0510, (79970.676, 99960.45] at 0.0950, (119950.224, 139939.998] at 0.0478, (159929.772, 179919.546] at 0.0741, and (179919.546, 199909.32] at 0.0977.
• Other flagged features are more limited: Balance has one reported flagged slice, (119193.78, 143032.536], with a gap of 0.0594. NumOfProducts has one flagged slice, (1.9, 2.2], with a gap of 0.0801, and IsActiveMember has one flagged slice, (-0.001, 0.1], with a gap of 0.0532.
• Several plotted binary features remain below threshold: In the displayed plots, HasCrCard, Geography_Germany, Geography_Spain, and Gender_Male do not cross the 0.04 cutoff in any shown slice.

The results show that overfit signals are concentrated in specific feature segments rather than uniformly across all features. The largest observed gaps occur in selected low-to-mid CreditScore bands, the Tenure interval (2.0, 3.0], and the lower EstimatedSalary interval (20001.354, 39991.128], with additional elevated gaps in several adjacent salary and score ranges. By contrast, other displayed binary features remain below the threshold, indicating a more localized pattern of train-test performance separation.

Tables

Overfit Diagnosis

Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
CreditScore	(400.0, 450.0]	48	14	0.7115	0.4583	0.2532
CreditScore	(450.0, 500.0]	114	26	0.6839	0.5858	0.0981
CreditScore	(500.0, 550.0]	270	70	0.7413	0.5451	0.1962
CreditScore	(550.0, 600.0]	378	89	0.6937	0.6398	0.0539
CreditScore	(750.0, 800.0]	234	67	0.7453	0.6432	0.1021
Tenure	(1.0, 2.0]	281	68	0.6395	0.5800	0.0595
Tenure	(2.0, 3.0]	272	68	0.6768	0.4874	0.1893
Tenure	(4.0, 5.0]	249	51	0.7378	0.6292	0.1086
Tenure	(7.0, 8.0]	258	53	0.7189	0.6457	0.0731
Balance	(119193.78, 143032.536]	542	140	0.7239	0.6644	0.0594
NumOfProducts	(1.9, 2.2]	907	227	0.6944	0.6143	0.0801
IsActiveMember	(-0.001, 0.1]	1389	353	0.6702	0.6170	0.0532
EstimatedSalary	(20001.354, 39991.128]	255	56	0.7469	0.5234	0.2235
EstimatedSalary	(39991.128, 59980.902]	255	59	0.6675	0.6165	0.0510
EstimatedSalary	(79970.676, 99960.45]	263	52	0.6762	0.5812	0.0950
EstimatedSalary	(119950.224, 139939.998]	257	71	0.7179	0.6702	0.0478
EstimatedSalary	(159929.772, 179919.546]	266	81	0.6584	0.5842	0.0741
EstimatedSalary	(179919.546, 199909.32]	238	71	0.7199	0.6222	0.0977

Figures

▶ Test Result: Robustness Diagnosis (validmind.model_validation.sklearn.RobustnessDiagnosis)

✅ Robustness Diagnosis

The Robustness Diagnosis test evaluates model resilience by measuring AUC decay after adding Gaussian noise to numeric input features at increasing perturbation sizes. Results are reported for both train_dataset_final and test_dataset_final from the baseline condition through perturbation sizes of 0.1 to 0.5 standard deviations. The table and plot show the corresponding AUC values, performance decay relative to baseline, and pass status at each perturbation level for both datasets.

Key insights:

• AUC declines as noise increases: AUC decreases from 0.6884 to 0.6680 on train_dataset_final and from 0.6664 to 0.6499 on test_dataset_final between baseline and perturbation size 0.5, indicating progressive performance reduction under higher noise levels.

• Train decay is larger overall: Performance decay reaches 0.0204 on train_dataset_final at perturbation size 0.5, compared with 0.0164 on test_dataset_final. Across all reported perturbation levels, the train dataset shows the larger cumulative decline from baseline.

• Test-set response is not strictly monotonic at low noise: On test_dataset_final, AUC moves from 0.6618 at perturbation size 0.1 to 0.6633 at 0.2 before declining again at higher perturbation levels. This contrasts with train_dataset_final, where AUC decreases at every successive perturbation level.

• All evaluated scenarios passed: Every baseline and perturbed evaluation is marked as passed in the results table for both datasets, including the highest perturbation size of 0.5.

The results show a consistent reduction in AUC as perturbation size increases, with the largest observed decay occurring at the highest noise level on both datasets. The train dataset exhibits a steeper overall decline than the test dataset, while the test dataset shows a small local fluctuation at lower perturbation levels before continuing downward. Across all evaluated perturbation sizes, all test instances remained within the passing threshold.

Tables

Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
Baseline (0.0)	train_dataset_final	2585	0.6884	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.6664	0.0000	True
0.1	train_dataset_final	2585	0.6865	0.0019	True
0.1	test_dataset_final	647	0.6618	0.0046	True
0.2	train_dataset_final	2585	0.6821	0.0063	True
0.2	test_dataset_final	647	0.6633	0.0031	True
0.3	train_dataset_final	2585	0.6777	0.0106	True
0.3	test_dataset_final	647	0.6605	0.0058	True
0.4	train_dataset_final	2585	0.6744	0.0139	True
0.4	test_dataset_final	647	0.6579	0.0085	True
0.5	train_dataset_final	2585	0.6680	0.0204	True
0.5	test_dataset_final	647	0.6499	0.0164	True

Figures

In summary

In this second notebook, you learned how to:

• Import a sample dataset
• Identify which tests you might want to run with ValidMind
• Initialize ValidMind datasets and model objects
• Run individual tests
• Utilize the output from tests you've run
• Log test results from sets of or individual tests as evidence to the ValidMind Platform
• Add supplementary individual test results to your documentation
• Assign predictions to your ValidMind model objects

Next steps

Integrate custom tests

Now that you're familiar with the basics of using the ValidMind Library to run and log tests to provide evidence for your documentation, let's learn how to incorporate your own custom tests into ValidMind: 3 — Integrate custom tests

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial