ValidMind for model development 2 — Start the model development process

Learn how to use ValidMind for your end-to-end model documentation process with our series of four introductory notebooks. In this second notebook, you'll run tests and investigate results, then add the results or evidence to your documentation.

You'll become familiar with the individual tests available in ValidMind, as well as how to run them and change parameters as necessary. Using ValidMind's repository of individual tests as building blocks helps you ensure that a model is being built appropriately.

For a full list of out-of-the-box tests, refer to our Test descriptions or try the interactive Test sandbox.

Learn by doing

Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Developer Fundamentals

Prerequisites

In order to log test results or evidence to your model documentation with this notebook, you'll need to first have:

• Registered a model within the ValidMind Platform with a predefined documentation template
• Installed the ValidMind Library in your local environment, allowing you to access all its features

Need help with the above steps?

Refer to the first notebook in this series: 1 — Set up the ValidMind Library

Setting up

Initialize the ValidMind Library

First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. In a browser, log in to ValidMind.

2. In the left sidebar, navigate to Inventory and select the model you registered for this "ValidMind for model development" series of notebooks.

3. Go to Getting Started and click Copy snippet to clipboard.

Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
)

Note: you may need to restart the kernel to use updated packages.

2026-01-10 02:02:06,518 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model development (ID: cmalgf3qi02ce199qm3rdkl46)
📁 Document Type: model_documentation

Import sample dataset

Then, let's import the public Bank Customer Churn Prediction dataset from Kaggle.

In our below example, note that:

• The target column, Exited has a value of 1 when a customer has churned and 0 otherwise.
• The ValidMind Library provides a wrapper to automatically load the dataset as a Pandas DataFrame object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns.

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()
raw_df.head()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

Identify qualitative tests

Next, let's say we want to do some data quality assessments by running a few individual tests.

Use the vm.tests.list_tests() function introduced by the first notebook in this series in combination with vm.tests.list_tags() and vm.tests.list_tasks() to find which prebuilt tests are relevant for data quality assessment:

• tasks represent the kind of modeling task associated with a test. Here we'll focus on classification tasks.
• tags are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the data_quality tag.

# Get the list of available task types
sorted(vm.tests.list_tasks())

['classification',
 'clustering',
 'data_validation',
 'feature_extraction',
 'monitoring',
 'nlp',
 'regression',
 'residual_analysis',
 'text_classification',
 'text_generation',
 'text_qa',
 'text_summarization',
 'time_series_forecasting',
 'visualization']

# Get the list of available tags
sorted(vm.tests.list_tags())

['AUC',
 'analysis',
 'anomaly_detection',
 'bias_and_fairness',
 'binary_classification',
 'calibration',
 'categorical_data',
 'classification',
 'classification_metrics',
 'clustering',
 'correlation',
 'credit_risk',
 'data_analysis',
 'data_distribution',
 'data_quality',
 'data_validation',
 'descriptive_statistics',
 'dimensionality_reduction',
 'distribution',
 'embeddings',
 'feature_importance',
 'feature_selection',
 'few_shot',
 'forecasting',
 'frequency_analysis',
 'kmeans',
 'linear_regression',
 'llm',
 'logistic_regression',
 'metadata',
 'model_comparison',
 'model_diagnosis',
 'model_explainability',
 'model_interpretation',
 'model_performance',
 'model_predictions',
 'model_selection',
 'model_training',
 'model_validation',
 'multiclass_classification',
 'nlp',
 'normality',
 'numerical_data',
 'outliers',
 'qualitative',
 'rag_performance',
 'ragas',
 'regression',
 'retrieval_performance',
 'scorecard',
 'seasonality',
 'senstivity_analysis',
 'sklearn',
 'stationarity',
 'statistical_test',
 'statistics',
 'statsmodels',
 'tabular_data',
 'text_data',
 'threshold_optimization',
 'time_series_data',
 'unit_root_test',
 'visualization',
 'zero_shot']

You can pass tags and tasks as parameters to the vm.tests.list_tests() function to filter the tests based on the tags and task types.

For example, to find tests related to tabular data quality for classification models, you can call list_tests() like this:

vm.tests.list_tests(task="classification", tags=["tabular_data", "data_quality"])

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.data_validation.ClassImbalance	Class Imbalance	Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model....	True	True	['dataset']	{'min_percent_threshold': {'type': 'int', 'default': 10}}	['tabular_data', 'binary_classification', 'multiclass_classification', 'data_quality']	['classification']
validmind.data_validation.DescriptiveStatistics	Descriptive Statistics	Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's...	False	True	['dataset']	{}	['tabular_data', 'time_series_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.Duplicates	Duplicates	Tests dataset for duplicate entries, ensuring model reliability via data quality verification....	False	True	['dataset']	{'min_threshold': {'type': '_empty', 'default': 1}}	['tabular_data', 'data_quality', 'text_data']	['classification', 'regression']
validmind.data_validation.HighCardinality	High Cardinality	Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting....	False	True	['dataset']	{'num_threshold': {'type': 'int', 'default': 100}, 'percent_threshold': {'type': 'float', 'default': 0.1}, 'threshold_type': {'type': 'str', 'default': 'percent'}}	['tabular_data', 'data_quality', 'categorical_data']	['classification', 'regression']
validmind.data_validation.HighPearsonCorrelation	High Pearson Correlation	Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity....	False	True	['dataset']	{'max_threshold': {'type': 'float', 'default': 0.3}, 'top_n_correlations': {'type': 'int', 'default': 10}, 'feature_columns': {'type': 'list', 'default': None}}	['tabular_data', 'data_quality', 'correlation']	['classification', 'regression']
validmind.data_validation.MissingValues	Missing Values	Evaluates dataset quality by ensuring missing value ratio across all features does not exceed a set threshold....	False	True	['dataset']	{'min_threshold': {'type': 'int', 'default': 1}}	['tabular_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.MissingValuesBarPlot	Missing Values Bar Plot	Assesses the percentage and distribution of missing values in the dataset via a bar plot, with emphasis on...	True	False	['dataset']	{'threshold': {'type': 'int', 'default': 80}, 'fig_height': {'type': 'int', 'default': 600}}	['tabular_data', 'data_quality', 'visualization']	['classification', 'regression']
validmind.data_validation.Skewness	Skewness	Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data...	False	True	['dataset']	{'max_threshold': {'type': '_empty', 'default': 1}}	['data_quality', 'tabular_data']	['classification', 'regression']
validmind.plots.BoxPlot	Box Plot	Generates customizable box plots for numerical features in a dataset with optional grouping using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'group_by': {'type': 'Optional', 'default': None}, 'width': {'type': 'int', 'default': 1800}, 'height': {'type': 'int', 'default': 1200}, 'colors': {'type': 'Optional', 'default': None}, 'show_outliers': {'type': 'bool', 'default': True}, 'title_prefix': {'type': 'str', 'default': 'Box Plot of'}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.plots.HistogramPlot	Histogram Plot	Generates customizable histogram plots for numerical features in a dataset using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'bins': {'type': 'Union', 'default': 30}, 'color': {'type': 'str', 'default': 'steelblue'}, 'opacity': {'type': 'float', 'default': 0.7}, 'show_kde': {'type': 'bool', 'default': True}, 'normalize': {'type': 'bool', 'default': False}, 'log_scale': {'type': 'bool', 'default': False}, 'title_prefix': {'type': 'str', 'default': 'Histogram of'}, 'width': {'type': 'int', 'default': 1200}, 'height': {'type': 'int', 'default': 800}, 'n_cols': {'type': 'int', 'default': 2}, 'vertical_spacing': {'type': 'float', 'default': 0.15}, 'horizontal_spacing': {'type': 'float', 'default': 0.1}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.stats.DescriptiveStats	Descriptive Stats	Provides comprehensive descriptive statistics for numerical features in a dataset....	False	True	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'include_advanced': {'type': 'bool', 'default': True}, 'confidence_level': {'type': 'float', 'default': 0.95}}	['tabular_data', 'statistics', 'data_quality']	['classification', 'regression', 'clustering']

Want to learn more about navigating ValidMind tests?

Refer to our notebook outlining the utilities available for viewing and understanding available ValidMind tests: Explore tests

Initialize the ValidMind datasets

With the individual tests we want to run identified, the next step is to connect your data with a ValidMind Dataset object. This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind, but you only need to do it once per dataset.

Initialize a ValidMind dataset object using the init_dataset function from the ValidMind (vm) module. For this example, we'll pass in the following arguments:

• dataset — The raw dataset that you want to provide as input to tests.
• input_id — A unique identifier that allows tracking what inputs are used when running each individual test.
• target_column — A required argument if tests require access to true values. This is the name of the target column in the dataset.

# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test
vm_raw_dataset = vm.init_dataset(
    dataset=raw_df,
    input_id="raw_dataset",
    target_column="Exited",
)

Running tests

Now that we know how to initialize a ValidMind dataset object, we're ready to run some tests!

You run individual tests by calling the run_test function provided by the validmind.tests module. For the examples below, we'll pass in the following arguments:

• test_id — The ID of the test to run, as seen in the ID column when you run list_tests.
• params — A dictionary of parameters for the test. These will override any default_params set in the test definition.

Run tabular data tests

The inputs expected by a test can also be found in the test definition — let's take validmind.data_validation.DescriptiveStatistics as an example.

Note that the output of the describe_test() function below shows that this test expects a dataset as input:

vm.tests.describe_test("validmind.data_validation.DescriptiveStatistics")

▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Descriptive Statistics

Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's dataset.

Purpose

The purpose of the Descriptive Statistics metric is to provide a comprehensive summary of both numerical and categorical data within a dataset. This involves statistics such as count, mean, standard deviation, minimum and maximum values for numerical data. For categorical data, it calculates the count, number of unique values, most common value and its frequency, and the proportion of the most frequent value relative to the total. The goal is to visualize the overall distribution of the variables in the dataset, aiding in understanding the model's behavior and predicting its performance.

Test Mechanism

The testing mechanism utilizes two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function pulls out several summary statistics, while value_counts() accounts for unique values. The resulting data is formatted into two distinct tables, one for numerical and another for categorical variable summaries. These tables provide a clear summary of the main characteristics of the variables, which can be instrumental in assessing the model's performance.

Signs of High Risk

• Skewed data or significant outliers can represent high risk. For numerical data, this may be reflected via a significant difference between the mean and median (50% percentile).
• For categorical data, a lack of diversity (low count of unique values), or overdominance of a single category (high frequency of the top value) can indicate high risk.

Strengths

• Provides a comprehensive summary of the dataset, shedding light on the distribution and characteristics of the variables under consideration.
• It is a versatile and robust method, applicable to both numerical and categorical data.
• Helps highlight crucial anomalies such as outliers, extreme skewness, or lack of diversity, which are vital in understanding model behavior during testing and validation.

Limitations

• While this metric offers a high-level overview of the data, it may fail to detect subtle correlations or complex patterns.
• Does not offer any insights on the relationship between variables.
• Alone, descriptive statistics cannot be used to infer properties about future unseen data.
• Should be used in conjunction with other statistical tests to provide a comprehensive understanding of the model's data.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DescriptiveStatistics", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Now, let's run a few tests to assess the quality of the dataset:

result = vm.tests.run_test(
    test_id="validmind.data_validation.DescriptiveStatistics",
    inputs={"dataset": vm_raw_dataset},
)

Descriptive Statistics

Descriptive Statistics is designed to provide a comprehensive summary of both numerical and categorical data within a dataset, offering key insights into the distribution, central tendency, variability, and diversity of the variables. The primary purpose of this test is to facilitate a clear understanding of the dataset’s structure and characteristics, which is essential for interpreting model behavior and anticipating performance.

The test operates by applying established statistical functions to the dataset. For numerical variables, it uses a summary statistics approach, calculating metrics such as count, mean, standard deviation, minimum, maximum, and several percentiles (including the 25th, 50th, 75th, 90th, and 95th). These metrics collectively describe the central tendency, spread, and range of the data. The mean provides an average value, while the standard deviation quantifies the typical deviation from the mean, indicating variability. Percentiles offer a view of the data’s distribution, highlighting where most values fall and identifying potential skewness or outliers. For categorical variables, the test counts the total number of entries, determines the number of unique categories, identifies the most frequent category (the mode), and calculates both the frequency and proportion of this top value. This approach reveals the diversity and dominance within categorical fields. The metrics for numerical data typically range from the minimum to the maximum observed values, while categorical metrics focus on counts and proportions, with the top value frequency percentage ranging from just above 0% to 100%. High proportions for a single category may indicate a lack of diversity, while large differences between mean and median in numerical data may suggest skewness or outliers.

The primary advantages of this test include its ability to deliver a thorough and accessible overview of the dataset, making it easier to detect patterns, anomalies, and potential data quality issues. By summarizing both numerical and categorical variables, the test supports a holistic understanding of the data landscape, which is particularly valuable during model development, validation, and monitoring. The inclusion of multiple percentiles and measures of spread allows for the identification of skewness, heavy tails, or clustering, which can impact model assumptions and performance. For categorical data, the test’s focus on unique values and dominant categories helps to quickly assess whether the data is sufficiently diverse or if certain categories are overrepresented, which could affect model fairness or generalizability. This versatility makes the test applicable across a wide range of scenarios, from initial data exploration to ongoing model risk management.

It should be noted that while this test provides a high-level summary of the data, it does not capture relationships or dependencies between variables, nor does it detect subtle or complex patterns that may influence model outcomes. The test is limited to univariate analysis, meaning it examines each variable in isolation. As a result, it cannot identify multicollinearity, interactions, or conditional distributions. Additionally, the test may not fully reveal the impact of rare but influential outliers, especially if they do not significantly affect summary statistics. For categorical variables, a high frequency of a single category or a low number of unique values may signal a lack of diversity, which could introduce bias or limit model robustness. For numerical variables, significant differences between the mean and median, or extreme values in the minimum and maximum, may indicate skewness or outliers, which could distort model training or evaluation. Therefore, while the test is a valuable starting point, it should be complemented by more detailed analyses to ensure a comprehensive understanding of the data.

This test shows the results in two tables: one summarizing numerical variables and the other summarizing categorical variables. The numerical table lists each variable alongside its count, mean, standard deviation, minimum, several percentiles (25th, 50th, 75th, 90th, 95th), and maximum. This format allows the reader to quickly assess the central tendency, spread, and range for each variable. For example, the “CreditScore” variable has a mean of 650.16, a standard deviation of 96.85, and values ranging from 350 to 850, with the 50th percentile (median) at 652, indicating a relatively symmetric distribution. The “Balance” variable shows a mean of 76,434.10 and a standard deviation of 62,612.25, with a minimum of 0 and a maximum of 250,898, suggesting a wide range and potential skewness, as the 25th percentile is 0 while the 75th is 128,045. The categorical table presents each variable with its total count, number of unique values, the most common value, its frequency, and the percentage this represents. For instance, “Geography” has three unique values, with “France” accounting for 50.12% of entries, while “Gender” has two unique values, with “Male” representing 54.95%. These tables provide a clear, at-a-glance summary of the dataset’s structure, highlighting both the diversity and concentration within each variable, as well as any notable patterns such as skewness, outliers, or dominance of specific categories.

The test results reveal the following key insights:

• Numerical variables exhibit varied distributions and ranges: Variables such as “CreditScore” and “Age” display relatively symmetric distributions, as indicated by close mean and median values, while “Balance” shows significant skewness with a large gap between the 25th percentile (0) and higher percentiles, and a high maximum value.
• Presence of potential outliers and skewness in certain variables: The “Balance” variable, with a minimum of 0 and a maximum of 250,898, and a mean substantially lower than the 75th and 90th percentiles, suggests a right-skewed distribution with a concentration of lower values and a long tail of higher balances.
• Categorical variables show moderate to high dominance of top categories: “Geography” is dominated by “France” at 50.12%, and “Gender” by “Male” at 54.95%, indicating that while there is some diversity, a single category accounts for over half of the observations in each case.
• Limited diversity in categorical variables: With only three unique values for “Geography” and two for “Gender,” the categorical variables are relatively limited in diversity, which may have implications for model generalizability and fairness.
• Stability in binary and low-cardinality variables: Variables such as “HasCrCard” and “IsActiveMember” are binary, with means close to 0.7 and 0.52, respectively, and standard deviations near 0.5, indicating balanced distributions without extreme dominance.

Based on these results, the dataset demonstrates a mix of symmetric and skewed distributions among numerical variables, with “CreditScore” and “Age” showing balanced central tendencies and moderate variability, while “Balance” stands out for its pronounced skewness and wide range. The presence of a substantial proportion of zero balances, as indicated by the 25th percentile, suggests a significant segment of the population with no account balance, while higher percentiles and the maximum highlight a smaller group with much larger balances. Categorical variables are characterized by moderate dominance of a single category, particularly in “Geography” and “Gender,” which may influence model behavior if these categories are associated with different outcomes. The limited number of unique values in categorical fields points to a relatively homogeneous dataset in these dimensions. Binary variables such as “HasCrCard” and “IsActiveMember” are well balanced, reducing the risk of bias from class imbalance. Overall, the descriptive statistics provide a clear and detailed view of the dataset’s structure, revealing both areas of stability and potential sources of skewness or concentration that could affect model performance and interpretation.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	8000.0	650.1596	96.8462	350.0	583.0	652.0	717.0	778.0	813.0	850.0
Age	8000.0	38.9489	10.4590	18.0	32.0	37.0	44.0	53.0	60.0	92.0
Tenure	8000.0	5.0339	2.8853	0.0	3.0	5.0	8.0	9.0	9.0	10.0
Balance	8000.0	76434.0965	62612.2513	0.0	0.0	97264.0	128045.0	149545.0	162488.0	250898.0
NumOfProducts	8000.0	1.5325	0.5805	1.0	1.0	1.0	2.0	2.0	2.0	4.0
HasCrCard	8000.0	0.7026	0.4571	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	8000.0	0.5199	0.4996	0.0	0.0	1.0	1.0	1.0	1.0	1.0
EstimatedSalary	8000.0	99790.1880	57520.5089	12.0	50857.0	99505.0	149216.0	179486.0	189997.0	199992.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	8000.0	3.0	France	4010.0	50.12
Gender	8000.0	2.0	Male	4396.0	54.95

result2 = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_raw_dataset},
    params={"min_percent_threshold": 30},
)

❌ Class Imbalance

Class Imbalance is designed to evaluate and quantify the distribution of target classes within a dataset used by a machine learning model, with the primary purpose of identifying whether any class is under-represented to a degree that could introduce bias into the model’s predictions. By systematically assessing the proportion of each class, the test aims to ensure that the dataset is sufficiently balanced to support robust and fair model training and evaluation.

The test operates by calculating the frequency of each class in the target column, expressing these frequencies as percentages of the total dataset. It then compares each class’s percentage to a predefined minimum threshold, which in this case is set at 30%. If any class falls below this threshold, it is flagged as failing the test, indicating a potential imbalance. The methodology involves straightforward counting of class occurrences, division by the total number of records, and conversion to a percentage. The resulting values range from 0% to 100%, where higher percentages indicate greater representation of a class. A class that meets or exceeds the threshold is considered adequately represented, while a class below the threshold is considered under-represented. The test outputs both a tabular summary and a visual plot, making it easy to interpret the distribution and identify any imbalances at a glance.

The primary advantages of this test include its ability to quickly and clearly identify under-represented classes that could impact model performance, especially in scenarios where class balance is critical for predictive accuracy and fairness. The test’s simplicity and speed make it suitable for routine use in data preprocessing pipelines. Its quantitative output, both in tabular and graphical form, provides immediate insight into the degree of imbalance, supporting transparent communication with stakeholders. The adjustable threshold parameter allows the test to be tailored to specific domain requirements, making it flexible for a wide range of applications. The visual plot enhances interpretability, enabling users to intuitively grasp the class proportions and spot potential issues without needing to parse raw numbers.

It should be noted that the test has several limitations. It may be less informative for datasets with a large number of classes, where some degree of imbalance is expected or unavoidable. The choice of threshold is subjective and can influence the test’s sensitivity; setting it too high may result in false positives for imbalance, while setting it too low may overlook meaningful disparities. The test does not account for the varying costs or consequences of misclassifying different classes, which can be significant in certain domains. Additionally, while the test highlights imbalances, it does not provide guidance or methods for addressing them. Its applicability is limited to classification tasks and does not extend to regression or clustering problems. Importantly, the test flags any class below the threshold as high risk, which should be interpreted in the context of the specific modeling objectives and domain requirements.

This test shows the results in both a tabular format and a bar plot. The table, titled "Exited Class Imbalance," lists each class in the target variable ("Exited"), the percentage of total rows that each class represents, and a pass/fail status based on the 30% minimum threshold. The first row corresponds to class 0, which comprises 79.80% of the dataset and passes the threshold. The second row corresponds to class 1, which comprises 20.20% of the dataset and fails the threshold. The accompanying bar plot visually represents these proportions, with the x-axis indicating the class (0 or 1) and the y-axis showing the percentage of the dataset each class occupies. The height of each bar directly corresponds to the class’s share of the data, making it easy to compare the relative sizes. The plot clearly shows a substantial difference between the two classes, with class 0 dominating the dataset and class 1 being significantly under-represented relative to the threshold. The scale of the y-axis ranges from 0 to 1 (or 0% to 100%), and the bars are colored for visual clarity. This dual presentation allows for both precise numerical interpretation and intuitive visual assessment of class distribution.

The test results reveal the following key insights:

• Majority Class Dominates Dataset: Class 0 constitutes 79.80% of the total records, indicating a strong dominance in the dataset.
• Minority Class Fails Threshold: Class 1 represents only 20.20% of the dataset, falling below the 30% minimum threshold and thus failing the test.
• Clear Visual Disparity in Class Distribution: The bar plot visually emphasizes the imbalance, with class 0’s bar being nearly four times the height of class 1’s bar.
• Binary Target Structure: The dataset contains only two classes, simplifying the interpretation but also highlighting the stark contrast in representation.
• Threshold Sensitivity Evident: The choice of a 30% threshold is critical, as class 1 would pass at a lower threshold but fails under the current setting, demonstrating the impact of parameter selection on test outcomes.

Based on these results, the dataset exhibits a pronounced class imbalance, with the majority class (class 0) substantially outnumbering the minority class (class 1). The minority class does not meet the minimum representation threshold of 30%, as specified by the test parameters, and is therefore flagged as under-represented. This imbalance is clearly reflected in both the tabular summary and the bar plot, which together provide a comprehensive view of the class distribution. The results suggest that the dataset’s current structure may influence the model’s ability to learn patterns associated with the minority class, potentially affecting predictive performance and fairness. The observed class proportions and the pass/fail outcomes for each class offer a transparent and quantitative basis for understanding the dataset’s composition and its implications for model development. The test’s sensitivity to the threshold parameter is also evident, underscoring the importance of aligning test settings with domain-specific requirements and modeling objectives.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	79.80%	Pass
1	20.20%	Fail

Figures

The output above shows that the class imbalance test did not pass according to the value we set for min_percent_threshold.

To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.

As this is technically a different dataset, remember to first initialize a new ValidMind Dataset object to pass in as input as required by run_test():

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

# Pass the initialized `balanced_raw_dataset` as input into the test run
result = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_balanced_raw_dataset},
    params={"min_percent_threshold": 30},
)

✅ Class Imbalance

Class Imbalance is designed to evaluate and quantify the distribution of target classes within a dataset used by a machine learning model, with the primary purpose of identifying whether any class is under-represented to a degree that could introduce bias into the model’s predictions. By ensuring that each class meets a minimum representation threshold, the test helps safeguard against the risk of the model favoring the majority class and underperforming on the minority class, which is critical for maintaining fairness and predictive reliability in classification tasks.

The test operates by calculating the frequency of each class in the target column, expressing these frequencies as percentages of the total dataset. It then compares each class’s percentage to a configurable minimum threshold, which in this instance is set at 30%. If any class falls below this threshold, it is flagged as not meeting the balance criterion. The methodology involves straightforward counting of class occurrences, division by the total number of records, and conversion to a percentage, making the process transparent and easy to interpret. The resulting percentages typically range from 0% to 100%, where higher values indicate greater representation. A class is considered adequately represented if its percentage meets or exceeds the threshold, while lower values signal potential imbalance. The test outputs both tabular and visual representations, providing a clear and immediate understanding of class proportions.

The primary advantages of this test include its ability to quickly and clearly identify under-represented classes, which is essential for preventing model bias and ensuring robust performance across all categories. The test’s simplicity and speed make it suitable for routine checks during data preparation and model development. Its quantitative output not only highlights the presence of imbalance but also measures its extent, supporting informed decision-making. The adjustable threshold parameter allows the test to be tailored to specific domain requirements or regulatory standards, and the inclusion of visual plots enhances interpretability, making it easier for stakeholders to grasp the class distribution at a glance.

It should be noted that the test has several limitations. It may be less informative for datasets with a large number of classes, where some degree of imbalance is often unavoidable due to natural class frequencies. The sensitivity of the test to the chosen threshold means that inappropriate settings could either mask genuine imbalance or overstate minor deviations. The test does not account for the varying costs or impacts of misclassifying different classes, which can be significant in certain applications. Additionally, while the test identifies imbalance, it does not provide solutions or corrective actions. Its applicability is limited to classification problems and does not extend to regression or clustering tasks. High risk is indicated when any class falls below the threshold, but this does not capture more nuanced aspects of class distribution or model performance.

This test shows the results in both tabular and graphical formats. The table titled "Exited Class Imbalance" lists each class in the target variable "Exited," displaying the percentage of rows each class represents and whether it passes the minimum threshold criterion. The columns include the class label, the percentage of total records attributed to that class, and a pass/fail indicator based on the 30% threshold. The accompanying bar plot visually depicts the proportion of each class, with the x-axis representing the class labels (0 and 1) and the y-axis showing the corresponding percentage, ranging from 0 to 0.5 (or 0% to 50%). Both classes are shown to occupy exactly 50% of the dataset, as indicated by the equal bar heights and the table values. This balanced distribution is visually apparent, with no class falling below the threshold. The plot and table together provide a comprehensive view of class representation, making it easy to assess whether the dataset meets the balance requirement.

The test results reveal the following key insights:

• Both classes meet the minimum representation threshold: Each class, labeled as 0 and 1, constitutes exactly 50% of the dataset, which is well above the 30% minimum threshold set for this test.
• No evidence of class imbalance: The pass/fail column in the table indicates that both classes pass the test, confirming that neither class is under-represented.
• Symmetrical class distribution: The bar plot visually reinforces the tabular data, showing two bars of equal height, which reflects a perfectly balanced class distribution.
• Stable and uniform dataset composition: The absence of variation between class proportions suggests that the dataset is stable with respect to the target variable, reducing the risk of model bias due to class imbalance.
• Clear interpretability of results: The combination of tabular and graphical outputs allows for immediate and unambiguous interpretation of class proportions and their compliance with the threshold.

Based on these results, the dataset used for the model demonstrates a perfectly balanced class distribution for the target variable "Exited," with both classes equally represented at 50%. This uniformity ensures that the model is not predisposed to favor one class over the other due to data imbalance, supporting fair and reliable predictive performance. The test’s outputs, both in tabular and graphical form, provide clear evidence that the dataset meets the specified minimum threshold for class representation, with no class falling below the 30% criterion. The stability and symmetry observed in the class proportions indicate that the risk of bias arising from class imbalance is minimal in this context. These results suggest that the dataset is well-suited for training classification models without the need for additional balancing interventions, and the observed characteristics support the integrity of subsequent modeling efforts.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

Utilize test output

You can utilize the output from a ValidMind test for further use, for example, if you want to remove highly correlated features. Removing highly correlated features helps make the model simpler, more stable, and easier to understand.

Below we demonstrate how to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.

First, we'll run validmind.data_validation.HighPearsonCorrelation with the balanced_raw_dataset we initialized previously as input as is for comparison with later runs:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

High Pearson Correlation is designed to identify pairs of features within a dataset that exhibit strong linear relationships, with the primary purpose of detecting potential feature redundancy or multicollinearity. This is crucial for ensuring that the predictive model remains interpretable and robust, as highly correlated features can obscure the true impact of individual variables and may lead to overfitting or instability in model estimates.

The test operates by calculating the Pearson correlation coefficient for every possible pair of features in the dataset. The Pearson correlation coefficient is a statistical measure that quantifies the strength and direction of a linear relationship between two continuous variables, producing values that range from -1 to 1. A value of 1 indicates a perfect positive linear relationship, -1 indicates a perfect negative linear relationship, and 0 indicates no linear relationship. The test systematically computes these coefficients for all feature pairs, removes self-correlations and duplicate pairs, and then sorts the results by the absolute value of the coefficient. Each pair is evaluated against a predefined threshold (in this case, 0.3), and a Pass or Fail status is assigned depending on whether the absolute correlation exceeds this threshold. The test then returns the top n pairs with the strongest correlations, providing a focused view of the most significant relationships in the data.

The primary advantages of this test include its efficiency and transparency in highlighting linear dependencies between features. By surfacing the most strongly correlated pairs, the test enables data scientists and risk managers to quickly identify areas where feature redundancy or multicollinearity may be present, which is particularly valuable during the early stages of model development and feature selection. The clear tabular output, which includes the feature pairs, their correlation coefficients, and Pass/Fail status, supports straightforward interpretation and documentation. This approach is especially useful in regulated environments where model interpretability and transparency are paramount, as it provides a defensible record of the relationships present in the data.

It should be noted that the test is limited to detecting linear relationships and does not capture nonlinear dependencies, which may also be relevant in some modeling contexts. The Pearson correlation coefficient is sensitive to outliers, meaning that a small number of extreme values can disproportionately influence the results and potentially mask or exaggerate true relationships. Additionally, the test only considers pairwise relationships and may not detect more complex interactions involving three or more features. High correlation coefficients, particularly those exceeding the set threshold, are indicative of potential multicollinearity, which can undermine the stability and interpretability of model coefficients. Care must be taken in interpreting these results, as the presence of high correlations does not necessarily imply causation or redundancy without further domain-specific analysis.

This test shows its results in a tabular format, where each row represents a unique pair of features from the dataset. The columns include the feature pair, the calculated Pearson correlation coefficient (rounded to four decimal places), and a Pass/Fail status indicating whether the absolute value of the coefficient exceeds the threshold of 0.3. The coefficients range from -0.3341 to 0.3341, with both positive and negative values indicating the direction of the linear relationship. The Pass/Fail column provides a quick reference for identifying which pairs surpass the threshold, with only one pair failing the test. The table is sorted by the absolute value of the coefficient, so the strongest relationships appear at the top. Notably, the pair (Age, Exited) has the highest absolute correlation at 0.3341 and is the only pair marked as Fail, indicating a moderate positive linear relationship that exceeds the threshold. All other pairs have coefficients below the threshold, with values ranging from -0.1984 to 0.0474, and are marked as Pass. This structure allows for rapid assessment of the most significant linear relationships in the dataset and highlights any pairs that may warrant further investigation.

The test results reveal the following key insights:

• Only One Feature Pair Exceeds the Correlation Threshold: The pair (Age, Exited) has a Pearson correlation coefficient of 0.3341, which is the only value exceeding the threshold of 0.3, resulting in a Fail status for this pair.
• All Other Feature Pairs Remain Below the Threshold: The remaining nine feature pairs have coefficients ranging from -0.1984 to 0.0474, all of which are below the 0.3 threshold and are marked as Pass, indicating no other strong linear relationships.
• Predominance of Weak Linear Relationships: Most feature pairs exhibit weak correlations, with absolute values well below the threshold, suggesting limited risk of multicollinearity among these variables.
• Balanced Distribution of Positive and Negative Correlations: The coefficients include both positive and negative values, reflecting a mix of direct and inverse linear relationships, but none are strong enough to raise immediate concerns except for the (Age, Exited) pair.
• Clear Tabular Presentation Facilitates Rapid Assessment: The sorted table format, with explicit Pass/Fail indicators, enables efficient identification of the most relevant feature relationships and supports transparent documentation.

Based on these results, the dataset demonstrates a generally low level of linear correlation among most feature pairs, with only the (Age, Exited) pair exhibiting a moderate positive relationship that exceeds the predefined threshold. This suggests that, aside from this single pair, the risk of feature redundancy or multicollinearity is minimal within the current feature set. The presence of both positive and negative coefficients across the pairs indicates a diverse range of relationships, but none approach the level that would typically warrant concern for model interpretability or stability, except for the identified pair. The clear separation between the one failing pair and the others supports the conclusion that the dataset is largely free from problematic linear dependencies, with the exception of the moderate association between Age and Exited. This observation provides a focused area for further review while affirming the overall suitability of the feature set for modeling from a linear correlation perspective.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3341	Fail
(IsActiveMember, Exited)	-0.1984	Pass
(Balance, NumOfProducts)	-0.1565	Pass
(Balance, Exited)	0.1376	Pass
(NumOfProducts, IsActiveMember)	0.0575	Pass
(Age, Balance)	0.0474	Pass
(HasCrCard, IsActiveMember)	-0.0426	Pass
(NumOfProducts, Exited)	-0.0399	Pass
(Tenure, IsActiveMember)	-0.0390	Pass
(Age, NumOfProducts)	-0.0345	Pass

The output above shows that the test did not pass according to the value we set for max_threshold.

corr_result is an object of type TestResult. We can inspect the result object to see what the test has produced:

print(type(corr_result))
print("Result ID: ", corr_result.result_id)
print("Params: ", corr_result.params)
print("Passed: ", corr_result.passed)
print("Tables: ", corr_result.tables)

<class 'validmind.vm_models.result.result.TestResult'>
Result ID:  validmind.data_validation.HighPearsonCorrelation
Params:  {'max_threshold': 0.3}
Passed:  False
Tables:  [ResultTable]

Let's remove the highly correlated features and create a new VM dataset object.

We'll begin by checking out the table in the result and extracting a list of features that failed the test:

# Extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3341	Fail
1	(IsActiveMember, Exited)	-0.1984	Pass
2	(Balance, NumOfProducts)	-0.1565	Pass
3	(Balance, Exited)	0.1376	Pass
4	(NumOfProducts, IsActiveMember)	0.0575	Pass
5	(Age, Balance)	0.0474	Pass
6	(HasCrCard, IsActiveMember)	-0.0426	Pass
7	(NumOfProducts, Exited)	-0.0399	Pass
8	(Tenure, IsActiveMember)	-0.0390	Pass
9	(Age, NumOfProducts)	-0.0345	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

Next, extract the feature names from the list of strings (example: (Age, Exited) > Age):

high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

Now, it's time to re-initialize the dataset with the highly correlated features removed.

Note the use of a different input_id. This allows tracking the inputs used when running each individual test.

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

Re-running the test with the reduced feature set should pass the test:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

High Pearson Correlation is designed to identify pairs of features within a dataset that exhibit strong linear relationships, with the primary purpose of detecting potential feature redundancy or multicollinearity. By highlighting highly correlated feature pairs, this test supports model developers and risk management teams in understanding dependencies that may affect model performance, interpretability, and robustness.

The test operates by calculating the Pearson correlation coefficient for every possible pair of features in the dataset. The Pearson correlation coefficient quantifies the strength and direction of the linear relationship between two variables, with values ranging from -1 (perfect negative linear relationship) to 1 (perfect positive linear relationship), and 0 indicating no linear relationship. The test systematically excludes self-correlations and duplicate pairs to ensure each unique feature pair is evaluated only once. It then compares the absolute value of each coefficient to a predefined threshold, which in this case is set at 0.3. Pairs with coefficients exceeding this threshold are flagged as potentially problematic due to high correlation, while those below the threshold are considered to pass. The test outputs the top n strongest correlations, regardless of whether they pass or fail, providing a transparent view of the most significant linear relationships in the data. This approach enables users to quickly assess the extent of linear dependencies and prioritize further investigation or mitigation steps as needed.

The primary advantages of this test include its efficiency and clarity in surfacing linear relationships between features, which is particularly valuable in the early stages of model development and risk assessment. By providing a ranked list of the strongest correlations, the test allows practitioners to focus on the most relevant feature pairs that may contribute to multicollinearity or redundancy. This transparency aids in model interpretability, as it becomes easier to understand which variables may be providing overlapping information. The test is also straightforward to implement and interpret, making it accessible for both technical and non-technical stakeholders. Its ability to quickly flag potential issues supports proactive risk management and helps ensure that models are built on a foundation of well-understood, non-redundant features.

It should be noted that the test is limited to detecting linear relationships and does not capture nonlinear dependencies, which may also be relevant in some modeling contexts. The Pearson correlation coefficient is sensitive to outliers, meaning that a few extreme values can disproportionately influence the results and potentially mask or exaggerate true relationships. Additionally, the test only evaluates pairwise relationships and may not identify more complex interactions involving three or more features. High correlation coefficients, particularly those exceeding the set threshold, are indicative of potential multicollinearity, which can undermine model stability and the interpretability of individual feature contributions. However, the presence of high correlations does not automatically imply a problem; further analysis is often required to determine the practical impact on model performance.

This test shows the results in a tabular format, where each row represents a unique pair of features, the calculated Pearson correlation coefficient for that pair, and a Pass/Fail status based on whether the absolute value of the coefficient exceeds the threshold of 0.3. The "Columns" field specifies the feature pair, "Coefficient" provides the numerical value of the correlation (ranging from -1 to 1), and "Pass/Fail" indicates whether the pair is below (Pass) or above (Fail) the threshold. In this particular output, all coefficients are well below the threshold, with the highest absolute value being -0.1984 for the pair (IsActiveMember, Exited). The coefficients span both positive and negative values, indicating both direct and inverse relationships, but none approach the threshold that would suggest a strong linear dependency. The table is sorted by the absolute value of the coefficient, presenting the top ten strongest correlations in the dataset. This allows for a straightforward assessment of the degree of linear association among the most relevant feature pairs, with all observed values falling within a relatively narrow range and no extreme outliers present.

The test results reveal the following key insights:

• No Feature Pairs Exceed Correlation Threshold: All observed Pearson correlation coefficients are below the 0.3 threshold, with the highest absolute value being -0.1984 for (IsActiveMember, Exited), indicating no strong linear relationships among the top feature pairs.
• Distribution of Correlation Coefficients Is Narrow: The coefficients for the top ten pairs range from -0.1984 to 0.0331, suggesting that the dataset does not contain any feature pairs with moderate or high linear association.
• Both Positive and Negative Relationships Present: The results include both positive and negative coefficients, such as 0.1376 for (Balance, Exited) and -0.1565 for (Balance, NumOfProducts), reflecting a mix of direct and inverse linear relationships, though all are weak in magnitude.
• Pass Status Uniform Across All Pairs: Every feature pair in the output is marked as "Pass," confirming that none of the evaluated relationships meet the criteria for high correlation as defined by the test parameters.
• No Evidence of Multicollinearity Among Top Features: The absence of coefficients near or above the threshold suggests that multicollinearity is not a significant characteristic among the most strongly correlated feature pairs in this dataset.

Based on these results, the dataset exhibits a low degree of linear association among its top feature pairs, as evidenced by the uniformly low Pearson correlation coefficients and the consistent "Pass" status across all evaluated pairs. The range of coefficients, from -0.1984 to 0.0331, indicates that none of the relationships approach the threshold that would signal potential redundancy or multicollinearity. Both positive and negative associations are present, but all are weak, suggesting that the features contribute largely independent information to the model. This pattern supports the interpretability and stability of the model, as it reduces the likelihood that any single feature's predictive power is confounded by strong linear dependencies with others. The results provide a clear and objective characterization of the dataset's feature relationships, confirming that, under the current threshold, the risk of linear redundancy or multicollinearity is minimal among the most relevant feature pairs.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1984	Pass
(Balance, NumOfProducts)	-0.1565	Pass
(Balance, Exited)	0.1376	Pass
(NumOfProducts, IsActiveMember)	0.0575	Pass
(HasCrCard, IsActiveMember)	-0.0426	Pass
(NumOfProducts, Exited)	-0.0399	Pass
(Tenure, IsActiveMember)	-0.0390	Pass
(Tenure, EstimatedSalary)	0.0331	Pass
(Balance, HasCrCard)	-0.0268	Pass
(HasCrCard, EstimatedSalary)	-0.0245	Pass

You can also plot the correlation matrix to visualize the new correlation between features:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.PearsonCorrelationMatrix",
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

Pearson Correlation Matrix

Pearson Correlation Matrix is designed to evaluate the extent of linear dependency between all pairs of numerical variables in a dataset. Its primary purpose is to identify potential redundancy among variables by quantifying the strength and direction of their linear relationships, thereby supporting dimensionality reduction and improving model interpretability.

The test operates by calculating the Pearson correlation coefficient for every pair of numerical variables in the dataset. This coefficient measures the degree to which two variables move together in a linear fashion, with values ranging from -1 to 1. A value of 1 indicates a perfect positive linear relationship, -1 indicates a perfect negative linear relationship, and 0 indicates no linear relationship. The test compiles these coefficients into a correlation matrix, which is then visualized as a heat map. The heat map uses color gradients to represent the magnitude and direction of each correlation, with a specific highlight (white) for coefficients whose absolute value exceeds 0.7, signaling a high degree of correlation. This approach allows for rapid identification of variable pairs that may be redundant or highly interdependent, which is particularly useful for feature selection and multicollinearity assessment in predictive modeling.

The primary advantages of this test include its ability to provide a clear, quantitative assessment of linear relationships between variables, which is essential for detecting redundancy and potential multicollinearity in datasets. The heat map visualization makes it accessible to a wide range of users, including those who may not be comfortable interpreting raw correlation matrices. By highlighting strong correlations, the test supports informed decisions about feature selection, helping to streamline models and potentially enhance their generalizability. Additionally, the test is computationally efficient and can be applied to datasets of varying sizes, making it a practical tool for exploratory data analysis and model diagnostics.

It should be noted that the Pearson Correlation Matrix is limited to detecting linear relationships and may not capture more complex, non-linear dependencies between variables. As a result, important associations could be overlooked if they do not manifest as linear patterns. The test also does not measure the causal influence or predictive power of one variable over another, focusing solely on the degree of co-movement. The threshold of 0.7 for highlighting high correlations is somewhat arbitrary and may not be appropriate for all contexts, potentially missing meaningful relationships with lower coefficients. Furthermore, a large number of highly correlated variables can indicate redundancy and increase the risk of overfitting, which may compromise model performance if not addressed.

This test shows a heat map representation of the Pearson correlation matrix for the dataset’s numerical variables. Each cell in the matrix corresponds to the correlation coefficient between a pair of variables, with the variable names listed along both the horizontal and vertical axes. The color scale ranges from deep blue (indicating strong positive correlation) through white (no correlation) to deep red (strong negative correlation), as shown by the color bar on the right. The diagonal cells, which compare each variable with itself, are always 1 and appear as the darkest blue. The off-diagonal cells display the pairwise correlations, with values annotated within each cell for precise interpretation. Notably, none of the off-diagonal cells are highlighted in white, indicating that no pair of variables exceeds the 0.7 absolute correlation threshold. The majority of the coefficients are close to zero, suggesting weak or negligible linear relationships. The largest absolute correlation observed is -0.20 between "IsActiveMember" and "Exited," and -0.16 between "NumOfProducts" and "Balance," both of which are well below the high-correlation threshold. The heat map provides a comprehensive visual summary, making it easy to spot any strong linear dependencies or lack thereof across the dataset.

The test results reveal the following key insights:

• No High Linear Correlations Detected: All off-diagonal correlation coefficients fall well below the 0.7 threshold, indicating an absence of strong linear relationships between any pair of variables.
• Predominance of Weak Relationships: Most correlation values are clustered near zero, with the majority ranging between -0.20 and 0.14, suggesting that the variables are largely independent in a linear sense.
• Notable Negative Associations: The most negative correlation is observed between "IsActiveMember" and "Exited" at -0.20, and between "NumOfProducts" and "Balance" at -0.16, indicating mild inverse relationships.
• Minimal Positive Associations: The highest positive correlation is 0.14 between "Balance" and "Exited," which remains weak and does not suggest redundancy.
• Diagonal Dominance: As expected, all diagonal elements are 1, confirming perfect self-correlation and serving as a reference for interpreting off-diagonal values.
• Uniform Distribution Across Variables: No variable stands out as being consistently highly correlated with multiple others, supporting the overall independence of features.

Based on these results, the dataset exhibits a low degree of linear dependency among its numerical variables, as evidenced by the uniformly low Pearson correlation coefficients and the absence of any values exceeding the 0.7 threshold. This pattern suggests that the variables are not redundant in a linear sense and are likely to contribute distinct information to any downstream modeling efforts. The weak correlations observed, both positive and negative, indicate that multicollinearity is not a significant concern in this dataset, reducing the risk of overfitting due to redundant features. The heat map visualization confirms that the relationships between variables are generally weak and dispersed, with no clusters or groupings of highly correlated variables. This structural independence among features supports the use of the full variable set in modeling, as each variable appears to capture unique aspects of the data. Overall, the results provide a clear and objective characterization of the dataset’s linear dependency structure, informing subsequent steps in feature selection and model development.

Figures

Documenting test results

Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it.

Every test result returned by the run_test() function has a .log() method that can be used to send the test results to the ValidMind Platform:

• When using run_documentation_tests(), documentation sections will be automatically populated with the results of all tests registered in the documentation template.
• When logging individual test results to the platform, you'll need to manually add those results to the desired section of the model documentation.

To demonstrate how to add test results to your model documentation, we'll populate the entire Data Preparation section of the documentation using the clean vm_raw_dataset_preprocessed dataset as input, and then document an additional individual result for the highly correlated dataset vm_balanced_raw_dataset.

Run and log multiple tests

run_documentation_tests() allows you to run multiple tests at once and automatically log the results to your documentation. Below, we'll run the tests using the previously initialized vm_raw_dataset_preprocessed as input — this will populate the entire Data Preparation section for every test that is part of the documentation template.

For this example, we'll pass in the following arguments:

• inputs: Any inputs to be passed to the tests.
• config: A dictionary <test_id>:<test_config> that allows configuring each test individually. Each test config requires the following:
  • params: Individual test parameters.
  • inputs: Individual test inputs. This overrides any inputs passed from the run_documentation_tests() function.

When including explicit configuration for individual tests, you'll need to specify the inputs even if they mirror what is included in your global configuration.

# Individual test config with inputs specified
test_config = {
    "validmind.data_validation.ClassImbalance": {
        "params": {"min_percent_threshold": 30},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "params": {"max_threshold": 0.3},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
}

# Global test config
tests_suite = vm.run_documentation_tests(
    inputs={
        "dataset": vm_raw_dataset_preprocessed,
    },
    config=test_config,
    section=["data_preparation"],
)

Test suite complete!

26/26 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Data Preparation

Data Preparation

▶ Test Result: Dataset Description (validmind.data_validation.DatasetDescription)

Dataset Description

Dataset Description is designed to provide a comprehensive analysis and statistical summary of each column in a machine learning model's dataset. The primary purpose of this test is to deliver a detailed overview of the dataset's structure, including the distribution, completeness, and uniqueness of data across all features, thereby enabling a thorough understanding of the data characteristics that underpin model development and evaluation.

The test operates by systematically examining each column in the dataset to infer its data type—such as numeric, categorical, boolean, or text—and then computing a suite of descriptive statistics for each. For every column, the test calculates the total number of entries, the count and proportion of missing values, and the count and proportion of unique values. For numeric columns, the test generates histograms to visualize the distribution of values, typically using a fixed number of bins to group data points and reveal patterns such as skewness or clustering. For categorical, boolean, and text columns, the test computes frequency distributions to show how often each unique value appears. The metrics produced—such as missing value percentage (ranging from 0% to 100%) and distinct value percentage—help identify data quality issues and the potential for overfitting or underfitting. Columns with unsupported data types are flagged, and those with all missing values are excluded from histogram generation. The results are aggregated into a summary table, which presents a holistic view of the dataset's structure and quality.

The primary advantages of this test include its ability to deliver a granular and versatile summary of dataset features, making it particularly valuable for initial data exploration and quality assessment. By quantifying missingness, uniqueness, and distributional properties for each column, the test enables practitioners to quickly identify potential data quality challenges, such as high cardinality or sparsity, that could impact model performance. Its flexibility in handling multiple data types ensures that both structured and semi-structured datasets can be analyzed effectively. The inclusion of histograms and frequency tables provides visual and quantitative insights into feature distributions, supporting informed decisions about feature engineering, data cleaning, and model selection. This comprehensive approach is especially useful in regulated environments where transparency and documentation of data characteristics are critical.

It should be noted that the test has several limitations and potential risks. The computational cost can be significant for large datasets with many columns, as generating detailed statistics and histograms for each feature may require substantial processing time and memory. The use of a fixed number of bins in histograms may not always capture the true underlying distribution, potentially obscuring important patterns or outliers. Columns with unsupported data types are not analyzed, which could leave gaps in the dataset summary. Additionally, the test does not directly assess model performance or predictive power; it focuses solely on data quality and structure. High proportions of missing values, extreme skewness, or a large number of unique values in certain columns may signal risks for downstream modeling, such as reduced predictive accuracy or increased model complexity. Interpretation challenges may arise if the summary statistics are not contextualized within the specific modeling objectives or business requirements.

This test shows the results in the form of a summary table, where each row represents a dataset column and each column in the table provides a specific metric: the feature name, data type, total count of entries, number and percentage of missing values, number and percentage of distinct values. The table allows users to quickly scan for completeness, uniqueness, and data type distribution across all features. For example, the "Count" column indicates the total number of non-missing entries per feature, while "Missing" and "Missing %" quantify the extent of missing data. "Distinct" and "Distinct %" reveal the diversity of values in each column, which is critical for understanding cardinality and potential overfitting risks. All features in this dataset have a count of 3,232 and zero missing values, indicating complete data coverage. Numeric columns such as "CreditScore," "Tenure," "Balance," "NumOfProducts," and "EstimatedSalary" are distinguished from categorical columns like "Geography," "Gender," "HasCrCard," "IsActiveMember," and "Exited." Notably, "EstimatedSalary" has a distinct value for every entry, while "Balance" also shows high uniqueness. Categorical features have low distinct counts, reflecting limited possible values. The table format enables straightforward comparison across features, highlighting both the breadth and depth of data available for modeling.

The test results reveal the following key insights:

• Dataset Completeness Is Absolute: All columns have a count of 3,232 and zero missing values, resulting in a missing value percentage of 0.0% across the entire dataset, which indicates no data loss or gaps in any feature.
• Feature Cardinality Varies Widely: Numeric columns such as "EstimatedSalary" and "Balance" exhibit extremely high cardinality, with "EstimatedSalary" having 3,232 distinct values (100% distinct) and "Balance" having 2,221 distinct values (68.72% distinct), suggesting these features are nearly or fully unique for each record.
• Categorical Features Are Well-Defined: Categorical columns like "Geography," "Gender," "HasCrCard," "IsActiveMember," and "Exited" have low distinct counts, ranging from 2 to 3 unique values, which is typical for features representing discrete categories or binary indicators.
• Numeric Features Show Moderate Uniqueness: "CreditScore" has 436 distinct values (13.49% distinct), "Tenure" has 11 (0.34%), and "NumOfProducts" has 4 (0.12%), indicating varying levels of granularity and potential for capturing nuanced patterns.
• No Unsupported or Problematic Data Types Detected: All columns are classified as either numeric or categorical, with no unsupported types or columns excluded from analysis, ensuring comprehensive coverage of the dataset.

Based on these results, the dataset demonstrates a high degree of completeness and structural clarity, with no missing values and well-defined data types across all features. The presence of features with very high cardinality, such as "EstimatedSalary" and "Balance," suggests that the dataset contains variables capable of capturing individual-level variation, which may enhance model expressiveness but also introduces the potential for overfitting if not managed appropriately. Categorical features are limited in their possible values, supporting straightforward interpretation and reducing the risk of sparsity. The moderate uniqueness observed in features like "CreditScore" and "Tenure" provides a balance between granularity and generalizability. The absence of unsupported data types or columns with all missing values indicates that the dataset is well-prepared for downstream modeling tasks. Overall, the dataset's structure supports robust model development, with clear delineation between feature types and no immediate data quality concerns evident from the summary statistics.

Tables

Dataset Description

Name	Type	Count	Missing	Missing %	Distinct	Distinct %
CreditScore	Numeric	3232.0	0	0.0	436	0.1349
Geography	Categorical	3232.0	0	0.0	3	0.0009
Gender	Categorical	3232.0	0	0.0	2	0.0006
Tenure	Numeric	3232.0	0	0.0	11	0.0034
Balance	Numeric	3232.0	0	0.0	2221	0.6872
NumOfProducts	Numeric	3232.0	0	0.0	4	0.0012
HasCrCard	Categorical	3232.0	0	0.0	2	0.0006
IsActiveMember	Categorical	3232.0	0	0.0	2	0.0006
EstimatedSalary	Numeric	3232.0	0	0.0	3232	1.0000
Exited	Categorical	3232.0	0	0.0	2	0.0006

▶ Test Result: Class Imbalance (validmind.data_validation.ClassImbalance)

✅ Class Imbalance

Class Imbalance is designed to evaluate and quantify the distribution of target classes within a dataset used by a machine learning model, with the primary purpose of identifying whether any class is under-represented to a degree that could introduce bias into the model’s predictions. By ensuring that each class meets a minimum representation threshold, the test helps safeguard against the risk of the model favoring the majority class and underperforming on the minority class, which is critical for maintaining fairness and predictive reliability in classification tasks.

The test operates by calculating the frequency of each class in the target column of the dataset, expressing these frequencies as percentages of the total number of records. It then compares each class’s percentage to a configurable minimum threshold, which in this instance is set to 30%. If any class falls below this threshold, it is flagged as not meeting the balance criterion. The methodology is straightforward: the test requires the target column as input, counts the occurrences of each unique class, divides by the total number of records to obtain the proportion, and multiplies by 100 to express this as a percentage. The results are typically presented in both tabular and graphical formats, with the table showing each class, its percentage, and a pass/fail indicator, and the plot visually depicting the class proportions. Values range from 0% to 100%, where higher values indicate greater representation. A class passing the threshold is considered adequately represented, while a class below the threshold signals potential imbalance.

The primary advantages of this test include its ability to quickly and transparently identify under-represented classes, which is essential for preventing model bias and ensuring robust predictive performance across all classes. The test’s simplicity and speed make it suitable for routine data quality checks, and its quantitative output provides clear, actionable information. The adjustable threshold parameter allows the test to be tailored to different domains or regulatory requirements, enhancing its flexibility. The inclusion of both tabular and visual outputs aids interpretability, making it easier for stakeholders to understand the class distribution at a glance. This is particularly valuable in regulated environments or high-stakes applications where transparency and fairness are paramount.

It should be noted that the test has several limitations. It may be less informative for datasets with a large number of classes, where some degree of imbalance is expected due to the natural distribution of the data. The choice of threshold is subjective and can influence the test’s sensitivity; setting it too high may result in false positives for imbalance, while setting it too low may overlook meaningful disparities. The test does not account for the varying costs or impacts of misclassifying different classes, which can be significant in certain domains. Additionally, while the test identifies imbalances, it does not provide solutions or corrective actions. Its applicability is limited to classification problems and does not extend to regression or clustering tasks. High risk is indicated when any class falls below the threshold, but this does not automatically imply a problem unless the imbalance is consequential for the model’s intended use.

This test shows the results in both a table and a bar plot, each providing a clear view of the class distribution for the target variable "Exited." The table lists each class (0 and 1), the percentage of rows corresponding to each class, and a pass/fail status based on the 30% minimum threshold. Both classes have exactly 50.00% representation, and both pass the threshold, as indicated in the "Pass/Fail" column. The accompanying bar plot visually reinforces this result, displaying two bars of equal height at the 0.5 mark on the y-axis, corresponding to 50% for each class. The x-axis represents the class labels (0 and 1), while the y-axis shows the proportion of the dataset each class occupies, ranging from 0 to 0.5. The plot’s symmetry and the equal bar heights make it immediately apparent that the dataset is perfectly balanced with respect to the "Exited" target variable. There are no classes with disproportionately low or high representation, and no regions of the plot suggest any deviation from balance. The results are straightforward to interpret: both the tabular and graphical outputs confirm that the dataset meets the specified class balance criterion.

The test results reveal the following key insights:

• Perfect Class Balance Achieved: Both classes in the "Exited" target variable are represented equally, each comprising exactly 50.00% of the dataset, which is well above the 30% minimum threshold.
• All Classes Pass the Balance Criterion: The "Pass/Fail" column in the results table indicates that both classes meet the required minimum representation, with no class flagged for imbalance.
• Visual Confirmation of Symmetry: The bar plot visually confirms the numerical results, showing two bars of identical height, which reinforces the observation of perfect balance between the classes.
• No Evidence of Under-Representation: There are no classes with a percentage below the threshold, and no indication of risk associated with class imbalance in this dataset.
• Stable and Uniform Distribution: The results demonstrate a stable and uniform class distribution, with no notable variations or dependencies between the classes.

Based on these results, the dataset used for the "Exited" target variable demonstrates a perfectly balanced class distribution, with both classes equally represented at 50.00%. This uniformity is confirmed by both the tabular and graphical outputs, which show no deviation from the balance criterion set at 30%. The absence of any class below the threshold indicates that the risk of model bias due to class imbalance is minimal in this context. The results suggest that the model trained on this dataset is unlikely to exhibit preferential behavior toward either class based on class frequency alone. The stability and symmetry observed in the class proportions provide a strong foundation for fair and reliable model performance, as the model will have equal exposure to both classes during training. This balanced distribution is particularly advantageous for classification tasks where equal treatment of all classes is desired, and it supports the interpretability and transparency of the model development process.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

▶ Test Result: Duplicates (validmind.data_validation.Duplicates)

✅ Duplicates

Duplicates is designed to identify and quantify duplicate rows within a dataset, with the primary purpose of ensuring data quality and supporting model reliability by preventing the influence of redundant or repeated information. This test is a critical component of data pre-processing, as it helps to mitigate the risk of overfitting and ensures that the model is trained on unique, representative data rather than memorizing repeated entries.

The test operates by systematically scanning each row in the dataset to detect exact duplicates. If a specific text column is designated, the test focuses on that column; otherwise, it evaluates all feature columns collectively. The process involves comparing each row to all others to determine if any are identical across the relevant columns. The test then calculates two key metrics: the absolute number of duplicate rows and the percentage of duplicates relative to the total number of rows. These metrics provide a quantitative assessment of data redundancy. The number of duplicates is a straightforward count, while the percentage contextualizes this count within the size of the dataset, typically ranging from 0% (no duplicates) to 100% (all rows are duplicates). A low or zero value is generally desirable, indicating high data integrity, whereas higher values may signal data collection or processing issues. The test is considered passed if the number of duplicates falls below a user-defined threshold, which can be tailored to the specific requirements of the modeling task.

The primary advantages of this test include its ability to provide a clear, quantitative overview of data redundancy, which is essential for maintaining the integrity of statistical analyses and model training. By reporting both the absolute and relative frequency of duplicate rows, the test enables users to quickly assess the extent of potential data contamination. This dual-metric approach is particularly useful in large-scale data environments, where manual inspection is impractical. The test’s flexibility in allowing users to specify a minimum threshold for acceptable duplicates further enhances its applicability across diverse datasets and modeling scenarios. Additionally, by focusing on exact duplicates, the test ensures that only truly redundant entries are flagged, reducing the risk of false positives and supporting more accurate downstream analyses.

It should be noted that the test is limited to detecting exact duplicates and does not account for semantically similar but non-identical entries, which may still introduce bias or redundancy into the model. The computational complexity of the test increases with dataset size, potentially impacting performance on very large datasets. Furthermore, the test does not differentiate between benign duplicates—such as legitimate repeated observations—and problematic duplicates arising from data processing errors. Interpretation of the results therefore requires domain knowledge to distinguish between these cases. High numbers or percentages of duplicates are a sign of elevated risk, as they may indicate underlying issues with data collection or preparation that could compromise model performance. Users should also be aware that the test’s focus on exact matches may overlook more subtle forms of data redundancy.

This test shows the results in a tabular format, specifically presenting a table titled "Duplicate Rows Results for Dataset." The table contains two columns: "Number of Duplicates" and "Percentage of Rows (%)". The "Number of Duplicates" column displays the total count of duplicate rows detected in the dataset, while the "Percentage of Rows (%)" column expresses this count as a proportion of the total dataset size, formatted as a percentage. In this instance, the table reports a value of 0 for the number of duplicates and 0.0% for the percentage of rows, indicating that no duplicate rows were found in the dataset. This output is straightforward to interpret: a value of zero in both columns confirms the absence of exact duplicate entries. The scale for both metrics is absolute, with the number of duplicates ranging from zero to the total number of rows minus one, and the percentage ranging from 0% to 100%. The results are notable for their clarity and simplicity, as the absence of duplicates is immediately apparent and requires no further breakdown or subgroup analysis.

The test results reveal the following key insights:

• No Duplicate Rows Detected: The dataset contains zero duplicate rows, as indicated by the "Number of Duplicates" value of 0.
• Complete Data Uniqueness: The "Percentage of Rows (%)" is 0.0%, confirming that every row in the dataset is unique with respect to the columns evaluated.
• High Data Integrity: The absence of duplicates suggests that the data collection and processing pipelines have maintained high standards of data quality, with no evidence of redundancy or repeated entries.
• Clear and Unambiguous Output: The results are presented in a simple, easily interpretable table, allowing for immediate assessment without the need for further analysis or visualization.

Based on these results, the dataset demonstrates complete uniqueness across all evaluated rows, with no evidence of exact duplicate entries. This observation indicates that the data preparation and ingestion processes have effectively prevented the introduction of redundant information, thereby supporting the reliability and validity of subsequent modeling efforts. The zero values for both the absolute number and percentage of duplicates provide strong assurance that the model will not be influenced by repeated data, reducing the risk of overfitting and ensuring that statistical analyses are based on independent observations. The clarity and simplicity of the results facilitate straightforward interpretation and confirm that the dataset meets a fundamental criterion for high-quality model development. This outcome supports confidence in the integrity of the data and its suitability for use in both training and evaluation contexts, with no immediate need for further duplicate-related data quality interventions.

Tables

Duplicate Rows Results for Dataset

Number of Duplicates	Percentage of Rows (%)
0	0.0

▶ Test Result: High Cardinality (validmind.data_validation.HighCardinality)

✅ High Cardinality

High Cardinality is designed to assess the number of unique values present in categorical columns within a dataset, with the primary purpose of detecting high cardinality that may indicate potential overfitting or the presence of unwanted noise. By systematically evaluating the distinctiveness of categorical features, this test helps to identify columns that may introduce complexity or instability into downstream modeling processes.

The test operates by first identifying all columns in the dataset that are classified as categorical. For each of these columns, it calculates two key metrics: the number of distinct values (n_distinct) and the percentage of distinct values relative to the total number of records (p_distinct). The number of distinct values provides a direct count of unique entries in a column, while the percentage contextualizes this count as a proportion of the dataset size, typically ranging from 0% (no unique values) to 100% (all values unique). The test then compares the number of distinct values in each column to a predefined numeric threshold, which is determined based on test parameters and dataset characteristics. If a column’s n_distinct is less than this threshold, it passes the test; otherwise, it fails. The results are compiled into a table that lists each categorical column, its number and percentage of distinct values, and its pass/fail status. This approach enables a clear, quantitative assessment of cardinality risk, with higher values indicating greater potential for overfitting or data quality issues.

The primary advantages of this test include its effectiveness in early detection of high cardinality, which is a common source of overfitting and model instability, especially in categorical features. By quantifying the uniqueness of values in each categorical column, the test provides actionable insights that can guide feature engineering and data preprocessing decisions. Its ability to highlight potential outliers and inconsistencies further enhances data quality, making it a valuable tool for both classification and regression tasks. The test’s straightforward methodology and clear output format facilitate rapid interpretation and integration into broader model risk management frameworks, supporting robust model development and validation processes.

It should be noted that the test is limited to categorical data types and does not evaluate numerical or continuous features, which restricts its applicability in datasets dominated by such variables. Additionally, the test does not account for the contextual importance or predictive relevance of unique values within categorical features, which means that some critical data points may be overlooked if they are rare but highly informative. The use of a static threshold for determining high cardinality may not be optimal across diverse datasets or business applications, potentially leading to false positives or negatives. Furthermore, columns that fail the test—those with a number of distinct values exceeding the threshold—are flagged as higher risk, but this does not necessarily imply a direct negative impact on model performance without further analysis.

This test shows the results in a tabular format, where each row corresponds to a categorical column in the dataset. The table includes four columns: the name of the categorical column, the number of distinct values observed in that column, the percentage of distinct values relative to the total dataset size, and the pass/fail status based on the predefined threshold. For the columns evaluated—“Geography” and “Gender”—the number of distinct values is 3 and 2, respectively, with corresponding percentages of 0.0928% and 0.0619%. Both columns are marked as “Pass,” indicating that their cardinality is below the threshold set by the test. The table is straightforward to interpret: higher numbers in the “Number of Distinct Values” and “Percentage of Distinct Values (%)” columns would indicate higher cardinality, and a “Fail” in the “Pass/Fail” column would flag a potential risk. In this case, all evaluated columns fall well below the threshold, with low percentages and a clear pass status, suggesting minimal risk of overfitting due to high cardinality in these features.

The test results reveal the following key insights:

• All Categorical Columns Exhibit Low Cardinality: Both “Geography” and “Gender” have a small number of distinct values, with counts of 3 and 2, respectively, indicating limited diversity within these features.
• Distinct Value Percentages Are Minimal: The percentage of distinct values for “Geography” is 0.0928% and for “Gender” is 0.0619%, both of which are substantially below typical thresholds for high cardinality, reflecting a high degree of repetition in these columns.
• No Columns Exceed the Cardinality Threshold: Both columns receive a “Pass” status, confirming that neither approaches the numeric threshold that would indicate elevated risk of overfitting or data instability.
• Uniformity Across Evaluated Features: The results show a consistent pattern of low cardinality across all assessed categorical columns, with no outliers or anomalous values present in the dataset.

Based on these results, the dataset’s categorical features—specifically “Geography” and “Gender”—demonstrate a stable and low-cardinality profile, with each column containing only a few unique values and a very small proportion of distinct entries relative to the total dataset size. This uniformity suggests that the risk of overfitting due to high cardinality in these features is minimal, and the data structure is unlikely to introduce instability or excessive complexity into downstream modeling. The clear “Pass” status for all evaluated columns further supports the conclusion that the dataset’s categorical variables are well within acceptable bounds for cardinality, providing a solid foundation for reliable model development. The absence of any columns near or above the threshold indicates that the dataset does not contain categorical features that would require additional scrutiny or specialized preprocessing to mitigate cardinality-related risks. Overall, the results reflect a dataset with categorical features that are well-suited for standard modeling approaches, with no immediate indications of cardinality-driven challenges.

Tables

Column	Number of Distinct Values	Percentage of Distinct Values (%)	Pass/Fail
Geography	3	0.0928	Pass
Gender	2	0.0619	Pass

▶ Test Result: Missing Values (validmind.data_validation.MissingValues)

✅ Missing Values

Missing Values is designed to assess the quality of a dataset by quantifying the proportion of missing values present in each feature, with the primary purpose of ensuring that the ratio of missing data does not exceed a predefined threshold. This process is essential for maintaining the integrity and reliability of data used in predictive modeling, as excessive missing values can compromise model performance and lead to biased or invalid results.

The test operates by systematically examining each column in the dataset, identifying and counting the number of missing entries, typically represented as "NaN" values. For each feature, the test calculates the percentage of missing values relative to the total number of records, providing a clear metric for data completeness. This percentage is then compared against a user-defined threshold, which defaults to 1 percent, to determine whether the feature passes or fails the test. The results are presented in a tabular format, with columns indicating the feature name, the absolute count of missing values, the corresponding percentage, and a pass/fail status. The metric of interest—the percentage of missing values—ranges from 0% (no missing data) to 100% (all data missing), with lower values indicating higher data quality. A result below the threshold is generally considered acceptable, while values at or above the threshold may signal potential data quality issues that could affect downstream modeling.

The primary advantages of this test include its ability to provide a rapid and detailed assessment of missing data across all features, enabling data scientists and model risk managers to quickly identify areas where data quality may be compromised. By offering a granular, column-level breakdown, the test supports targeted data cleaning and preprocessing efforts, which are critical for building robust and reliable machine learning models. The straightforward nature of the test makes it accessible and easy to interpret, facilitating communication of data quality status to both technical and non-technical stakeholders. Additionally, the test's configurable threshold allows for flexibility in adapting to varying data quality requirements across different modeling contexts.

It should be noted that the test is limited in scope, as it does not provide insights into the underlying causes of missing data or offer guidance on appropriate imputation or handling strategies. Features with missing value percentages just below the threshold may still pose risks to model performance, yet these are not flagged by the test. Furthermore, the test only detects standard missing value representations such as "NaN" and may overlook alternative encodings like "-999" or "None," which could have similar implications for data quality. High risk is indicated when one or more features exceed the missing value threshold, or when missing data is distributed across multiple features, potentially compounding the impact on model reliability.

This test shows the results in a structured table format, where each row corresponds to a feature in the dataset and columns display the feature name, the number of missing values, the percentage of missing values, and the pass/fail status based on the threshold. The table provides a comprehensive overview of data completeness for each feature, with all values for the number and percentage of missing values reported as zero. The "Pass/Fail" column indicates that every feature has passed the test, as none exceed the missing value threshold. The scale for the percentage of missing values is from 0% to 100%, but in this case, all features are at 0%. This uniformity across all features suggests a high level of data completeness, with no features exhibiting any missing values. The absence of variation in the results and the consistent "Pass" status across all features are notable, indicating that the dataset is free from missing data according to the criteria of this test.

The test results reveal the following key insights:

• All Features Exhibit Complete Data: Every feature in the dataset, including CreditScore, Geography, Gender, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, has zero missing values, corresponding to a 0.0% missing value rate.
• Uniform Pass Status Across Features: All features meet the missing value threshold criteria, resulting in a "Pass" status for each, with no exceptions or borderline cases observed.
• No Evidence of Data Quality Risk from Missing Values: The absence of missing values in any feature indicates that, based on this test, there are no immediate data quality risks related to missing data that could impact model development or performance.
• Consistent Data Completeness Across Feature Types: Both categorical and numerical features display the same level of completeness, with no variation in missing value rates between different types of variables.

Based on these results, the dataset demonstrates complete data integrity with respect to missing values, as every feature contains a full set of observed values and passes the established threshold criteria. The uniformity of zero missing values across all features, regardless of type, suggests that the data collection and preprocessing processes have been effective in ensuring data completeness. This high level of data quality provides a strong foundation for subsequent modeling activities, as the absence of missing data eliminates the need for imputation or special handling strategies and reduces the risk of introducing bias or instability into the model. The results also indicate that, at least in terms of missing values, there are no observable patterns or dependencies that could affect the model's behavior, and the dataset is well-suited for use in predictive modeling without further concern for missing data.

Tables

Column	Number of Missing Values	Percentage of Missing Values (%)	Pass/Fail
CreditScore	0	0.0	Pass
Geography	0	0.0	Pass
Gender	0	0.0	Pass
Tenure	0	0.0	Pass
Balance	0	0.0	Pass
NumOfProducts	0	0.0	Pass
HasCrCard	0	0.0	Pass
IsActiveMember	0	0.0	Pass
EstimatedSalary	0	0.0	Pass
Exited	0	0.0	Pass

▶ Test Result: Skewness (validmind.data_validation.Skewness)

❌ Skewness

Skewness is designed to evaluate the asymmetry in the distribution of numerical data within a dataset, with the primary purpose of identifying deviations from a normal distribution that may impact the performance and reliability of machine learning models. By quantifying the degree of skewness in each numeric column, this test helps to ensure data quality and supports the optimization of predictive modeling processes.

The test operates by systematically calculating the skewness statistic for each numerical column in the dataset. Skewness is a measure that captures the extent to which the distribution of values in a variable deviates from perfect symmetry. A skewness value of zero indicates a perfectly symmetrical distribution, while positive values suggest a longer or fatter right tail, and negative values indicate a longer or fatter left tail. The test compares each calculated skewness value against a predefined maximum threshold, typically set at 1. If the skewness of a column is less than this threshold, the column is considered to have acceptable symmetry and passes the test; otherwise, it fails. The test requires only the raw numerical data as input and outputs a table listing each column, its skewness value, and the pass/fail result. The skewness metric can theoretically range from negative to positive infinity, but in practice, values between -1 and 1 are generally considered acceptable, with values outside this range indicating substantial asymmetry that may warrant further attention.

The primary advantages of this test include its efficiency and clarity in identifying unequal data distributions that could undermine model assumptions or performance. The test is computationally lightweight, making it suitable for large datasets and routine data quality checks. Its adjustable threshold allows practitioners to tailor the sensitivity of the test to the specific requirements of their modeling context, accommodating both strict and lenient standards as needed. By providing a clear, quantitative measure of distributional asymmetry, the test enables early detection of potential data quality issues, supporting robust model development and risk mitigation. This is particularly valuable in regulated environments or applications where model interpretability and reliability are paramount.

It should be noted that the test is limited to evaluating only numeric columns, potentially overlooking skewness or bias present in categorical or text-based data. The underlying assumption that data should approximate a normal distribution may not always hold, especially in domains where naturally skewed distributions are expected or acceptable. The threshold for acceptable skewness is inherently subjective and may require iterative refinement based on domain expertise and empirical model performance. High skewness values, particularly those that persist across multiple columns or datasets, may signal underlying data quality challenges or violations of model assumptions, which could manifest as suboptimal predictions or biased outcomes.

This test shows the results in a tabular format, where each row corresponds to a numerical column in the dataset, and the columns display the variable name, its calculated skewness value, and the pass/fail outcome based on the threshold of 1. The skewness values are presented as real numbers, with negative values indicating left-skewed distributions and positive values indicating right-skewed distributions. The pass/fail column provides a clear binary assessment of whether each variable meets the symmetry criterion. Notably, the table covers a range of variables, including CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited. The skewness values observed range from -0.8455 to 1.2661, with most variables exhibiting values close to zero, indicating near-symmetrical distributions. Only the NumOfProducts column exceeds the threshold, with a skewness of 1.2661, resulting in a fail outcome. All other columns pass the test, suggesting that their distributions do not exhibit substantial asymmetry. The table format allows for straightforward comparison across variables, highlighting both the overall distributional characteristics and any specific deviations from the expected range.

The test results reveal the following key insights:

• Most Variables Exhibit Acceptable Symmetry: The majority of numerical columns, including CreditScore, Tenure, Balance, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, have skewness values well within the acceptable range of -1 to 1, indicating distributions that are approximately symmetrical.
• NumOfProducts Shows Substantial Right Skew: The NumOfProducts column has a skewness value of 1.2661, exceeding the threshold and indicating a pronounced right-skewed distribution, which is the only variable to fail the test.
• Range of Skewness Values Is Narrow: Excluding NumOfProducts, the skewness values for all other columns fall between -0.8455 and 0.1265, suggesting limited variation and a general tendency toward symmetry in the dataset.
• Binary Variables Display Minimal Skewness: Columns such as HasCrCard, IsActiveMember, and Exited, which are likely binary, show skewness values close to zero, reflecting balanced distributions or equal representation of categories.
• No Extreme Left Skew Observed: None of the columns display skewness values less than -1, indicating an absence of substantial left-skewed distributions in the dataset.

Based on these results, the dataset demonstrates a high degree of symmetry across most numerical variables, with only a single exception. The NumOfProducts column stands out with a skewness value above the threshold, indicating a distribution with a heavier right tail, which may reflect a concentration of observations at lower product counts and a smaller number of customers with higher product counts. This pattern is consistent with typical customer behavior in financial datasets, where most individuals hold a limited number of products. The remaining variables, including key financial and demographic indicators, exhibit skewness values close to zero, suggesting that their distributions are well-balanced and unlikely to introduce significant bias or instability into downstream modeling processes. The narrow range of skewness values across the dataset supports the conclusion that the data is generally well-behaved with respect to distributional symmetry, reducing the likelihood of model performance degradation due to skewed inputs. The clear pass/fail outcomes facilitate rapid identification of variables that may require further investigation or transformation, while the overall results indicate that the dataset is suitable for modeling approaches that assume or benefit from approximately normal input distributions.

Tables

Skewness Results for Dataset

Column	Skewness	Pass/Fail
CreditScore	-0.1129	Pass
Tenure	0.0310	Pass
Balance	-0.2973	Pass
NumOfProducts	1.2661	Fail
HasCrCard	-0.8455	Pass
IsActiveMember	0.1265	Pass
EstimatedSalary	-0.0191	Pass
Exited	0.0000	Pass

▶ Test Result: Unique Rows (validmind.data_validation.UniqueRows)

❌ Unique Rows

Unique Rows is designed to assess the diversity of the dataset by verifying that the number of unique rows in each column exceeds a specified threshold, thereby ensuring that the data used for model development is sufficiently varied. The primary purpose of this test is to identify whether the dataset contains enough distinct values across its features to support robust and unbiased model training, which is critical for generalization and reducing the risk of overfitting.

The test operates by first determining the total number of rows in the dataset and then calculating the number of unique values present in each column. For each column, the percentage of unique values is computed by dividing the number of unique values by the total row count and multiplying by 100 to express the result as a percentage. This percentage is then compared to a predefined minimum threshold, which serves as a benchmark for acceptable diversity. If a column’s percentage of unique values meets or exceeds the threshold, it is marked as passing; otherwise, it is marked as failing. The test provides a pass/fail verdict for each column, and the overall assessment is based on whether all columns meet the required standard. The typical range for the percentage of unique values is from 0% (no diversity) to 100% (all values unique), with higher percentages generally indicating greater diversity. Low percentages may signal redundancy or limited variability, which can impact model performance and generalizability.

The primary advantages of this test include its efficiency and systematic approach to evaluating data diversity across all columns in a dataset. By quantifying the uniqueness of each feature, the test quickly highlights areas where data may be lacking in variability, which is essential for developing models that are both effective and unbiased. This method is particularly useful in scenarios where data quality and representativeness are critical, such as in regulated environments or when building models intended for deployment in dynamic, real-world settings. The test’s straightforward metrics and clear pass/fail outcomes make it easy to interpret and act upon, supporting rapid data quality assessments during model development and validation cycles.

It should be noted that the Unique Rows test has several limitations. It assumes that higher uniqueness directly correlates with better data quality, which may not always be the case, especially in domains where certain repeated values are meaningful or required. The test treats all columns equally, without considering their relative importance or predictive power in the modeling process. This can lead to misleading interpretations, particularly for categorical variables with inherently limited unique values, such as binary indicators or demographic categories. Additionally, a low count of unique values in some columns may not necessarily indicate a problem if those features are not expected to be diverse. The test also does not account for the relationships between columns, focusing solely on univariate uniqueness, which may overlook more complex data quality issues.

This test shows the results in a tabular format, where each row corresponds to a column in the dataset and includes the column name, the number of unique values, the percentage of unique values relative to the total row count, and a pass/fail indicator. The table allows for straightforward comparison across columns, with the “Number of Unique Values” and “Percentage of Unique Values (%)” columns providing quantitative measures of diversity. The “Pass/Fail” column indicates whether each feature meets the prescribed uniqueness threshold. Notable observations from the table include a wide range in the percentage of unique values, from as low as 0.0619% for several binary or categorical columns to as high as 100% for the “EstimatedSalary” column. Columns such as “CreditScore” and “Balance” also exhibit relatively high uniqueness, while features like “Geography,” “Gender,” “NumOfProducts,” “HasCrCard,” “IsActiveMember,” and “Exited” show very low uniqueness and do not pass the test. The scale of values ranges from 2 unique values in binary columns up to 3,232 in “EstimatedSalary,” highlighting significant variability in feature diversity across the dataset.

The test results reveal the following key insights:

• High Uniqueness in Continuous Features: Columns such as “EstimatedSalary” (100.0%), “Balance” (68.7191%), and “CreditScore” (13.4901%) demonstrate substantial diversity, with “EstimatedSalary” containing a unique value for every row and “Balance” and “CreditScore” also showing high counts of unique values relative to the dataset size.
• Low Uniqueness in Categorical and Binary Features: Features like “Geography” (0.0928%), “Gender” (0.0619%), “NumOfProducts” (0.1238%), “HasCrCard” (0.0619%), “IsActiveMember” (0.0619%), and “Exited” (0.0619%) have very low percentages of unique values, reflecting their categorical or binary nature and resulting in a fail outcome for the uniqueness threshold.
• Intermediate Uniqueness in Tenure: The “Tenure” column, with 11 unique values and a percentage of 0.3403%, sits between the highly unique continuous features and the low-uniqueness categorical features, but still does not meet the threshold for passing.
• Distinct Pass/Fail Patterns by Feature Type: All continuous or quasi-continuous features pass the uniqueness test, while all categorical or binary features fail, indicating a clear pattern based on feature type.
• Wide Range of Unique Value Counts: The number of unique values spans from as few as 2 in binary columns to over 3,000 in “EstimatedSalary,” illustrating significant heterogeneity in feature diversity across the dataset.

Based on these results, the dataset exhibits a pronounced contrast in uniqueness between continuous and categorical features. Continuous variables such as “EstimatedSalary,” “Balance,” and “CreditScore” provide high levels of diversity, which supports robust model training and reduces the risk of overfitting for these features. In contrast, categorical and binary features consistently show low uniqueness, which is expected given their limited set of possible values but results in these columns failing the uniqueness threshold. This pattern suggests that while the dataset is well-equipped to capture nuanced variation in continuous attributes, it may be constrained in its ability to represent diversity in categorical dimensions. The clear separation in pass/fail outcomes by feature type highlights the importance of interpreting uniqueness metrics in the context of feature semantics and intended model use. Overall, the results indicate that the dataset’s diversity is primarily driven by its continuous features, with categorical features contributing limited variability, a characteristic that should be considered when evaluating the model’s capacity for generalization and its potential sensitivity to categorical feature distributions.

Tables

Column	Number of Unique Values	Percentage of Unique Values (%)	Pass/Fail
CreditScore	436	13.4901	Pass
Geography	3	0.0928	Fail
Gender	2	0.0619	Fail
Tenure	11	0.3403	Fail
Balance	2221	68.7191	Pass
NumOfProducts	4	0.1238	Fail
HasCrCard	2	0.0619	Fail
IsActiveMember	2	0.0619	Fail
EstimatedSalary	3232	100.0000	Pass
Exited	2	0.0619	Fail

▶ Test Result: Too Many Zero Values (validmind.data_validation.TooManyZeroValues)

❌ Too Many Zero Values

Too Many Zero Values is designed to identify numerical columns within a dataset that contain an excessive proportion of zero values, with the primary purpose of highlighting potential data sparsity or lack of variation that could impact the effectiveness of machine learning models. By flagging columns where the percentage of zero values exceeds a predefined threshold, this test supports the detection of features that may not contribute meaningful information to predictive modeling or may indicate underlying data quality issues.

The test operates by systematically evaluating each numerical column in the dataset, calculating both the absolute count and the percentage of zero values relative to the total number of rows. For each column, the test compares the computed percentage of zero values to a preset threshold, which is set by default at 0.03%. If the proportion of zeros in a column surpasses this threshold, the column is marked as having failed the test. The output includes, for each numerical variable, the total row count, the number of zero values, the percentage of zero values, and a pass/fail status. The percentage metric ranges from 0% to 100%, where higher values indicate a greater prevalence of zeros. A low percentage suggests that zeros are rare and likely not problematic, while a high percentage may signal data sparsity or a lack of feature variability. The test is designed to be flexible, allowing the threshold to be adjusted to suit the specific requirements of different analyses or modeling contexts.

The primary advantages of this test include its ability to efficiently surface columns with a high concentration of zero values, which might otherwise be overlooked in large or complex datasets. By providing both the count and percentage of zeros, the test enables a nuanced understanding of the distribution of zero values within each feature. This dual reporting facilitates targeted data quality assessments and supports informed feature selection decisions. The test’s focus on numerical columns ensures that it is applied only where zeros are meaningful, reducing the risk of misinterpretation that could arise from analyzing categorical or non-numeric data. Additionally, the customizable threshold parameter allows practitioners to tailor the sensitivity of the test to the specific needs of their modeling objectives or domain requirements.

It should be noted that this test is limited to detecting the prevalence of zero values and does not account for other potentially problematic data characteristics, such as extreme values, missing data, or outliers. The test does not evaluate the context or meaning of zeros within a given feature, which may be entirely appropriate or expected in some cases. As a result, columns flagged as having too many zeros may not necessarily be unsuitable for modeling, and further domain-specific analysis may be required. The test also does not identify patterns or sequences of zeros, which could be relevant in time-series or longitudinal data. Furthermore, the exclusive focus on numerical columns means that issues in categorical or text-based features are not addressed. High ratios of zero values, especially those approaching or exceeding 100%, are considered high risk as they indicate a lack of variation and potential redundancy in the feature.

This test shows the results in a tabular format, where each row corresponds to a numerical variable from the dataset and columns display the variable name, total row count, number of zero values, percentage of zero values, and the pass/fail status based on the 0.03% threshold. To interpret the table, one should examine the "Percentage of Zero Values (%)" column to assess the prevalence of zeros in each feature, with higher percentages indicating greater sparsity. The "Pass/Fail" column provides a direct indication of whether the feature exceeds the threshold. In this particular output, all four variables—Tenure, Balance, HasCrCard, and IsActiveMember—are shown with their respective row counts of 3,232. The number of zero values ranges from 132 for Tenure to 1,718 for IsActiveMember. The corresponding percentages of zero values are 4.08% for Tenure, 31.28% for Balance, 30.54% for HasCrCard, and 53.16% for IsActiveMember. All variables are marked as "Fail," indicating that each exceeds the 0.03% threshold. Notably, IsActiveMember has the highest proportion of zeros, exceeding half the dataset, while Tenure, though lower, still surpasses the threshold by a significant margin. The table provides a clear, quantitative summary of zero value prevalence across key numerical features, enabling straightforward comparison and assessment.

The test results reveal the following key insights:

• All Numerical Features Exceed Zero Threshold: Every numerical column assessed—Tenure, Balance, HasCrCard, and IsActiveMember—has a percentage of zero values that surpasses the 0.03% threshold, resulting in a "Fail" status for each.
• Substantial Zero Value Prevalence in IsActiveMember: The IsActiveMember variable exhibits the highest proportion of zero values at 53.16%, indicating that more than half of the entries for this feature are zeros.
• High Zero Concentration in Balance and HasCrCard: Both Balance and HasCrCard display similar and notably high percentages of zero values, at 31.28% and 30.54% respectively, suggesting a significant portion of the dataset contains zeros for these features.
• Lower, Yet Significant, Zero Rate in Tenure: While Tenure has a lower percentage of zero values at 4.08% compared to the other features, it still exceeds the threshold by a wide margin and is flagged as failing the test.
• Consistent Row Counts Across Features: All variables are evaluated over the same total row count of 3,232, ensuring comparability of zero value proportions across features.

Based on these results, the dataset’s numerical features are characterized by a consistently high prevalence of zero values, with all assessed columns exceeding the established threshold for acceptable zero value frequency. The IsActiveMember feature stands out with over half of its entries being zeros, indicating a pronounced lack of variation that could limit its utility in predictive modeling or signal a structural aspect of the data collection process. Balance and HasCrCard also demonstrate substantial zero value concentrations, each with nearly one-third of their values as zeros, which may reflect underlying business processes or data encoding conventions. Tenure, while exhibiting a lower rate, still presents a non-trivial proportion of zeros relative to the threshold. The uniformity in row counts across features supports direct comparison and highlights that the observed patterns are not artifacts of differing sample sizes. Collectively, these observations suggest that zero values are a prominent characteristic of the dataset’s numerical features, which may have implications for model interpretability, feature selection, and downstream analytical tasks. The results provide a clear, quantitative basis for understanding the distribution of zero values and their potential impact on modeling efforts.

Tables

Variable	Row Count	Number of Zero Values	Percentage of Zero Values (%)	Pass/Fail
Tenure	3232	132	4.0842	Fail
Balance	3232	1011	31.2809	Fail
HasCrCard	3232	987	30.5384	Fail
IsActiveMember	3232	1718	53.1559	Fail

▶ Test Result: IQR Outliers Table (validmind.data_validation.IQROutliersTable)

IQR Outliers Table

IQR Outliers Table is designed to identify and summarize outliers in numerical features of a dataset using the Interquartile Range (IQR) method. The primary purpose of this test is to detect data points that deviate significantly from the central tendency of each variable, as these outliers can influence statistical analyses and model performance. By systematically quantifying and describing the extent and characteristics of outliers, the test supports robust data pre-processing and quality assessment.

The test operates by calculating the first quartile (25th percentile) and third quartile (75th percentile) for each numerical feature in the dataset, then determining the IQR as the difference between these two values. Outliers are defined as data points that fall below the lower bound (first quartile minus 1.5 times the IQR) or above the upper bound (third quartile plus 1.5 times the IQR). For each feature, the test counts the number of outliers and provides summary statistics for these outlier values, including the minimum, 25th percentile, median, 75th percentile, and maximum. The test is applicable to all numerical features unless specific variables are selected, and the outlier threshold can be adjusted as needed. The output typically includes a table listing each variable, the total count of outliers, the mean value of the variable, and the distribution of outlier values. The number of outliers can range from zero to the total number of observations, with higher counts indicating greater deviation from typical values. Outlier values that are far from the mean or the main data distribution may signal data quality issues or unusual patterns.

The primary advantages of this test include its robustness to extreme values, as the IQR method is based on quartile calculations rather than mean and standard deviation, making it less sensitive to the presence of outliers themselves. The test provides a clear and comprehensive summary of outlier characteristics for each numerical feature, enabling users to quickly identify variables that may require further investigation or special handling. Its flexibility allows for customization of both the features analyzed and the threshold for outlier detection, making it suitable for a wide range of datasets and use cases. This approach is particularly useful in scenarios where data quality is critical, such as in regulated environments or when preparing data for machine learning models that are sensitive to outliers.

It should be noted that the IQR Outliers Table has certain limitations. The method may produce false positives in variables with non-normal or highly skewed distributions, as the IQR-based thresholds may not align with the natural variability of such data. The test does not provide guidance on how to address detected outliers, leaving interpretation and remediation to the user. It is only applicable to numerical features, excluding categorical variables from analysis. Additionally, the default threshold may not be optimal for all datasets, particularly those with heavy tails or that have undergone significant pre-processing. Signs of high risk include a large number of outliers across multiple features, outliers that are substantially distant from the mean, and extremely high or low values that may indicate data entry errors or other quality concerns.

This test shows the results in a tabular format, presenting a summary of outliers detected by the IQR method for each numerical variable. The table includes columns for the variable name, total count of outliers, mean value of the variable, and the minimum, 25th percentile, median, 75th percentile, and maximum values among the outliers. For example, the "CreditScore" variable has 9 outliers, with a mean value of 647.61, and outlier values ranging from a minimum of 350 to a maximum of 373. The quartile values for these outliers are tightly clustered, with the 25th and 50th percentiles at 350 and the 75th percentile at 365, indicating that most outliers are at the lower end of the distribution. In contrast, the "NumOfProducts" variable has 44 outliers, all with a value of 4, as indicated by the minimum, quartiles, and maximum all being 4. The mean value of this variable is 1.50, suggesting that the outlier value of 4 is significantly higher than the typical value. The table allows users to quickly assess the prevalence and characteristics of outliers in each feature, compare the distribution of outlier values, and identify variables with concentrated or dispersed outlier patterns. The scale of the values is determined by the original units of each variable, and the table provides a clear reference for further analysis or data cleaning.

The test results reveal the following key insights:

• Outliers Are Concentrated in Specific Features: Only two numerical variables, "CreditScore" and "NumOfProducts," exhibit outliers according to the IQR method, indicating that outlier presence is not widespread across all features.
• CreditScore Outliers Are Low and Clustered: The 9 outliers in "CreditScore" are all at the lower end of the distribution, with values tightly grouped between 350 and 373, and quartile values indicating little variation among these outliers.
• NumOfProducts Outliers Are Uniform and High: All 44 outliers for "NumOfProducts" have the same value of 4, which is substantially higher than the mean of 1.50 for this variable, suggesting a distinct separation between typical and outlier values.
• Magnitude of Outliers Differs by Feature: While "CreditScore" outliers are relatively close to each other, "NumOfProducts" outliers are uniform, highlighting different outlier behaviors between features.
• Potential Data Quality Patterns: The clustering of "CreditScore" outliers at the minimum value and the uniformity of "NumOfProducts" outliers may indicate specific data entry patterns or business rules affecting these variables.

Based on these results, the outlier analysis using the IQR method reveals that outlier presence is limited to specific numerical features, with "NumOfProducts" showing a notably higher count and uniformity of outlier values compared to "CreditScore." The tight clustering of "CreditScore" outliers at the lower end suggests a possible threshold effect or a group of observations with unusually low scores, while the uniformity of "NumOfProducts" outliers at the maximum observed value points to a potential cap or business rule influencing this variable. The mean values for both variables further emphasize the separation between typical and outlier values, with outliers in "NumOfProducts" being significantly higher than the average. These patterns provide a clear characterization of the outlier landscape in the dataset, highlighting areas where data distribution deviates from the norm and where further scrutiny may be warranted to understand the underlying causes. The results offer a detailed quantitative basis for understanding the distribution and nature of outliers, supporting informed decisions about subsequent data processing or model development steps.

Tables

Summary of Outliers Detected by IQR Method

Variable	Total Count of Outliers	Mean Value of Variable	Minimum Outlier Value	Outlier Value at 25th Percentile	Outlier Value at 50th Percentile	Outlier Value at 75th Percentile	Maximum Outlier Value
CreditScore	9	647.6074	350	350.0	350.0	365.0	373
NumOfProducts	44	1.4988	4	4.0	4.0	4.0	4

▶ Test Result: IQR Outliers Bar Plot (validmind.data_validation.IQROutliersBarPlot)

IQR Outliers Bar Plot

IQR Outliers Bar Plot is designed to visually analyze and evaluate the extent and distribution of outliers in numeric variables using the Interquartile Range (IQR) method. Its primary purpose is to clarify the dataset’s distribution, flag possible abnormalities, and gauge potential risks associated with processing skewed data, which may impact the predictive performance of machine learning models.

The test operates by systematically examining each numeric feature in the dataset. For each feature, it calculates the 25th percentile (Q1) and the 75th percentile (Q3) to determine the interquartile range, which is the difference between Q3 and Q1. Using a default threshold multiplier of 1.5, the test establishes lower and upper bounds for typical values. Any data point falling below the lower bound or above the upper bound is classified as an outlier. The test then segments the data into four percentile ranges: 0-25, 25-50, 50-75, and 75-100. It counts the number of outliers within each percentile segment and visualizes these counts in a bar plot for each feature. The resulting plots provide a clear depiction of where outliers are concentrated within the distribution. The outlier count is a non-negative integer, and higher values in upper percentiles may indicate the presence of extreme values that could disproportionately influence model behavior. The interpretation of these results depends on the context, but generally, a concentration of outliers in higher percentiles signals a potential risk for model robustness.

The primary advantages of this test include its ability to effectively identify and visualize outliers in numeric data, making it easier to comprehend the distribution and potential impact of these outliers on model performance. The method is flexible, as it can be applied to all numeric features or a selected subset, and is agnostic to the specific modeling task, making it suitable for both classification and regression problems. Its computational efficiency allows it to handle large datasets without significant performance overhead. By providing a visual summary, the test supports rapid assessment of data quality and highlights features that may require further investigation or preprocessing before modeling.

It should be noted that the test is limited to numerical variables and does not extend to categorical data. While it reveals the presence and distribution of outliers, it does not directly assess their impact on model predictive performance. The underlying assumption that the data is unimodal and symmetric may not always hold, especially in cases of non-normal distributions, which can lead to misleading interpretations. High concentrations of outliers, particularly in the upper percentiles, may indicate the presence of extreme values that could skew model training and predictions. Additionally, if a feature contains a majority of its values as outliers, it may not contribute positively to the model and could introduce instability.

This test shows the results as a series of bar plots, each corresponding to a numeric feature in the dataset. The x-axis of each plot represents the percentile ranges (0-25, 25-50, 50-75, 75-100), while the y-axis indicates the count of outliers detected within each range. For the feature "CreditScore," the plot displays zero outliers in the lower half (0-50 percentiles), with a notable increase to six outliers in the 50-75 percentile range and three outliers in the 75-100 percentile range. This suggests that outliers are concentrated in the upper half of the distribution, with a peak in the 50-75 percentile segment. For the feature "NumOfProducts," the plot shows a starkly different pattern: there are no outliers in the first three percentile ranges (0-75), but a significant spike to 44 outliers in the 75-100 percentile range. This indicates that outliers for this feature are almost exclusively present in the highest quartile, suggesting the presence of extreme values at the upper end of the distribution. The bar plots are straightforward to interpret, with each bar’s height directly reflecting the number of outliers in the corresponding percentile range. The scale of the y-axis varies by feature, allowing for direct comparison of outlier prevalence within each feature but not across features unless normalized.

The test results reveal the following key insights:

• Outliers Concentrated in Upper Percentiles for Both Features: Both "CreditScore" and "NumOfProducts" exhibit outliers predominantly in the upper half of their distributions, with no outliers detected in the lower 50 percentiles for either feature.
• Distinct Outlier Distribution Patterns Between Features: "CreditScore" shows a moderate number of outliers in the 50-75 percentile range (6) and a smaller number in the 75-100 range (3), while "NumOfProducts" has a pronounced concentration of outliers exclusively in the 75-100 percentile range (44).
• Absence of Outliers in Lower Percentiles: Both features have zero outliers in the 0-25 and 25-50 percentile ranges, indicating that lower values for these features are within expected bounds and do not exhibit abnormal behavior.
• Extreme Values in "NumOfProducts": The sharp increase to 44 outliers in the top quartile for "NumOfProducts" suggests the presence of a substantial number of extreme values at the upper end, which could have a significant impact on model training and predictions if not addressed.
• Moderate Outlier Presence in "CreditScore": The distribution of outliers in "CreditScore" is less extreme, with a gradual increase in the upper percentiles, indicating a more balanced but still notable presence of higher-end outliers.

Based on these results, the data exhibits clear patterns in outlier distribution across the examined numeric features. "CreditScore" demonstrates a moderate accumulation of outliers in the upper half of its distribution, with the majority appearing between the 50th and 100th percentiles, suggesting that higher credit scores are more likely to be flagged as outliers. In contrast, "NumOfProducts" displays a highly concentrated outlier presence exclusively in the top quartile, indicating that only the highest values for this feature deviate significantly from the expected range. The absence of outliers in the lower percentiles for both features suggests that lower values are well-behaved and conform to the anticipated distribution. These patterns highlight the importance of monitoring and potentially addressing the influence of upper-end outliers, particularly for "NumOfProducts," where the magnitude and exclusivity of outliers in the highest percentile may introduce instability or bias in downstream modeling. The results provide a clear, visual summary of where outlier risks are most pronounced, supporting targeted data quality assessments and informing further analysis of feature behavior within the modeling process.

Figures

▶ Test Result: Descriptive Statistics (validmind.data_validation.DescriptiveStatistics)

Descriptive Statistics

Descriptive Statistics is designed to provide a comprehensive summary of both numerical and categorical variables within a dataset, offering a detailed overview of the data’s distribution, central tendency, and variability. The primary purpose of this test is to facilitate a clear understanding of the dataset’s structure, highlight the main characteristics of each variable, and support the assessment of the model’s behavior and potential performance by identifying patterns, anomalies, and potential data quality issues.

The test operates by applying established statistical functions to the dataset, specifically leveraging the describe() function for numerical variables and the value_counts() function for categorical variables. For numerical data, the test calculates key metrics such as count, mean, standard deviation, minimum, maximum, and several percentiles (25th, 50th, 75th, 90th, and 95th), which collectively describe the central tendency, spread, and range of the data. The mean provides the average value, while the standard deviation quantifies the amount of variation or dispersion. Percentiles indicate the value below which a given percentage of observations fall, with the 50th percentile representing the median. For categorical variables, the test reports the total count, the number of unique categories, the most frequent category (top value), its frequency, and the proportion of this top value relative to the total. These metrics help to identify the diversity and dominance of categories within each variable. The values for these metrics are typically non-negative, with proportions ranging from 0% to 100%. High proportions for a single category or large differences between mean and median can signal potential data risks such as lack of diversity or skewness.

The primary advantages of this test include its ability to quickly and effectively summarize large and complex datasets, making it easier to detect broad patterns, outliers, and potential data quality issues. By providing a side-by-side comparison of both numerical and categorical variables, the test enables users to assess the overall health and representativeness of the data. This is particularly useful in scenarios where understanding the underlying data distribution is critical for model development, validation, and monitoring. The test’s versatility allows it to be applied across a wide range of datasets and domains, and its robust statistical summaries can highlight important anomalies such as extreme values, skewed distributions, or imbalanced categories, all of which are essential for informed model risk management and regulatory compliance.

It should be noted that while this test offers a thorough high-level overview of the dataset, it does not capture relationships or dependencies between variables, nor does it detect subtle or complex patterns that may influence model performance. The test is limited to univariate analysis, meaning it examines each variable independently. As a result, it cannot identify multivariate outliers or interactions that may be critical in certain modeling contexts. Additionally, the test does not provide predictive insights about future or unseen data, and its results should be interpreted in conjunction with other statistical analyses. Signs of high risk, such as significant differences between mean and median (indicating skewness), large standard deviations (suggesting outliers), or a high proportion of a single category in categorical variables (indicating lack of diversity), should be carefully considered, as they may impact model fairness, stability, and generalizability.

This test shows the results in two tabular formats: one summarizing numerical variables and the other summarizing categorical variables. The numerical variables table presents each variable’s name, count of observations, mean, standard deviation, minimum and maximum values, and several percentiles (25th, 50th, 75th, 90th, and 95th). This allows the reader to assess the central tendency, spread, and range of each variable. For example, the “CreditScore” variable has a mean of 647.6, a standard deviation of 98.1, and ranges from 350 to 850, with the 50th percentile (median) at 650. The “Balance” variable shows a mean of 82,641.6, a standard deviation of 61,121.2, and a wide range from 0 to 250,898, with the 50th percentile at 103,755. Categorical variables are summarized in a separate table, listing each variable’s name, total count, number of unique values, the most frequent category, its frequency, and the percentage this represents of the total. For instance, “Geography” has three unique values, with “France” being the most common at 45.95% of the data, while “Gender” is split between two categories, with “Male” at 51.11%. These tables provide a clear, structured view of the dataset’s composition, making it straightforward to identify the distribution, diversity, and potential dominance of specific values or categories.

The test results reveal the following key insights:

• Numerical variables exhibit broad ranges and varying degrees of dispersion: Variables such as “CreditScore” and “EstimatedSalary” display wide ranges, with “CreditScore” spanning from 350 to 850 and “EstimatedSalary” from 12 to 199,953. Standard deviations are substantial, particularly for “Balance” (61,121.2) and “EstimatedSalary” (57,452.1), indicating significant variability within these features.
• Central tendency and skewness are evident in several variables: For “Balance,” the mean (82,641.6) is notably lower than the median (103,755), suggesting a left-skewed distribution with a substantial proportion of zero balances. “Tenure” and “NumOfProducts” show medians equal to their means, indicating more symmetric distributions.
• Categorical variables show limited diversity and category dominance: “Geography” is dominated by “France” (45.95%), while “Gender” is nearly evenly split, with “Male” at 51.11%. The number of unique values is low for both variables, with three for “Geography” and two for “Gender,” indicating limited categorical diversity.
• Binary variables are prevalent and display imbalanced distributions: Variables such as “HasCrCard” and “IsActiveMember” are binary, with “HasCrCard” showing a mean of 0.6946 (indicating 69.46% have a credit card) and “IsActiveMember” at 0.4684 (46.84% active members), reflecting moderate imbalance in these features.
• Percentile analysis highlights concentration and outlier potential: The 95th percentile for “Balance” (164,540) and “EstimatedSalary” (189,132) is close to their respective maximums, suggesting the presence of high-value outliers. The 25th and 75th percentiles for most variables are well separated, indicating a broad spread in the data.

Based on these results, the dataset demonstrates a mix of well-distributed and skewed variables, with numerical features such as “Balance” and “EstimatedSalary” showing substantial variability and evidence of outliers, as indicated by high standard deviations and large gaps between percentiles. The presence of a significant proportion of zero values in “Balance” and the dominance of specific categories in “Geography” and “Gender” suggest that certain features may have limited diversity or be subject to class imbalance. Binary variables like “HasCrCard” and “IsActiveMember” are not perfectly balanced, which could influence model sensitivity to these features. The overall structure of the data, as revealed by the descriptive statistics, indicates that while most variables cover a broad range and provide sufficient granularity, some features may require further scrutiny for skewness, outlier impact, or lack of diversity. These characteristics are important for understanding the model’s potential behavior, as they can affect model stability, fairness, and generalizability across different segments of the data.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	3232.0	647.6074	98.0944	350.0	580.0	650.0	716.0	777.0	812.0	850.0
Tenure	3232.0	4.9985	2.9052	0.0	2.0	5.0	8.0	9.0	9.0	10.0
Balance	3232.0	82641.5746	61121.2268	0.0	0.0	103755.0	129682.0	150351.0	164540.0	250898.0
NumOfProducts	3232.0	1.4988	0.6676	1.0	1.0	1.0	2.0	2.0	3.0	4.0
HasCrCard	3232.0	0.6946	0.4606	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	3232.0	0.4684	0.4991	0.0	0.0	0.0	1.0	1.0	1.0	1.0
EstimatedSalary	3232.0	100558.4368	57452.1222	12.0	51711.0	101049.0	150000.0	179539.0	189132.0	199953.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	3232.0	3.0	France	1485.0	45.95
Gender	3232.0	2.0	Male	1652.0	51.11

▶ Test Result: Pearson Correlation Matrix (validmind.data_validation.PearsonCorrelationMatrix)

Pearson Correlation Matrix

Pearson Correlation Matrix is designed to evaluate the extent of linear dependency between all pairs of numerical variables in a dataset. Its primary purpose is to identify potential redundancy by quantifying the strength and direction of linear relationships, thereby supporting dimensionality reduction and improving model interpretability by highlighting variables that may not contribute unique information.

The test operates by calculating the Pearson correlation coefficient for every pair of numerical variables in the dataset. This coefficient measures the degree to which two variables move together in a linear fashion, with values ranging from -1 to 1. A value of 1 indicates a perfect positive linear relationship, -1 indicates a perfect negative linear relationship, and 0 indicates no linear relationship. The test compiles these coefficients into a correlation matrix, which is then visualized as a heat map. The heat map uses color gradients to represent the magnitude and direction of each correlation, with a specific highlight (white) for coefficients whose absolute value exceeds 0.7, signaling a high degree of correlation. The matrix is symmetric, with each cell representing the correlation between a pair of variables, and the diagonal always showing a value of 1, as each variable is perfectly correlated with itself. This approach allows for rapid identification of both strong and weak linear relationships, supporting further analysis of variable redundancy and potential multicollinearity.

The primary advantages of this test include its ability to provide a clear, quantitative assessment of linear relationships between variables, which is essential for identifying redundant features that may unnecessarily complicate a model. The heat map visualization makes it accessible to a wide range of users, including those less comfortable with raw numerical matrices, by offering an intuitive overview of the correlation structure. This test is particularly useful in scenarios where model simplicity and interpretability are priorities, as it helps to streamline feature selection and reduce the risk of overfitting by flagging variables that do not contribute independent information. Additionally, the test supports robust model development by ensuring that highly correlated variables are identified early in the modeling process.

It should be noted that the Pearson Correlation Matrix is limited to detecting linear relationships and may not capture more complex, non-linear dependencies between variables. This means that some forms of redundancy or interaction may go undetected if they do not manifest as linear associations. The test also does not measure the strength of causal influence, only the degree of co-movement. The threshold of 0.7 for highlighting high correlations is somewhat arbitrary and may not be appropriate for all datasets or modeling contexts. Furthermore, a large number of highly correlated variables can indicate redundancy and increase the risk of overfitting, but the test does not provide guidance on which variables to remove or retain. Interpretation challenges may arise if users conflate correlation with causation or overlook important non-linear relationships.

This test shows a heat map of the Pearson correlation coefficients for all pairs of numerical variables in the dataset. The heat map is structured as a square matrix, with both axes listing the variable names in the same order. Each cell in the matrix represents the correlation coefficient between the corresponding pair of variables, with the color indicating the strength and direction of the relationship: blue shades for positive correlations, red shades for negative correlations, and white for coefficients with an absolute value above 0.7. The diagonal cells, which compare each variable to itself, are always dark blue, indicating perfect correlation (value of 1). The color bar on the right provides a reference for interpreting the color scale, ranging from -1 (strong negative correlation) to 1 (strong positive correlation). The numerical values within the cells further clarify the exact strength of each relationship. Notably, the matrix reveals that most variable pairs have correlation coefficients close to zero, indicating weak or negligible linear relationships. The only moderate correlation observed is between "Balance" and "NumOfProducts" at -0.16, which is still well below the high-correlation threshold. No cell is highlighted in white, confirming the absence of strong linear dependencies. The overall distribution of values is tightly clustered around zero, with no significant outliers or regions of high correlation.

The test results reveal the following key insights:

• No Strong Linear Dependencies Detected: All off-diagonal correlation coefficients fall well below the 0.7 threshold in absolute value, indicating an absence of strong linear relationships between any pair of variables.
• Predominance of Weak Correlations: Most variable pairs exhibit correlation coefficients close to zero, with values typically ranging from -0.2 to 0.2, suggesting that the variables are largely independent in a linear sense.
• Moderate Negative Correlation Between Balance and NumOfProducts: The most notable relationship is a moderate negative correlation of -0.16 between "Balance" and "NumOfProducts," which, while not strong, may warrant further exploration in the context of the business domain.
• Symmetry and Consistency Across the Matrix: The matrix is symmetric, and the diagonal values are all 1, as expected, confirming the correct computation and visualization of the correlation structure.
• No Evidence of Redundancy or Multicollinearity: The lack of high correlation coefficients suggests that the dataset does not contain redundant numerical variables that would pose a risk of multicollinearity or unnecessary complexity.

Based on these results, the dataset demonstrates a high degree of linear independence among its numerical variables, with no pairs exhibiting strong positive or negative correlations. This pattern suggests that each variable is likely contributing unique information to the model, reducing the risk of redundancy and multicollinearity. The absence of high-correlation regions in the heat map supports the conclusion that dimensionality reduction through the removal of highly correlated variables is not necessary in this case. The moderate negative correlation between "Balance" and "NumOfProducts" is the only relationship of note, but its magnitude is insufficient to raise concerns about redundancy. Overall, the correlation structure observed here is favorable for model development, as it supports the inclusion of all numerical variables without introducing significant risks related to overfitting or loss of interpretability due to overlapping information. The results provide a clear and objective basis for proceeding with further modeling steps, confident that the numerical features are not linearly redundant.

Figures

▶ Test Result: High Pearson Correlation (validmind.data_validation.HighPearsonCorrelation)

✅ High Pearson Correlation

High Pearson Correlation is designed to identify pairs of features within a dataset that exhibit strong linear relationships, with the primary purpose of detecting potential feature redundancy or multicollinearity. This is crucial for ensuring that the predictive model remains interpretable and robust, as highly correlated features can obscure the true influence of individual variables and may lead to overfitting or instability in model estimates.

The test operates by calculating the Pearson correlation coefficient for every possible pair of features in the dataset. The Pearson correlation coefficient is a statistical measure that quantifies the strength and direction of a linear relationship between two continuous variables, ranging from -1 (perfect negative linear relationship) to +1 (perfect positive linear relationship), with 0 indicating no linear relationship. The test systematically computes these coefficients for all feature pairs, removes self-correlations and duplicate pairs, and then sorts the results by the absolute value of the coefficient. A pre-defined threshold, set at 0.3 in this case, is used to determine whether a pair is considered highly correlated. Each pair is assigned a "Pass" if the absolute value of the coefficient is below the threshold, or a "Fail" if it exceeds the threshold. The test then returns the top n pairs with the strongest correlations, providing a clear view of the most significant linear relationships present in the data.

The primary advantages of this test include its efficiency and transparency in highlighting linear dependencies between features, which is particularly valuable during the early stages of model development and risk assessment. By surfacing the most strongly correlated feature pairs, the test enables practitioners to quickly identify and address potential sources of multicollinearity, which can otherwise compromise model interpretability and predictive stability. The clear tabular output, which lists feature pairs alongside their correlation coefficients and pass/fail status, facilitates straightforward communication of results to both technical and non-technical stakeholders. This makes the test especially useful in regulated environments where model transparency and documentation are paramount.

It should be noted that the test is limited to detecting only linear relationships, as measured by the Pearson correlation coefficient, and does not capture more complex or nonlinear dependencies that may exist among features. The metric is also sensitive to outliers, which can disproportionately influence the calculated coefficients and potentially mask or exaggerate true relationships. Additionally, the test focuses exclusively on pairwise correlations, meaning it may overlook higher-order interactions or dependencies involving three or more features. High correlation coefficients, particularly those exceeding the threshold, are indicative of potential multicollinearity, which can undermine the reliability of model parameter estimates and complicate the interpretation of individual feature effects.

This test shows its results in a tabular format, where each row represents a unique pair of features from the dataset. The columns include the feature pair, the calculated Pearson correlation coefficient, and a pass/fail status based on whether the absolute value of the coefficient exceeds the threshold of 0.3. The coefficients are presented as decimal values, typically ranging from -1 to 1, with negative values indicating inverse relationships and positive values indicating direct relationships. In the provided results, all feature pairs have coefficients well below the threshold, with the highest absolute value being -0.1984 for the pair (IsActiveMember, Exited). The table is sorted by the strength of the correlation, allowing for quick identification of the most strongly related pairs. Notably, all pairs in this output are marked as "Pass," indicating that none of the observed linear relationships are strong enough to warrant immediate concern for multicollinearity under the current threshold. The range of coefficients spans from -0.1984 to 0.0331, suggesting generally weak linear associations among the top feature pairs. This output provides a clear and interpretable summary of the linear dependencies present in the dataset, with no values approaching the threshold that would indicate a high risk of redundancy.

The test results reveal the following key insights:

• No Feature Pairs Exceed Correlation Threshold: All observed Pearson correlation coefficients are below the set threshold of 0.3, with the highest absolute value being -0.1984, indicating an absence of strong linear relationships among the top feature pairs.
• Weak Linear Associations Dominate: The majority of feature pairs exhibit weak correlations, with coefficients ranging from -0.1984 to 0.0331, suggesting that the features are largely independent in terms of linear relationships.
• Negative Correlations Slightly More Pronounced: The strongest correlations observed are negative, such as between IsActiveMember and Exited (-0.1984) and Balance and NumOfProducts (-0.1565), indicating mild inverse relationships in these cases.
• Pass Status Uniform Across All Pairs: Every feature pair in the top ten is marked as "Pass," reflecting that none of the relationships approach the threshold for high correlation and thus do not signal immediate multicollinearity concerns.
• Distribution of Correlations Across Feature Types: The feature pairs include a mix of behavioral, financial, and demographic variables, yet none display strong linear dependencies, highlighting the diversity and relative independence of the dataset's features.

Based on these results, the dataset demonstrates a low degree of linear association among its primary features, as evidenced by the uniformly low Pearson correlation coefficients and the absence of any pairs exceeding the 0.3 threshold. The weak correlations observed suggest that the features are not redundant in a linear sense and are unlikely to introduce multicollinearity into subsequent modeling efforts. The negative correlations, while present, are modest and do not approach levels that would compromise model interpretability or stability. The consistent "Pass" status across all top feature pairs further supports the conclusion that the dataset's structure is conducive to robust model development, with minimal risk of linear redundancy affecting parameter estimation or predictive performance. These observations collectively indicate that the feature set is well-suited for use in predictive modeling applications where independence among predictors is desirable, and that the risk of linear multicollinearity is minimal under the current configuration and threshold.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1984	Pass
(Balance, NumOfProducts)	-0.1565	Pass
(Balance, Exited)	0.1376	Pass
(NumOfProducts, IsActiveMember)	0.0575	Pass
(HasCrCard, IsActiveMember)	-0.0426	Pass
(NumOfProducts, Exited)	-0.0399	Pass
(Tenure, IsActiveMember)	-0.0390	Pass
(Tenure, EstimatedSalary)	0.0331	Pass
(Balance, HasCrCard)	-0.0268	Pass
(HasCrCard, EstimatedSalary)	-0.0245	Pass

Run and log an individual test

Next, we'll use the previously initialized vm_balanced_raw_dataset (that still has a highly correlated Age column) as input to run an individual test, then log the result to the ValidMind Platform.

When running individual tests, you can use a custom result_id to tag the individual result with a unique identifier:

• This result_id can be appended to test_id with a : separator.
• The balanced_raw_dataset result identifier will correspond to the balanced_raw_dataset input, the dataset that still has the Age column.

result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)
result.log()

❌ High Pearson Correlation Balanced Raw Dataset

High Pearson Correlation is designed to identify highly correlated feature pairs in a dataset, with the primary purpose of detecting potential feature redundancy or multicollinearity. This is crucial for ensuring that the features used in a machine learning model do not exhibit strong linear relationships that could compromise model interpretability or performance. By systematically evaluating the linear associations between all pairs of features, the test provides transparency into the structure of the dataset and highlights areas where further feature engineering or selection may be warranted.

The test operates by calculating the Pearson correlation coefficient for every possible pair of features in the dataset. The Pearson correlation coefficient is a statistical measure that quantifies the strength and direction of a linear relationship between two continuous variables, ranging from -1 (perfect negative linear relationship) to 1 (perfect positive linear relationship), with 0 indicating no linear relationship. The test removes self-correlations and duplicate pairs, then sorts the results by the absolute value of the correlation coefficient. Each pair is evaluated against a predefined threshold (in this case, 0.3), and a Pass or Fail status is assigned depending on whether the absolute value of the coefficient exceeds this threshold. The test outputs the top n strongest correlations, providing a clear view of the most significant linear relationships present in the data. This approach enables users to quickly identify pairs of features that may introduce multicollinearity or redundancy, which can affect model stability and interpretability.

The primary advantages of this test include its simplicity and transparency in revealing linear dependencies between features. It provides a direct and interpretable output that lists the most correlated feature pairs, their correlation coefficients, and a clear Pass or Fail status based on the specified threshold. This makes it particularly useful in the early stages of data exploration and preprocessing, where understanding the relationships between variables is essential for effective feature selection and engineering. The test is also computationally efficient, making it suitable for large datasets, and its results can be easily communicated to both technical and non-technical stakeholders. By highlighting potential multicollinearity, the test supports the development of more robust and interpretable models.

It should be noted that the test is limited to detecting linear relationships and does not capture nonlinear dependencies between features. As a result, important associations that are not linear in nature may go undetected. The Pearson correlation coefficient is also sensitive to outliers, which can distort the measure and lead to misleading conclusions about the strength of relationships. Additionally, the test only considers pairwise relationships and does not account for more complex interactions involving three or more variables. High correlation coefficients, particularly those exceeding the threshold, may indicate a risk of multicollinearity, which can undermine the reliability of model coefficients and reduce interpretability. Care must be taken in interpreting the results, as the presence of high correlations does not necessarily imply causation or redundancy without further analysis.

This test shows its results in the form of a table, where each row represents a unique pair of features from the dataset. The columns include the feature pair, the Pearson correlation coefficient (rounded to four decimal places), and a Pass or Fail status based on whether the absolute value of the coefficient exceeds the threshold of 0.3. The coefficients range from -0.3341 to 0.3341, indicating both positive and negative linear relationships of varying strengths. The table is sorted by the absolute value of the correlation coefficient, with the strongest relationships listed first. Notably, only one feature pair, (Age, Exited), exceeds the threshold and is marked as Fail, while all other pairs are marked as Pass. The remaining coefficients are relatively low, with most values clustered between -0.1984 and 0.0474, suggesting generally weak linear relationships among the other feature pairs. The Pass/Fail column provides an immediate visual cue for identifying pairs that may warrant further investigation. The table format allows for straightforward comparison of the strength and direction of relationships across all evaluated pairs.

The test results reveal the following key insights:

• Only One Feature Pair Exceeds the Correlation Threshold: The pair (Age, Exited) has a Pearson correlation coefficient of 0.3341, surpassing the threshold of 0.3 and resulting in a Fail status, indicating a moderate positive linear relationship between these features.
• All Other Feature Pairs Show Weak Linear Relationships: The remaining nine feature pairs have coefficients ranging from -0.1984 to 0.0474, all below the threshold, and are marked as Pass, suggesting minimal risk of multicollinearity among these pairs.
• Negative and Positive Correlations Are Both Present: The coefficients include both positive and negative values, with the strongest negative correlation observed between (IsActiveMember, Exited) at -0.1984, though still below the threshold.
• No Evidence of Widespread Redundancy: The distribution of coefficients indicates that, aside from the (Age, Exited) pair, the dataset does not exhibit strong linear dependencies among the top feature pairs evaluated.
• Clear Pass/Fail Delineation Facilitates Interpretation: The Pass/Fail status provides an immediate and unambiguous indication of which feature pairs may require further scrutiny, streamlining the review process.

Based on these results, the dataset demonstrates a generally low level of linear association among its features, with only the (Age, Exited) pair exhibiting a moderate correlation that exceeds the predefined threshold. This suggests that, with the exception of this pair, the risk of multicollinearity or feature redundancy is minimal for the evaluated features. The presence of both positive and negative coefficients reflects a balanced distribution of relationships, and the clear Pass/Fail delineation aids in quickly identifying areas of potential concern. The overall pattern indicates that the dataset is well-structured with respect to linear dependencies, supporting the development of interpretable and stable models. The single pair exceeding the threshold may warrant further examination to assess its impact on model behavior, but the absence of additional high correlations suggests that the feature set is largely free from problematic linear relationships.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3341	Fail
(IsActiveMember, Exited)	-0.1984	Pass
(Balance, NumOfProducts)	-0.1565	Pass
(Balance, Exited)	0.1376	Pass
(NumOfProducts, IsActiveMember)	0.0575	Pass
(Age, Balance)	0.0474	Pass
(HasCrCard, IsActiveMember)	-0.0426	Pass
(NumOfProducts, Exited)	-0.0399	Pass
(Tenure, IsActiveMember)	-0.0390	Pass
(Age, NumOfProducts)	-0.0345	Pass

2026-01-10 02:06:45,733 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset does not exist in model's document

Note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for this particular test ID.

That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.

Add individual test results to model documentation

With the test results logged, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation (Need more help?):

1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

2. In the left sidebar that appears for your model, click Documentation under Documents.

3. Locate the Data Preparation section and click on 2.3. Correlations and Interactions to expand that section.

4. Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a + button appears, indicating that you can insert a new block.

   
   
   

5. Click + and then select Test-Driven Block under FROM LIBRARY:

   • Click on VM Library under TEST-DRIVEN in the left sidebar.
   • In the search bar, type in HighPearsonCorrelation.
   • Select HighPearsonCorrelation:balanced_raw_dataset as the test.

   A preview of the test gets shown:

   
   
   

6. Finally, click Insert 1 Test Result to Document to add the test result to the documentation.

   Confirm that the individual results for the high correlation test has been correctly inserted into section 2.3. Correlations and Interactions of the documentation.

7. Finalize the documentation by editing the test result's description block to explain the changes you made to the raw data and the reasons behind them as shown in the screenshot below:

   

Model testing

So far, we've focused on the data assessment and pre-processing that usually occurs prior to any models being built. Now, let's instead assume we have already built a model and we want to incorporate some model results into our documentation.

Train simple logistic regression model

Using ValidMind tests, we'll train a simple logistic regression model on our dataset and evaluate its performance by using the LogisticRegression class from the sklearn.linear_model.

To start, let's grab the first few rows from the balanced_raw_no_age_df dataset with the highly correlated features removed we initialized earlier:

balanced_raw_no_age_df.head()

	CreditScore	Geography	Gender	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
3610	548	Spain	Male	0	178056.54	2	1	0	38434.73	0
3064	646	Spain	Male	1	0.00	2	1	0	183289.22	0
404	638	Spain	Male	9	77637.35	2	1	1	111346.22	0
7772	625	France	Female	3	0.00	2	1	0	41295.10	1
1635	654	France	Male	6	0.00	1	0	0	183872.88	1

Before training the model, we need to encode the categorical features in the dataset:

• Use the OneHotEncoder class from the sklearn.preprocessing module to encode the categorical features.
• The categorical features in the dataset are Geography and Gender.

balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
3610	548	0	178056.54	2	1	0	38434.73	0	False	True	True
3064	646	1	0.00	2	1	0	183289.22	0	False	True	True
404	638	9	77637.35	2	1	1	111346.22	0	False	True	True
7772	625	3	0.00	2	1	0	41295.10	1	False	False	False
1635	654	6	0.00	1	0	0	183872.88	1	False	False	True

We'll split our preprocessed dataset into training and testing, to help assess how well the model generalizes to unseen data:

• We start by dividing our balanced_raw_no_age_df dataset into training and test subsets using train_test_split, with 80% of the data allocated to training (train_df) and 20% to testing (test_df).
• From each subset, we separate the features (all columns except "Exited") into X_train and X_test, and the target column ("Exited") into y_train and y_test.

from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

Then using GridSearchCV, we'll find the best-performing hyperparameters or settings and save them:

from sklearn.linear_model import LogisticRegression

# Logistic Regression grid params
log_reg_params = {
    "penalty": ["l1", "l2"],
    "C": [0.001, 0.01, 0.1, 1, 10, 100, 1000],
    "solver": ["liblinear"],
}

# Grid search for Logistic Regression
from sklearn.model_selection import GridSearchCV

grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)
grid_log_reg.fit(X_train, y_train)

# Logistic Regression best estimator
log_reg = grid_log_reg.best_estimator_

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

Initialize model evaluation objects

The last step for evaluating the model's performance is to initialize the ValidMind Dataset and Model objects in preparation for assigning model predictions to each dataset.

# Initialize the datasets into their own dataset objects
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

You'll also need to initialize a ValidMind model object (vm_model) that can be passed to other functions for analysis and tests on the data for each of our three models.

You simply initialize this model object with vm.init_model():

# Register the model
vm_model = vm.init_model(log_reg, input_id="log_reg_model_v1")

Assign predictions

Once the model has been registered you can assign model predictions to the training and testing datasets.

• The assign_predictions() method from the Dataset object can link existing predictions to any number of models.
• This method links the model's class prediction values and probabilities to our vm_train_ds and vm_test_ds datasets.

If no prediction values are passed, the method will compute predictions automatically:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

2026-01-10 02:06:46,929 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-01-10 02:06:46,931 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-01-10 02:06:46,931 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-01-10 02:06:46,934 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-01-10 02:06:46,936 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-01-10 02:06:46,938 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-01-10 02:06:46,940 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-01-10 02:06:46,940 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Run the model evaluation tests

In this next example, we'll focus on running the tests within the Model Development section of the model documentation. Only tests associated with this section will be executed, and the corresponding results will be updated in the model documentation.

• Note the additional config that is passed to run_documentation_tests() — this allows you to override inputs or params in certain tests.
• In our case, we want to explicitly use the vm_train_ds for the validmind.model_validation.sklearn.ClassifierPerformance:in_sample test, since it's supposed to run on the training dataset and not the test dataset.

test_config = {
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "dataset": vm_train_ds,
            "model": vm_model,
        },
    }
}
results = vm.run_documentation_tests(
    section=["model_development"],
    inputs={
        "dataset": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds
        "model": vm_model,
        "datasets": (
            vm_train_ds,
            vm_test_ds,
        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds
    },
    config=test_config,
)

Test suite complete!

34/34 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Development

Model Development

▶ Test Result: Model Metadata (validmind.model_validation.ModelMetadata)

Model Metadata

Model Metadata is designed to compare the metadata of different models and generate a summary table with the results, focusing on key attributes such as architecture, framework, framework version, and programming language. The primary purpose of this test is to provide a standardized overview of essential model characteristics, enabling clear comparison and identification of potential compatibility or consistency issues across multiple models.

The test operates by retrieving metadata for each model using a dedicated function that extracts relevant information, such as the modeling technique, the framework used, the specific version of that framework, and the programming language in which the model is implemented. This information is then standardized by renaming columns according to a predefined set of labels, ensuring consistency in presentation. The compiled metadata is organized into a summary table, which serves as a centralized reference for model documentation. The test does not compute statistical or quantitative metrics but instead focuses on the qualitative comparison of categorical metadata fields. The summary table allows users to quickly assess similarities and differences between models, which is particularly important for integration, deployment, and ongoing model management. The typical values for each field are categorical, such as the name of the framework or the version number, and the interpretation centers on identifying uniformity or divergence in these attributes.

The primary advantages of this test include its ability to provide a clear and concise comparison of essential model metadata, which is critical for effective model governance and lifecycle management. By standardizing metadata labels and presenting them in a unified format, the test facilitates easier interpretation and comparison across models, reducing the risk of miscommunication or oversight. This approach is particularly useful in environments where multiple models are developed and maintained simultaneously, as it helps stakeholders quickly identify potential compatibility issues, such as mismatched framework versions or programming languages. The test also supports regulatory and audit requirements by ensuring that all relevant metadata is documented and readily accessible, thereby enhancing transparency and traceability in model management processes.

It should be noted that the test relies on the completeness and accuracy of the metadata provided by each model, as it assumes that the underlying function returns all necessary fields. If any metadata is missing or inconsistent, the summary table may not fully capture the relevant characteristics, potentially leading to gaps in documentation. Additionally, the test focuses exclusively on high-level metadata and does not include detailed parameter information or model-specific configurations, which may be important for more granular analysis. Significant differences in framework versions or programming languages, as highlighted by the test, could pose challenges in model integration and deployment, especially in environments with strict compatibility requirements. Interpretation of the results should therefore consider the broader context of model usage and deployment constraints.

This test shows a single summary table that presents the metadata for the evaluated model. The table includes columns for Modeling Technique, Modeling Framework, Framework Version, and Programming Language. Each row corresponds to a specific model, with the current result displaying one model entry. The Modeling Technique column indicates the type of model used, in this case, "SKlearnModel." The Modeling Framework column specifies the software library, here "sklearn," while the Framework Version column provides the exact version number, "1.8.0." The Programming Language column identifies the language used for implementation, which is "Python." To read the table, users can scan across each row to compare the attributes of different models, although in this instance, only one model is represented. The values are categorical and descriptive, with no numerical ranges or units of measurement. The table format allows for straightforward identification of any inconsistencies or notable characteristics, such as the use of a specific framework version or programming language. In this result, all metadata fields are populated, and there are no missing or inconsistent entries, indicating a well-documented model instance.

The test results reveal the following key insights:

• Complete Metadata Coverage for Single Model: The summary table displays all required metadata fields for the evaluated model, with no missing or incomplete entries, ensuring comprehensive documentation.
• Consistent Use of Framework and Language: The model utilizes the "sklearn" framework at version "1.8.0" and is implemented in "Python," reflecting a standard and widely adopted configuration.
• Single Model Representation: Only one model is included in the current results, providing a focused view but limiting comparative analysis across multiple models.
• Standardized Metadata Presentation: The use of predefined labels and a structured table format ensures that the metadata is presented in a clear and consistent manner, facilitating easy interpretation.

Based on these results, the metadata comparison test provides a clear and structured overview of the evaluated model's key attributes, confirming that all essential metadata fields are present and accurately documented. The use of the "sklearn" framework at version "1.8.0" and implementation in "Python" aligns with common industry practices, suggesting compatibility with standard deployment environments. The absence of missing or inconsistent metadata entries indicates robust model documentation, which supports effective model management and regulatory compliance. However, the current results are limited to a single model, which restricts the ability to draw broader conclusions about consistency or variation across a portfolio of models. The standardized presentation of metadata ensures that any future additions to the table will be easily comparable, enabling efficient identification of potential integration or compatibility challenges as more models are evaluated. Overall, the test demonstrates the value of systematic metadata documentation in supporting transparent and reliable model governance.

Tables

Modeling Technique	Modeling Framework	Framework Version	Programming Language
SKlearnModel	sklearn	1.8.0	Python

▶ Test Result: Dataset Split (validmind.data_validation.DatasetSplit)

Dataset Split

Dataset Split is designed to evaluate and visualize the distribution of data among the training, testing, and validation datasets within a machine learning model. Its primary purpose is to ensure that the data is partitioned in a manner that supports effective model training, validation, and testing, thereby promoting robust learning and reliable generalization to new, unseen data.

The test operates by first determining the total number of data points across all available datasets, which typically include training, testing, and, if present, validation sets. For each dataset, the test calculates the absolute size, representing the count of data points, and then computes the proportion of each dataset relative to the total. These proportions are expressed as percentages, providing a clear view of how the data is allocated. The methodology involves aggregating the dataset sizes and then dividing each individual dataset’s size by the total to derive its proportion. The results are presented in a tabular format, with columns for dataset name, size, and proportion. The proportions range from 0% to 100%, where a balanced split is generally considered to be around 60-80% for training, 10-20% for testing, and 10-20% for validation, depending on the specific modeling context. A disproportionately small training set may hinder the model’s ability to learn, while a very small test or validation set may limit the assessment of model generalization.

The primary advantages of this test include its ability to provide a transparent and easily interpretable summary of dataset allocation, which is critical for identifying potential imbalances that could affect model performance. The test is versatile, applicable across various data types such as tabular, time series, and text, and is suitable for a wide range of machine learning tasks, including classification and regression. By offering a straightforward visualization of dataset splits, it enables practitioners to quickly assess whether the data partitioning aligns with best practices, supporting both model development and regulatory review. The test’s simplicity and general applicability make it a valuable tool for routine model documentation and governance, ensuring that data allocation is not overlooked during model development.

It should be noted that the Dataset Split test does not assess the quality, representativeness, or diversity of the data within each split; it focuses solely on the quantitative distribution. The test does not provide recommendations for correcting imbalances or optimizing splits, nor does it account for more complex splitting strategies such as stratified or time-based splits, which may be necessary for certain data types or modeling objectives. Additionally, the absence of a validation set in the results may complicate the evaluation of model performance during development. High-risk scenarios include a very small training set, which can impede learning, or a disproportionately small test set, which may not provide a reliable measure of generalization. The test’s reliance on size and proportion alone means that it should be complemented by other assessments to ensure comprehensive data quality and suitability for modeling.

This test shows the results in a tabular format, listing each dataset alongside its absolute size and its proportion of the total dataset. The table includes three rows: one for the training dataset, one for the test dataset, and one summarizing the total. The “Dataset” column identifies each split, the “Size” column provides the number of data points in each, and the “Proportion” column expresses this as a percentage of the total dataset size. The training dataset, labeled “train_dataset_final,” contains 2,585 data points, accounting for 79.98% of the total. The test dataset, “test_dataset_final,” includes 647 data points, representing 20.02% of the total. The total row confirms that the combined size is 3,232 data points, with proportions summing to 100%. The proportions are rounded to two decimal places, facilitating precise interpretation. There is no validation dataset present in the results. The table allows for immediate visual assessment of the data allocation, with the training set comprising the majority and the test set making up the remainder. The absence of a validation set is notable, as is the relatively high proportion allocated to training.

The test results reveal the following key insights:

• Training Dataset Dominates Allocation: The training dataset comprises 2,585 out of 3,232 total data points, representing 79.98% of the data, which is at the upper end of typical training set proportions.
• Test Dataset Proportion Is Moderate: The test dataset contains 647 data points, accounting for 20.02% of the total, which is within the common range for test splits but leaves no data for validation.
• No Validation Dataset Present: The results do not include a validation dataset, indicating that all data is allocated between training and testing, which may affect the ability to monitor model performance during development.
• Clear and Balanced Split Between Two Sets: The split between training and test datasets is clear and straightforward, with proportions summing precisely to 100%, and no overlap or missing data points.
• Proportions Are Consistent With Standard Practices: The allocation of approximately 80% to training and 20% to testing aligns with common practices, though the lack of a validation set is a notable deviation.

Based on these results, the dataset is divided into two distinct splits: a large training set and a moderately sized test set, with no separate validation set included. The training set’s proportion of nearly 80% provides ample data for model learning, while the test set’s 20% allocation is sufficient for evaluating generalization performance. The absence of a validation set means that model tuning and early stopping may need to rely on alternative strategies or cross-validation within the training data. The proportions are consistent with standard data partitioning practices, supporting robust model development and evaluation. The results indicate a straightforward and transparent data allocation, with no evidence of missing or overlapping data. The clear delineation between training and test sets facilitates reliable assessment of model performance, though the lack of a validation set may limit the ability to monitor and optimize the model during iterative development. Overall, the dataset split supports effective model training and testing, with proportions that are well-aligned with established norms for machine learning workflows.

Tables

Dataset	Size	Proportion
train_dataset_final	2585	79.98%
test_dataset_final	647	20.02%
Total	3232	100%

▶ Test Result: Population Stability Index (validmind.model_validation.sklearn.PopulationStabilityIndex)

Population Stability Index

Population Stability Index is designed to assess the stability of a machine learning model’s predictions by quantifying shifts in the distribution of model outputs between two datasets, typically a development (initial) and a validation (new) sample. The primary purpose of this test is to detect whether the population characteristics or the model’s predictive behavior have changed over time or across different data samples, which could indicate potential model drift or changes in the underlying data environment.

The test operates by dividing the model’s prediction outputs from both the initial and new datasets into a series of bins, with bin boundaries determined based on the distribution of the initial dataset. For each bin, the proportion of records from both datasets is calculated. The Population Stability Index (PSI) is then computed for each bin by applying a logarithmic transformation to the ratio of these proportions, which quantifies the degree of divergence between the two distributions. The PSI values for all bins are summed to yield an overall PSI score, which typically ranges from 0 upwards, with values below 0.1 generally considered to indicate no significant population shift, values between 0.1 and 0.25 suggesting moderate shift, and values above 0.25 indicating a significant shift. The test outputs both tabular and graphical representations, including a summary table of bin counts, proportions, and PSI values, as well as a grouped bar chart and a line plot to visualize the distributional changes and the magnitude of PSI across bins.

The primary advantages of this test include its ability to provide a clear, quantitative measure of model stability over time or across different samples, making it especially valuable for ongoing model monitoring and risk management. The PSI is straightforward to interpret and enables direct comparison across features or model outputs, facilitating early detection of population or model drift. The use of both tabular summaries and visualizations enhances interpretability, allowing stakeholders to quickly identify where and how distributional changes occur. This makes the PSI particularly useful in regulated environments or in production settings where model performance consistency is critical, as it supports transparent and repeatable monitoring processes.

It should be noted that the PSI test has several limitations. It does not account for dependencies between features, so correlated features may exhibit similar shifts, potentially overstating the extent of change. The test does not provide insight into the causes of distributional changes, nor does it distinguish between model drift and changes in the underlying data relationships (concept drift). Outliers or rare events may not be adequately captured, especially if binning is coarse or not tailored to the data’s characteristics. Additionally, because the PSI is calculated on model predictions rather than raw features, shifts may reflect changes in model behavior, data, or both, making interpretation more complex. High PSI values are a clear sign of risk, indicating that the model’s predictive environment has changed and that its reliability may be compromised.

This test shows the results in both tabular and graphical formats. The summary table presents, for each bin, the count and percentage of records from the initial and new datasets, as well as the PSI contribution for that bin. The final row aggregates the total counts and the overall PSI value. The accompanying plot displays the population ratios for each bin as grouped bars (with initial and new datasets shown in different colors), while the PSI value for each bin is overlaid as a line plot on a secondary axis. This dual-axis visualization allows for simultaneous assessment of both the magnitude and direction of distributional changes and the corresponding PSI contributions. The table reveals that the bin-wise PSI values are all very low, with the highest individual bin PSI at 0.0084 and the overall PSI at 0.0205, well below the commonly accepted threshold for significant population shift. The population ratios for each bin between the initial and new datasets are closely aligned, as seen in both the table and the bar chart, with only minor deviations in certain bins. The line plot of PSI values further confirms that no single bin exhibits a substantial divergence. The range of population ratios across bins is consistent, and the total sample sizes for both datasets are clearly indicated, supporting a robust comparison.

The test results reveal the following key insights:

• Overall population stability is maintained: The total PSI value is 0.0205, which is well below the threshold of 0.1, indicating no significant shift in the distribution of model predictions between the initial and new datasets.
• Bin-level distributional alignment is strong: Across all ten bins, the percentage of records from the initial and new datasets remains closely matched, with differences generally within a few percentage points, and the highest bin-level PSI at only 0.0084.
• No bins exhibit substantial divergence: The PSI values for individual bins are all below 0.01, suggesting that no specific region of the prediction distribution is experiencing notable change.
• Population ratios are consistent across bins: Both the table and the bar chart show that the distribution of records across bins is similar for both datasets, with the largest bin containing approximately 15% of records in each dataset and the smallest bins around 5%.
• Visualizations confirm quantitative results: The grouped bar chart and PSI line plot visually reinforce the close alignment between datasets and the minimal contribution of each bin to the overall PSI, with no outliers or abrupt changes observed.

Based on these results, the model’s prediction distribution demonstrates a high degree of stability between the initial and new datasets, as evidenced by the low overall PSI and the close alignment of population ratios across all bins. The absence of any bins with elevated PSI values or marked differences in population proportions suggests that the underlying population characteristics and the model’s predictive behavior have remained consistent over time or across samples. The graphical and tabular outputs provide a clear, corroborative view of this stability, with both formats indicating that the model is not experiencing significant drift or instability in its output distribution. These observations collectively support the conclusion that, for the period and datasets evaluated, the model’s predictions remain robust and reliable, with no evidence of material change in the population or model performance as measured by the PSI.

Tables

Population Stability Index for train_dataset_final and test_dataset_final Datasets

Bin	Count Initial	Percent Initial (%)	Count New	Percent New (%)	PSI
0	131	5.0677	35	5.4096	0.0002
1	271	10.4836	50	7.7280	0.0084
2	272	10.5222	77	11.9011	0.0017
3	349	13.5010	91	14.0649	0.0002
4	379	14.6615	91	14.0649	0.0002
5	311	12.0309	98	15.1468	0.0072
6	362	14.0039	87	13.4467	0.0002
7	226	8.7427	48	7.4189	0.0022
8	133	5.1451	34	5.2550	0.0000
9	151	5.8414	36	5.5641	0.0001
Total	2585	100.0000	647	100.0000	0.0205

Figures

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

Confusion Matrix is designed to evaluate and visually represent the predictive performance of a classification machine learning model by quantifying the number of correct and incorrect predictions across each class. Its primary purpose is to provide a detailed breakdown of the model’s ability to correctly identify true positives, true negatives, false positives, and false negatives, which are fundamental to understanding model accuracy and error types.

The test operates by comparing the predicted class labels generated by the model against the actual class labels from the test dataset. Using these two sets of labels, a confusion matrix is constructed, which is a two-dimensional table where each row represents the actual class and each column represents the predicted class. The matrix is populated with counts of predictions falling into each category: true positives (correctly predicted positive cases), true negatives (correctly predicted negative cases), false positives (incorrectly predicted positive cases), and false negatives (incorrectly predicted negative cases). The confusion matrix is then visualized as a heatmap, where the intensity of each cell’s color corresponds to the count, making it easy to identify areas where the model performs well or poorly. The values in the matrix are non-negative integers, and higher values along the diagonal (true positives and true negatives) generally indicate better model performance, while higher off-diagonal values (false positives and false negatives) suggest areas where the model is making more errors.

The primary advantages of this test include its ability to provide a clear and comprehensive visual summary of a classification model’s performance, making it straightforward to identify the types and frequencies of errors the model is making. The confusion matrix is particularly useful for multi-class problems, as it scales naturally to more than two classes and allows for a granular inspection of model behavior across all categories. By explicitly displaying the counts of true positives, true negatives, false positives, and false negatives, the test enables practitioners to focus on specific error types, such as distinguishing between Type I and Type II errors, and to identify whether the model is biased toward certain classes. This level of detail supports targeted model refinement and is valuable for both technical evaluation and communication with stakeholders.

It should be noted that the confusion matrix has several limitations and potential risks. In situations where the dataset is imbalanced, the matrix may give a misleading impression of model performance, as high counts in the majority class can mask poor performance in minority classes. The confusion matrix does not provide a single summary statistic, such as accuracy or F1-score, so additional calculations are required to derive these metrics. Furthermore, the matrix is descriptive and does not support statistical hypothesis testing, which limits its use for formal model validation. There is also a risk of misinterpretation, especially if users focus solely on the diagonal values without considering the distribution of errors or the underlying class balance. High numbers of false positives or false negatives, as indicated in the test description, are signs of increased risk and may signal that the model is not effectively distinguishing between classes.

This test shows a confusion matrix heatmap with four key cells representing the model’s classification outcomes: true negatives (209), false positives (106), false negatives (123), and true positives (209). The matrix is displayed as a 2x2 grid, with the rows corresponding to the true class labels (0 and 1) and the columns to the predicted class labels (0 and 1). Each cell contains both a label and the corresponding count, making it easy to interpret the results visually. The diagonal cells (top-left and bottom-right) represent correct predictions, while the off-diagonal cells (top-right and bottom-left) represent errors. The color intensity in each cell reflects the magnitude of the count, with darker shades indicating higher values. The matrix reveals that the model correctly identifies 209 instances each of both class 0 and class 1, but also misclassifies 106 instances as false positives and 123 as false negatives. The distribution of values suggests a moderate balance between correct and incorrect predictions, with a slightly higher number of false negatives compared to false positives. The range of values in the matrix spans from 106 to 209, and the total number of predictions evaluated is 647. This visualization provides a direct and interpretable summary of the model’s classification behavior, highlighting both strengths and areas for potential improvement.

The test results reveal the following key insights:

• Balanced Correct Predictions Across Classes: The model achieves an equal number of true positives and true negatives, with 209 correct predictions for each class, indicating no apparent bias toward either class in terms of correct identification.
• Moderate Error Rates in Both Directions: There are 106 false positives and 123 false negatives, showing that the model makes a comparable number of errors in both directions, though false negatives are slightly more prevalent.
• Comparable Distribution of Actual and Predicted Classes: The sum of true positives and false negatives (332) closely matches the sum of true negatives and false positives (315), suggesting that the dataset is relatively balanced between the two classes.
• Error Magnitude Relative to Correct Predictions: The number of incorrect predictions (229 in total) is substantial compared to the number of correct predictions (418 in total), indicating that the model’s overall accuracy is moderate and that there is room for improvement in reducing both types of errors.
• Visual Emphasis on Error Types: The heatmap’s color intensity makes it easy to distinguish between high and low counts, with the darkest cells corresponding to correct predictions and lighter cells highlighting the areas where the model is less reliable.

Based on these results, the confusion matrix demonstrates that the model exhibits a balanced ability to correctly classify both classes, as evidenced by the equal counts of true positives and true negatives. However, the presence of a significant number of false positives and false negatives indicates that the model’s predictive performance is moderate, with a non-negligible proportion of errors in both directions. The slightly higher count of false negatives suggests that the model is marginally more likely to miss positive cases than to incorrectly flag negatives as positives. The overall distribution of actual and predicted classes appears balanced, which supports the interpretation that the model is not systematically favoring one class over the other. The visualization effectively highlights the areas where the model performs well and where it is prone to misclassification, providing a clear basis for further analysis of model behavior. These insights collectively characterize the model as having reasonable but not exceptional classification performance, with a need to address both types of errors to achieve higher reliability.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

Classifier Performance: In Sample is designed to evaluate the effectiveness of classification models by quantifying their ability to correctly identify and distinguish between different classes. The primary purpose of this test is to provide a comprehensive assessment of a model’s predictive performance using a suite of standard metrics, including precision, recall, F1-Score, accuracy, and ROC AUC, which together offer a multi-faceted view of how well the model performs on the data it was trained on.

The test operates by applying the model to a labeled dataset and comparing the predicted class labels to the true labels. It calculates precision, which measures the proportion of positive identifications that are actually correct, and recall, which assesses the proportion of actual positives that are correctly identified. The F1-Score is the harmonic mean of precision and recall, providing a balanced measure that accounts for both false positives and false negatives. Accuracy represents the overall proportion of correct predictions out of all predictions made. For multiclass problems, the test computes both macro and weighted averages of these metrics: macro averages treat all classes equally, while weighted averages account for class imbalance by weighting each class’s contribution according to its frequency. The ROC AUC metric evaluates the model’s ability to discriminate between classes by measuring the area under the receiver operating characteristic curve, which plots the true positive rate against the false positive rate at various threshold settings. ROC AUC values range from 0 to 1, with values closer to 1 indicating strong discriminatory power and values near 0.5 suggesting performance no better than random guessing. These metrics are derived from the model’s predictions and the true class labels, providing a robust statistical summary of classification performance.

The primary advantages of this test include its versatility and comprehensiveness in evaluating classification models. By incorporating multiple metrics, it captures different aspects of model performance, such as the trade-off between precision and recall, the overall accuracy, and the model’s ability to distinguish between classes regardless of class imbalance. The inclusion of both macro and weighted averages ensures that the test remains informative even in the presence of class imbalance, which is common in real-world datasets. The ROC AUC metric is particularly valuable for assessing models on unbalanced datasets, as it provides a threshold-independent measure of discriminatory power. This holistic approach enables practitioners to identify not only how well the model performs overall but also how it behaves with respect to each class, making it especially useful for applications where certain types of errors are more costly than others.

It should be noted that this test is specifically designed for classification models and is not applicable to regression tasks. The accuracy and interpretability of the results depend on the quality and representativeness of the test dataset; if the dataset does not reflect real-world conditions, the metrics may not generalize well. The test assumes that class labels are correctly identified and that the model’s outputs are properly calibrated for probability-based metrics like ROC AUC. Low values for precision, recall, F1-Score, accuracy, or ROC AUC indicate poor model performance, and significant imbalances between precision and recall may signal issues such as overfitting or underfitting. Additionally, a ROC AUC score close to 0.5 or lower suggests that the model lacks discriminatory power and may be failing to learn meaningful patterns. Interpretation challenges may arise if the dataset is highly imbalanced or if the model’s predictions are not well-calibrated, potentially leading to misleading conclusions about model effectiveness.

This test shows the results in the form of two tables. The first table, titled "Precision, Recall, and F1," presents the precision, recall, and F1-Score for each class (Class 0 and Class 1), as well as the weighted and macro averages of these metrics. Each row corresponds to a class or an averaging method, and each column displays the respective metric value, which ranges from 0 to 1, with higher values indicating better performance. The second table, titled "Accuracy and ROC AUC," summarizes the overall accuracy and the ROC AUC score for the model. Accuracy reflects the proportion of correct predictions, while ROC AUC measures the model’s ability to distinguish between the two classes. Notable observations include the close alignment of precision, recall, and F1-Score values across both classes, with all metrics falling in the range of approximately 0.61 to 0.65. The weighted and macro averages are nearly identical, indicating balanced class representation. The overall accuracy is 0.6329, and the ROC AUC is 0.6805, suggesting moderate discriminatory power. These results provide a clear, quantitative summary of the model’s in-sample classification performance, allowing for straightforward interpretation and comparison.

The test results reveal the following key insights:

• Balanced Class Performance: Both Class 0 and Class 1 exhibit similar precision, recall, and F1-Score values, with Class 0 showing a precision of 0.6304, recall of 0.6541, and F1-Score of 0.642, while Class 1 has a precision of 0.6356, recall of 0.6114, and F1-Score of 0.6233, indicating no significant disparity in model performance between the two classes.
• Consistent Averaged Metrics: The weighted and macro averages for precision, recall, and F1-Score are all approximately 0.633, reflecting a balanced dataset and consistent model behavior across classes, with minimal impact from class imbalance.
• Moderate Overall Accuracy: The model achieves an accuracy of 0.6329, meaning that approximately 63% of predictions are correct, which is above random chance but does not indicate high performance.
• Moderate Discriminatory Power: The ROC AUC score of 0.6805 suggests that the model has moderate ability to distinguish between the two classes, performing better than random guessing but not reaching strong discriminatory levels.
• No Extreme Metric Values: All reported metrics fall within a narrow range, with no values approaching 0 or 1, indicating stable but unexceptional model performance without evidence of overfitting or severe underfitting.

Based on these results, the model demonstrates moderate and balanced in-sample classification performance, with similar precision, recall, and F1-Score values for both classes and nearly identical macro and weighted averages. The accuracy of 0.6329 indicates that the model correctly predicts the class label in roughly two-thirds of cases, while the ROC AUC of 0.6805 reflects a moderate capacity to distinguish between the two classes. The close alignment of metrics across classes and averaging methods suggests that the model does not favor one class over the other and that class imbalance is not a significant factor in this dataset. The absence of extreme metric values points to stable model behavior, with no evidence of erratic predictions or severe misclassification. Collectively, these insights indicate that the model is able to capture some meaningful patterns in the data but does not achieve high levels of predictive accuracy or discriminatory power, suggesting that there may be limitations in the model’s ability to generalize or that the underlying data may not be highly separable. The results provide a clear and objective summary of the model’s in-sample performance, serving as a baseline for further evaluation or comparison with alternative models or datasets.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6304	0.6541	0.6420
1	0.6356	0.6114	0.6233
Weighted Average	0.6330	0.6329	0.6327
Macro Average	0.6330	0.6327	0.6326

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6329
ROC AUC	0.6805

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

Classifier Performance: Out of Sample is designed to evaluate the effectiveness of classification models by quantifying their ability to correctly identify and distinguish between different classes. The primary purpose of this test is to provide a comprehensive assessment of a model’s predictive performance using a suite of standard metrics, including precision, recall, F1-Score, accuracy, and ROC AUC, which together offer a multi-faceted view of how well the model performs on unseen data.

The test operates by applying the trained classification model to an out-of-sample dataset and calculating several key performance metrics. Precision measures the proportion of positive identifications that are actually correct, reflecting the model’s ability to avoid false positives. Recall quantifies the proportion of actual positives that are correctly identified, indicating the model’s sensitivity to true positives. The F1-Score is the harmonic mean of precision and recall, providing a balanced measure that accounts for both false positives and false negatives. Accuracy represents the overall proportion of correct predictions among all cases, offering a general sense of model correctness. For multiclass or binary classification, macro and weighted averages of these metrics are computed to account for class imbalances and provide a holistic view. The ROC AUC score evaluates the model’s ability to discriminate between classes across all possible thresholds, with values ranging from 0.5 (no discrimination) to 1.0 (perfect discrimination). These metrics are derived from the model’s predictions and the true class labels, and their values are interpreted in the context of the specific application, with higher values generally indicating better performance.

The primary advantages of this test include its versatility and comprehensiveness in evaluating classification models. By incorporating multiple metrics, the test captures different aspects of model performance, such as precision for minimizing false positives, recall for capturing true positives, and F1-Score for balancing both. The inclusion of macro and weighted averages ensures that the evaluation remains robust even in the presence of class imbalance, which is common in real-world datasets. The ROC AUC metric is particularly valuable for assessing the model’s discriminatory power, especially when dealing with unbalanced classes, as it summarizes the trade-off between sensitivity and specificity across all thresholds. This multi-metric approach enables practitioners to identify specific strengths and weaknesses in model behavior, supporting informed decision-making in model selection and deployment.

It should be noted that this test is specifically designed for classification models and is not applicable to regression tasks. The accuracy of the results depends on the representativeness of the test dataset; if the data does not reflect real-world distributions, the metrics may not generalize well. The test assumes that class labels are correctly identified and that the model outputs are properly formatted for binary or multiclass evaluation. Interpretation challenges may arise if there is a significant imbalance between precision and recall, or if the ROC AUC score is close to 0.5, which would indicate that the model is performing no better than random guessing. Low values in any of the key metrics, or substantial discrepancies between them, may signal areas where the model’s predictive capabilities are limited or where further investigation is warranted.

This test shows the results in the form of two tables. The first table, titled "Precision, Recall, and F1," presents the precision, recall, and F1-Score for each class (Class 0 and Class 1), as well as the weighted and macro averages of these metrics. Each row corresponds to a class or an averaging method, and each column displays the respective metric value, which ranges from 0 to 1, with higher values indicating better performance. The second table, titled "Accuracy and ROC AUC," summarizes the overall accuracy and the ROC AUC score for the model on the out-of-sample dataset. Accuracy reflects the proportion of correct predictions, while ROC AUC measures the model’s ability to distinguish between classes across all thresholds. Notable observations include the close alignment of precision and recall values for both classes, the consistency between macro and weighted averages, and a ROC AUC score that is moderately above the random baseline, suggesting some discriminatory power but not a high level of separation between classes.

The test results reveal the following key insights:

• Balanced Class Performance: Both Class 0 and Class 1 exhibit similar precision (0.6295 for Class 0, 0.6635 for Class 1) and recall (0.6635 for Class 0, 0.6295 for Class 1), resulting in identical F1-Scores of 0.6461 for both classes, indicating balanced predictive behavior across classes.
• Consistent Averaged Metrics: The macro and weighted averages for precision, recall, and F1-Score are closely aligned (all approximately 0.646), demonstrating that the model’s performance is stable and not unduly influenced by class imbalance.
• Moderate Overall Accuracy: The model achieves an accuracy of 0.6461, meaning that approximately 65% of predictions are correct, which is consistent with the individual class metrics and suggests moderate overall effectiveness.
• ROC AUC Indicates Modest Discriminatory Power: The ROC AUC score of 0.6862 is above the random baseline of 0.5, indicating that the model has some ability to distinguish between classes, but the value does not approach the high end of the scale, reflecting only moderate separation.
• No Extreme Metric Values: All reported metrics fall within a narrow range (0.6295 to 0.6862), with no values indicating either very strong or very weak performance, and no significant disparities between classes or metrics.

Based on these results, the model demonstrates a consistent and balanced performance across both classes, with precision, recall, and F1-Score values that are nearly identical for Class 0 and Class 1. The close agreement between macro and weighted averages further supports the observation that class imbalance does not significantly affect the model’s predictive behavior. The overall accuracy of approximately 65% aligns with the class-level metrics, reinforcing the model’s moderate effectiveness in making correct predictions. The ROC AUC score, while above the threshold for random guessing, suggests that the model’s ability to discriminate between classes is present but not strong, indicating that there is room for improvement in terms of class separation. The absence of extreme values or large discrepancies among the metrics points to a stable and uniform model performance, with no single class dominating or lagging behind. Collectively, these insights characterize the model as moderately effective, with balanced predictive capabilities and a consistent pattern of results across all evaluated metrics.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6295	0.6635	0.6461
1	0.6635	0.6295	0.6461
Weighted Average	0.6470	0.6461	0.6461
Macro Average	0.6465	0.6465	0.6461

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6461
ROC AUC	0.6862

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

Precision Recall Curve is designed to evaluate the trade-off between precision and recall for binary classification models, providing a visual representation of how well the model balances the ability to correctly identify positive cases (recall) with the accuracy of those positive predictions (precision). The primary purpose of this test is to assess the model’s effectiveness in scenarios where both false positives and false negatives carry significant consequences, offering a nuanced view of model performance beyond simple accuracy metrics.

The test operates by extracting the ground truth labels and the predicted probabilities for the positive class from the model’s test dataset. Using these inputs, the test applies the precision-recall curve methodology, which systematically varies the decision threshold used to classify positive cases. For each threshold, the test calculates the corresponding precision (the proportion of true positives among all positive predictions) and recall (the proportion of true positives among all actual positives). These pairs of precision and recall values are then plotted to form the Precision-Recall Curve. The curve typically ranges from the bottom left (low precision and recall) to the top right (high precision and recall), with the area under the curve serving as an indicator of overall model performance. High values near the top right suggest a strong model, while curves closer to the diagonal or bottom left indicate weaker performance. The curve provides a comprehensive view of how the model’s predictive quality changes as the threshold is adjusted, making it particularly useful for understanding the trade-offs involved in different operational settings.

The primary advantages of this test include its ability to clearly illustrate the balance between minimizing false positives and false negatives, which is especially important in domains where both types of errors are costly or consequential. The visual nature of the Precision-Recall Curve allows stakeholders to intuitively grasp how the model’s performance shifts across different thresholds, supporting informed decision-making about threshold selection. This test is particularly valuable in imbalanced datasets, where traditional accuracy metrics may be misleading, as it focuses specifically on the positive class and provides a more granular understanding of model behavior. Additionally, the curve can highlight regions where the model performs consistently well or poorly, aiding in the identification of optimal operating points.

It should be noted that this test is limited to binary classification models and is not applicable to multiclass or foundation models. The interpretation of the curve can be challenging in cases where the costs of false positives and false negatives are highly asymmetric or when the dataset is extremely imbalanced, as the curve may not fully capture the practical implications of model errors. A low area under the curve is a sign of high risk, indicating that the model struggles to achieve both high precision and high recall simultaneously. Furthermore, if the curve remains close to the bottom left of the plot, it suggests that the model is not effective at distinguishing between positive and negative cases, which may necessitate further investigation or model refinement. Care must be taken to contextualize the results within the specific business or operational requirements, as the optimal balance between precision and recall can vary significantly depending on the application.

This test shows a single Precision-Recall Curve plotted with recall on the x-axis and precision on the y-axis, both ranging from 0 to 1. The curve provides a continuous view of the trade-off between precision and recall as the classification threshold is varied. To interpret the plot, one should observe how the curve trends across the range of recall values. At low recall values (left side of the plot), precision is initially high, peaking above 0.9, indicating that when the model is very selective, most positive predictions are correct. As recall increases, precision gradually decreases, with the curve exhibiting a downward trend and some fluctuations. Around recall values of 0.2 to 0.7, precision stabilizes in the range of 0.7 to 0.6, suggesting a moderate balance between the two metrics. As recall approaches 1.0, precision drops further, reaching values close to 0.5, which indicates that when the model attempts to capture nearly all positive cases, the proportion of correct positive predictions decreases. The overall shape of the curve, with a steep initial decline followed by a more gradual slope, reflects the model’s ability to maintain reasonable precision at moderate recall levels but highlights a trade-off as recall increases. Notable observations include the initial peak in precision, the relatively stable midsection, and the eventual decline as recall maximizes, all of which provide insight into the model’s operational characteristics across different thresholds.

The test results reveal the following key insights:

• Precision Peaks at Low Recall: The model achieves its highest precision, exceeding 0.9, when recall is very low, indicating that highly selective thresholds yield very accurate positive predictions but capture only a small fraction of actual positives.
• Stable Precision at Moderate Recall: Between recall values of approximately 0.2 and 0.7, precision remains relatively stable, fluctuating between 0.7 and 0.6, suggesting that the model maintains a moderate balance between precision and recall across a broad range of thresholds.
• Precision Declines at High Recall: As recall increases beyond 0.7 and approaches 1.0, precision steadily declines, reaching values near 0.5, which indicates that the model’s ability to correctly identify positive cases diminishes as it attempts to capture all positives.
• Curve Shape Indicates Trade-Off: The overall shape of the curve, with an initial sharp drop followed by a gradual decline, highlights the inherent trade-off between precision and recall, with no region where both metrics are simultaneously maximized.
• No Evidence of Severe Model Failure: The curve does not remain near the bottom left, and there are no abrupt drops to zero, indicating that the model avoids extreme cases of poor performance across the tested thresholds.

Based on these results, the model demonstrates a clear trade-off between precision and recall, with the ability to achieve high precision at the cost of low recall and vice versa. The stable midsection of the curve suggests that the model can maintain a moderate balance between the two metrics across a range of thresholds, which may be suitable for applications where neither false positives nor false negatives can be entirely prioritized. The decline in precision at high recall levels indicates that the model becomes less reliable when attempting to identify all positive cases, which is a common characteristic in many binary classifiers. The absence of regions with extremely low precision or recall suggests that the model does not exhibit severe deficiencies in distinguishing between classes. The overall pattern of the curve provides a comprehensive view of the model’s operational characteristics, supporting informed decisions about threshold selection based on the specific requirements of the use case. The results highlight the importance of considering both precision and recall in evaluating model performance, particularly in contexts where the costs of different types of errors must be carefully balanced.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

ROC Curve is designed to evaluate the performance of binary classification models by illustrating the trade-off between the true positive rate and the false positive rate across a range of classification thresholds. Its primary purpose is to provide a comprehensive visual and quantitative assessment of a model’s ability to distinguish between two classes, such as default versus non-default, using the Area Under the Curve (AUC) as a summary metric of discriminative power.

The test operates by first generating predicted probabilities for each observation in the test dataset using the selected binary classification model. These probabilities, along with the true class labels, are used to calculate the true positive rate (the proportion of actual positives correctly identified) and the false positive rate (the proportion of actual negatives incorrectly identified as positives) at various threshold levels. The ROC curve is then plotted with the false positive rate on the x-axis and the true positive rate on the y-axis, providing a visual representation of model performance across all possible thresholds. A reference line representing random classification (AUC of 0.5) is included for context. The AUC score, which ranges from 0 to 1, is computed as the area under the ROC curve; a value closer to 1 indicates strong discriminative ability, while a value near 0.5 suggests performance no better than random guessing. The test also ensures that any infinite values in the threshold calculations are removed to maintain result integrity.

The primary advantages of this test include its ability to provide a holistic view of model performance across all classification thresholds, rather than being limited to a single operating point. The ROC curve visually demonstrates how the model’s sensitivity and specificity change as the decision threshold varies, making it particularly useful for applications where the costs of false positives and false negatives differ or are not known in advance. The AUC score offers a robust, threshold-independent summary of model discrimination, which remains consistent regardless of class distribution in the dataset. This makes the ROC-AUC approach especially valuable in scenarios with imbalanced classes or when comparing models across different datasets.

It should be noted that the ROC Curve test is specifically designed for binary classification tasks and does not generalize to multi-class or regression models. Its effectiveness may be reduced when model outputs are highly skewed toward the extremes of 0 or 1, as this can distort the curve and the AUC score. Additionally, the ROC curve can sometimes present an overly optimistic view of model performance in the presence of severe class imbalance, as it focuses on ranking rather than absolute classification accuracy. A key sign of high risk is an AUC score near or below 0.5, which indicates that the model lacks discriminative power and performs no better than random chance. If the ROC curve closely follows the diagonal line of randomness, this further signals limited or no ability to distinguish between classes.

This test shows the results in the form of a ROC curve plot, where the x-axis represents the false positive rate (ranging from 0 to 1) and the y-axis represents the true positive rate (also ranging from 0 to 1). The solid magenta line traces the model’s performance across all thresholds, while the dashed gray line represents the performance of a random classifier with an AUC of 0.5. The legend indicates that the model’s ROC curve achieves an AUC of 0.69, which is also displayed in the plot title. To interpret the plot, one should observe how far the ROC curve rises above the diagonal line; the greater the area between the curve and the line of randomness, the better the model’s ability to distinguish between positive and negative classes. The curve for this model consistently stays above the random line, with the true positive rate increasing more rapidly than the false positive rate at lower thresholds, indicating some degree of discriminative power. The AUC value of 0.69 quantifies this performance, suggesting that the model is able to correctly rank a randomly chosen positive instance higher than a randomly chosen negative instance approximately 69% of the time. There are no abrupt drops or irregularities in the curve, and the plot covers the full range of possible false positive and true positive rates, providing a complete view of model behavior.

The test results reveal the following key insights:

• Model demonstrates moderate discriminative ability: The ROC curve for the logistic regression model on the test dataset achieves an AUC of 0.69, indicating that the model is able to distinguish between the two classes better than random, but does not reach a high level of separation.
• Curve consistently outperforms random baseline: Across the entire range of thresholds, the ROC curve remains above the diagonal line representing random classification, confirming that the model maintains some predictive value throughout.
• True positive rate increases steadily with false positive rate: The shape of the ROC curve shows a gradual and consistent rise in true positive rate as the false positive rate increases, without sharp inflections or plateaus, suggesting stable model behavior across thresholds.
• No evidence of severe class imbalance distortion: The ROC curve does not exhibit the extreme skew or flatness that can occur with highly imbalanced datasets, implying that the model’s ranking ability is not unduly affected by class proportions in this test set.
• AUC value provides clear summary of performance: The single AUC metric of 0.69 offers a concise, interpretable summary of the model’s overall ability to rank positive instances above negative ones, facilitating straightforward comparison with other models or datasets.

Based on these results, the logistic regression model evaluated on the test dataset displays a moderate level of discriminative power, as evidenced by an AUC of 0.69 and a ROC curve that consistently outperforms the random baseline across all thresholds. The model’s ability to separate positive from negative cases is stable and does not show signs of erratic behavior or threshold sensitivity. The absence of curve flattening or abrupt changes suggests that the model’s probability outputs are well-calibrated for ranking, and there is no indication of severe class imbalance affecting the results. The AUC score, while not approaching the ideal value of 1, is significantly above 0.5, confirming that the model provides meaningful predictive information. These observations collectively indicate that the model is capable of providing useful discrimination between classes, though there remains room for improvement in achieving higher separation. The ROC curve and AUC metric together offer a comprehensive and interpretable assessment of the model’s classification performance on this dataset.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

Training Test Degradation is designed to assess whether the model’s performance on unseen test data degrades beyond an acceptable threshold compared to its performance on the training data. The primary purpose of this test is to evaluate the model’s generalization capability by quantifying the difference in key classification metrics—precision, recall, and F1 score—between the training and test datasets, ensuring that the model remains robust and reliable when applied to new data.

The test operates by calculating precision, recall, and F1 score for each class on both the training and test datasets. Precision measures the proportion of true positive predictions among all positive predictions, indicating the model’s accuracy in identifying relevant instances. Recall quantifies the proportion of true positives identified out of all actual positives, reflecting the model’s ability to capture all relevant cases. The F1 score is the harmonic mean of precision and recall, providing a balanced measure that accounts for both false positives and false negatives. Each metric is computed separately for the training and test datasets, and the degradation is determined by the relative difference between the two scores, expressed as a percentage of the training score. A negative degradation indicates improved performance on the test set, while a positive value signals a drop. The test flags any metric where degradation exceeds a preset threshold (typically 10%), marking it as a potential risk. The results are summarized in a table, with columns for class, metric, training score, test score, degradation percentage, and pass/fail status, allowing for straightforward interpretation of the model’s stability across datasets.

The primary advantages of this test include its ability to provide a clear, quantitative assessment of the model’s generalization performance, which is critical for evaluating real-world applicability. By examining multiple metrics across all classes, the test offers a comprehensive view of the model’s strengths and weaknesses, rather than relying on a single performance indicator. The flexible threshold for acceptable degradation allows the test to be tailored to different risk tolerances and business requirements, making it suitable for a wide range of classification tasks. This multi-metric, multi-class approach ensures that subtle issues in model behavior are detected, supporting robust model monitoring and governance.

It should be noted that the test’s effectiveness depends on the representativeness and balance of the training and test datasets; if either dataset is unbalanced or lacks sufficient coverage, the results may not accurately reflect the model’s true generalization ability. The test does not account for the underlying data distribution or the presence of rare or underrepresented classes, which may lead to misleading conclusions if the data is not well-prepared. Additionally, the test is limited to classification tasks and does not extend to regression or other modeling paradigms. High degradation percentages, especially those exceeding the 10% threshold, are signs of potential overfitting or instability, but the test does not diagnose the root cause of such patterns. Interpretation challenges may arise if degradation is negative, as this could indicate either improved generalization or data artifacts, requiring further investigation.

This test shows the results in a tabular format, with each row representing a specific class and metric combination. The columns include the class label, metric name (precision, recall, F1-score), the model’s score on the training dataset, the score on the test dataset, the calculated degradation percentage, and a pass/fail indicator based on whether the degradation exceeds the 10% threshold. All metric scores are presented as values between 0 and 1, representing the proportion of correct predictions or balanced accuracy for each class and metric. The degradation percentage quantifies the relative change from training to test performance, with negative values indicating improvement and positive values indicating decline. Notably, all degradation percentages in the table are well below the 10% threshold, and several are negative, suggesting that the model performs slightly better on the test set for those metrics. The pass/fail column shows “Pass” for all metrics and classes, indicating that the model’s generalization is within acceptable limits. The range of metric scores is relatively narrow, with precision, recall, and F1 scores for both classes falling between approximately 0.61 and 0.66, and degradation percentages ranging from -4.38% to 0.14%. No metric exhibits a substantial drop in performance, and there are no outliers or extreme values in the results.

The test results reveal the following key insights:

• Model Generalization Remains Stable Across Datasets: All evaluated metrics for both classes show degradation percentages well below the 10% threshold, with values ranging from -4.38% to 0.14%, indicating consistent performance between training and test datasets.
• Test Set Performance Matches or Exceeds Training: Several metrics, including recall and F1-score for both classes, display negative degradation percentages, meaning the model performs slightly better on the test set than on the training set for these measures.
• Balanced Performance Across Classes and Metrics: Precision, recall, and F1 scores for both classes are closely grouped, with scores between 0.61 and 0.66, suggesting that the model does not favor one class over the other and maintains balanced predictive capability.
• No Metrics Exceed Degradation Threshold: The pass/fail status is “Pass” for all metrics and classes, confirming that the model’s performance degradation is within the acceptable range and there are no immediate signs of overfitting or instability.
• Consistent Metric Behavior Across Classes: Both classes exhibit similar patterns in metric scores and degradation, with no class showing disproportionate changes or instability, supporting the model’s robustness across the classification task.

Based on these results, the model demonstrates strong generalization capability, with minimal performance degradation between the training and test datasets across all evaluated metrics and classes. The close alignment of precision, recall, and F1 scores for both classes, along with the consistently low degradation percentages, indicates that the model is neither overfitting to the training data nor suffering from instability when exposed to new data. The absence of any metrics exceeding the 10% degradation threshold further supports the model’s reliability and suitability for deployment in scenarios where stable predictive performance is required. The observed negative degradation values for some metrics suggest that the model may even generalize slightly better to the test data, though this should be interpreted cautiously and may warrant further investigation into data characteristics. Overall, the results provide clear evidence that the model maintains balanced and robust performance across both datasets, with no notable disparities or areas of concern in its classification behavior.

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
0	Precision	0.6304	0.6295	0.1352	Pass
0	Recall	0.6541	0.6635	-1.4340	Pass
0	F1-Score	0.6420	0.6461	-0.6288	Pass
1	Precision	0.6356	0.6635	-4.3838	Pass
1	Recall	0.6114	0.6295	-2.9683	Pass
1	F1-Score	0.6233	0.6461	-3.6574	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

Minimum Accuracy is designed to assess whether the model’s prediction accuracy on a given dataset meets or exceeds a specified minimum threshold. The primary purpose of this test is to ensure that the model achieves a baseline level of correctness in its predictions, which is essential for reliable deployment in both binary and multiclass classification scenarios. By establishing a minimum acceptable accuracy, this test acts as a safeguard against deploying models that may not perform adequately in real-world applications.

The test operates by calculating the proportion of correct predictions made by the model compared to the total number of predictions, a metric known as accuracy. This is achieved by comparing the true labels of the dataset with the predicted labels generated by the model. The accuracy score is computed using a standard method that counts the number of exact matches between the true and predicted labels, then divides this count by the total number of predictions. The resulting value ranges from 0 to 1, where 1 indicates perfect accuracy and 0 indicates no correct predictions. The computed accuracy is then compared to a predefined threshold, which in this case is set at 0.7. If the model’s accuracy meets or exceeds this threshold, the test is marked as passed; otherwise, it is marked as failed. This approach provides a clear, quantitative measure of the model’s overall predictive performance, making it straightforward to interpret and communicate.

The primary advantages of this test include its simplicity and directness, offering a holistic measure of model performance that is easy to understand and communicate to both technical and non-technical stakeholders. It is particularly effective when the dataset has balanced classes, as it treats all classes equally and provides a single summary statistic that reflects the model’s ability to correctly classify instances across the board. The test’s versatility allows it to be applied to both binary and multiclass classification problems, making it a broadly applicable tool for initial model evaluation. Its straightforward nature also facilitates rapid assessment and comparison of different models or model versions, supporting efficient model selection and monitoring processes.

It should be noted that the Minimum Accuracy test has several limitations that must be considered when interpreting its results. One key limitation is its potential to provide misleading assessments when the dataset is imbalanced, as high accuracy can be achieved simply by predicting the majority class, even if the model performs poorly on minority classes. The test does not account for other important aspects of model performance, such as precision, recall, or the ability to manage false positives and false negatives, which can be critical in many applications. Additionally, persistent failure to meet the threshold is a sign of high risk, indicating that the model may not be suitable for deployment without further refinement. The test’s focus on overall correctness means it may overlook nuanced performance issues that could impact specific use cases or subpopulations.

This test shows the results in a tabular format, presenting three columns: Score, Threshold, and Pass/Fail. The Score column displays the model’s computed accuracy, which in this case is 0.6461. The Threshold column indicates the minimum required accuracy for the model to pass the test, set at 0.7. The Pass/Fail column provides a binary outcome based on whether the Score meets or exceeds the Threshold. To interpret the table, one should compare the Score to the Threshold; if the Score is greater than or equal to the Threshold, the model passes, otherwise it fails. In this instance, the Score of 0.6461 falls below the Threshold of 0.7, resulting in a Fail outcome. The values are presented as decimals, representing the proportion of correct predictions, and the table format allows for clear and immediate assessment of the model’s performance relative to the established criterion. Notably, the Score is relatively close to the Threshold, suggesting that the model is underperforming but not drastically so, which may warrant further investigation into the sources of error or potential areas for improvement.

The test results reveal the following key insights:

• Model Accuracy Falls Short of Threshold: The model achieves an accuracy score of 0.6461, which is below the required threshold of 0.7, resulting in a Fail outcome for the test.
• Clear Pass/Fail Determination: The tabular output provides an unambiguous assessment, with the Pass/Fail column directly reflecting whether the model meets the minimum accuracy requirement.
• Narrow Margin to Passing: The difference between the achieved accuracy and the threshold is 0.0539, indicating that the model is relatively close to meeting the criterion but does not satisfy the minimum standard.
• Single Metric Focus: The results are based solely on the overall accuracy metric, without additional breakdowns by class or subgroup, emphasizing the holistic nature of the test.

Based on these results, the model’s current accuracy of 0.6461 does not meet the established minimum threshold of 0.7, as clearly indicated by the Fail outcome in the results table. The proximity of the score to the threshold suggests that the model is approaching the desired level of performance but remains insufficient for passing this specific test. The results provide a straightforward and easily interpretable assessment of the model’s overall predictive capability, highlighting that while the model is not drastically underperforming, it does not yet achieve the baseline standard required for acceptance. The exclusive focus on overall accuracy means that the test does not provide information about performance on individual classes or the nature of errors, but it does establish a clear benchmark for model adequacy. The observed results underscore the importance of considering both the absolute accuracy value and its relation to the predefined threshold when evaluating model readiness for deployment or further development.

Tables

Score	Threshold	Pass/Fail
0.6461	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

Minimum F1 Score is designed to evaluate whether the model’s F1 score on the validation dataset meets or exceeds a predefined minimum threshold, ensuring that the model achieves a balanced trade-off between precision and recall. This test is particularly important in classification tasks where the distribution of classes may be imbalanced, as it provides a more informative measure of model performance than accuracy alone by considering both false positives and false negatives.

The test operates by calculating the F1 score on the validation dataset using scikit-learn’s metrics. For binary classification problems, the standard F1 score is computed, which is the harmonic mean of precision and recall. For multi-class problems, macro averaging is applied, which involves calculating the F1 score for each class independently and then averaging these scores to provide an overall measure. The F1 score ranges from 0 to 1, where 1 indicates perfect precision and recall, and 0 indicates the worst possible balance. The computed F1 score is then compared to a predefined threshold value, which represents the minimum acceptable performance standard for the model. If the F1 score meets or exceeds this threshold, the model passes the test; otherwise, it fails. This approach ensures that the model does not sacrifice recall for precision or vice versa, and that it maintains a minimum level of balanced performance.

The primary advantages of this test include its ability to provide a single, interpretable metric that captures both the model’s ability to correctly identify positive cases (recall) and its ability to avoid false positives (precision). This is especially valuable in scenarios with imbalanced class distributions, where traditional accuracy metrics can be misleading. The flexibility to set the minimum threshold allows practitioners to tailor the test to the specific risk tolerance and business requirements of the application. Additionally, the use of macro averaging in multi-class settings ensures that performance is not dominated by the majority class, providing a more equitable assessment across all classes. This makes the test particularly useful for regulatory and risk-sensitive environments where balanced performance is critical.

It should be noted that the F1 score assumes equal importance for precision and recall, which may not align with all business or regulatory contexts where the cost of false positives and false negatives differs. The test does not provide insight into the individual contributions of precision and recall, nor does it account for other important metrics such as the area under the ROC curve or class-specific performance. In cases where the threshold is set too low, the test may not be sufficiently discriminating, while an excessively high threshold could result in the rejection of otherwise acceptable models. Furthermore, a failing result is considered a sign of high risk, indicating that the model may not be effectively balancing the identification of positive cases with the minimization of false positives, which could have significant operational or regulatory implications.

This test shows the results in a tabular format, presenting three columns: Score, Threshold, and Pass/Fail. The Score column displays the computed F1 score for the model on the validation dataset, which in this case is 0.6461. The Threshold column indicates the minimum acceptable F1 score, set at 0.5 for this test. The Pass/Fail column provides a categorical assessment of whether the model’s F1 score meets or exceeds the threshold, with a value of "Pass" indicating that the model satisfies the minimum performance requirement. The table is straightforward to interpret: each row corresponds to a single test instance, and the reader can quickly compare the model’s actual F1 score to the threshold to determine compliance. The F1 score is presented as a decimal value between 0 and 1, with higher values indicating better balance between precision and recall. In this instance, the model’s F1 score of 0.6461 is well above the threshold of 0.5, resulting in a passing outcome. There are no additional breakdowns by class or subgroup, as the test is designed to provide a single summary measure of performance.

The test results reveal the following key insights:

• Model Achieves Required F1 Score: The model’s F1 score on the validation set is 0.6461, which exceeds the minimum threshold of 0.5, indicating that the model meets the predefined standard for balanced precision and recall.
• Clear Pass Outcome: The Pass/Fail assessment is "Pass," confirming that the model’s performance is sufficient according to the established criteria and that no further action is required based on this metric.
• Sufficient Margin Above Threshold: The difference between the model’s F1 score and the threshold is 0.1461, providing a comfortable margin that suggests the model is not at risk of narrowly failing the test under similar conditions.
• Single Metric Focus: The results are presented as a single summary statistic, emphasizing the overall balance of precision and recall without delving into class-specific or subgroup performance.

Based on these results, the model demonstrates a balanced performance on the validation dataset, as evidenced by an F1 score of 0.6461 that comfortably exceeds the minimum threshold of 0.5. The clear "Pass" outcome indicates that the model achieves the required standard for the trade-off between precision and recall, with a sufficient margin to suggest stability in this aspect of performance. The use of a single summary metric provides a straightforward assessment of compliance with the minimum F1 score requirement, and the results do not indicate any immediate risk of failing to meet this standard. The absence of class-level breakdowns means that the test focuses exclusively on overall model behavior, providing a high-level view of its ability to balance false positives and false negatives in the validation context. This supports the conclusion that, with respect to the F1 score criterion, the model maintains the expected level of balanced classification performance.

Tables

Score	Threshold	Pass/Fail
0.6461	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

Minimum ROC AUC Score is designed to validate the model’s ability to distinguish between classes by ensuring that the Receiver Operating Characteristic Area Under the Curve (ROC AUC) score on the validation dataset meets or exceeds a specified minimum threshold. This test is essential for assessing the model’s discriminatory power in both binary and multiclass classification settings, providing a quantitative benchmark for acceptable model performance.

The test operates by calculating the multiclass ROC AUC score using the true target values and the model’s predicted probabilities. To accommodate multiclass scenarios, the test first transforms the categorical target variables into a binary format using a label binarization process, which allows the ROC AUC metric—originally designed for binary classification—to be applied in a multiclass context. The ROC AUC score itself measures the model’s ability to correctly rank positive instances higher than negative ones across all possible classification thresholds, with values ranging from 0 to 1. A score of 0.5 indicates random performance, while a score of 1.0 reflects perfect discrimination. The test compares the computed ROC AUC score to a predefined threshold, typically set at 0.5, and records whether the model passes or fails based on this comparison. The results, including the actual score, the threshold, and the pass/fail status, are captured for documentation and review.

The primary advantages of this test include its comprehensive assessment of model performance across all possible classification thresholds, making it robust to the choice of any single decision boundary. The ROC AUC metric is particularly valuable because it simultaneously considers both the true positive rate and the false positive rate, offering a balanced view of the model’s ability to distinguish between classes. This threshold-independent evaluation is especially useful in applications where the optimal classification threshold is not known in advance or may vary over time. Additionally, the test’s compatibility with both binary and multiclass problems, through the use of macro-averaging, ensures broad applicability across a range of classification tasks.

It should be noted that the test has several limitations and potential risks. The ROC AUC metric may not provide meaningful insights in cases of severe class imbalance, as it can yield deceptively high scores even when the model fails to predict the minority class effectively. The macro-averaging approach, which gives equal weight to each class, may not be appropriate if the class distribution is highly skewed, potentially masking poor performance on underrepresented classes. Furthermore, while the test indicates whether the model meets a minimum standard of discriminatory power, it does not diagnose the underlying causes of suboptimal performance or identify which classes are most affected. A low ROC AUC score, particularly one below the threshold, signals that the model is not effectively distinguishing between classes, which is a high-risk scenario for most classification applications.

This test shows the results in a tabular format, presenting three columns: Score, Threshold, and Pass/Fail. The Score column displays the calculated ROC AUC value for the model on the validation dataset, which in this case is 0.6862. The Threshold column indicates the minimum acceptable ROC AUC score, set at 0.5 for this test. The Pass/Fail column records whether the model’s performance meets or exceeds the threshold, with a value of "Pass" indicating success. The table is straightforward to interpret: each row corresponds to a single test run, and the reader can quickly compare the model’s actual ROC AUC score to the required threshold. The ROC AUC score of 0.6862 falls within the typical range of 0 to 1, where higher values reflect better discriminatory ability. The fact that the score exceeds the threshold and results in a "Pass" status suggests that the model demonstrates acceptable performance in distinguishing between classes on the validation data. No additional breakdowns or subgroup analyses are presented in this result, and the focus remains on the overall ROC AUC metric as a summary measure of model quality.

The test results reveal the following key insights:

• Model Achieves Acceptable Discriminatory Power: The ROC AUC score of 0.6862 indicates that the model is able to distinguish between classes at a level above random chance, surpassing the minimum threshold of 0.5.
• Threshold Surpassed with Margin: The observed score exceeds the threshold by 0.1862, providing a clear margin of safety and suggesting that the model’s performance is not borderline.
• Test Outcome is Positive: The Pass/Fail status is "Pass," confirming that the model meets the predefined standard for ROC AUC on the validation dataset.
• Single Metric Focus: The results are presented as a single summary statistic, emphasizing overall model performance without class-specific or subgroup breakdowns.

Based on these results, the model demonstrates a level of discriminatory power that is considered acceptable according to the predefined ROC AUC threshold. The score of 0.6862, while not approaching perfect discrimination, is notably above the random baseline and provides evidence that the model is capable of ranking instances with a reasonable degree of accuracy. The margin by which the score exceeds the threshold suggests that the model’s performance is stable and not at risk of failing this criterion under minor fluctuations in data or model parameters. The test’s focus on a single, aggregate metric means that the results provide a high-level view of model quality, suitable for initial validation or regulatory reporting. However, the absence of class-specific analysis means that further investigation may be warranted if more granular insights are required. Overall, the model’s ability to pass this test supports its use in applications where a minimum standard of classification performance is necessary, and the results provide a clear, objective benchmark for ongoing monitoring and comparison.

Tables

Score	Threshold	Pass/Fail
0.6862	0.5	Pass

▶ Test Result: Permutation Feature Importance (validmind.model_validation.sklearn.PermutationFeatureImportance)

Permutation Feature Importance

Permutation Feature Importance is designed to assess the significance of each feature in a machine learning model by quantifying the impact on model performance when the values of individual features are randomly rearranged. The primary purpose of this test is to identify which features the model relies on most for its predictions, thereby providing transparency into the model’s decision-making process and highlighting the relative contribution of each input variable.

The test operates by systematically permuting, or shuffling, the values of each feature in the dataset one at a time and then measuring the resulting change in the model’s predictive performance. This is accomplished using the permutation_importance method from the sklearn.inspection module, which evaluates the decrease in a chosen performance metric—such as accuracy, F1 score, or another relevant measure—after each feature is permuted. The underlying logic is that if a feature is important, disrupting its relationship with the target variable will lead to a noticeable drop in model performance, whereas permuting an unimportant feature will have little to no effect. The output is typically a set of importance scores, often visualized as a bar plot, where higher values indicate greater importance. These scores are non-negative and generally range from zero upwards, with larger values signifying a more substantial impact on model performance. Interpretation of these values should consider the context of the model and the expected influence of each feature.

The primary advantages of this test include its model-agnostic nature, allowing it to be applied to any predictive model that provides a performance metric, regardless of the underlying algorithm. It offers clear, interpretable insights into which features drive model predictions, which can be particularly valuable for model validation, regulatory review, and stakeholder communication. The test can also help detect overfitting by revealing if the model is overly dependent on a small subset of features, and it may uncover unexpected patterns or relationships in the data that warrant further investigation. Additionally, because the method does not require retraining the model, it is computationally efficient and straightforward to implement in most machine learning workflows.

It should be noted that permutation feature importance does not imply causality; it only measures the extent to which a feature contributes to the model’s predictive accuracy. The method does not account for interactions or correlations between features, which can result in misleading importance scores if two or more features are highly correlated—one may appear important while the other does not, even though both are relevant. The test may also be less reliable for features with high cardinality or those that are easily permuted without affecting the target. Furthermore, the approach is limited to models compatible with the sklearn ecosystem and may not be directly applicable to models built with other libraries such as statsmodels, pytorch, or catboost. Signs of high risk include heavy reliance on features with unstable or easily manipulated values, or the absence of importance for features that domain knowledge suggests should be influential.

This test shows a horizontal bar plot titled "Permutation Importances," which visually represents the relative importance of each feature in the model. Each bar corresponds to a feature, with the length of the bar indicating the magnitude of the decrease in model performance when that feature is permuted. The x-axis represents the importance score, which quantifies the impact on the model’s predictive accuracy, while the y-axis lists the feature names. The scale of the x-axis ranges from 0 to approximately 0.08, with higher values denoting greater importance. Notable observations include three features—IsActiveMember, Geography_Germany, and Gender_Male—showing the highest importance scores, with IsActiveMember leading at just above 0.07, followed closely by Geography_Germany at slightly above 0.06, and Gender_Male at just above 0.04. Other features such as NumOfProducts, Tenure, and Geography_Spain display much lower importance, with scores below 0.01. Features like EstimatedSalary, Balance, CreditScore, and HasCrCard have minimal to negligible importance, as indicated by their very short bars. This distribution suggests a strong reliance on a small subset of features, with the majority contributing little to the model’s predictive capability.

The test results reveal the following key insights:

• Model relies most on IsActiveMember, Geography_Germany, and Gender_Male: These three features have the highest permutation importance scores, with IsActiveMember at approximately 0.07, Geography_Germany at just over 0.06, and Gender_Male at just above 0.04, indicating that the model’s predictions are most sensitive to changes in these variables.
• Limited contribution from remaining features: Features such as NumOfProducts, Tenure, and Geography_Spain show much lower importance scores, all below 0.01, suggesting that their influence on the model’s predictions is minimal.
• Negligible impact from EstimatedSalary, Balance, CreditScore, and HasCrCard: These features have importance scores close to zero, indicating that permuting their values does not meaningfully affect model performance.
• Clear separation between high and low importance features: The results display a distinct gap between the top three features and the rest, highlighting a concentrated reliance on a small subset of inputs.
• No evidence of strong feature interaction effects: The absence of moderate importance scores among the remaining features suggests that the model does not capture significant interactions or dependencies involving these variables.

Based on these results, the model demonstrates a pronounced dependence on IsActiveMember, Geography_Germany, and Gender_Male, with these features collectively accounting for the majority of the model’s predictive power. The sharp drop-off in importance among the remaining features indicates that the model’s decision-making process is driven by a narrow set of variables, while most other features contribute little to the outcome. This pattern may reflect the underlying data structure or the model’s ability to extract relevant information from these key features. The negligible importance of features such as EstimatedSalary, Balance, CreditScore, and HasCrCard suggests that, within the context of this model and dataset, these variables do not provide additional predictive value. The results also imply that the model does not leverage complex interactions among the less important features, as evidenced by the lack of intermediate importance scores. Overall, the observed distribution of feature importances provides a clear and interpretable view of the model’s internal logic, highlighting both the dominant drivers of prediction and the limited role of other inputs.

Figures

▶ Test Result: SHAP Global Importance (validmind.model_validation.sklearn.SHAPGlobalImportance)

SHAP Global Importance

SHAP Global Importance is designed to evaluate and visualize the global importance of features in a predictive model by leveraging SHAP (SHapley Additive exPlanations) values. The primary purpose of this test is to provide a transparent and quantifiable breakdown of how each input feature contributes to the model’s overall predictions, thereby supporting model interpretability and risk identification in both classification and regression contexts.

The test operates by first selecting an appropriate SHAP explainer based on the underlying model architecture, such as TreeExplainer for tree-based models or LinearExplainer for linear models. The explainer computes Shapley values for each feature across all instances in the dataset, quantifying the marginal contribution of each feature to the model’s output. These values are then aggregated to produce two key visualizations: the mean importance plot and the summary plot. The mean importance plot displays the average absolute Shapley value for each feature, normalized to a percentage scale, which highlights the relative global importance of each feature. The summary plot provides a more granular view, showing the distribution of Shapley values for each feature across all samples, with color gradients representing the feature values. The SHAP value itself measures the impact of a feature on the model’s prediction, with values typically ranging from negative to positive, where the magnitude indicates the strength of the effect and the sign indicates the direction. High absolute SHAP values suggest strong influence, while values near zero indicate minimal impact. These visualizations collectively allow for the identification of dominant features, potential overreliance, and the overall distribution of feature effects.

The primary advantages of this test include its ability to deliver both global and local interpretability, making it possible to understand not only which features are most influential overall but also how they affect individual predictions. By quantifying feature importance in a consistent and theoretically grounded manner, SHAP values facilitate transparent model governance and support regulatory compliance. The visualizations produced by this test are intuitive and actionable, enabling stakeholders to quickly identify which features drive model behavior and to detect any unexpected patterns. This is particularly valuable in high-stakes domains where model decisions must be explainable and justifiable. Furthermore, the test’s methodology is model-agnostic, allowing it to be applied across a wide range of machine learning algorithms, and its ability to highlight both the magnitude and direction of feature effects provides a nuanced understanding of model logic.

It should be noted that the test has several limitations and potential risks. In high-dimensional datasets, the interpretability of SHAP plots can become challenging, as the presence of many features may obscure meaningful patterns and make it difficult to discern which features are truly important. Additionally, while SHAP values provide a robust measure of feature importance, the translation of these values into real-world business impact remains subjective and context-dependent. There is also a risk of overemphasis on certain features, as indicated by disproportionately high SHAP values, which may signal model overfitting or reliance on spurious correlations. Anomalies such as unexpected features with high importance or highly scattered summary plots may indicate that the model is basing its decisions on undesirable or illogical reasoning. Careful interpretation is required to ensure that the observed feature importances align with domain knowledge and business objectives.

This test shows the results in the form of two primary visualizations: a normalized mean SHAP importance bar plot and a SHAP summary plot. The mean importance plot ranks features by their normalized average absolute SHAP values, expressed as percentages, with the most influential features at the top. The summary plot displays the distribution of SHAP values for each feature, with the horizontal axis representing the SHAP value (impact on model output) and the vertical axis listing the features in order of importance. Each point in the summary plot corresponds to a single instance’s SHAP value for a given feature, colored according to the feature’s value (from low to high). The range of SHAP values on the summary plot typically spans from negative to positive, indicating whether a feature pushes the prediction higher or lower. In the provided results, "IsActiveMember" stands out as the most important feature, followed by "Geography_Germany" and "Gender_Male," each with normalized SHAP values approaching or exceeding 60%. The remaining features, such as "Balance," "NumOfProducts," and "Geography_Spain," show progressively lower importance, with normalized values below 30%. The summary plot reveals the spread and directionality of each feature’s impact, with some features exhibiting a wide range of SHAP values, indicating variable influence across instances, while others cluster tightly around zero, suggesting limited effect. Notably, the color gradients in the summary plot help to identify whether high or low feature values are associated with positive or negative impacts on the model output.

The test results reveal the following key insights:

• Dominance of IsActiveMember in Model Decisions: The feature "IsActiveMember" exhibits the highest normalized SHAP value, reaching 100%, indicating that it is the most influential factor in the model’s predictions.
• Significant Impact of Geography_Germany and Gender_Male: "Geography_Germany" and "Gender_Male" follow closely, with normalized SHAP values of approximately 80% and 60%, respectively, highlighting their substantial roles in shaping model outcomes.
• Secondary Importance of Balance and NumOfProducts: "Balance" and "NumOfProducts" display moderate importance, with normalized SHAP values around 30% and 15%, suggesting they contribute meaningfully but are not primary drivers.
• Limited Influence of Remaining Features: Features such as "Geography_Spain," "EstimatedSalary," "HasCrCard," "Tenure," and "CreditScore" all have normalized SHAP values below 15%, indicating minimal impact on the model’s overall predictions.
• Consistent Directionality and Spread in SHAP Summary Plot: The summary plot shows that the most important features have a wide spread of SHAP values, with both positive and negative impacts, while less important features cluster near zero, reflecting limited variability in their influence.
• Color Gradient Reveals Feature Value Effects: The color mapping in the summary plot demonstrates that for certain features, higher or lower values are consistently associated with either increasing or decreasing the model output, providing additional interpretive depth.

Based on these results, the model’s predictive behavior is primarily driven by a small subset of features, with "IsActiveMember," "Geography_Germany," and "Gender_Male" accounting for the majority of the explained variance in model output. The normalized mean SHAP importance plot clearly delineates the hierarchy of feature influence, while the summary plot provides a detailed view of how individual feature values affect predictions across the dataset. The concentration of high SHAP values among a few features suggests that the model relies heavily on these variables, which may warrant further scrutiny to ensure that this reliance is justified and not indicative of overfitting or spurious correlations. The limited importance of other features indicates that their contribution to model decisions is minimal, and their effects are largely stable and consistent across instances. The observed color gradients in the summary plot further clarify the relationship between feature values and their impact on predictions, supporting a nuanced understanding of model logic. Overall, the results provide a transparent and quantifiable account of feature importance, enabling stakeholders to assess the alignment of model behavior with domain expectations and regulatory requirements.

Figures

▶ Test Result: Weakspots Diagnosis (validmind.model_validation.sklearn.WeakspotsDiagnosis)

❌ Weakspots Diagnosis

Weakspots Diagnosis is designed to identify and visualize regions within the feature space of a machine learning model where performance metrics fall below specified thresholds, thereby highlighting areas where the model may be underperforming. The primary purpose of this test is to provide a granular assessment of model behavior across different segments of input features, enabling a detailed understanding of where the model’s predictions may be less reliable or consistent.

The test operates by partitioning the feature space into discrete bins for each selected feature, such as Balance, CreditScore, EstimatedSalary, Tenure, NumOfProducts, and categorical indicators like Gender_Male, Geography_Germany, Geography_Spain, HasCrCard, and IsActiveMember. For each bin, the test computes key performance metrics—accuracy, precision, recall, and F1 score—on both the training and test datasets. Accuracy measures the proportion of correct predictions, ranging from 0 (no correct predictions) to 1 (all correct). Precision quantifies the proportion of positive identifications that are actually correct, also ranging from 0 to 1, with higher values indicating fewer false positives. Recall assesses the proportion of actual positives that are correctly identified, again from 0 to 1, with higher values reflecting fewer false negatives. The F1 score is the harmonic mean of precision and recall, providing a balanced measure that penalizes extreme values in either metric. Each metric is compared against a user-defined threshold (e.g., 0.75 for accuracy, 0.5 for precision and recall, 0.7 for F1), and bins where the test set performance falls below these thresholds are flagged as weak spots. The results are visualized as bar charts, with separate bars for training and test performance in each bin, and a dashed line indicating the threshold for each metric, facilitating direct visual comparison.

The primary advantages of this test include its ability to pinpoint specific regions of the feature space where the model’s predictive performance is suboptimal, which is particularly valuable for risk-sensitive applications or regulatory review. By providing a breakdown of performance by feature bin, the test enables targeted investigation into the causes of low performance, such as data sparsity, feature interactions, or model bias. The use of multiple metrics ensures a comprehensive evaluation, capturing different aspects of model behavior. The graphical output makes it easy to identify patterns, such as overfitting (where training performance is high but test performance is low) or systematic underperformance in certain feature ranges. The flexibility to set custom thresholds for each metric allows the test to be tailored to the specific requirements and risk tolerances of different business contexts.

It should be noted that the test’s reliance on binning can introduce arbitrariness, as the choice of bin edges and the number of bins may affect the granularity and interpretability of the results. Bins with very few records may yield unstable or unrepresentative metrics, potentially leading to misleading conclusions about model performance in those regions. The effectiveness of the test is also contingent on the appropriateness of the chosen thresholds, which may not be universally applicable and should be selected with domain knowledge. The test is limited to numerical and categorical features, excluding text data, and does not provide direct guidance on how to remediate observed weak spots. Additionally, significant disparities between training and test performance within a bin may indicate overfitting, but the test does not diagnose the underlying causes or suggest corrective actions. Users should interpret results with caution, especially in bins with low sample sizes or where feature distributions differ markedly between training and test sets.

This test shows a comprehensive set of bar charts and tabular data summarizing model performance across bins for each feature. Each plot displays the metric value (accuracy, precision, recall, F1) on the y-axis and the feature bin on the x-axis, with separate bars for training and test datasets. A red dashed line marks the threshold for each metric, making it easy to see which bins fall below the required standard. The tables provide the exact metric values, number of records per bin, and dataset split, allowing for detailed inspection. For example, in the CreditScore feature, accuracy in the test set ranges from 0.5455 to 0.8235 across bins, with only the (450.0, 500.0] bin exceeding the 0.75 threshold. Precision and recall also vary, with some bins falling below the 0.5 threshold, particularly in lower and higher score ranges. Similar patterns are observed for other features: Balance shows low accuracy and F1 in mid-range bins, while EstimatedSalary and Tenure exhibit more stable but generally sub-threshold performance. Categorical features like Gender_Male and Geography_Germany reveal disparities in recall and F1 between groups, with some bins consistently underperforming. The visualizations make it clear where the model’s predictions are less reliable, and the tables provide the quantitative detail needed for further analysis. Notably, bins with very few records (e.g., Balance or NumOfProducts at extreme values) often show volatile metrics, underscoring the importance of considering sample size when interpreting results.

The test results reveal the following key insights:

• Model performance is consistently below threshold across most feature bins: For all major features, including CreditScore, Balance, EstimatedSalary, and Tenure, the majority of bins in the test set do not meet the specified accuracy (0.75) or F1 (0.7) thresholds, with only isolated bins exceeding these standards.
• Performance metrics vary substantially across feature bins: There is significant variation in accuracy, precision, recall, and F1 scores within each feature. For example, CreditScore test accuracy ranges from 0.5455 to 0.8235, and F1 from 0.5455 to 0.8235, indicating that model reliability is highly dependent on the input value range.
• Disparities between training and test performance suggest overfitting in certain regions: In several bins, particularly for Balance and NumOfProducts, training metrics are notably higher than test metrics, especially in bins with more records, indicating potential overfitting or data distribution shifts.
• Bins with low record counts exhibit unstable metrics: Bins with very few records, such as extreme values of Balance or NumOfProducts, show highly variable and sometimes extreme metric values (e.g., precision or recall of 1.0 or 0.0), which may not be representative of true model performance.
• Categorical features reveal group-specific performance differences: For features like Gender_Male and Geography_Germany, recall and F1 scores are substantially higher in one group compared to the other, with, for example, recall for Geography_Germany at (0.9, 1.0] reaching 0.8786 in the test set, while the other bin is much lower.
• Some feature bins show relatively stable performance across datasets: In features such as EstimatedSalary and Tenure, the difference between training and test metrics is less pronounced, suggesting more consistent model behavior in these regions, though still generally below threshold.
• No single feature or bin achieves uniformly high performance across all metrics: Even in bins where one metric exceeds the threshold, others often do not, indicating that model strengths are localized and do not generalize across all aspects of predictive quality.

Based on these results, the model demonstrates a pattern of sub-threshold performance across the majority of the feature space, with only a few isolated bins achieving the desired accuracy or F1 levels. The observed variability in metrics across bins highlights that the model’s predictive reliability is highly dependent on specific input ranges, and that certain regions—particularly those with low sample sizes—are prone to unstable or extreme metric values. Disparities between training and test performance in several bins suggest that the model may be overfitting to the training data in those regions, especially where the training metrics are substantially higher than those on the test set. Categorical features reveal that group membership can have a pronounced effect on model performance, with some groups consistently outperforming others in recall and F1. The overall pattern indicates that while the model may perform adequately in select regions, it does not achieve consistent, high-quality predictions across the full feature space. These insights collectively suggest that the model’s behavior is heterogeneous, with localized strengths and weaknesses that are clearly delineated by feature value, and that further investigation into the causes of low performance in specific bins—such as data sparsity, feature interactions, or distributional differences—would be necessary to fully understand and address the observed patterns.

Tables

Slice	Number of Records	Feature	Accuracy	Precision	Recall	F1	Dataset
(-238.388, 23838.756]	203	Balance	0.6650	0.5909	0.3421	0.4333	test_dataset_final
(23838.756, 47677.512]	6	Balance	0.3333	0.5000	0.2500	0.3333	test_dataset_final
(47677.512, 71516.268]	19	Balance	0.4737	0.5556	0.4545	0.5000	test_dataset_final
(71516.268, 95355.024]	70	Balance	0.6714	0.6667	0.6857	0.6761	test_dataset_final
(95355.024, 119193.78]	136	Balance	0.6985	0.7882	0.7444	0.7657	test_dataset_final
(119193.78, 143032.536]	136	Balance	0.6397	0.6404	0.7703	0.6994	test_dataset_final
(143032.536, 166871.292]	55	Balance	0.5636	0.5294	0.6923	0.6000	test_dataset_final
(166871.292, 190710.048]	18	Balance	0.5556	0.7143	0.7143	0.7143	test_dataset_final
(190710.048, 214548.804]	3	Balance	0.6667	0.5000	1.0000	0.6667	test_dataset_final
(214548.804, 238387.56]	0	Balance	0.0000	0.0000	0.0000	0.0000	test_dataset_final
(-238.388, 23838.756]	811	Balance	0.6436	0.5928	0.3538	0.4432	train_dataset_final
(23838.756, 47677.512]	18	Balance	0.5556	0.8000	0.3636	0.5000	train_dataset_final
(47677.512, 71516.268]	73	Balance	0.6301	0.7391	0.4474	0.5574	train_dataset_final
(71516.268, 95355.024]	224	Balance	0.5670	0.5596	0.5545	0.5571	train_dataset_final
(95355.024, 119193.78]	542	Balance	0.6587	0.6552	0.7782	0.7114	train_dataset_final
(119193.78, 143032.536]	531	Balance	0.6460	0.6656	0.7331	0.6977	train_dataset_final
(143032.536, 166871.292]	270	Balance	0.6185	0.6242	0.7153	0.6667	train_dataset_final
(166871.292, 190710.048]	89	Balance	0.5955	0.5849	0.6889	0.6327	train_dataset_final
(190710.048, 214548.804]	24	Balance	0.3750	0.7000	0.3684	0.4828	train_dataset_final
(214548.804, 238387.56]	3	Balance	0.6667	1.0000	0.6667	0.8000	train_dataset_final
(349.5, 400.0]	5	CreditScore	0.6000	1.0000	0.6000	0.7500	test_dataset_final
(400.0, 450.0]	11	CreditScore	0.5455	0.5000	0.6000	0.5455	test_dataset_final
(450.0, 500.0]	34	CreditScore	0.8235	0.8750	0.7778	0.8235	test_dataset_final
(500.0, 550.0]	64	CreditScore	0.5625	0.5882	0.5882	0.5882	test_dataset_final
(550.0, 600.0]	93	CreditScore	0.6774	0.7143	0.6863	0.7000	test_dataset_final
(600.0, 650.0]	112	CreditScore	0.5446	0.5745	0.4655	0.5143	test_dataset_final
(650.0, 700.0]	132	CreditScore	0.6591	0.6154	0.6667	0.6400	test_dataset_final
(700.0, 750.0]	87	CreditScore	0.6897	0.7429	0.5909	0.6582	test_dataset_final
(750.0, 800.0]	55	CreditScore	0.6909	0.7143	0.6897	0.7018	test_dataset_final
(800.0, 850.0]	54	CreditScore	0.6667	0.6562	0.7500	0.7000	test_dataset_final
(349.5, 400.0]	9	CreditScore	0.6667	1.0000	0.6667	0.8000	train_dataset_final
(400.0, 450.0]	55	CreditScore	0.5636	0.5862	0.5862	0.5862	train_dataset_final
(450.0, 500.0]	116	CreditScore	0.6552	0.6290	0.6964	0.6610	train_dataset_final
(500.0, 550.0]	261	CreditScore	0.6858	0.7308	0.6690	0.6985	train_dataset_final
(550.0, 600.0]	363	CreditScore	0.6501	0.6611	0.6432	0.6521	train_dataset_final
(600.0, 650.0]	508	CreditScore	0.6181	0.5887	0.6134	0.6008	train_dataset_final
(650.0, 700.0]	490	CreditScore	0.6245	0.6147	0.5992	0.6068	train_dataset_final
(700.0, 750.0]	404	CreditScore	0.6312	0.6404	0.5729	0.6048	train_dataset_final
(750.0, 800.0]	236	CreditScore	0.6271	0.6316	0.6102	0.6207	train_dataset_final
(800.0, 850.0]	143	CreditScore	0.5944	0.6140	0.4930	0.5469	train_dataset_final
(-108.112, 20077.908]	60	EstimatedSalary	0.6833	0.6786	0.6552	0.6667	test_dataset_final
(20077.908, 40064.066]	58	EstimatedSalary	0.7241	0.6552	0.7600	0.7037	test_dataset_final
(40064.066, 60050.224]	57	EstimatedSalary	0.6140	0.6667	0.6667	0.6667	test_dataset_final
(60050.224, 80036.382]	57	EstimatedSalary	0.5439	0.4643	0.5417	0.5000	test_dataset_final
(80036.382, 100022.54]	91	EstimatedSalary	0.7253	0.6792	0.8182	0.7423	test_dataset_final
(100022.54, 120008.698]	62	EstimatedSalary	0.6129	0.6452	0.6061	0.6250	test_dataset_final
(120008.698, 139994.856]	69	EstimatedSalary	0.6957	0.7308	0.5758	0.6441	test_dataset_final
(139994.856, 159981.014]	58	EstimatedSalary	0.6207	0.7083	0.5312	0.6071	test_dataset_final
(159981.014, 179967.172]	72	EstimatedSalary	0.6111	0.6486	0.6154	0.6316	test_dataset_final
(179967.172, 199953.33]	63	EstimatedSalary	0.5873	0.7692	0.5000	0.6061	test_dataset_final
(-108.112, 20077.908]	266	EstimatedSalary	0.6654	0.6689	0.7122	0.6899	train_dataset_final
(20077.908, 40064.066]	244	EstimatedSalary	0.6598	0.6935	0.6565	0.6745	train_dataset_final
(40064.066, 60050.224]	247	EstimatedSalary	0.6478	0.6471	0.6311	0.6390	train_dataset_final
(60050.224, 80036.382]	288	EstimatedSalary	0.6493	0.6597	0.6463	0.6529	train_dataset_final
(80036.382, 100022.54]	230	EstimatedSalary	0.5522	0.5304	0.5545	0.5422	train_dataset_final
(100022.54, 120008.698]	274	EstimatedSalary	0.6058	0.5781	0.5781	0.5781	train_dataset_final
(120008.698, 139994.856]	259	EstimatedSalary	0.6332	0.6500	0.5954	0.6215	train_dataset_final
(139994.856, 159981.014]	257	EstimatedSalary	0.6304	0.6034	0.5882	0.5957	train_dataset_final
(159981.014, 179967.172]	268	EstimatedSalary	0.6381	0.6972	0.5429	0.6104	train_dataset_final
(179967.172, 199953.33]	252	EstimatedSalary	0.6389	0.6161	0.5897	0.6026	train_dataset_final
(-0.001, 0.1]	307	Gender_Male	0.6645	0.6965	0.7692	0.7311	test_dataset_final
(0.9, 1.0]	340	Gender_Male	0.6294	0.6053	0.4600	0.5227	test_dataset_final
(-0.001, 0.1]	1273	Gender_Male	0.6363	0.6451	0.7815	0.7068	train_dataset_final
(0.9, 1.0]	1312	Gender_Male	0.6296	0.6135	0.3982	0.4830	train_dataset_final
(-0.001, 0.1]	439	Geography_Germany	0.6378	0.6187	0.4479	0.5196	test_dataset_final
(0.9, 1.0]	208	Geography_Germany	0.6635	0.6989	0.8786	0.7785	test_dataset_final
(-0.001, 0.1]	1789	Geography_Germany	0.6182	0.5759	0.4286	0.4914	train_dataset_final
(0.9, 1.0]	796	Geography_Germany	0.6658	0.6873	0.8852	0.7738	train_dataset_final
(-0.001, 0.1]	504	Geography_Spain	0.6488	0.6844	0.6569	0.6704	test_dataset_final
(0.9, 1.0]	143	Geography_Spain	0.6364	0.5577	0.5000	0.5273	test_dataset_final
(-0.001, 0.1]	1985	Geography_Spain	0.6443	0.6515	0.6502	0.6508	train_dataset_final
(0.9, 1.0]	600	Geography_Spain	0.5950	0.5644	0.4669	0.5111	train_dataset_final
(-0.001, 0.1]	179	HasCrCard	0.6201	0.5816	0.6786	0.6264	test_dataset_final
(0.9, 1.0]	468	HasCrCard	0.6560	0.7005	0.6129	0.6538	test_dataset_final
(-0.001, 0.1]	808	HasCrCard	0.6126	0.6198	0.6122	0.6160	train_dataset_final
(0.9, 1.0]	1777	HasCrCard	0.6421	0.6434	0.6110	0.6268	train_dataset_final
(-0.001, 0.1]	352	IsActiveMember	0.6449	0.6680	0.8048	0.7300	test_dataset_final
(0.9, 1.0]	295	IsActiveMember	0.6475	0.6452	0.3279	0.4348	test_dataset_final
(-0.001, 0.1]	1366	IsActiveMember	0.6208	0.6448	0.8010	0.7144	train_dataset_final
(0.9, 1.0]	1219	IsActiveMember	0.6464	0.5957	0.2884	0.3887	train_dataset_final
(0.997, 1.3]	382	NumOfProducts	0.6518	0.7285	0.6880	0.7077	test_dataset_final
(1.9, 2.2]	208	NumOfProducts	0.6827	0.3235	0.5238	0.4000	test_dataset_final
(2.8, 3.1]	42	NumOfProducts	0.4524	1.0000	0.4390	0.6102	test_dataset_final
(3.7, 4.0]	15	NumOfProducts	0.5333	1.0000	0.5333	0.6957	test_dataset_final
(0.997, 1.3]	1510	NumOfProducts	0.6245	0.6980	0.6359	0.6655	train_dataset_final
(1.9, 2.2]	904	NumOfProducts	0.6637	0.3881	0.5677	0.4610	train_dataset_final
(2.8, 3.1]	142	NumOfProducts	0.5775	0.9877	0.5755	0.7273	train_dataset_final
(3.7, 4.0]	29	NumOfProducts	0.3793	1.0000	0.3793	0.5500	train_dataset_final
(-0.01, 1.0]	95	Tenure	0.6842	0.7500	0.6964	0.7222	test_dataset_final
(1.0, 2.0]	67	Tenure	0.6119	0.5278	0.6786	0.5938	test_dataset_final
(2.0, 3.0]	60	Tenure	0.5167	0.5667	0.5152	0.5397	test_dataset_final
(3.0, 4.0]	68	Tenure	0.6471	0.7027	0.6667	0.6842	test_dataset_final
(4.0, 5.0]	57	Tenure	0.7018	0.6207	0.7500	0.6792	test_dataset_final
(5.0, 6.0]	70	Tenure	0.5143	0.5161	0.4571	0.4848	test_dataset_final
(6.0, 7.0]	64	Tenure	0.7188	0.7931	0.6571	0.7188	test_dataset_final
(7.0, 8.0]	59	Tenure	0.7119	0.7273	0.5926	0.6531	test_dataset_final
(8.0, 9.0]	72	Tenure	0.7222	0.7647	0.6842	0.7222	test_dataset_final
(9.0, 10.0]	35	Tenure	0.6000	0.6000	0.5294	0.5625	test_dataset_final
(-0.01, 1.0]	377	Tenure	0.6472	0.6667	0.6429	0.6545	train_dataset_final
(1.0, 2.0]	280	Tenure	0.5929	0.5597	0.5769	0.5682	train_dataset_final
(2.0, 3.0]	267	Tenure	0.6217	0.6449	0.6312	0.6380	train_dataset_final
(3.0, 4.0]	247	Tenure	0.5789	0.6129	0.5758	0.5938	train_dataset_final
(4.0, 5.0]	281	Tenure	0.6441	0.6268	0.6544	0.6403	train_dataset_final
(5.0, 6.0]	244	Tenure	0.6639	0.6637	0.6303	0.6466	train_dataset_final
(6.0, 7.0]	236	Tenure	0.6144	0.5556	0.5392	0.5473	train_dataset_final
(7.0, 8.0]	267	Tenure	0.6929	0.7257	0.6165	0.6667	train_dataset_final
(8.0, 9.0]	260	Tenure	0.6269	0.6160	0.6111	0.6135	train_dataset_final
(9.0, 10.0]	126	Tenure	0.6429	0.7069	0.5942	0.6457	train_dataset_final

Figures

▶ Test Result: Overfit Diagnosis (validmind.model_validation.sklearn.OverfitDiagnosis)

Overfit Diagnosis

Overfit Diagnosis is designed to assess potential overfitting in a model’s predictions by identifying regions where performance between training and testing sets deviates significantly. The primary purpose of this test is to pinpoint specific feature segments or regions where the model may be overfitting, thereby providing targeted insights into the model’s generalization behavior across different data partitions.

The test operates by comparing the model’s performance on training and test datasets, grouped by feature columns. For each feature, the data is segmented into bins or slices, and the model’s predictive performance is evaluated separately for each segment using a relevant metric—Area Under the Curve (AUC) for classification models and Mean Squared Error (MSE) for regression models. The test then calculates the difference in performance between the training and test sets for each segment, referred to as the “gap.” This gap quantifies the extent to which the model’s performance on the training data exceeds that on the test data within each feature segment. The typical range for the AUC metric is from 0 to 1, where higher values indicate better discrimination between classes, and a gap close to zero suggests similar performance on both datasets. A positive gap above the default threshold of 0.04 is interpreted as a potential sign of overfitting in that segment, as it indicates the model is performing substantially better on the training data than on the test data. The results are visualized as bar plots, with the AUC gap on the y-axis and feature segments on the x-axis, and a horizontal line marking the overfitting threshold for easy identification of high-risk regions.

The primary advantages of this test include its ability to localize overfitting to specific feature segments, which is particularly useful for diagnosing and understanding model behavior in a granular manner. By supporting both classification and regression models and allowing for flexible metric selection, the test can be adapted to a wide range of modeling scenarios. The visualization of overfitting regions provides an intuitive and accessible means for stakeholders to interpret the results, facilitating communication and debugging. This targeted approach enables practitioners to focus their attention on problematic areas rather than relying solely on global performance metrics, which may obscure localized issues. Additionally, the test’s design allows for the identification of both severe and moderate overfitting, depending on the chosen threshold, making it a versatile tool for model risk management and validation workflows.

It should be noted that the test’s effectiveness is influenced by several limitations and potential risks. The default threshold for identifying overfitting regions may not be universally appropriate and may require adjustment based on the specific context or risk tolerance of the application. The test assumes that the binning of features adequately captures meaningful data segments; poor binning choices can lead to misleading results or mask subtle forms of overfitting. Furthermore, the test may not detect more nuanced or distributed overfitting patterns that do not exceed the threshold in any single segment but may still impact overall model reliability. Interpretation challenges can arise when the number of records in certain segments is low, as small sample sizes can lead to unstable or noisy performance estimates. Signs of high risk include significant gaps between training and test performance metrics for specific feature segments, multiple regions with gaps exceeding the threshold, and higher than expected differences in predicted versus actual values in the test set compared to the training set.

This test shows the results in both tabular and graphical formats. The table presents, for each feature and its respective segment, the number of training and test records, the AUC for both datasets, and the calculated gap. The plots visualize the AUC gap for each segment, with the x-axis representing the feature bins and the y-axis showing the magnitude of the gap. A red dashed line marks the threshold of 0.04, above which segments are flagged for potential overfitting. The range of AUC gaps observed spans from negative values (indicating better test performance) to positive values well above the threshold, with the highest observed gap being 0.2484 for the “Balance” feature in the (166871.292, 190710.048] segment. Notable regions with substantial gaps include “CreditScore” (0.1467 in the (500.0, 550.0] segment), “Tenure” (0.129 in the (9.0, 10.0] segment), and “EstimatedSalary” (0.1472 in the (60050.224, 80036.382] segment). The plots provide a clear visual indication of which feature segments exceed the overfitting threshold, with the majority of flagged regions showing positive gaps, suggesting the model is overfitting in those areas. The number of records per segment varies, with some segments containing relatively few records, which may affect the stability of the gap estimates. Overall, the results highlight both the distribution and magnitude of overfitting across multiple features and segments.

The test results reveal the following key insights:

• Overfitting is concentrated in specific feature segments: Several feature segments exhibit AUC gaps significantly above the 0.04 threshold, indicating localized overfitting rather than a uniform pattern across all features.
• Highest overfitting observed in Balance and CreditScore segments: The “Balance” feature segment (166871.292, 190710.048] shows the largest AUC gap at 0.2484, while “CreditScore” (500.0, 550.0] and “EstimatedSalary” (60050.224, 80036.382] also display substantial gaps of 0.1467 and 0.1472, respectively.
• Tenure and EstimatedSalary show multiple overfit regions: The “Tenure” feature has several segments with gaps above the threshold, notably (5.0, 6.0] at 0.125 and (9.0, 10.0] at 0.129, while “EstimatedSalary” also has multiple segments exceeding the threshold.
• Most features have at least one segment near or above the threshold: Features such as “NumOfProducts,” “Geography_Germany,” and “Balance” each have at least one segment with a gap close to or above 0.04, suggesting that overfitting is not isolated to a single feature.
• Some segments show negative or negligible gaps: Certain feature segments, such as “HasCrCard” and “IsActiveMember,” display negative or near-zero gaps, indicating either no overfitting or slightly better performance on the test set.
• Segment size impacts gap stability: Segments with fewer records, such as those in “Balance” and “Tenure,” tend to show more extreme gap values, which may reflect higher variance due to limited data.

Based on these results, the model demonstrates a pattern of localized overfitting, with the most pronounced effects observed in specific segments of the “Balance,” “CreditScore,” “Tenure,” and “EstimatedSalary” features. The presence of multiple segments with AUC gaps well above the 0.04 threshold indicates that the model’s predictive performance is substantially better on the training data than on the test data in these regions, suggesting that the model may be capturing noise or spurious patterns specific to the training set. The distribution of overfitting is not uniform; rather, it is concentrated in certain feature-value ranges, while other segments and features show minimal or no overfitting. The variability in segment sizes also contributes to the observed gaps, with smaller segments exhibiting greater fluctuations. These insights collectively indicate that while the model may generalize adequately in some areas, there are distinct regions where its reliability is compromised, particularly in higher-value segments of “Balance” and lower-to-mid segments of “CreditScore” and “EstimatedSalary.” The results underscore the importance of segment-level analysis for understanding model behavior and highlight the need for careful interpretation of performance metrics, especially in regions with limited data.

Tables

Overfit Diagnosis

Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
CreditScore	(500.0, 550.0]	261	64	0.7016	0.5549	0.1467
CreditScore	(600.0, 650.0]	508	112	0.6657	0.5859	0.0798
Tenure	(2.0, 3.0]	267	60	0.6484	0.5623	0.0861
Tenure	(5.0, 6.0]	244	70	0.7152	0.5902	0.1250
Tenure	(7.0, 8.0]	267	59	0.7451	0.7049	0.0402
Tenure	(9.0, 10.0]	126	35	0.7107	0.5817	0.1290
Balance	(47677.512, 71516.268]	73	19	0.6346	0.5682	0.0664
Balance	(143032.536, 166871.292]	270	55	0.6686	0.6260	0.0426
Balance	(166871.292, 190710.048]	89	18	0.6056	0.3571	0.2484
NumOfProducts	(1.9, 2.2]	904	208	0.6930	0.6460	0.0470
EstimatedSalary	(40064.066, 60050.224]	247	57	0.6942	0.6465	0.0478
EstimatedSalary	(60050.224, 80036.382]	288	57	0.6850	0.5379	0.1472
EstimatedSalary	(179967.172, 199953.33]	252	63	0.6909	0.6500	0.0409
Geography_Germany	(0.9, 1.0]	796	208	0.6457	0.6055	0.0402

Figures

▶ Test Result: Robustness Diagnosis (validmind.model_validation.sklearn.RobustnessDiagnosis)

✅ Robustness Diagnosis

Robustness Diagnosis is designed to assess the resilience of a machine learning model by quantifying how its predictive performance degrades when the input data is subjected to controlled perturbations, specifically through the addition of Gaussian noise. The primary purpose of this test is to simulate real-world scenarios where input data may be imperfect or corrupted, thereby providing a measure of the model’s robustness and reliability under less-than-ideal conditions.

The test operates by systematically introducing Gaussian noise to the numerical input features of the dataset at varying levels of standard deviation, referred to as perturbation sizes. For each perturbation level, the model’s performance is evaluated using the Area Under the Receiver Operating Characteristic Curve (AUC), a metric that measures the model’s ability to distinguish between classes. AUC values range from 0 to 1, where 1 indicates perfect discrimination and 0.5 suggests no better than random guessing. The test calculates the performance decay by comparing the AUC at each noise level to the baseline (no noise) AUC, quantifying the relative drop in performance. This process is repeated for both training and test datasets, and the results are aggregated into tables and visualized in plots, allowing for a clear depiction of how model performance changes as input noise increases. The test also includes a pass/fail indicator based on whether the performance decay exceeds predefined thresholds, providing an additional layer of interpretability.

The primary advantages of this test include its ability to provide actionable insights into the model’s robustness against noisy or corrupted data, which is critical for deployment in real-world environments where data quality cannot always be guaranteed. By leveraging the AUC metric, the test offers a standardized and interpretable measure of classification performance that is widely recognized in the field. The use of multiple perturbation scales enables a granular analysis of performance decay, helping to identify specific thresholds at which the model’s reliability may be compromised. The inclusion of both tabular and graphical outputs facilitates comprehensive analysis, making it easier to communicate results to both technical and non-technical stakeholders. Additionally, the test’s methodology is adaptable to both training and test datasets, supporting a thorough evaluation of model generalizability under adverse conditions.

It should be noted that the test’s reliance on Gaussian noise as the sole form of perturbation may not fully capture the diversity of real-world data corruptions, such as outliers, missing values, or non-Gaussian noise patterns. The performance thresholds used to determine pass/fail status are somewhat arbitrary and may require adjustment to align with specific business or regulatory requirements. Interpretation of the results should consider that a significant drop in performance with minimal noise, or consistent failure to meet performance standards across multiple perturbation scales, may indicate heightened risk. Furthermore, the test does not account for the impact of noise on categorical or unstructured features, potentially overlooking vulnerabilities in those areas. The visualization and aggregation of results, while informative, may mask localized effects or interactions between features and noise, necessitating further investigation for comprehensive risk assessment.

This test shows the results in both tabular and graphical formats, with the table presenting detailed metrics for each perturbation size and dataset, including the number of rows, AUC, performance decay, and pass/fail status. The plot visualizes AUC as a function of perturbation size for both the training and test datasets, with separate lines for each and horizontal dashed lines indicating the performance thresholds. The x-axis represents the perturbation size, expressed as a multiple of the standard deviation of the input features, while the y-axis shows the AUC values, where higher values indicate better model discrimination. The plot allows for easy comparison of performance trends between the training and test datasets as noise increases. Notably, the AUC remains relatively stable for small perturbations but begins to decline more noticeably at higher noise levels, particularly for the test dataset at a perturbation size of 0.5, where the AUC drops to 0.6514. The performance decay column in the table quantifies the relative reduction in AUC compared to the baseline, with values ranging from near zero at low noise levels to 0.0348 for the test dataset at the highest perturbation. All results remain above the specified performance thresholds, as indicated by the “Passed” status for each row.

The test results reveal the following key insights:

• Model performance remains stable under low noise: For both training and test datasets, AUC values show minimal decline at perturbation sizes of 0.1 and 0.2, with performance decay remaining below 0.005 for all cases, indicating strong robustness to small input perturbations.
• Performance decay accelerates at higher noise levels: At perturbation sizes of 0.3 and above, the AUC begins to decrease more rapidly, particularly in the test dataset, where the performance decay reaches 0.0154 at 0.3 and 0.0348 at 0.5, suggesting increased sensitivity to larger perturbations.
• Test dataset exhibits greater sensitivity to noise: While both datasets experience some performance degradation, the test dataset shows a sharper decline in AUC at higher perturbation sizes, dropping from 0.6862 at baseline to 0.6514 at 0.5, compared to a smaller relative decrease in the training dataset.
• All results remain above performance thresholds: Despite the observed decay, both training and test datasets maintain AUC values above the defined thresholds across all perturbation sizes, as indicated by the consistent “Passed” status in the results table.
• No abrupt performance failures observed: There are no instances of sudden or catastrophic drops in AUC, and the performance decay progresses in a gradual, predictable manner, with no evidence of instability or erratic behavior across the tested noise levels.

Based on these results, the model demonstrates a high degree of robustness to moderate levels of Gaussian noise in its numerical input features, maintaining stable AUC values and passing all performance thresholds across both training and test datasets. The observed performance decay is minimal at low perturbation sizes, indicating that the model is resilient to small input corruptions that may occur in real-world data. As the noise level increases, the test dataset shows a more pronounced decline in AUC, highlighting a potential area of increased sensitivity that warrants attention, particularly for applications where data quality may be variable. However, the absence of abrupt or excessive performance drops, even at the highest tested perturbation size, suggests that the model’s predictive capabilities remain reliable within the tested range. The results provide a clear, quantitative characterization of the model’s behavior under noisy conditions, supporting confidence in its deployment for scenarios where input data may be subject to moderate levels of corruption.

Tables

Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
Baseline (0.0)	train_dataset_final	2585	0.6805	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.6862	0.0000	True
0.1	train_dataset_final	2585	0.6794	0.0011	True
0.1	test_dataset_final	647	0.6861	0.0001	True
0.2	train_dataset_final	2585	0.6766	0.0040	True
0.2	test_dataset_final	647	0.6819	0.0043	True
0.3	train_dataset_final	2585	0.6737	0.0069	True
0.3	test_dataset_final	647	0.6707	0.0154	True
0.4	train_dataset_final	2585	0.6618	0.0187	True
0.4	test_dataset_final	647	0.6804	0.0058	True
0.5	train_dataset_final	2585	0.6676	0.0129	True
0.5	test_dataset_final	647	0.6514	0.0348	True

Figures

In summary

In this second notebook, you learned how to:

• Import a sample dataset
• Identify which tests you might want to run with ValidMind
• Initialize ValidMind datasets and model objects
• Run individual tests
• Utilize the output from tests you've run
• Log test results from sets of or individual tests as evidence to the ValidMind Platform
• Add supplementary individual test results to your documentation
• Assign model predictions to your ValidMind model objects

Next steps

Integrate custom tests

Now that you're familiar with the basics of using the ValidMind Library to run and log tests to provide evidence for your model documentation, let's learn how to incorporate your own custom tests into ValidMind: 3 — Integrate custom tests