ValidMind for development 4 — Finalize testing and documentation

Learn how to use ValidMind for your end-to-end documentation process with our introductory notebook series. In this last notebook, finalize the testing and documentation of your model and have a fully documented sample model ready for review.

We'll first use run_documentation_tests() previously covered in 2 — Start the development process to ensure that your custom test results generated in 3 — Integrate custom tests are included in your documentation. Then, we'll view and update the configuration for the entire documentation template to suit your needs.

Learn by doing

Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Developer Fundamentals

Prerequisites

In order to finalize the testing and documentation for your sample model, you'll need to first have:

• Registered a model within the ValidMind Platform with a predefined documentation template
• Installed the ValidMind Library in your local environment, allowing you to access all its features
• Learned how to import and initialize datasets for use with ValidMind
• Learned how to run and log default and custom tests with ValidMind, including from external test providers
• Inserted test-driven blocks for the results of the following tests into your model's documentation:
  • HighPearsonCorrelation:balanced_raw_dataset
  • my_test_provider.ConfusionMatrix
  • my_custom_tests.ConfusionMatrix:test_dataset_normalized

Need help with the above steps?

Refer to the first three notebooks in this series:

• 1 — Set up the ValidMind Library
• 2 — Start the development process
• 3 — Integrate custom tests

Setting up

This section should be very familiar to you now — as we performed the same actions in the previous two notebooks in this series.

Initialize the ValidMind Library

As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Development from the Document drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="documentation",
)

Note: you may need to restart the kernel to use updated packages.

2026-09-11 23:48:58,137 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model development (ID: cmalgf3qi02ce199qm3rdkl46)
📁 Document Type: model_documentation

Import sample dataset

Next, we'll import the same public Bank Customer Churn Prediction dataset from Kaggle we used in the last notebooks so that we have something to work with:

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

We'll apply a simple rebalancing technique to the dataset before continuing:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

Remove highly correlated features

Let's also quickly remove highly correlated features from the dataset using the output from a ValidMind test.

As you learned previously, before we can run tests you'll need to initialize a ValidMind dataset object:

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:

# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships between features to identify potentially redundant variables or concentrations of linear dependence. The result table lists the top correlations observed in the dataset, along with each pair’s Pearson coefficient and its Pass/Fail status relative to the configured absolute correlation threshold of 0.3. Across the 10 reported pairs, coefficients range from -0.1857 to 0.3499. Only one pair exceeds the threshold, while the remaining reported relationships are below the cutoff.

Key insights:

• Single threshold breach identified: The pair (Age, Exited) has a Pearson correlation coefficient of 0.3499, which exceeds the 0.3 threshold and is the only relationship in the reported output marked Fail.
• All other reported pairs remain below threshold: The other nine reported correlations have absolute values between 0.0369 and 0.1857, and each is marked Pass under the configured threshold.
• Reported relationships are generally weak: Aside from (Age, Exited), the largest absolute correlations in the table are (Balance, NumOfProducts) = -0.1857, (IsActiveMember, Exited) = -0.1831, and (Balance, Exited) = 0.1549, all materially below the fail threshold.
• Both positive and negative associations appear: The reported coefficients include positive values such as (Age, Exited) = 0.3499 and (Balance, Exited) = 0.1549, as well as negative values such as (Balance, NumOfProducts) = -0.1857 and (CreditScore, Exited) = -0.0369.

The reported correlation structure is concentrated around low-magnitude pairwise linear relationships, with one exception. Among the top 10 listed pairs, only Age and Exited exceed the configured threshold, while all remaining relationships are substantially smaller in absolute magnitude and pass the test criterion. Overall, the result indicates a largely limited level of pairwise linear dependence in the reported feature pairs, with a single flagged association in the output.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3499	Fail
(Balance, NumOfProducts)	-0.1857	Pass
(IsActiveMember, Exited)	-0.1831	Pass
(Balance, Exited)	0.1549	Pass
(Age, Balance)	0.0568	Pass
(NumOfProducts, Exited)	-0.0420	Pass
(Tenure, IsActiveMember)	-0.0413	Pass
(HasCrCard, IsActiveMember)	-0.0386	Pass
(NumOfProducts, IsActiveMember)	0.0380	Pass
(CreditScore, Exited)	-0.0369	Pass

# From result object, extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3499	Fail
1	(Balance, NumOfProducts)	-0.1857	Pass
2	(IsActiveMember, Exited)	-0.1831	Pass
3	(Balance, Exited)	0.1549	Pass
4	(Age, Balance)	0.0568	Pass
5	(NumOfProducts, Exited)	-0.0420	Pass
6	(Tenure, IsActiveMember)	-0.0413	Pass
7	(HasCrCard, IsActiveMember)	-0.0386	Pass
8	(NumOfProducts, IsActiveMember)	0.0380	Pass
9	(CreditScore, Exited)	-0.0369	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

# Extract feature names from the list of strings
high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

We can then re-initialize the dataset with a different input_id and the highly correlated features removed and re-run the test for confirmation:

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

# Re-run the test with the reduced feature set
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships between features to identify potentially redundant variables or multicollinearity. The result table reports the top 10 feature pairs ranked by Pearson correlation coefficient after removing duplicate and self-correlations, using an absolute threshold of 0.3 for pass/fail classification. In this run, all reported coefficients fall within a narrow range from -0.1857 to 0.1549, and each listed pair is classified as Pass under the configured threshold.

Key insights:

• No pair exceeds threshold: All 10 reported feature pairs are marked Pass, with absolute correlation values below the configured 0.3 threshold.
• Weak relationships throughout: The largest absolute coefficient is 0.1857 for (Balance, NumOfProducts), followed by 0.1831 for (IsActiveMember, Exited), indicating that even the strongest observed linear relationships are weak in magnitude.
• Top correlations are mixed in sign: The reported coefficients include both negative and positive values, with the strongest negative relationships observed for (Balance, NumOfProducts) at -0.1857 and (IsActiveMember, Exited) at -0.1831, and the strongest positive relationship observed for (Balance, Exited) at 0.1549.
• Target-related correlations remain low: Feature pairs involving Exited in the reported top 10 show coefficients of -0.1831, 0.1549, -0.0420, -0.0369, and -0.0321, all below the threshold and low in absolute magnitude.

The reported correlation structure shows low pairwise linear dependence across the top-ranked feature combinations under the 0.3 threshold. None of the listed relationships meet the test’s criterion for high Pearson correlation, and the strongest observed coefficients remain materially below the threshold. Within the reported results, the linear associations are limited in magnitude and do not indicate concentrated pairwise correlation.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Balance, NumOfProducts)	-0.1857	Pass
(IsActiveMember, Exited)	-0.1831	Pass
(Balance, Exited)	0.1549	Pass
(NumOfProducts, Exited)	-0.0420	Pass
(Tenure, IsActiveMember)	-0.0413	Pass
(HasCrCard, IsActiveMember)	-0.0386	Pass
(NumOfProducts, IsActiveMember)	0.0380	Pass
(CreditScore, Exited)	-0.0369	Pass
(CreditScore, EstimatedSalary)	-0.0352	Pass
(Tenure, Exited)	-0.0321	Pass

Train the model

We'll then use ValidMind tests to train a simple logistic regression model on our prepared dataset:

# First encode the categorical features in our dataset with the highly correlated features removed
balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
3315	652	5	0.00	2	1	1	937.15	0	False	False	False
2148	696	9	149777.00	1	1	1	198032.93	0	False	False	True
5862	789	6	0.00	2	1	0	103603.65	0	False	False	True
6403	603	6	105000.85	2	1	1	87412.24	1	False	True	True
136	704	7	113034.22	1	1	0	162503.48	1	True	False	False

# Split the processed dataset into train and test
from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

from sklearn.linear_model import LogisticRegression

# Logistic Regression grid params
log_reg_params = {
    "penalty": ["l1", "l2"],
    "C": [0.001, 0.01, 0.1, 1, 10, 100, 1000],
    "solver": ["liblinear"],
}

# Grid search for Logistic Regression
from sklearn.model_selection import GridSearchCV

grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)
grid_log_reg.fit(X_train, y_train)

# Logistic Regression best estimator
log_reg = grid_log_reg.best_estimator_

Initialize the ValidMind objects

Let's initialize the ValidMind Dataset and Model objects in preparation for assigning model predictions to each dataset:

# Initialize the datasets into their own ValidMind dataset objects
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

# Initialize the ValidMind model object
vm_model = vm.init_model(log_reg, input_id="log_reg_model_v1")

Assign predictions

Once the model is registered, we'll assign predictions to the training and test datasets:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

2026-09-11 23:49:08,497 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-09-11 23:49:08,500 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-09-11 23:49:08,500 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-09-11 23:49:08,502 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-09-11 23:49:08,504 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-09-11 23:49:08,504 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-09-11 23:49:08,505 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-09-11 23:49:08,506 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Add custom tests

We'll also add the same custom tests we implemented in the previous notebook so that this session has access to the same custom inline test and local test provider.

Implement custom inline test

Let's set up a custom inline test that calculates the confusion matrix for a binary classification model:

# First create a confusion matrix plot
import matplotlib.pyplot as plt
from sklearn import metrics

# Get the predicted classes
y_pred = log_reg.predict(vm_test_ds.x)

confusion_matrix = metrics.confusion_matrix(y_test, y_pred)

cm_display = metrics.ConfusionMatrixDisplay(
    confusion_matrix=confusion_matrix, display_labels=[False, True]
)
cm_display.plot()

# Create the reusable ConfusionMatrix inline test with normalized matrix
@vm.test("my_custom_tests.ConfusionMatrix")
def confusion_matrix(dataset, model, normalize=False):
    """The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.

    The confusion matrix is a 2x2 table that contains 4 values:

    - True Positive (TP): the number of correct positive predictions
    - True Negative (TN): the number of correct negative predictions
    - False Positive (FP): the number of incorrect positive predictions
    - False Negative (FN): the number of incorrect negative predictions

    The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.
    """
    y_true = dataset.y
    y_pred = dataset.y_pred(model=model)

    if normalize:
        confusion_matrix = metrics.confusion_matrix(y_true, y_pred, normalize="all")
    else:
        confusion_matrix = metrics.confusion_matrix(y_true, y_pred)

    cm_display = metrics.ConfusionMatrixDisplay(
        confusion_matrix=confusion_matrix, display_labels=[False, True]
    )
    cm_display.plot()

    plt.close()  # close the plot to avoid displaying it

    return cm_display.figure_  # return the figure object itself

# Test dataset with normalize=True
result = vm.tests.run_test(
    "my_custom_tests.ConfusionMatrix:test_dataset_normalized",
    inputs={"model": vm_model, "dataset": vm_test_ds},
    params={"normalize": True},
)

Confusion Matrix Test Dataset Normalized

The ConfusionMatrix test evaluates classification performance by comparing predicted labels with true labels, and this result presents the normalized confusion matrix for the test dataset. The figure shows the proportion of observations in each true/predicted label combination, with rows corresponding to true labels and columns to predicted labels. The normalized cell values are 0.31 for true negatives, 0.22 for false positives, 0.16 for false negatives, and 0.31 for true positives.

Key insights:

• Correct classifications are balanced: The true negative and true positive cells are both 0.31, indicating equal normalized proportions of correct classifications for the negative and positive classes.

• False positives exceed false negatives: The false positive cell is 0.22, compared with 0.16 for false negatives, showing that incorrect positive predictions occur more frequently than incorrect negative predictions in the normalized matrix.

• Diagonal exceeds off-diagonal mass: The diagonal cells sum to 0.62 (0.31 + 0.31), while the off-diagonal cells sum to 0.38 (0.22 + 0.16), indicating that correct classifications account for a larger share of the normalized outcomes than misclassifications.

The normalized confusion matrix shows evenly distributed correct classification rates across both classes, with identical values for true negatives and true positives. Misclassifications are lower in aggregate than correct classifications, although the error distribution is asymmetric, with a higher normalized share of false positives than false negatives. Overall, the result reflects a balanced pattern in correct predictions alongside modestly higher positive overprediction error.

Parameters:

{
  "normalize": true
}
            

Figures

Add a local test provider

Finally, let's save our custom inline test to our local test provider:

# Create custom tests folder
tests_folder = "my_tests"

import os

# create tests folder
os.makedirs(tests_folder, exist_ok=True)

# remove existing tests
for f in os.listdir(tests_folder):
    # remove files and pycache
    if f.endswith(".py") or f == "__pycache__":
        os.system(f"rm -rf {tests_folder}/{f}")

# Save custom inline test to custom tests folder
confusion_matrix.save(
    tests_folder,
    imports=["import matplotlib.pyplot as plt", "from sklearn import metrics"],
)

2026-09-11 23:49:12,758 - INFO(validmind.tests.decorator): Saved to /home/runner/work/documentation/documentation/site/notebooks/EXECUTED/development/my_tests/ConfusionMatrix.py!Be sure to add any necessary imports to the top of the file.
2026-09-11 23:49:12,758 - INFO(validmind.tests.decorator): This metric can be run with the ID: <test_provider_namespace>.ConfusionMatrix

# Register local test provider
from validmind.tests import LocalTestProvider

# initialize the test provider with the tests folder we created earlier
my_test_provider = LocalTestProvider(tests_folder)

vm.tests.register_test_provider(
    namespace="my_test_provider",
    test_provider=my_test_provider,
)

Reconnect to ValidMind

After you insert test-driven blocks into your model documentation, changes should persist and become available every time you call vm.preview_template().

However, you'll need to reload the connection to the ValidMind Platform if you have added test-driven blocks when the connection was already established using reload():

vm.reload()

Now, when you run preview_template() again, the three test-driven blocks you added to your documentation in the last two notebooks in should show up in the template in sections 2.3 Correlations and Interactions and 3.2 Model Evaluation:

vm.preview_template()

▶ 1. Conceptual Soundness ('conceptual_soundness')

▶ 1.1. Model Overview ('model_overview')

Text Block: 'model_overview'

▶ 1.2. Intended Use and Business Use Case ('intended_use_business_use_case')

▶ 1.2.1. Intended Use ('intended_use')

Empty Section

▶ 1.2.2. Regulatory Requirements ('regulatory_requirements')

Empty Section

▶ 1.2.3. Model Limitations ('model_limitations')

Empty Section

▶ 1.3. Model Selection ('model_selection')

Empty Section

▶ 2. Data Preparation ('data_preparation')

▶ 2.1. Data description ('data_description')

Text Block: 'dataset_summary_text'

▶ Test: Dataset Description ('validmind.data_validation.DatasetDescription')

Dataset Description

Provides comprehensive analysis and statistical summaries of each column in a machine learning model's dataset.

Purpose

The test depicted in the script is meant to run a comprehensive analysis on a Machine Learning model's datasets. The test or metric is implemented to obtain a complete summary of the columns in the dataset, including vital statistics of each column such as count, distinct values, missing values, histograms for numerical, categorical, boolean, and text columns. This summary gives a comprehensive overview of the dataset to better understand the characteristics of the data that the model is trained on or evaluates.

Test Mechanism

The DatasetDescription class accomplishes the purpose as follows: firstly, the test method "run" infers the data type of each column in the dataset and stores the details (id, column type). For each column, the "describe_column" method is invoked to collect statistical information about the column, including count, missing value count and its proportion to the total, unique value count, and its proportion to the total. Depending on the data type of a column, histograms are generated that reflect the distribution of data within the column. Numerical columns use the "get_numerical_histograms" method to calculate histogram distribution, whereas for categorical, boolean and text columns, a histogram is computed with frequencies of each unique value in the datasets. For unsupported types, an error is raised. Lastly, a summary table is built to aggregate all the statistical insights and histograms of the columns in a dataset.

Signs of High Risk

• High ratio of missing values to total values in one or more columns which may impact the quality of the predictions.
• Unsupported data types in dataset columns.
• Large number of unique values in the dataset's columns which might make it harder for the model to establish patterns.
• Extreme skewness or irregular distribution of data as reflected in the histograms.

Strengths

• Provides a detailed analysis of the dataset with versatile summaries like count, unique values, histograms, etc.
• Flexibility in handling different types of data: numerical, categorical, boolean, and text.
• Useful in detecting problems in the dataset like missing values, unsupported data types, irregular data distribution, etc.
• The summary gives a comprehensive understanding of dataset features allowing developers to make informed decisions.

Limitations

• The computation can be expensive from a resource standpoint, particularly for large datasets with numerous columns.
• The histograms use an arbitrary number of bins which may not be the optimal number of bins for specific data distribution.
• Unsupported data types for columns will raise an error which may limit evaluating the dataset.
• Columns with all null or missing values are not included in histogram computation.
• This test only validates the quality of the dataset but doesn't address the model's performance directly.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DatasetDescription", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'data_quality_tests_text'

▶ Test: Class Imbalance ('validmind.data_validation.ClassImbalance')

Class Imbalance

Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model.

Purpose

The Class Imbalance test is designed to evaluate the distribution of target classes in a dataset that's utilized by a machine learning model. Specifically, it aims to ensure that the classes aren't overly skewed, which could lead to bias in the model's predictions. It's crucial to have a balanced training dataset to avoid creating a model that's biased with high accuracy for the majority class and low accuracy for the minority class.

Test Mechanism

This Class Imbalance test operates by calculating the frequency (expressed as a percentage) of each class in the target column of the dataset. It then checks whether each class appears in at least a set minimum percentage of the total records. This minimum percentage is a modifiable parameter, but the default value is set to 10%.

Signs of High Risk

• Any class that represents less than the pre-set minimum percentage threshold is marked as high risk, implying a potential class imbalance.
• The function provides a pass/fail outcome for each class based on this criterion.
• Fundamentally, if any class fails this test, it's highly likely that the dataset possesses imbalanced class distribution.

Strengths

• The test can spot under-represented classes that could affect the efficiency of a machine learning model.
• The calculation is straightforward and swift.
• The test is highly informative because it not only spots imbalance, but it also quantifies the degree of imbalance.
• The adjustable threshold enables flexibility and adaptation to differing use-cases or domain-specific needs.
• The test creates a visually insightful plot showing the classes and their corresponding proportions, enhancing interpretability and comprehension of the data.

Limitations

• The test might struggle to perform well or provide vital insights for datasets with a high number of classes. In such cases, the imbalance could be inevitable due to the inherent class distribution.
• Sensitivity to the threshold value might result in faulty detection of imbalance if the threshold is set excessively high.
• Regardless of the percentage threshold, it doesn't account for varying costs or impacts of misclassifying different classes, which might fluctuate based on specific applications or domains.
• While it can identify imbalances in class distribution, it doesn't provide direct methods to address or correct these imbalances.
• The test is only applicable for classification operations and unsuitable for regression or clustering tasks.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_percent_threshold	10

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_percent_threshold": "my_vm_min_percent_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.ClassImbalance", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Duplicates ('validmind.data_validation.Duplicates')

Duplicates

Tests dataset for duplicate entries, ensuring model reliability via data quality verification.

Purpose

The 'Duplicates' test is designed to check for duplicate rows within the dataset provided to the model. It serves as a measure of data quality, ensuring that the model isn't merely memorizing duplicate entries or being swayed by redundant information. This is an important step in the pre-processing of data for both classification and regression tasks.

Test Mechanism

This test operates by checking each row for duplicates in the dataset. If a text column is specified in the dataset, the test is conducted on this column; if not, the test is run on all feature columns. The number and percentage of duplicates are calculated and returned in a DataFrame. Additionally, a test is passed if the total count of duplicates falls below a specified minimum threshold.

Signs of High Risk

• A high number of duplicate rows in the dataset, which can lead to overfitting where the model performs well on the training data but poorly on unseen data.
• A high percentage of duplicate rows in the dataset, indicating potential problems with data collection or processing.

Strengths

• Assists in improving the reliability of the model's training process by ensuring the training data is not contaminated with duplicate entries, which can distort statistical analyses.
• Provides both absolute numbers and percentage values of duplicate rows, giving a thorough overview of data quality.
• Highly customizable as it allows for setting a user-defined minimum threshold to determine if the test has been passed.

Limitations

• Does not distinguish between benign duplicates (i.e., coincidental identical entries in different rows) and problematic duplicates originating from data collection or processing errors.
• The test becomes more computationally intensive as the size of the dataset increases, which might not be suitable for very large datasets.
• Can only check for exact duplicates and may miss semantically similar information packaged differently.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_threshold	1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.Duplicates", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: High Cardinality ('validmind.data_validation.HighCardinality')

High Cardinality

Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting.

Purpose

The “High Cardinality” test is used to evaluate the number of unique values present in the categorical columns of a dataset. In this context, high cardinality implies the presence of a large number of unique, non-repetitive values in the dataset.

Test Mechanism

The test first infers the dataset's type and then calculates an initial numeric threshold based on the test parameters. It only considers columns classified as "Categorical". For each of these columns, the number of distinct values (n_distinct) and the percentage of distinct values (p_distinct) are calculated. The test will pass if n_distinct is less than the calculated numeric threshold. Lastly, the results, which include details such as column name, number of distinct values, and pass/fail status, are compiled into a table.

Signs of High Risk

• A large number of distinct values (high cardinality) in one or more categorical columns implies a high risk.
• A column failing the test (n_distinct >= num_threshold) is another indicator of high risk.

Strengths

• The High Cardinality test is effective in early detection of potential overfitting and unwanted noise.
• It aids in identifying potential outliers and inconsistencies, thereby improving data quality.
• The test can be applied to both classification and regression task types, demonstrating its versatility.

Limitations

• The test is restricted to only "Categorical" data types and is thus not suitable for numerical or continuous features, limiting its scope.
• The test does not consider the relevance or importance of unique values in categorical features, potentially causing it to overlook critical data points.
• The threshold (both number and percent) used for the test is static and may not be optimal for diverse datasets and varied applications. Further mechanisms to adjust and refine this threshold could enhance its effectiveness.

Required Inputs: dataset

Parameters:

Parameter	Default Value
num_threshold	100
percent_threshold	0.1
threshold_type	'percent'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "num_threshold": "my_vm_num_threshold",
    "percent_threshold": "my_vm_percent_threshold",
    "threshold_type": "my_vm_threshold_type"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.HighCardinality", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Missing Values ('validmind.data_validation.MissingValues')

Missing Values

Evaluates dataset quality by ensuring missing value percentage across all features does not exceed a set threshold.

Purpose

The Missing Values test is designed to evaluate the quality of a dataset by measuring the number of missing values across all features. The objective is to ensure that the ratio of missing data to total data is less than a predefined threshold (as a percentage), defaulting to 1.0, in order to maintain the data quality necessary for reliable predictive strength in a machine learning model.

Test Mechanism

The mechanism for this test involves iterating through each column of the dataset, counting missing values (represented as NaNs), and calculating the percentage they represent against the total number of rows. The test then checks if the missing value percentage is less than or equal to the predefined min_percentage_threshold. The results are shown in a table summarizing each column, the number of missing values, the percentage of missing values in each column, and a Pass/Fail status based on the threshold comparison.

Signs of High Risk

• When the missing value percentage in any column exceeds the min_percentage_threshold value.
• Presence of missing values across many columns, leading to multiple instances of failing the threshold.

Strengths

• Quick and granular identification of missing data across each feature in the dataset.
• Provides an effective and straightforward means of maintaining data quality, essential for constructing efficient machine learning models.

Limitations

• Does not suggest the root causes of the missing values or recommend ways to impute or handle them.
• May overlook features with significant missing data but still less than the min_percentage_threshold, potentially impacting the model.
• Does not account for data encoded as values like "-999" or "None," which might not technically classify as missing but could bear similar implications.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_percentage_threshold	1.0

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_percentage_threshold": "my_vm_min_percentage_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.MissingValues", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Skewness ('validmind.data_validation.Skewness')

Skewness

Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data quality and optimize model performance.

Purpose

The purpose of the Skewness test is to measure the asymmetry in the distribution of data within a predictive machine learning model. Specifically, it evaluates the divergence of said distribution from a normal distribution. Understanding the level of skewness helps identify data quality issues, which are crucial for optimizing the performance of traditional machine learning models in both classification and regression settings.

Test Mechanism

This test calculates the skewness of numerical columns in the dataset, focusing specifically on numerical data types. The calculated skewness value is then compared against a predetermined maximum threshold, which is set by default to 1. If the skewness value is less than this maximum threshold, the test passes; otherwise, it fails. The test results, along with the skewness values and column names, are then recorded for further analysis.

Signs of High Risk

• Substantial skewness levels that significantly exceed the maximum threshold.
• Persistent skewness in the data, indicating potential issues with the foundational assumptions of the machine learning model.
• Subpar model performance, erroneous predictions, or biased inferences due to skewed data distributions.

Strengths

• Fast and efficient identification of unequal data distributions within a machine learning model.
• Adjustable maximum threshold parameter, allowing for customization based on user needs.
• Provides a clear quantitative measure to mitigate model risks related to data skewness.

Limitations

• Only evaluates numeric columns, potentially missing skewness or bias in non-numeric data.
• Assumes that data should follow a normal distribution, which may not always be applicable to real-world data.
• Subjective threshold for risk grading, requiring expert input and recurrent iterations for refinement.

Required Inputs: dataset

Parameters:

Parameter	Default Value
max_threshold	1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "max_threshold": "my_vm_max_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.Skewness", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Unique Rows ('validmind.data_validation.UniqueRows')

Unique Rows

Verifies the diversity of the dataset by ensuring that the count of unique rows exceeds a prescribed threshold.

Purpose

The UniqueRows test is designed to gauge the quality of the data supplied to the machine learning model by verifying that the count of distinct rows in the dataset exceeds a specific threshold, thereby ensuring a varied collection of data. Diversity in data is essential for training an unbiased and robust model that excels when faced with novel data.

Test Mechanism

The testing process starts with calculating the total number of rows in the dataset. Subsequently, the count of unique rows is determined for each column in the dataset. If the percentage of unique rows (calculated as the ratio of unique rows to the overall row count) is less than the prescribed minimum percentage threshold given as a function parameter, the test passes. The results are cached and a final pass or fail verdict is given based on whether all columns have successfully passed the test.

Signs of High Risk

• A lack of diversity in data columns, demonstrated by a count of unique rows that falls short of the preset minimum percentage threshold, is indicative of high risk.
• This lack of variety in the data signals potential issues with data quality, possibly leading to overfitting in the model and issues with generalization, thus posing a significant risk.

Strengths

• The UniqueRows test is efficient in evaluating the data's diversity across each information column in the dataset.
• This test provides a quick, systematic method to assess data quality based on uniqueness, which can be pivotal in developing effective and unbiased machine learning models.

Limitations

• A limitation of the UniqueRows test is its assumption that the data's quality is directly proportionate to its uniqueness, which may not always hold true. There might be contexts where certain non-unique rows are essential and should not be overlooked.
• The test does not consider the relative 'importance' of each column in predicting the output, treating all columns equally.
• This test may not be suitable or useful for categorical variables, where the count of unique categories is inherently limited.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_percent_threshold	1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_percent_threshold": "my_vm_min_percent_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.UniqueRows", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Too Many Zero Values ('validmind.data_validation.TooManyZeroValues')

Too Many Zero Values

Identifies numerical columns in a dataset that contain an excessive number of zero values, defined by a threshold percentage.

Purpose

The 'TooManyZeroValues' test is utilized to identify numerical columns in the dataset that may present a quantity of zero values considered excessive. The aim is to detect situations where these may implicate data sparsity or a lack of variation, limiting their effectiveness within a machine learning model. The definition of 'too many' is quantified as a percentage of total values, with a default set to 0.03%.

Test Mechanism

This test is conducted by looping through each column in the dataset and categorizing those that pertain to numerical data. On identifying a numerical column, the function computes the total quantity of zero values and their ratio to the total row count. Should the proportion exceed a pre-set threshold parameter, set by default at 0.03%, the column is considered to have failed the test. The results for each column are summarized and reported, indicating the count and percentage of zero values for each numerical column, alongside a status indicating whether the column has passed or failed the test.

Signs of High Risk

• Numerical columns showing a high ratio of zero values when compared to the total count of rows (exceeding the predetermined threshold).
• Columns characterized by zero values across the board suggest a complete lack of data variation, signifying high risk.

Strengths

• Assists in highlighting columns featuring an excess of zero values that could otherwise go unnoticed within a large dataset.
• Provides the flexibility to alter the threshold that determines when the quantity of zero values becomes 'too many', thus catering to specific needs of a particular analysis or model.
• Offers feedback in the form of both counts and percentages of zero values, which allows a closer inspection of the distribution and proportion of zeros within a column.
• Targets specifically numerical data, thereby avoiding inappropriate application to non-numerical columns and mitigating the risk of false test failures.

Limitations

• Is exclusively designed to check for zero values and doesn’t assess the potential impact of other values that could affect the dataset, such as extremely high or low figures, missing values, or outliers.
• Lacks the ability to detect a repetitive pattern of zeros, which could be significant in time-series or longitudinal data.
• Zero values can actually be meaningful in some contexts; therefore, tagging them as 'too many' could potentially misinterpret the data to some extent.
• This test does not take into consideration the context of the dataset, and fails to recognize that within certain columns, a high number of zero values could be quite normal and not necessarily an indicator of poor data quality.
• Cannot evaluate non-numerical or categorical columns, which might bring with them different types of concerns or issues.

Required Inputs: dataset

Parameters:

Parameter	Default Value
max_percent_threshold	0.03

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "max_percent_threshold": "my_vm_max_percent_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.TooManyZeroValues", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: IQR Outliers Table ('validmind.data_validation.IQROutliersTable')

IQR Outliers Table

Determines and summarizes outliers in numerical features using the Interquartile Range method.

Purpose

The "Interquartile Range Outliers Table" (IQROutliersTable) metric is designed to identify and summarize outliers within numerical features of a dataset using the Interquartile Range (IQR) method. This exercise is crucial in the pre-processing of data because outliers can substantially distort statistical analysis and impact the performance of machine learning models.

Test Mechanism

The IQR, which is the range separating the first quartile (25th percentile) from the third quartile (75th percentile), is calculated for each numerical feature within the dataset. An outlier is defined as a data point falling below the "Q1 - 1.5 * IQR" or above "Q3 + 1.5 * IQR" range. The test computes the number of outliers and their summary statistics (minimum, 25th percentile, median, 75th percentile, and maximum values) for each numerical feature. If no specific features are chosen, the test applies to all numerical features in the dataset. The default outlier threshold is set to 1.5 but can be customized by the user.

Signs of High Risk

• A large number of outliers in multiple features.
• Outliers significantly distanced from the mean value of variables.
• Extremely high or low outlier values indicative of data entry errors or other data quality issues.

Strengths

• Provides a comprehensive summary of outliers for each numerical feature, helping pinpoint features with potential quality issues.
• The IQR method is robust to extremely high or low outlier values as it is based on quartile calculations.
• Can be customized to work on selected features and set thresholds for outliers.

Limitations

• Might cause false positives if the variable deviates from a normal or near-normal distribution, especially for skewed distributions.
• Does not provide interpretation or recommendations for addressing outliers, relying on further analysis by users or data scientists.
• Only applicable to numerical features, not categorical data.
• Default thresholds may not be optimal for data with heavy pre-processing, manipulation, or inherently high kurtosis (heavy tails).

Required Inputs: dataset

Parameters:

Parameter	Default Value
threshold	1.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "threshold": "my_vm_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.IQROutliersTable", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: IQR Outliers Bar Plot ('validmind.data_validation.IQROutliersBarPlot')

IQR Outliers Bar Plot

Visualizes outlier distribution across percentiles in numerical data using the Interquartile Range (IQR) method.

Purpose

The InterQuartile Range Outliers Bar Plot (IQROutliersBarPlot) metric aims to visually analyze and evaluate the extent of outliers in numeric variables based on percentiles. Its primary purpose is to clarify the dataset's distribution, flag possible abnormalities in it, and gauge potential risks associated with processing potentially skewed data, which can affect the machine learning model's predictive prowess.

Test Mechanism

The examination invokes a series of steps:

1. For every numeric feature in the dataset, the 25th percentile (Q1) and 75th percentile (Q3) are calculated before deriving the Interquartile Range (IQR), the difference between Q1 and Q3.
2. Subsequently, the metric calculates the lower and upper thresholds by subtracting Q1 from the threshold times IQR and adding Q3 to threshold times IQR, respectively. The default threshold is set at 1.5.
3. Any value in the feature that falls below the lower threshold or exceeds the upper threshold is labeled as an outlier.
4. The number of outliers are tallied for different percentiles, such as [0-25], [25-50], [50-75], and [75-100].
5. These counts are employed to construct a bar plot for the feature, showcasing the distribution of outliers across different percentiles.

Signs of High Risk

• A prevalence of outliers in the data, potentially skewing its distribution.
• Outliers dominating higher percentiles (75-100) which implies the presence of extreme values, capable of severely influencing the model's performance.
• Certain features harboring most of their values as outliers, which signifies that these features might not contribute positively to the model's forecasting ability.

Strengths

• Effectively identifies outliers in the data through visual means, facilitating easier comprehension and offering insights into the outliers' possible impact on the model.
• Provides flexibility by accommodating all numeric features or a chosen subset.
• Task-agnostic in nature; it is viable for both classification and regression tasks.
• Can handle large datasets as its operation does not hinge on computationally heavy operations.

Limitations

• Its application is limited to numerical variables and does not extend to categorical ones.
• Only reveals the presence and distribution of outliers and does not provide insights into how these outliers might affect the model's predictive performance.
• The assumption that data is unimodal and symmetric may not always hold true. In cases with non-normal distributions, the results can be misleading.

Required Inputs: dataset

Parameters:

Parameter	Default Value
threshold	1.5
fig_width	800

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "threshold": "my_vm_threshold",
    "fig_width": "my_vm_fig_width"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.IQROutliersBarPlot", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 2.2. Descriptive Statistics ('descriptive_statistics')

▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Descriptive Statistics

Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's dataset.

Purpose

The purpose of the Descriptive Statistics metric is to provide a comprehensive summary of both numerical and categorical data within a dataset. This involves statistics such as count, mean, standard deviation, minimum and maximum values for numerical data. For categorical data, it calculates the count, number of unique values, most common value and its frequency, and the proportion of the most frequent value relative to the total. The goal is to visualize the overall distribution of the variables in the dataset, aiding in understanding the model's behavior and predicting its performance.

Test Mechanism

The testing mechanism utilizes two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function pulls out several summary statistics, while value_counts() accounts for unique values. The resulting data is formatted into two distinct tables, one for numerical and another for categorical variable summaries. These tables provide a clear summary of the main characteristics of the variables, which can be instrumental in assessing the model's performance.

Signs of High Risk

• Skewed data or significant outliers can represent high risk. For numerical data, this may be reflected via a significant difference between the mean and median (50% percentile).
• For categorical data, a lack of diversity (low count of unique values), or overdominance of a single category (high frequency of the top value) can indicate high risk.

Strengths

• Provides a comprehensive summary of the dataset, shedding light on the distribution and characteristics of the variables under consideration.
• It is a versatile and robust method, applicable to both numerical and categorical data.
• Helps highlight crucial anomalies such as outliers, extreme skewness, or lack of diversity, which are vital in understanding model behavior during testing and validation.

Limitations

• While this metric offers a high-level overview of the data, it may fail to detect subtle correlations or complex patterns.
• Does not offer any insights on the relationship between variables.
• Alone, descriptive statistics cannot be used to infer properties about future unseen data.
• Should be used in conjunction with other statistical tests to provide a comprehensive understanding of the model's data.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DescriptiveStatistics", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 2.3. Correlations and Interactions ('correlations')

▶ Test: Pearson Correlation Matrix ('validmind.data_validation.PearsonCorrelationMatrix')

Pearson Correlation Matrix

Evaluates linear dependency between numerical variables in a dataset via a Pearson Correlation coefficient heat map.

Purpose

This test is intended to evaluate the extent of linear dependency between all pairs of numerical variables in the given dataset. It provides the Pearson Correlation coefficient, which reveals any high correlations present. The purpose of doing this is to identify potential redundancy, as variables that are highly correlated can often be removed to reduce the dimensionality of the dataset without significantly impacting the model's performance.

Test Mechanism

This metric test generates a correlation matrix for all numerical variables in the dataset using the Pearson correlation formula. A heat map is subsequently created to visualize this matrix effectively. The color of each point on the heat map corresponds to the magnitude and direction (positive or negative) of the correlation, with a range from -1 (perfect negative correlation) to 1 (perfect positive correlation). Any correlation coefficients higher than 0.7 (in absolute terms) are indicated in white in the heat map, suggesting a high degree of correlation.

Signs of High Risk

• A large number of variables in the dataset showing a high degree of correlation (coefficients approaching ±1). This indicates redundancy within the dataset, suggesting that some variables may not be contributing new information to the model.
• Potential risk of overfitting.

Strengths

• Detects and quantifies the linearity of relationships between variables, aiding in identifying redundant variables to simplify models and potentially improve performance.
• The heatmap visualization provides an easy-to-understand overview of correlations, beneficial for users not comfortable with numerical matrices.

Limitations

• Limited to detecting linear relationships, potentially missing non-linear relationships which impede opportunities for dimensionality reduction.
• Measures only the degree of linear relationship, not the strength of one variable's effect on another.
• The 0.7 correlation threshold is arbitrary and might exclude valid dependencies with lower coefficients.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.PearsonCorrelationMatrix", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: High Pearson Correlation ('validmind.data_validation.HighPearsonCorrelation')

High Pearson Correlation

Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity.

Purpose

The High Pearson Correlation test measures the linear relationship between features in a dataset, with the main goal of identifying high correlations that might indicate feature redundancy or multicollinearity. Identification of such issues allows developers and risk management teams to properly deal with potential impacts on the machine learning model's performance and interpretability.

Test Mechanism

The test works by generating pairwise Pearson correlations for all features in the dataset, then sorting and eliminating duplicate and self-correlations. It assigns a Pass or Fail based on whether the absolute value of the correlation coefficient surpasses a pre-set threshold (defaulted at 0.3). It lastly returns the top n strongest correlations regardless of passing or failing status (where n is 10 by default but can be configured by passing the top_n_correlations parameter).

Signs of High Risk

• A high risk indication would be the presence of correlation coefficients exceeding the threshold.
• If the features share a strong linear relationship, this could lead to potential multicollinearity and model overfitting.
• Redundancy of variables can undermine the interpretability of the model due to uncertainty over the authenticity of individual variable's predictive power.

Strengths

• Provides a quick and simple means of identifying relationships between feature pairs.
• Generates a transparent output that displays pairs of correlated variables, the Pearson correlation coefficient, and a Pass or Fail status for each.
• Aids in early identification of potential multicollinearity issues that may disrupt model training.

Limitations

• Can only delineate linear relationships, failing to shed light on nonlinear relationships or dependencies.
• Sensitive to outliers where a few outliers could notably affect the correlation coefficient.
• Limited to identifying redundancy only within feature pairs; may fail to spot more complex relationships among three or more variables.

Required Inputs: dataset

Parameters:

Parameter	Default Value
max_threshold	0.3
top_n_correlations	10
feature_columns	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "max_threshold": "my_vm_max_threshold",
    "top_n_correlations": "my_vm_top_n_correlations",
    "feature_columns": "my_vm_feature_columns"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.HighPearsonCorrelation", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 2.4. Feature Selection and Engineering ('feature_selection')

Text Block: 'feature_selection'

▶ 3. Model Development ('model_development')

▶ 3.1. Model Training ('model_training')

▶ Test: Model Metadata ('validmind.model_validation.ModelMetadata')

Model Metadata

Compare metadata of different models and generate a summary table with the results.

Purpose: The purpose of this function is to compare the metadata of different models, including information about their architecture, framework, framework version, and programming language.

Test Mechanism: The function retrieves the metadata for each model using get_model_info, renames columns according to a predefined set of labels, and compiles this information into a summary table.

Signs of High Risk:

• Inconsistent or missing metadata across models can indicate potential issues in model documentation or management.
• Significant differences in framework versions or programming languages might pose challenges in model integration and deployment.

Strengths:

• Provides a clear comparison of essential model metadata.
• Standardizes metadata labels for easier interpretation and comparison.
• Helps identify potential compatibility or consistency issues across models.

Limitations:

• Assumes that the get_model_info function returns all necessary metadata fields.
• Relies on the correctness and completeness of the metadata provided by each model.
• Does not include detailed parameter information, focusing instead on high-level metadata.

Required Inputs: model

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.ModelMetadata", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Dataset Split ('validmind.data_validation.DatasetSplit')

Dataset Split

Evaluates and visualizes the distribution proportions among training, testing, and validation datasets of an ML model.

Purpose

The DatasetSplit test is designed to evaluate and visualize the distribution of data among training, testing, and validation datasets, if available, within a given machine learning model. The main purpose is to assess whether the model's datasets are split appropriately, as an imbalanced split might affect the model's ability to learn from the data and generalize to unseen data.

Test Mechanism

The DatasetSplit test first calculates the total size of all available datasets in the model. Then, for each individual dataset, the methodology involves determining the size of the dataset and its proportion relative to the total size. The results are then conveniently summarized in a table that shows dataset names, sizes, and proportions. Absolute size and proportion of the total dataset size are displayed for each individual dataset.

Signs of High Risk

• A very small training dataset, which may result in the model not learning enough from the data.
• A very large training dataset and a small test dataset, which may lead to model overfitting and poor generalization to unseen data.
• A small or non-existent validation dataset, which might complicate the model's performance assessment.

Strengths

• The DatasetSplit test provides a clear, understandable visualization of dataset split proportions, which can highlight any potential imbalance in dataset splits quickly.
• It covers a wide range of task types including classification, regression, and text-related tasks.
• The metric is not tied to any specific data type and is applicable to tabular data, time series data, or text data.

Limitations

• The DatasetSplit test does not provide any insight into the quality or diversity of the data within each split, just the size and proportion.
• The test does not give any recommendations or adjustments for imbalanced datasets.
• Potential lack of compatibility with more complex modes of data splitting (for example, stratified or time-based splits) could limit the applicability of this test.

Required Inputs: datasets

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DatasetSplit", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Population Stability Index ('validmind.model_validation.sklearn.PopulationStabilityIndex')

Population Stability Index

Assesses the Population Stability Index (PSI) to quantify the stability of an ML model's predictions across different datasets.

Purpose

The Population Stability Index (PSI) serves as a quantitative assessment for evaluating the stability of a machine learning model's output distributions when comparing two different datasets. Typically, these would be a development and a validation dataset or two datasets collected at different periods. The PSI provides a measurable indication of any significant shift in the model's performance over time or noticeable changes in the characteristics of the population the model is making predictions for.

Test Mechanism

The implementation of the PSI in this script involves calculating the PSI for each feature between the training and test datasets. Data from both datasets is sorted and placed into either a predetermined number of bins or quantiles. The boundaries for these bins are initially determined based on the distribution of the training data. The contents of each bin are calculated and their respective proportions determined. Subsequently, the PSI is derived for each bin through a logarithmic transformation of the ratio of the proportions of data for each feature in the training and test datasets. The PSI, along with the proportions of data in each bin for both datasets, are displayed in a summary table, a grouped bar chart, and a scatter plot.

Signs of High Risk

• A high PSI value is a clear indicator of high risk. Such a value suggests a significant shift in the model predictions or severe changes in the characteristics of the underlying population.
• This ultimately suggests that the model may not be performing as well as expected and that it may be less reliable for making future predictions.

Strengths

• The PSI provides a quantitative measure of the stability of a model over time or across different samples, making it an invaluable tool for evaluating changes in a model's performance.
• It allows for direct comparisons across different features based on the PSI value.
• The calculation and interpretation of the PSI are straightforward, facilitating its use in model risk management.
• The use of visual aids such as tables and charts further simplifies the comprehension and interpretation of the PSI.

Limitations

• The PSI test does not account for the interdependence between features: features that are dependent on one another may show similar shifts in their distributions, which in turn may result in similar PSI values.
• The PSI test does not inherently provide insights into why there are differences in distributions or why the PSI values may have changed.
• The test may not handle features with significant outliers adequately.
• Additionally, the PSI test is performed on model predictions, not on the underlying data distributions which can lead to misinterpretations. Any changes in PSI could be due to shifts in the model (model drift), changes in the relationships between features and the target variable (concept drift), or both. However, distinguishing between these causes is non-trivial.
• For multiclass models the PSI is computed one-vs-rest (one table/plot per class), which requires per-class probabilities from the model's predict_proba. Models that cannot produce a full per-class probability matrix (e.g. metadata-only models, or predictions supplied as a single precomputed probability column) are skipped for the multiclass case.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
num_bins	10
mode	'fixed'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "num_bins": "my_vm_num_bins",
    "mode": "my_vm_mode"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.PopulationStabilityIndex", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 3.2. Model Evaluation ('model_evaluation')

▶ Test: Confusion Matrix ('validmind.model_validation.sklearn.ConfusionMatrix')

Confusion Matrix

Evaluates and visually represents the classification ML model's predictive performance using a Confusion Matrix heatmap.

Purpose

The Confusion Matrix tester is designed to assess the performance of a classification Machine Learning model. This performance is evaluated based on how well the model is able to correctly classify True Positives, True Negatives, False Positives, and False Negatives - fundamental aspects of model accuracy.

Test Mechanism

The mechanism used involves taking the predicted results (y_test_predict) from the classification model and comparing them against the actual values (y_test_true). A confusion matrix is built using the unique labels extracted from y_test_true, employing scikit-learn's metrics. The matrix is then visually rendered with the help of a Plotly heatmap. A heatmap is created which provides a two-dimensional graphical representation of the model's performance, showcasing distributions of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).

Signs of High Risk

• High numbers of False Positives (FP) and False Negatives (FN), depicting that the model is not effectively classifying the values.
• Low numbers of True Positives (TP) and True Negatives (TN), implying that the model is struggling with correctly identifying class labels.

Strengths

• It provides a simplified yet comprehensive visual snapshot of the classification model's predictive performance.
• It distinctly brings out True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN), thus making it easier to focus on potential areas of improvement.
• The matrix is beneficial in dealing with multi-class classification problems as it can provide a simple view of complex model performances.
• It aids in understanding the different types of errors that the model could potentially make, as it provides in-depth insights into Type-I and Type-II errors.

Limitations

• In cases of unbalanced classes, the effectiveness of the confusion matrix might be lessened. It may wrongly interpret the accuracy of a model that is essentially just predicting the majority class.
• It does not provide a single unified statistic that could evaluate the overall performance of the model. Different aspects of the model's performance are evaluated separately instead.
• It mainly serves as a descriptive tool and does not offer the capability for statistical hypothesis testing.
• Risks of misinterpretation exist because the matrix doesn't directly provide precision, recall, or F1-score data. These metrics have to be computed separately.
• The threshold parameter only applies to binary classification (it splits a single positive-class probability into two classes). For multiclass targets the model's argmax class predictions are used and threshold is ignored.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
threshold	0.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "threshold": "my_vm_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ConfusionMatrix", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Classifier Performance In Sample ('validmind.model_validation.sklearn.ClassifierPerformance:in_sample')

Classifier Performance In Sample

Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy, and ROC AUC scores.

Purpose

The Classifier Performance test is designed to evaluate the performance of Machine Learning classification models. It accomplishes this by computing precision, recall, F1-Score, and accuracy, as well as the ROC AUC (Receiver operating characteristic - Area under the curve) scores, thereby providing a comprehensive analytic view of the models' performance. The test is adaptable, handling binary and multiclass models equally effectively.

Test Mechanism

The test produces a report that includes precision, recall, F1-Score, and accuracy, by leveraging the classification_report from scikit-learn's metrics module. For multiclass models, macro and weighted averages for these scores are also calculated. Additionally, the ROC AUC scores are calculated and included in the report using the multiclass_roc_auc_score function. The outcome of the test (report format) differs based on whether the model is binary or multiclass.

Signs of High Risk

• Low values for precision, recall, F1-Score, accuracy, and ROC AUC, indicating poor performance.
• Imbalance in precision and recall scores.
• A low ROC AUC score, especially scores close to 0.5 or lower, suggesting a failing model.

Strengths

• Versatile, capable of assessing both binary and multiclass models.
• Utilizes a variety of commonly employed performance metrics, offering a comprehensive view of model performance.
• The use of ROC-AUC as a metric is beneficial for evaluating unbalanced datasets.

Limitations

• Assumes correctly identified labels for binary classification models.
• Specifically designed for classification models and not suitable for regression models.
• May provide limited insights if the test dataset does not represent real-world scenarios adequately.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
average	'macro'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "average": "my_vm_average"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ClassifierPerformance:in_sample", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Classifier Performance Out Of Sample ('validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample')

Classifier Performance Out Of Sample

Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy, and ROC AUC scores.

Purpose

The Classifier Performance test is designed to evaluate the performance of Machine Learning classification models. It accomplishes this by computing precision, recall, F1-Score, and accuracy, as well as the ROC AUC (Receiver operating characteristic - Area under the curve) scores, thereby providing a comprehensive analytic view of the models' performance. The test is adaptable, handling binary and multiclass models equally effectively.

Test Mechanism

The test produces a report that includes precision, recall, F1-Score, and accuracy, by leveraging the classification_report from scikit-learn's metrics module. For multiclass models, macro and weighted averages for these scores are also calculated. Additionally, the ROC AUC scores are calculated and included in the report using the multiclass_roc_auc_score function. The outcome of the test (report format) differs based on whether the model is binary or multiclass.

Signs of High Risk

• Low values for precision, recall, F1-Score, accuracy, and ROC AUC, indicating poor performance.
• Imbalance in precision and recall scores.
• A low ROC AUC score, especially scores close to 0.5 or lower, suggesting a failing model.

Strengths

• Versatile, capable of assessing both binary and multiclass models.
• Utilizes a variety of commonly employed performance metrics, offering a comprehensive view of model performance.
• The use of ROC-AUC as a metric is beneficial for evaluating unbalanced datasets.

Limitations

• Assumes correctly identified labels for binary classification models.
• Specifically designed for classification models and not suitable for regression models.
• May provide limited insights if the test dataset does not represent real-world scenarios adequately.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
average	'macro'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "average": "my_vm_average"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Precision Recall Curve ('validmind.model_validation.sklearn.PrecisionRecallCurve')

Precision Recall Curve

Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve.

Purpose

The Precision Recall Curve metric is intended to evaluate the trade-off between precision and recall in classification models, particularly binary classification models. It assesses the model's capacity to produce accurate results (high precision), as well as its ability to capture a majority of all positive instances (high recall).

Test Mechanism

The test extracts ground truth labels and prediction probabilities from the model's test dataset. It applies the precision_recall_curve method from the sklearn metrics module to these extracted labels and predictions, which computes a precision-recall pair for each possible threshold. This calculation results in an array of precision and recall scores that can be plotted against each other to form the Precision-Recall Curve. This curve is then visually represented by using Plotly's scatter plot.

Signs of High Risk

• A lower area under the Precision-Recall Curve signifies high risk.
• This corresponds to a model yielding a high amount of false positives (low precision) and/or false negatives (low recall).
• If the curve is closer to the bottom left of the plot, rather than being closer to the top right corner, it can be a sign of high risk.

Strengths

• This metric aptly represents the balance between precision (minimizing false positives) and recall (minimizing false negatives), which is especially critical in scenarios where both values are significant.
• Through the graphic representation, it enables an intuitive understanding of the model's performance across different threshold levels.

Limitations

• For multiclass models the curve is computed one-vs-rest (one curve per class plus a micro-average), which requires per-class probabilities from the model's predict_proba. Models that cannot produce a full per-class probability matrix (e.g. Foundation/metadata-only models, or predictions supplied as a single precomputed probability column) are skipped for the multiclass case.
• It may not fully represent the overall accuracy of the model if the cost of false positives and false negatives are extremely different, or if the dataset is heavily imbalanced.

Required Inputs: model, dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.PrecisionRecallCurve", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: ROC Curve ('validmind.model_validation.sklearn.ROCCurve')

ROC Curve

Evaluates classification model performance by generating and plotting the Receiver Operating Characteristic (ROC) curve and calculating the Area Under Curve (AUC) score, for both binary and multiclass models.

Purpose

The Receiver Operating Characteristic (ROC) curve evaluates the performance of classification models. This curve illustrates the balance between the True Positive Rate (TPR) and False Positive Rate (FPR) across various threshold levels. In combination with the Area Under the Curve (AUC), the ROC curve measures the model's discrimination ability between classes. For binary problems (e.g., default vs non-default) a single curve is drawn for the positive class. For multiclass problems the curve is computed one-vs-rest — one curve and AUC per class, plus a micro-average across all classes — so the model's discrimination ability can be assessed for every class. Ideally, a higher AUC score signifies superior model performance in accurately distinguishing between classes.

Test Mechanism

This test selects the target model and dataset and determines the number of classes from the true labels. For binary targets it computes the predicted probabilities for the positive class and, together with the true outcomes, generates and plots a single ROC curve. For multiclass targets it obtains the full per-class probability matrix from the model and plots a one-vs-rest curve for each class along with a micro-average curve. In both cases a line signifying randomness (AUC of 0.5) is included, and the AUC score(s) are computed as a numerical estimation of performance. If any Infinite values are detected in the ROC threshold, these are effectively eliminated. The resulting ROC curves, AUC scores, and thresholds are consequently saved for future reference.

Signs of High Risk

• A high risk is potentially linked to the model's performance if the AUC score drops below or nears 0.5.
• Another warning sign would be the ROC curve lying closer to the line of randomness, indicating no discriminative ability.
• For the model to be deemed competent at its classification tasks, it is crucial that the AUC score is significantly above 0.5.

Strengths

• The ROC Curve offers an inclusive visual depiction of a model's discriminative power throughout all conceivable classification thresholds, unlike other metrics that solely disclose model performance at one fixed threshold.
• Despite the proportions of the dataset, the AUC Score, which represents the entire ROC curve as a single data point, continues to be consistent, proving to be the ideal choice for such situations.

Limitations

• For multiclass models the curve is computed one-vs-rest (one curve per class plus a micro-average), which requires per-class probabilities from the model's predict_proba. Models that cannot produce a full per-class probability matrix (e.g. metadata-only models, or predictions supplied as a single precomputed probability column) are skipped for the multiclass case.
• Furthermore, its performance might be subpar with models that output probabilities highly skewed towards 0 or 1.
• At the extreme, the ROC curve could reflect high performance even when the majority of classifications are incorrect, provided that the model's ranking format is retained. This phenomenon is commonly termed the "Class Imbalance Problem".

Required Inputs: model, dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ROCCurve", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'model_validation_tests_text'

▶ Test: Training Test Degradation ('validmind.model_validation.sklearn.TrainingTestDegradation')

Training Test Degradation

Tests if model performance degradation between training and test datasets exceeds a predefined threshold.

Purpose

The TrainingTestDegradation class serves as a test to verify that the degradation in performance between the training and test datasets does not exceed a predefined threshold. This test measures the model's ability to generalize from its training data to unseen test data, assessing key classification metrics such as precision, recall, and f1 score to verify the model's robustness and reliability.

Test Mechanism

The code applies several predefined metrics, including precision, recall, and f1 scores, to the model's predictions for both the training and test datasets. It calculates the degradation as the difference between the training score and test score divided by the training score. The test is considered successful if the degradation for each metric is less than the preset maximum threshold (default: 0.10). The results are summarized in a table showing each metric's train score, test score, degradation percentage, and pass/fail status.

Signs of High Risk

• A degradation percentage that exceeds the maximum allowed threshold of 10% for any of the evaluated metrics.
• A high difference or gap between the metric scores on the training and the test datasets.
• The 'Pass/Fail' column displaying 'Fail' for any of the evaluated metrics.

Strengths

• Provides a quantitative measure of the model's ability to generalize to unseen data, which is key for predicting its practical real-world performance.
• By evaluating multiple metrics, it takes into account different facets of model performance and enables a more holistic evaluation.
• The use of a variable predefined threshold allows the flexibility to adjust the acceptability criteria for different scenarios.

Limitations

• The test compares raw performance on training and test data but does not factor in the nature of the data. Areas with less representation in the training set might still perform poorly on unseen data.
• It requires good coverage and balance in the test and training datasets to produce reliable results, which may not always be available.
• The test is currently only designed for classification tasks.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
max_threshold	0.1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "max_threshold": "my_vm_max_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.TrainingTestDegradation", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Minimum Accuracy ('validmind.model_validation.sklearn.MinimumAccuracy')

Minimum Accuracy

Checks if the model's prediction accuracy meets or surpasses a specified threshold.

Purpose

The Minimum Accuracy test’s objective is to verify whether the model's prediction accuracy on a specific dataset meets or surpasses a predetermined minimum threshold. Accuracy, which is simply the ratio of correct predictions to total predictions, is a key metric for evaluating the model's performance. Considering binary as well as multiclass classifications, accurate labeling becomes indispensable.

Test Mechanism

The test mechanism involves contrasting the model's accuracy score with a preset minimum threshold value, with the default being 0.7. The accuracy score is computed utilizing sklearn’s accuracy_score method, where the true labels y_true and predicted labels class_pred are compared. If the accuracy score is above the threshold, the test receives a passing mark. The test returns the result along with the accuracy score and threshold used for the test.

Signs of High Risk

• Model fails to achieve or surpass the predefined score threshold.
• Persistent scores below the threshold, indicating a high risk of inaccurate predictions.

Strengths

• Simplicity, presenting a straightforward measure of holistic model performance across all classes.
• Particularly advantageous when classes are balanced.
• Versatile, as it can be implemented on both binary and multiclass classification tasks.

Limitations

• Misleading accuracy scores when classes in the dataset are highly imbalanced.
• Favoritism towards the majority class, giving an inaccurate perception of model performance.
• Inability to measure the model's precision, recall, or capacity to manage false positives or false negatives.
• Focused on overall correctness and may not be sufficient for all types of model analytics.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
min_threshold	0.7

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.MinimumAccuracy", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Minimum F1 Score ('validmind.model_validation.sklearn.MinimumF1Score')

Minimum F1 Score

Assesses if the model's F1 score on the validation set meets a predefined minimum threshold, ensuring balanced performance between precision and recall.

Purpose

The main objective of this test is to ensure that the F1 score, a balanced measure of precision and recall, of the model meets or surpasses a predefined threshold on the validation dataset. The F1 score is highly useful for gauging model performance in classification tasks, especially in cases where the distribution of positive and negative classes is skewed.

Test Mechanism

The F1 score for the validation dataset is computed through scikit-learn's metrics in Python. The scoring mechanism differs based on the classification problem: for multi-class problems, macro averaging is used, and for binary classification, the built-in f1_score calculation is used. The obtained F1 score is then assessed against the predefined minimum F1 score that is expected from the model.

Signs of High Risk

• If a model returns an F1 score that is less than the established threshold, it is regarded as high risk.
• A low F1 score might suggest that the model is not finding an optimal balance between precision and recall, failing to effectively identify positive classes while minimizing false positives.

Strengths

• Provides a balanced measure of a model's performance by accounting for both false positives and false negatives.
• Particularly advantageous in scenarios with imbalanced class distribution, where accuracy can be misleading.
• Flexibility in setting the threshold value allows tailored minimum acceptable performance standards.

Limitations

• May not be suitable for all types of models and machine learning tasks.
• The F1 score assumes an equal cost for false positives and false negatives, which may not be true in some real-world scenarios.
• Practitioners might need to rely on other metrics such as precision, recall, or the ROC-AUC score that align more closely with specific requirements.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
min_threshold	0.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.MinimumF1Score", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Minimum ROCAUC Score ('validmind.model_validation.sklearn.MinimumROCAUCScore')

Minimum ROCAUC Score

Validates model by checking if the ROC AUC score meets or surpasses a specified threshold.

Purpose

The Minimum ROC AUC Score test is used to determine the model's performance by ensuring that the Receiver Operating Characteristic Area Under the Curve (ROC AUC) score on the validation dataset meets or exceeds a predefined threshold. The ROC AUC score indicates how well the model can distinguish between different classes, making it a crucial measure in binary and multiclass classification tasks.

Test Mechanism

This test implementation calculates the multiclass ROC AUC score on the true target values and the model's predictions. The test converts the multi-class target variables into binary format using LabelBinarizer before computing the score. If this ROC AUC score is higher than the predefined threshold (defaulted to 0.5), the test passes; otherwise, it fails. The results, including the ROC AUC score, the threshold, and whether the test passed or failed, are then stored in a ThresholdTestResult object.

Signs of High Risk

• A high risk or failure in the model's performance as related to this metric would be represented by a low ROC AUC score, specifically any score lower than the predefined minimum threshold. This suggests that the model is struggling to distinguish between different classes effectively.

Strengths

• The test considers both the true positive rate and false positive rate, providing a comprehensive performance measure.
• ROC AUC score is threshold-independent meaning it measures the model's quality across various classification thresholds.
• Works robustly with binary as well as multi-class classification problems.

Limitations

• ROC AUC may not be useful if the class distribution is highly imbalanced; it could perform well in terms of AUC but still fail to predict the minority class.
• The test does not provide insight into what specific aspects of the model are causing poor performance if the ROC AUC score is unsatisfactory.
• The use of macro average for multiclass ROC AUC score implies equal weightage to each class, which might not be appropriate if the classes are imbalanced.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
min_threshold	0.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.MinimumROCAUCScore", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 3.3. Model Explainability and Interpretability ('explainability')

▶ Test: Permutation Feature Importance ('validmind.model_validation.sklearn.PermutationFeatureImportance')

Permutation Feature Importance

Assesses the significance of each feature in a model by evaluating the impact on model performance when feature values are randomly rearranged.

Purpose

The Permutation Feature Importance (PFI) metric aims to assess the importance of each feature used by the Machine Learning model. The significance is measured by evaluating the decrease in the model's performance when the feature's values are randomly arranged.

Test Mechanism

PFI is calculated via the permutation_importance method from the sklearn.inspection module. This method shuffles the columns of the feature dataset and measures the impact on the model's performance. A significant decrease in performance after permutating a feature's values deems the feature as important. On the other hand, if performance remains the same, the feature is likely not important. The output of the PFI metric is a figure illustrating the importance of each feature.

Signs of High Risk

• The model heavily relies on a feature with highly variable or easily permutable values, indicating instability.
• A feature deemed unimportant by the model but expected to have a significant effect on the outcome based on domain knowledge is not influencing the model's predictions.

Strengths

• Provides insights into the importance of different features and may reveal underlying data structure.
• Can indicate overfitting if a particular feature or set of features overly impacts the model's predictions.
• Model-agnostic and can be used with any classifier that provides a measure of prediction accuracy before and after feature permutation.

Limitations

• Does not imply causality; it only presents the amount of information that a feature provides for the prediction task.
• Does not account for interactions between features. If features are correlated, the permutation importance may allocate importance to one and not the other.
• Cannot interact with certain libraries like statsmodels, pytorch, catboost, etc., thus limiting its applicability.

Required Inputs: model, dataset

Parameters:

Parameter	Default Value
fontsize	None
figure_height	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {
    "fontsize": "my_vm_fontsize",
    "figure_height": "my_vm_figure_height"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.PermutationFeatureImportance", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'validmind.model_validation.sklearn.SHAPGlobalImportance_global_importance_text'

▶ Test: SHAP Global Importance ('validmind.model_validation.sklearn.SHAPGlobalImportance')

SHAP Global Importance

Evaluates and visualizes global feature importance using SHAP values for model explanation and risk identification.

Purpose

The SHAP (SHapley Additive exPlanations) Global Importance metric aims to elucidate model outcomes by attributing them to the contributing features. It assigns a quantifiable global importance to each feature via their respective absolute Shapley values, thereby making it suitable for tasks like classification (both binary and multiclass). This metric forms an essential part of model risk management.

Test Mechanism

The exam begins with the selection of a suitable explainer which aligns with the model's type. For tree-based models like XGBClassifier, RandomForestClassifier, CatBoostClassifier, TreeExplainer is used whereas for linear models like LogisticRegression, XGBRegressor, LinearRegression, it is the LinearExplainer. Once the explainer calculates the Shapley values, these values are visualized using two specific graphical representations:

1. Mean Importance Plot: This graph portrays the significance of individual features based on their absolute Shapley values. It calculates the average of these absolute Shapley values across all instances to highlight the global importance of features.

2. Summary Plot: This visual tool combines the feature importance with their effects. Every dot on this chart represents a Shapley value for a certain feature in a specific case. The vertical axis is denoted by the feature whereas the horizontal one corresponds to the Shapley value. A color gradient indicates the value of the feature, gradually changing from low to high. Features are systematically organized in accordance with their importance.

Signs of High Risk

• Overemphasis on certain features in SHAP importance plots, thus hinting at the possibility of model overfitting
• Anomalies such as unexpected or illogical features showing high importance, which might suggest that the model's decisions are rooted in incorrect or undesirable reasoning
• A SHAP summary plot filled with high variability or scattered data points, indicating a cause for concern

Strengths

• SHAP does more than just illustrating global feature significance, it offers a detailed perspective on how different features shape the model's decision-making logic for each instance.
• It provides clear insights into model behavior.

Limitations

• High-dimensional data can convolute interpretations.
• Associating importance with tangible real-world impact still involves a certain degree of subjectivity.

Required Inputs: model, dataset

Parameters:

Parameter	Default Value
kernel_explainer_samples	10
tree_or_linear_explainer_samples	200
class_of_interest	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {
    "kernel_explainer_samples": "my_vm_kernel_explainer_samples",
    "tree_or_linear_explainer_samples": "my_vm_tree_or_linear_explainer_samples",
    "class_of_interest": "my_vm_class_of_interest"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.SHAPGlobalImportance", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 3.4. Model Diagnosis ('model_diagnosis')

Text Block: 'model_weak_spots_description'

▶ Test: Weakspots Diagnosis ('validmind.model_validation.sklearn.WeakspotsDiagnosis')

Weakspots Diagnosis

Identifies and visualizes weak spots in a machine learning model's performance across various sections of the feature space.

Purpose

The weak spots test is applied to evaluate the performance of a machine learning model within specific regions of its feature space. This test slices the feature space into various sections, evaluating the model's outputs within each section against specific performance metrics (e.g., accuracy, precision, recall, and F1 scores). The ultimate aim is to identify areas where the model's performance falls below the set thresholds, thereby exposing its possible weaknesses and limitations.

Test Mechanism

The test mechanism adopts an approach of dividing the feature space of the training dataset into numerous bins. The model's performance metrics (accuracy, precision, recall, F1 scores) are then computed for each bin on both the training and test datasets. A "weak spot" is identified if any of the performance metrics fall below a predetermined threshold for a particular bin on the test dataset. The test results are visually plotted as bar charts for each performance metric, indicating the bins which fail to meet the established threshold.

Signs of High Risk

• Any performance metric of the model dropping below the set thresholds.
• Significant disparity in performance between the training and test datasets within a bin could be an indication of overfitting.
• Regions or slices with consistently low performance metrics. Such instances could mean that the model struggles to handle specific types of input data adequately, resulting in potentially inaccurate predictions.

Strengths

• The test helps pinpoint precise regions of the feature space where the model's performance is below par, allowing for more targeted improvements to the model.
• The graphical presentation of the performance metrics offers an intuitive way to understand the model's performance across different feature areas.
• The test exhibits flexibility, letting users set different thresholds for various performance metrics according to the specific requirements of the application.

Limitations

• The binning system utilized for the feature space in the test could over-simplify the model's behavior within each bin. The granularity of this slicing depends on the chosen 'bins' parameter and can sometimes be arbitrary.
• The effectiveness of this test largely hinges on the selection of thresholds for the performance metrics, which may not hold universally applicable and could be subjected to the specifications of a particular model and application.
• The test is unable to handle datasets with a text column, limiting its application to numerical or categorical data types only.
• Despite its usefulness in highlighting problematic regions, the test does not offer direct suggestions for model improvement.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
features_columns	None
metrics	None
thresholds	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "features_columns": "my_vm_features_columns",
    "metrics": "my_vm_metrics",
    "thresholds": "my_vm_thresholds"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.WeakspotsDiagnosis", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'model_overfit_regions_description'

▶ Test: Overfit Diagnosis ('validmind.model_validation.sklearn.OverfitDiagnosis')

Overfit Diagnosis

Assesses potential overfitting in a model's predictions, identifying regions where performance between training and testing sets deviates significantly.

Purpose

The Overfit Diagnosis test aims to identify areas in a model's predictions where there is a significant difference in performance between the training and testing sets. This test helps to pinpoint specific regions or feature segments where the model may be overfitting.

Test Mechanism

This test compares the model's performance on training versus test data, grouped by feature columns. It calculates the difference between the training and test performance for each group and identifies regions where this difference exceeds a specified threshold:

• The test works for both classification and regression models.
• It defaults to using the AUC metric for classification models and the MSE metric for regression models.
• The threshold for identifying overfitting regions is set to 0.04 by default.
• The test calculates the performance metrics for each feature segment and plots regions where the performance gap exceeds the threshold.

Signs of High Risk

• Significant gaps between training and test performance metrics for specific feature segments.
• Multiple regions with performance gaps exceeding the defined threshold.
• Higher than expected differences in predicted versus actual values in the test set compared to the training set.

Strengths

• Identifies specific areas where overfitting occurs.
• Supports multiple performance metrics, providing flexibility.
• Applicable to both classification and regression models.
• Visualization of overfitting segments aids in better understanding and debugging.

Limitations

• The default threshold may not be suitable for all use cases and requires tuning.
• May not capture more subtle forms of overfitting that do not exceed the threshold.
• Assumes that the binning of features adequately represents the data segments.

Required Inputs: model, datasets

Parameters:

Parameter	Default Value
metric	None
cut_off_threshold	0.04

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "datasets": "my_vm_datasets"
}
params = {
    "metric": "my_vm_metric",
    "cut_off_threshold": "my_vm_cut_off_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.OverfitDiagnosis", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'model_robustness_description'

▶ Test: Robustness Diagnosis ('validmind.model_validation.sklearn.RobustnessDiagnosis')

Robustness Diagnosis

Assesses the robustness of a machine learning model by evaluating performance decay under noisy conditions.

Purpose

The Robustness Diagnosis test aims to evaluate the resilience of a machine learning model when subjected to perturbations or noise in its input data. This is essential for understanding the model's ability to handle real-world scenarios where data may be imperfect or corrupted.

Test Mechanism

This test introduces Gaussian noise to the numeric input features of the datasets at varying scales of standard deviation. The performance of the model is then measured using a specified metric. The process includes:

• Adding Gaussian noise to numerical input features based on scaling factors.
• Evaluating the model's performance on the perturbed data using metrics like AUC for classification tasks and MSE for regression tasks.
• Aggregating and plotting the results to visualize performance decay relative to perturbation size.

Signs of High Risk

• A significant drop in performance metrics with minimal noise.
• Performance decay values exceeding the specified threshold.
• Consistent failure to meet performance standards across multiple perturbation scales.

Strengths

• Provides insights into the model's robustness against noisy or corrupted data.
• Utilizes a variety of performance metrics suitable for both classification and regression tasks.
• Visualization helps in understanding the extent of performance degradation.

Limitations

• Gaussian noise might not adequately represent all types of real-world data perturbations.
• Performance thresholds are somewhat arbitrary and might need tuning.
• The test may not account for more complex or unstructured noise patterns that could affect model robustness.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
metric	None
scaling_factor_std_dev_list	[0.1, 0.2, 0.3, 0.4, 0.5]
performance_decay_threshold	0.05

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "metric": "my_vm_metric",
    "scaling_factor_std_dev_list": "my_vm_scaling_factor_std_dev_list",
    "performance_decay_threshold": "my_vm_performance_decay_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.RobustnessDiagnosis", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 4. Monitoring and Governance ('monitoring_governance')

▶ 4.1. Monitoring Plan ('monitoring_plan')

Text Block: 'monitoring_plan'

▶ 4.2. Monitoring Implementation ('monitoring_implementation')

Text Block: 'monitoring_implementation'

▶ 4.3. Governance Plan ('governance_plan')

Text Block: 'governance_plan'

Include custom test results

Since your custom test IDs are now part of your documentation template, you can now run tests for an entire section and all additional custom tests should be loaded without any issues.

Let's run all tests in the Model Evaluation section of the documentation. Note that we have been running the sample custom confusion matrix with normalize=True to demonstrate the ability to provide custom parameters.

In the Run the model evaluation tests section of 2 — Start the development process, you learned how to assign inputs to individual tests with run_documentation_tests(). Assigning parameters is similar, you only need to provide assign a params dictionary to a given test ID, my_test_provider.ConfusionMatrix in this case.

test_config = {
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "dataset": vm_train_ds,
            "model": vm_model,
        },
    },
    "my_test_provider.ConfusionMatrix": {
        "params": {"normalize": True},
        "inputs": {"dataset": vm_test_ds, "model": vm_model},
    },
}
results = vm.run_documentation_tests(
    section=["model_evaluation"],
    inputs={
        "dataset": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds
        "model": vm_model,
        "datasets": (
            vm_train_ds,
            vm_test_ds,
        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds
    },
    config=test_config,
)

2026-09-11 23:49:13,030 - WARNING(validmind.vm_models.test_suite.runner): Config key 'my_test_provider.ConfusionMatrix' does not match a test_id in the template.
    Ensure you registered a content block with the correct content_id in the template
    The configuration for this test will be ignored.

Test suite complete!

18/18 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Evaluation

Model Evaluation

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

The Confusion Matrix test evaluates classification performance by comparing predicted class labels against observed class labels. The result is presented as a 2×2 matrix showing counts for true positives, true negatives, false positives, and false negatives for the binary target classes 0 and 1. In this test run, the matrix reports 202 true positives, 198 true negatives, 141 false positives, and 106 false negatives. These values show how model predictions are distributed across correct and incorrect classifications for both classes.

Key insights:

• Correct classifications exceed errors: The model records 202 true positives and 198 true negatives, for a combined 400 correct classifications, compared with 141 false positives and 106 false negatives, for a combined 247 misclassifications.
• False positives are more frequent: False positives total 141, exceeding false negatives at 106. This indicates more cases with true label 0 were classified as 1 than cases with true label 1 classified as 0.
• Positive-class detection is the largest correct segment: True positives are the single largest cell in the matrix at 202, slightly above true negatives at 198. This reflects a marginally higher count of correctly identified class 1 cases than correctly identified class 0 cases.
• Error distribution is asymmetric across classes: For true class 1, the matrix shows 202 correct predictions versus 106 false negatives. For true class 0, it shows 198 correct predictions versus 141 false positives, indicating a higher error count on the class 0 side of the matrix.

The confusion matrix shows that correct classifications are more numerous than misclassifications overall, with true positives and true negatives both near 200 observations. The largest source of error is false positives, which occur more often than false negatives. Collectively, the results indicate stronger observed classification counts in the correctly predicted cells than in the error cells, with misclassification more concentrated in predictions of class 1 for observations belonging to class 0.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

The Classifier Performance test evaluates classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The in-sample results are reported for both classes individually and as macro and weighted averages, alongside overall accuracy and ROC AUC. Class-level precision ranges from 0.6378 to 0.6425, recall ranges from 0.6288 to 0.6514, and F1-scores range from 0.6333 to 0.6469. Aggregate metrics show weighted average precision, recall, and F1 of 0.6402, macro averages near 0.6401 to 0.6402, accuracy of 0.6402, and ROC AUC of 0.6898.

Key insights:

• Class metrics are closely aligned: Performance is similar across the two classes. Precision differs by 0.0047 between Class 0 and Class 1, recall differs by 0.0226, and F1 differs by 0.0136.
• Class 1 shows slightly stronger recall: Class 1 recall is 0.6514 compared with 0.6288 for Class 0. Class 1 also has the highest class-level F1-score at 0.6469.
• Aggregate averages are nearly identical: Weighted average and macro average metrics are effectively the same, with all reported values clustered around 0.6401 to 0.6402.
• ROC AUC exceeds point-metric levels: ROC AUC is 0.6898, which is higher than the reported accuracy and average F1 values of 0.6402.

The in-sample results show a broadly even classification profile across both classes, with only small differences between class-level precision, recall, and F1-score. Overall summary metrics are tightly concentrated around 0.64, and the near equivalence of weighted and macro averages indicates limited divergence between aggregate views of performance. ROC AUC of 0.6898 provides a higher ranking-based measure than the reported threshold-based summary metrics.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6378	0.6288	0.6333
1	0.6425	0.6514	0.6469
Weighted Average	0.6402	0.6402	0.6402
Macro Average	0.6402	0.6401	0.6401

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6402
ROC AUC	0.6898

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

The Classifier Performance test evaluates classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The out-of-sample results are presented for both classes individually, along with weighted and macro averages across classes. Class-level precision ranges from 0.5889 to 0.6513, recall ranges from 0.5841 to 0.6558, and F1 ranges from 0.6159 to 0.6206. Overall, the table reports weighted-average precision of 0.6216, weighted-average recall of 0.6182, weighted-average F1 of 0.6181, accuracy of 0.6182, and ROC AUC of 0.6627.

Key insights:

• Similar F1 across classes: F1-score is 0.6159 for Class 0 and 0.6206 for Class 1, indicating closely aligned harmonic mean performance across the two classes.
• Precision-recall tradeoff by class: Class 0 shows higher precision than recall (0.6513 vs. 0.5841), while Class 1 shows higher recall than precision (0.6558 vs. 0.5889), reflecting different error balance by class.
• Aggregate averages are closely aligned: Weighted and macro averages are nearly identical across all three metrics, with precision at 0.6216 vs. 0.6201, recall at 0.6182 vs. 0.6200, and F1 at 0.6181 vs. 0.6182.
• ROC AUC exceeds accuracy: ROC AUC is 0.6627 compared with accuracy of 0.6182, showing stronger ranking performance than threshold-based classification accuracy.

The out-of-sample results show broadly even performance across the two classes, with nearly identical class-level F1-scores and minimal differences between macro and weighted averages. At the same time, the class-specific precision and recall values differ in opposite directions, indicating that the distribution of false positives and false negatives varies by class. The overall summary metrics cluster around 0.62, while ROC AUC of 0.6627 is higher than the reported accuracy, indicating better discrimination than the single-threshold classification outcome.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6513	0.5841	0.6159
1	0.5889	0.6558	0.6206
Weighted Average	0.6216	0.6182	0.6181
Macro Average	0.6201	0.6200	0.6182

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6182
ROC AUC	0.6627

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

The PrecisionRecallCurve test evaluates the trade-off between precision and recall across classification thresholds for a binary classification model. The result is presented as a precision-recall curve with recall on the x-axis and precision on the y-axis. In this run, the curve begins at very high precision at minimal recall, rises into the low-to-mid 0.8 range at low recall levels, and then declines gradually as recall increases. At higher recall levels, precision continues to trend downward and finishes below 0.50 near the right edge of the plot.

Key insights:

• High precision at low recall: Precision is highest at very low recall, reaching approximately 1.0 at the left edge and then remaining in roughly the 0.80 to 0.85 range across the earliest portion of the curve.

• Early decline after initial peak: After the initial high-precision region, the curve drops to around 0.70 to 0.75 by low-to-moderate recall levels before showing a brief recovery toward the mid-0.70 range.

• Gradual degradation across mid recall: From approximately the mid-range of recall onward, precision trends downward in a relatively smooth manner, moving from about 0.63 to around 0.60 as recall approaches 0.60.

• Lower precision at high recall: In the higher-recall region, precision declines further into the low 0.50s and ends below 0.50 near maximum recall, indicating reduced positive predictive accuracy as more positives are captured.

The curve shows a clear trade-off between precision and recall across thresholds. Precision is strongest when recall is limited, followed by a progressive decline as recall increases. The overall shape indicates that the model maintains moderate precision through much of the middle recall range, with more pronounced deterioration in precision at higher recall levels.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

The ROCCurve test evaluates classification performance by plotting the Receiver Operating Characteristic curve and quantifying discrimination with the Area Under the Curve (AUC). The result shown for log_reg_model_v1 on test_dataset_final includes a single ROC curve against the random-classification reference line. The plotted curve traces the trade-off between true positive rate and false positive rate across thresholds, and the reported AUC is 0.66.

Key insights:

• AUC indicates above-random discrimination: The reported AUC is 0.66, which is above the random benchmark of 0.50 shown in the figure legend.
• ROC curve remains above baseline: Across most of the false positive rate range, the ROC curve lies above the diagonal random reference line, indicating positive separation between classes over a range of thresholds.
• Discrimination appears moderate: The curve shows improvement over random classification without approaching the upper-left corner closely, reflecting moderate rather than strong class separation in the test sample.

The ROC results show that log_reg_model_v1 demonstrates measurable discriminatory ability on test_dataset_final, with an AUC of 0.66 and a curve that stays above the random baseline through most thresholds. The overall pattern indicates performance better than chance, while the curve shape and distance from the upper-left corner reflect moderate separation between the target classes rather than strong discrimination.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

The TrainingTestDegradation test evaluates whether performance degradation between the training and test datasets remains within the predefined threshold for classification metrics. The results are reported separately for classes 1 and 0 across precision, recall, and F1-score, showing train score, test score, degradation percentage, and pass/fail status. Observed degradation values range from -2.1179% to 8.3439%, and all six metric-class combinations are marked as Pass.

Key insights:

• All evaluated metrics passed: Each of the six assessed metric-class combinations has a Pass status, indicating that all observed degradation values remain below the configured 10% threshold.
• Largest degradation in class 1 precision: For class 1, precision declines from 0.6425 on the training dataset to 0.5889 on the test dataset, corresponding to the highest observed degradation at 8.3439%.
• Recall improved for two class views: Class 1 recall increases from 0.6514 to 0.6558, producing a degradation value of -0.6859%, and class 0 precision increases from 0.6378 to 0.6513, with degradation of -2.1179%.
• F1-score degradation remained limited: F1-score decreases are moderate for both classes, with class 1 falling from 0.6469 to 0.6206 (4.0717%) and class 0 falling from 0.6333 to 0.6159 (2.7504%).

The test results show that train-to-test performance differences remained bounded across all evaluated classification metrics and both classes. The largest reduction is concentrated in class 1 precision, while class 1 recall and class 0 precision are higher on the test dataset than on the training dataset. Overall, the reported degradation profile is mixed across metrics but remains within the stated acceptance threshold for every evaluated result.

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
1	Precision	0.6425	0.5889	8.3439	Pass
1	Recall	0.6514	0.6558	-0.6859	Pass
1	F1-Score	0.6469	0.6206	4.0717	Pass
0	Precision	0.6378	0.6513	-2.1179	Pass
0	Recall	0.6288	0.5841	7.1160	Pass
0	F1-Score	0.6333	0.6159	2.7504	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

The Minimum Accuracy test evaluates whether the model’s prediction accuracy on the evaluated dataset meets or exceeds a predefined minimum threshold. The result table reports an accuracy score of 0.6182 against a threshold of 0.7, along with the corresponding pass/fail outcome. The recorded outcome for this test is Fail.

Key insights:

• Accuracy is below threshold: The observed accuracy score is 0.6182, which is lower than the configured minimum threshold of 0.7.
• Test outcome is fail: Because the reported score does not meet the threshold, the test result is recorded as Fail.
• Gap to threshold is 0.0818: The difference between the observed accuracy and the minimum threshold is 0.0818.

The result indicates that the model’s observed classification accuracy on the tested dataset did not satisfy the minimum criterion defined for this test. The measured score of 0.6182 remained below the 0.7 threshold, and the test therefore produced a failing outcome.

Tables

Score	Threshold	Pass/Fail
0.6182	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

The MinimumF1Score test evaluates whether the model’s F1 score on the validation dataset meets a predefined minimum threshold. The result table reports the observed F1 score, the configured threshold, and the corresponding pass/fail outcome for this comparison. In this test run, the validation F1 score is 0.6206 against a threshold of 0.5, and the recorded outcome is Pass.

Key insights:

• F1 score exceeds threshold: The observed validation F1 score is 0.6206, which is higher than the minimum threshold of 0.5 used in the test.
• Positive test outcome recorded: The result is marked as Pass, reflecting that the observed score satisfied the threshold condition.
• Margin above minimum is 0.1206: The difference between the observed F1 score and the threshold is 0.1206, indicating performance above the configured minimum level in this test.

The test result shows that the model’s validation F1 score met the predefined acceptance criterion. The observed score of 0.6206 exceeded the threshold of 0.5 by 0.1206, and the test output was recorded as Pass.

Tables

Score	Threshold	Pass/Fail
0.6206	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

The Minimum ROC AUC Score test evaluates whether the model’s ROC AUC score on the validation dataset meets or exceeds a predefined minimum threshold. The reported result includes the observed ROC AUC score, the threshold applied in the test, and the resulting pass/fail status. In this run, the model produced a ROC AUC score of 0.6627 against a threshold of 0.5, and the test result is recorded as Pass.

Key insights:

• Score exceeds threshold: The observed ROC AUC score of 0.6627 is above the configured minimum threshold of 0.5 by 0.1627.
• Test outcome is pass: The result is explicitly recorded as Pass, indicating the validation criterion defined by this test was met.
• Moderate separation reflected: The reported ROC AUC value of 0.6627 quantifies the model’s class discrimination performance under the test methodology used.

The result shows that the model met the minimum ROC AUC criterion established for this test. The observed score of 0.6627 exceeded the threshold of 0.5, and the test therefore concluded with a pass outcome. Collectively, the result documents that the model satisfied this specific validation check based on ROC AUC.

Tables

Score	Threshold	Pass/Fail
0.6627	0.5	Pass

Configuring documentation template tests

Let's call the utility function vm.get_test_suite().get_default_config() which will return the default configuration for the entire documentation template as a dictionary:

• This configuration will contain all the test IDs and their default parameters.
• You can then modify this configuration as needed and pass it to run_documentation_tests() to run all tests in the documentation template if needed.
• You still have the option to continue running tests for one section at a time; get_default_config() simply provides a useful reference for providing default parameters to every test.

import json

model_test_suite = vm.get_test_suite()
config = model_test_suite.get_default_config()
print("Suite Config: \n", json.dumps(config, indent=2))

Suite Config: 
 {
  "validmind.data_validation.DatasetDescription": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.data_validation.ClassImbalance": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_percent_threshold": 10
    }
  },
  "validmind.data_validation.Duplicates": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_threshold": 1
    }
  },
  "validmind.data_validation.HighCardinality": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "num_threshold": 100,
      "percent_threshold": 0.1,
      "threshold_type": "percent"
    }
  },
  "validmind.data_validation.MissingValues": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_percentage_threshold": 1.0
    }
  },
  "validmind.data_validation.Skewness": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "max_threshold": 1
    }
  },
  "validmind.data_validation.UniqueRows": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_percent_threshold": 1
    }
  },
  "validmind.data_validation.TooManyZeroValues": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "max_percent_threshold": 0.03
    }
  },
  "validmind.data_validation.IQROutliersTable": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "threshold": 1.5
    }
  },
  "validmind.data_validation.IQROutliersBarPlot": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "threshold": 1.5,
      "fig_width": 800
    }
  },
  "validmind.data_validation.DescriptiveStatistics": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.data_validation.PearsonCorrelationMatrix": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.data_validation.HighPearsonCorrelation": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "max_threshold": 0.3,
      "top_n_correlations": 10,
      "feature_columns": null
    }
  },
  "validmind.model_validation.ModelMetadata": {
    "inputs": {
      "model": "model"
    },
    "params": {}
  },
  "validmind.data_validation.DatasetSplit": {
    "inputs": {
      "datasets": "datasets"
    },
    "params": {}
  },
  "validmind.model_validation.sklearn.PopulationStabilityIndex": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "num_bins": 10,
      "mode": "fixed"
    }
  },
  "validmind.model_validation.sklearn.ConfusionMatrix": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "threshold": 0.5
    }
  },
  "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "average": "macro"
    }
  },
  "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "average": "macro"
    }
  },
  "validmind.model_validation.sklearn.PrecisionRecallCurve": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.model_validation.sklearn.ROCCurve": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.model_validation.sklearn.TrainingTestDegradation": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "max_threshold": 0.1
    }
  },
  "validmind.model_validation.sklearn.MinimumAccuracy": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "min_threshold": 0.7
    }
  },
  "validmind.model_validation.sklearn.MinimumF1Score": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "min_threshold": 0.5
    }
  },
  "validmind.model_validation.sklearn.MinimumROCAUCScore": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "min_threshold": 0.5
    }
  },
  "validmind.model_validation.sklearn.PermutationFeatureImportance": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {
      "fontsize": null,
      "figure_height": null
    }
  },
  "validmind.model_validation.sklearn.SHAPGlobalImportance": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {
      "kernel_explainer_samples": 10,
      "tree_or_linear_explainer_samples": 200,
      "class_of_interest": null
    }
  },
  "validmind.model_validation.sklearn.WeakspotsDiagnosis": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "features_columns": null,
      "metrics": null,
      "thresholds": null
    }
  },
  "validmind.model_validation.sklearn.OverfitDiagnosis": {
    "inputs": {
      "model": "model",
      "datasets": "datasets"
    },
    "params": {
      "metric": null,
      "cut_off_threshold": 0.04
    }
  },
  "validmind.model_validation.sklearn.RobustnessDiagnosis": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "metric": null,
      "scaling_factor_std_dev_list": [
        0.1,
        0.2,
        0.3,
        0.4,
        0.5
      ],
      "performance_decay_threshold": 0.05
    }
  }
}

Preview test configuration

The default config does not assign any inputs to a test, but you can assign inputs to individual tests as needed depending on the datasets and records (models) you want to pass to individual tests.

For this particular documentation template (binary classification), the ValidMind Library provides a sample configuration that can be used to populate the entire documentation using the following inputs as placeholders:

• A raw_dataset raw dataset
• A train_dataset training dataset
• A test_dataset test dataset
• A trained model instance

As part of updating the config you will need to ensure the correct input_ids are used in the final config passed to run_documentation_tests().

from validmind.datasets.classification import customer_churn
from validmind.utils import preview_test_config

test_config = customer_churn.get_demo_test_config()
preview_test_config(test_config)

{
    "validmind.data_validation.DatasetDescription": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {}
    },
    "validmind.data_validation.ClassImbalance": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_percent_threshold": 10
        }
    },
    "validmind.data_validation.Duplicates": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_threshold": 1
        }
    },
    "validmind.data_validation.HighCardinality": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "num_threshold": 100,
            "percent_threshold": 0.1,
            "threshold_type": "percent"
        }
    },
    "validmind.data_validation.MissingValues": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_percentage_threshold": 1.0
        }
    },
    "validmind.data_validation.Skewness": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "max_threshold": 1
        }
    },
    "validmind.data_validation.UniqueRows": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_percent_threshold": 1
        }
    },
    "validmind.data_validation.TooManyZeroValues": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "max_percent_threshold": 0.03
        }
    },
    "validmind.data_validation.IQROutliersTable": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "threshold": 1.5
        }
    },
    "validmind.data_validation.IQROutliersBarPlot": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "threshold": 1.5,
            "fig_width": 800
        }
    },
    "validmind.data_validation.DescriptiveStatistics": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {}
    },
    "validmind.data_validation.PearsonCorrelationMatrix": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {}
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "max_threshold": 0.3,
            "top_n_correlations": 10,
            "feature_columns": null
        }
    },
    "validmind.model_validation.ModelMetadata": {
        "inputs": {
            "model": "model"
        },
        "params": {}
    },
    "validmind.data_validation.DatasetSplit": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ]
        },
        "params": {}
    },
    "validmind.model_validation.sklearn.PopulationStabilityIndex": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "num_bins": 10,
            "mode": "fixed"
        }
    },
    "validmind.model_validation.sklearn.ConfusionMatrix": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "threshold": 0.5
        }
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "model": "model",
            "dataset": "train_dataset"
        }
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        }
    },
    "validmind.model_validation.sklearn.PrecisionRecallCurve": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {}
    },
    "validmind.model_validation.sklearn.ROCCurve": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {}
    },
    "validmind.model_validation.sklearn.TrainingTestDegradation": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "max_threshold": 0.1
        }
    },
    "validmind.model_validation.sklearn.MinimumAccuracy": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "min_threshold": 0.7
        }
    },
    "validmind.model_validation.sklearn.MinimumF1Score": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "min_threshold": 0.5
        }
    },
    "validmind.model_validation.sklearn.MinimumROCAUCScore": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "min_threshold": 0.5
        }
    },
    "validmind.model_validation.sklearn.PermutationFeatureImportance": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {
            "fontsize": null,
            "figure_height": null
        }
    },
    "validmind.model_validation.sklearn.SHAPGlobalImportance": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {
            "kernel_explainer_samples": 10,
            "tree_or_linear_explainer_samples": 200,
            "class_of_interest": null
        }
    },
    "validmind.model_validation.sklearn.WeakspotsDiagnosis": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "features_columns": null,
            "metrics": null,
            "thresholds": null
        }
    },
    "validmind.model_validation.sklearn.OverfitDiagnosis": {
        "inputs": {
            "model": "model",
            "datasets": [
                "train_dataset",
                "test_dataset"
            ]
        },
        "params": {
            "metric": null,
            "cut_off_threshold": 0.04
        }
    },
    "validmind.model_validation.sklearn.RobustnessDiagnosis": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "metric": null,
            "scaling_factor_std_dev_list": [
                0.1,
                0.2,
                0.3,
                0.4,
                0.5
            ],
            "performance_decay_threshold": 0.05
        }
    }
}

Run updated documentation section tests

Using this sample configuration, let's finish populating documentation by running all tests for the Model Development section of the documentation.

Recall that the training and test datasets in our exercise have the following input_id values:

• train_dataset_final for the training dataset
• test_dataset_final for the test dataset

config = {
    "validmind.model_validation.ModelMetadata": {
        "inputs": {"model": "log_reg_model_v1"},
    },
    "validmind.data_validation.DatasetSplit": {
        "inputs": {"datasets": ["train_dataset_final", "test_dataset_final"]},
    },
    "validmind.model_validation.sklearn.PopulationStabilityIndex": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {"num_bins": 10, "mode": "fixed"},
    },
    "validmind.model_validation.sklearn.ConfusionMatrix": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "my_test_provider.ConfusionMatrix": {
        "inputs": {"dataset": "test_dataset_final", "model": "log_reg_model_v1"},
    },
    "my_custom_tests.ConfusionMatrix:test_dataset_normalized": {
        "inputs": {"dataset": "test_dataset_final", "model": "log_reg_model_v1"},
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "train_dataset_final"}
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"}
    },
    "validmind.model_validation.sklearn.PrecisionRecallCurve": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "validmind.model_validation.sklearn.ROCCurve": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "validmind.model_validation.sklearn.TrainingTestDegradation": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {
            "metrics": ["accuracy", "precision", "recall", "f1"],
            "max_threshold": 0.1,
        },
    },
    "validmind.model_validation.sklearn.MinimumAccuracy": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"min_threshold": 0.7},
    },
    "validmind.model_validation.sklearn.MinimumF1Score": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"min_threshold": 0.5},
    },
    "validmind.model_validation.sklearn.MinimumROCAUCScore": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"min_threshold": 0.5},
    },
    "validmind.model_validation.sklearn.PermutationFeatureImportance": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "validmind.model_validation.sklearn.SHAPGlobalImportance": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"kernel_explainer_samples": 10},
    },
    "validmind.model_validation.sklearn.WeakspotsDiagnosis": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {
            "thresholds": {"accuracy": 0.75, "precision": 0.5, "recall": 0.5, "f1": 0.7}
        },
    },
    "validmind.model_validation.sklearn.OverfitDiagnosis": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {"cut_off_percentage": 4},
    },
    "validmind.model_validation.sklearn.RobustnessDiagnosis": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {
            "scaling_factor_std_dev_list": [0.0, 0.1, 0.2, 0.3, 0.4, 0.5],
            "accuracy_decay_threshold": 4,
        },
    },
}


full_suite = vm.run_documentation_tests(
    section="model_development",
    config=config,
)

2026-09-11 23:49:29,838 - WARNING(validmind.vm_models.test_suite.runner): Config key 'my_test_provider.ConfusionMatrix' does not match a test_id in the template.
    Ensure you registered a content block with the correct content_id in the template
    The configuration for this test will be ignored.
2026-09-11 23:49:29,838 - WARNING(validmind.vm_models.test_suite.runner): Config key 'my_custom_tests.ConfusionMatrix:test_dataset_normalized' does not match a test_id in the template.
    Ensure you registered a content block with the correct content_id in the template
    The configuration for this test will be ignored.

Test suite complete!

34/34 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Development

Model Development

▶ Test Result: Model Metadata (validmind.model_validation.ModelMetadata)

Model Metadata

The ModelMetadata test compares core metadata fields across models to summarize their implementation attributes. The result is presented as a metadata table with columns for modeling technique, modeling framework, framework version, and programming language. In this execution, the table contains one model entry with values reported for all four metadata fields: SKlearnModel, sklearn, version 1.7.2, and Python.

Key insights:

• Single model entry reported: The result table contains one model record, indicating that the comparison output for this execution is based on a single model metadata profile.
• All core metadata fields populated: Values are present for modeling technique, modeling framework, framework version, and programming language, with no missing entries shown in the reported table.
• Implementation stack explicitly identified: The model is documented as using the SKlearnModel technique within the sklearn framework, framework version 1.7.2, and Python as the programming language.

The reported metadata provides a complete high-level implementation profile for the model included in this test output. The result identifies the model’s technique, framework, framework version, and programming language without gaps in the displayed fields. As presented, the output supports clear documentation of the model’s core technical metadata for the single model record shown.

Tables

Modeling Technique	Modeling Framework	Framework Version	Programming Language
SKlearnModel	sklearn	1.7.2	Python

▶ Test Result: Dataset Split (validmind.data_validation.DatasetSplit)

Dataset Split

The DatasetSplit test evaluates the distribution of records across the model’s available datasets. The results table reports the absolute size and proportion of each split relative to the total dataset size. In this run, the table includes training and testing datasets along with the overall total, showing 2,585 records in train_dataset_final, 647 records in test_dataset_final, and 3,232 records in total.

Key insights:

• Training data comprises most records: train_dataset_final contains 2,585 records, representing 79.98% of the total dataset.
• Test split represents one-fifth: test_dataset_final contains 647 records, accounting for 20.02% of the total dataset.
• Two-way split is documented: The reported dataset structure includes training and testing splits only, with no validation dataset shown in the result table.

The reported split is composed of 3,232 total records divided between a training dataset of 2,585 observations and a testing dataset of 647 observations. The distribution is concentrated in the training split at 79.98%, with the remaining 20.02% allocated to testing. The result documents a two-way dataset partition in this test output.

Tables

Dataset	Size	Proportion
train_dataset_final	2585	79.98%
test_dataset_final	647	20.02%
Total	3232	100%

▶ Test Result: Population Stability Index (validmind.model_validation.sklearn.PopulationStabilityIndex)

Population Stability Index

The Population Stability Index (PSI) test evaluates the stability of model output distributions by comparing the binned population shares of the initial and new datasets. The results are presented across 10 fixed bins for train_dataset_final and test_dataset_final, showing counts, percentage shares, and bin-level PSI contributions. The total PSI reported is 0.0357, with the table and plot indicating how the initial and new population ratios differ within each bin.

Key insights:

• Low overall PSI: The total PSI is 0.0357 across the 10 fixed bins, reflecting a small aggregate difference between the initial and new distributions.
• Largest shift in bin 1: Bin 1 has the highest PSI contribution at 0.0265, where the population share declines from 9.2070% in the initial dataset to 4.9459% in the new dataset.
• Most bins are closely aligned: Several bins have negligible PSI contributions, including bins 4 and 5 at 0.0000, with very similar population shares between datasets in those ranges.
• Secondary differences are limited: Outside bin 1, the next largest PSI contributions are 0.0032 in bin 6 and 0.0031 in bin 8, indicating that remaining distribution differences are comparatively small.
• Mid-range bins remain concentrated: The highest population shares in both datasets are located in the middle bins, with bin 4 representing 15.2031% of the initial dataset and 14.9923% of the new dataset, and adjacent bins also showing similar concentration levels.

The PSI result shows that the initial and new distributions are broadly similar, with the aggregate divergence concentrated primarily in one lower-index bin. Middle bins retain closely matching population shares, and most other bin-level PSI contributions are minimal. Overall, the observed distributional change is limited in magnitude and unevenly concentrated rather than broadly distributed across bins.

Parameters:

{
  "num_bins": 10,
  "mode": "fixed"
}
            

Tables

Population Stability Index for train_dataset_final and test_dataset_final Datasets

Bin	Count Initial	Percent Initial (%)	Count New	Percent New (%)	PSI
0	166	6.4217	43	6.6461	0.0001
1	238	9.2070	32	4.9459	0.0265
2	297	11.4894	81	12.5193	0.0009
3	338	13.0754	91	14.0649	0.0007
4	393	15.2031	97	14.9923	0.0000
5	341	13.1915	86	13.2921	0.0000
6	285	11.0251	84	12.9830	0.0032
7	255	9.8646	70	10.8192	0.0009
8	139	5.3772	27	4.1731	0.0031
9	133	5.1451	36	5.5641	0.0003
Total	2585	100.0000	647	100.0000	0.0357

Figures

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

The Confusion Matrix test evaluates classification performance by comparing predicted labels with observed labels and displaying the counts of true positives, true negatives, false positives, and false negatives. The heatmap shows a binary classification result with 202 true positives, 198 true negatives, 141 false positives, and 106 false negatives. The matrix therefore captures both correctly classified observations and the distribution of the two error types across the test sample.

Key insights:

• Correct classifications exceed errors: The model records 202 true positives and 198 true negatives, for a total of 400 correct classifications, compared with 247 combined misclassifications.
• False positives are more frequent: False positives total 141, exceeding the 106 false negatives by 35 cases, indicating more negative cases were classified as positive than positive cases classified as negative.
• True positive count is highest: The largest single cell in the matrix is true positives at 202, followed closely by true negatives at 198, while both error cells are lower in count.
• Error distribution is asymmetric: Misclassification is not evenly split across error types, with false positives representing the larger share of total errors.

The confusion matrix shows that correct predictions outnumber incorrect predictions, with true positive and true negative counts at similar levels. Errors are present in both directions, with false positives occurring more often than false negatives. Overall, the observed classification pattern reflects stronger concentration in the correct-classification cells than in the error cells, alongside a moderate skew toward positive overprediction among misclassified cases.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

The Classifier Performance test evaluates classification performance using precision, recall, F1-score, accuracy, and ROC AUC. In this in-sample result, the report presents class-level precision, recall, and F1 values for classes 0 and 1, along with macro and weighted averages, and separate summary metrics for overall accuracy and ROC AUC. The class-level metrics are closely aligned across the two classes, while the aggregate metrics provide a consolidated view of overall in-sample classification performance.

Key insights:

• Balanced class-level performance: Class 0 and class 1 show similar results across all three metrics. Precision is 0.6378 for class 0 and 0.6425 for class 1, recall is 0.6288 and 0.6514 respectively, and F1-scores are 0.6333 and 0.6469.
• Aggregate metrics are tightly clustered: Weighted-average precision, recall, and F1 are all 0.6402, while macro-average precision, recall, and F1 are 0.6402, 0.6401, and 0.6401. The near-equality of macro and weighted averages indicates little variation between class-specific performance and aggregate performance.
• Recall is higher for class 1: Recall for class 1 is 0.6514 compared with 0.6288 for class 0, a difference of 0.0226. This indicates a modestly higher true positive capture rate for class 1 within the in-sample evaluation.
• ROC AUC exceeds accuracy: Overall accuracy is 0.6402, while ROC AUC is 0.6898. These metrics indicate that the ranking-based discrimination measure is higher than the threshold-dependent accuracy measure in this test run.

The in-sample performance results show a broadly even classification profile across both classes, with class-specific precision, recall, and F1 values remaining close to one another. Aggregate averages align almost exactly with each other and with the overall accuracy value of 0.6402, indicating consistent summary performance across reporting methods. ROC AUC of 0.6898 is higher than accuracy, while the main class-level difference is the somewhat stronger recall observed for class 1.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6378	0.6288	0.6333
1	0.6425	0.6514	0.6469
Weighted Average	0.6402	0.6402	0.6402
Macro Average	0.6402	0.6401	0.6401

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6402
ROC AUC	0.6898

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

The Classifier Performance test evaluates out-of-sample classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The reported results present class-level precision, recall, and F1 for classes 0 and 1, along with macro and weighted averages across classes. The table shows class 0 precision of 0.6513, recall of 0.5841, and F1 of 0.6159, while class 1 precision is 0.5889, recall is 0.6558, and F1 is 0.6206. Overall summary metrics are accuracy of 0.6182 and ROC AUC of 0.6627, with macro-average F1 of 0.6182 and weighted-average F1 of 0.6181.

Key insights:

• Balanced aggregate performance: Macro-average and weighted-average metrics are closely aligned, with F1 values of 0.6182 and 0.6181 respectively, and precision and recall also differing only marginally across the two averaging methods.
• Class-specific precision-recall tradeoff: Class 0 has higher precision than class 1 (0.6513 vs. 0.5889), while class 1 has higher recall than class 0 (0.6558 vs. 0.5841), indicating different error behavior across the two classes.
• Similar class-level F1 scores: F1 scores are close across classes, at 0.6159 for class 0 and 0.6206 for class 1, showing comparable combined precision-recall performance by class.
• Moderate discrimination in ROC AUC: The ROC AUC is 0.6627, providing a summary measure of ranking performance distinct from the threshold-based classification metrics.

The out-of-sample results show closely aligned aggregate metrics and similar F1 performance across both classes. Differences between classes are primarily expressed through the precision-recall tradeoff, with class 0 showing higher precision and class 1 showing higher recall. Accuracy is 0.6182, and the ROC AUC of 0.6627 indicates the model’s discrimination performance as measured by the reported test.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6513	0.5841	0.6159
1	0.5889	0.6558	0.6206
Weighted Average	0.6216	0.6182	0.6181
Macro Average	0.6201	0.6200	0.6182

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6182
ROC AUC	0.6627

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

The PrecisionRecallCurve test evaluates the trade-off between precision and recall across classification thresholds for the binary classification model. The result is presented as a precision-recall curve with recall on the x-axis and precision on the y-axis, showing how precision changes as recall increases. In this run, the curve starts at very high precision at very low recall, rises quickly above 0.8 in the low-recall region, and then declines gradually across the remainder of the recall range. By high recall levels near the right side of the plot, precision is approximately in the 0.5 range.

Key insights:

• High precision at low recall: At very low recall levels, the curve reaches precision values near 1.0 and remains above 0.8 in the earliest portion of the curve, indicating stronger precision in the most selective threshold region.
• Gradual precision decline: As recall increases, precision declines in a mostly gradual pattern rather than collapsing abruptly, moving from the low-to-mid 0.7 range into the low 0.6 range through the middle of the curve.
• Mid-range stability visible: Across the approximate mid-recall region, the curve stays relatively stable around 0.60 to 0.64, with only modest local fluctuation.
• Lower precision at broad coverage: At higher recall levels, precision trends downward further and approaches roughly 0.5 near the far right of the curve, reflecting lower precision when a larger share of positive cases is captured.

The curve shows a clear trade-off between precision and recall across thresholds. Precision is strongest in the low-recall region, remains around the low 0.6 range through much of the middle of the curve, and declines toward approximately 0.5 at high recall. Overall, the result reflects progressively lower precision as recall broadens, with no abrupt discontinuities in the observed curve shape.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

The ROCCurve test evaluates classification performance by plotting the Receiver Operating Characteristic curve and calculating the Area Under the Curve (AUC). The result for log_reg_model_v1 on test_dataset_final shows a single ROC curve for the binary classification task, plotted against the random-classification reference line. The chart reports an AUC of 0.66, with the ROC curve remaining above the 45-degree baseline across most of the false positive rate range.

Key insights:

• AUC indicates moderate discrimination: The reported AUC is 0.66, which places the model above the random benchmark of 0.50 and indicates measurable separation between the two classes on the test dataset.
• ROC curve stays above baseline: The ROC curve lies above the random reference line through most of the plot, showing that true positive rate generally increases faster than false positive rate across thresholds.
• Performance gain varies by threshold region: The curve rises gradually rather than sharply, indicating that the improvement in true positive rate is distributed across thresholds rather than concentrated in a small low-false-positive region.

The ROC result shows that log_reg_model_v1 demonstrates positive discriminatory ability on test_dataset_final, with an AUC of 0.66 and a curve consistently above the random baseline. The shape of the curve indicates that classification separation is present but not strong, as gains in true positive rate accumulate progressively across the false positive rate range rather than through a steep early rise.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

The TrainingTestDegradation test evaluates whether classification performance declines from the training dataset to the test dataset by more than the configured maximum threshold of 10% across precision, recall, and F1-score. The result table reports train and test scores separately for classes 1 and 0, along with the calculated degradation percentage and pass/fail status for each metric. Observed degradation values range from -2.1179% to 8.3439%, and each evaluated metric is marked as Pass.

Key insights:

• All metrics passed threshold: All six class-metric combinations have pass statuses of Pass, with no degradation percentage exceeding the 10% threshold.
• Largest decline in class 1 precision: Class 1 precision decreased from 0.6425 on training to 0.5889 on test, corresponding to the highest reported degradation at 8.3439%.
• Recall improved for two cases: Class 1 recall increased from 0.6514 to 0.6558, producing a degradation value of -0.6859%, and class 0 precision increased from 0.6378 to 0.6513, producing a degradation value of -2.1179%.
• F1-score changes were moderate: F1-score degradation remained limited for both classes, at 4.0717% for class 1 and 2.7504% for class 0, with train and test scores remaining relatively close in each case.
• Class 0 recall showed the largest class 0 gap: For class 0, recall declined from 0.6288 on training to 0.5841 on test, corresponding to a degradation of 7.1160%, which is the largest decrease among class 0 metrics.

The results show that train-to-test performance differences remained within the configured tolerance for every evaluated metric. The largest observed reductions were in class 1 precision and class 0 recall, while class 1 recall and class 0 precision improved on the test dataset relative to training. Overall, the reported precision, recall, and F1-score comparisons indicate limited performance degradation between the training and test datasets under this test definition.

Parameters:

{
  "max_threshold": 0.1
}
            

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
1	Precision	0.6425	0.5889	8.3439	Pass
1	Recall	0.6514	0.6558	-0.6859	Pass
1	F1-Score	0.6469	0.6206	4.0717	Pass
0	Precision	0.6378	0.6513	-2.1179	Pass
0	Recall	0.6288	0.5841	7.1160	Pass
0	F1-Score	0.6333	0.6159	2.7504	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

The Minimum Accuracy test evaluates whether the model’s prediction accuracy meets or exceeds a predefined minimum threshold. The result table reports an accuracy score of 0.6182 against a threshold of 0.7000, along with the associated pass/fail outcome. In this execution, the recorded outcome is Fail based on the comparison of the observed score to the configured threshold.

Key insights:

• Accuracy is below threshold: The observed accuracy score is 0.6182, which is lower than the configured minimum threshold of 0.7000 by 0.0818.
• Test outcome is Fail: The result table records a Fail outcome, reflecting that the measured accuracy does not meet the threshold criterion used in this test.
• Single-metric assessment applied: The evaluation is based on accuracy only, using the configured minimum threshold parameter of 0.7 shown in the test result.

The test result shows that the model’s measured accuracy on the evaluated dataset is 0.6182, which falls short of the 0.7000 minimum threshold defined for this assessment. This gap produces a Fail outcome in the test record. Collectively, the result documents that the model did not satisfy the minimum accuracy criterion under the conditions of this run.

Parameters:

{
  "min_threshold": 0.7
}
            

Tables

Score	Threshold	Pass/Fail
0.6182	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

The MinimumF1Score test evaluates whether the model’s validation F1 score meets a predefined minimum threshold. The result table reports a validation F1 score of 0.6206 alongside a threshold of 0.5, with the outcome recorded as Pass. These values provide the basis for assessing whether the model achieved the minimum balanced precision-recall performance level defined for this test.

Key insights:

• F1 score exceeds threshold: The validation F1 score is 0.6206 versus a minimum threshold of 0.5, placing the observed result 0.1206 above the required level.
• Test outcome is pass: The result is explicitly marked as Pass, indicating the observed validation F1 score met the criterion defined by the test parameter.

The test result shows that the model’s validation F1 score exceeded the configured minimum threshold and satisfied the pass criterion. The observed margin above the threshold was 0.1206, indicating that the minimum performance level specified for this test was achieved on the validation dataset.

Parameters:

{
  "min_threshold": 0.5
}
            

Tables

Score	Threshold	Pass/Fail
0.6206	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

The Minimum ROC AUC Score test evaluates whether the model’s ROC AUC on the validation dataset meets or exceeds a predefined minimum threshold. The result table reports a ROC AUC score of 0.6627, alongside the configured threshold of 0.5 and the resulting pass/fail outcome. This output provides a direct comparison between the observed discrimination metric and the minimum acceptance level defined for the test.

Key insights:

• Threshold exceeded: The observed ROC AUC score is 0.6627 versus a minimum threshold of 0.5, placing the result 0.1627 above the defined cutoff.
• Test status is pass: The recorded outcome is "Pass," indicating the validation result satisfied the test condition based on the configured threshold.
• Moderate separation reflected: The reported ROC AUC value of 0.6627 indicates classification performance above the baseline threshold used in this test.

The result shows that the model met the Minimum ROC AUC Score criterion on the evaluated validation data. The observed score of 0.6627 exceeded the configured threshold of 0.5, and the test therefore returned a passing outcome. Collectively, the result and supporting values indicate that the model satisfied this specific discrimination threshold under the test configuration provided.

Parameters:

{
  "min_threshold": 0.5
}
            

Tables

Score	Threshold	Pass/Fail
0.6627	0.5	Pass

▶ Test Result: Permutation Feature Importance (validmind.model_validation.sklearn.PermutationFeatureImportance)

Permutation Feature Importance

The Permutation Feature Importance test evaluates the relative contribution of each input feature by measuring the change in model performance after randomly permuting that feature’s values. The results are presented as a horizontal bar chart of permutation importances across the model inputs. Importance values span both positive and negative ranges, with the largest positive values observed for Geography_Germany, IsActiveMember, and Gender_Male, while CreditScore and NumOfProducts show negative values and several remaining features are near zero.

Key insights:

• Three features dominate importance: Geography_Germany shows the highest permutation importance at approximately 0.04, followed by IsActiveMember at roughly 0.038 and Gender_Male at about 0.03. These features are separated from the rest of the variables by a clear margin.

• Balance has moderate contribution: Balance has a positive importance near 0.01, placing it below the top three features but above the remaining variables.

• Most remaining features are low-impact: Tenure has a small positive importance around 0.003, while EstimatedSalary and HasCrCard are only slightly above zero. These results indicate limited measured effect on model performance under permutation.

• Negative importances are present: CreditScore has the most negative importance at approximately -0.009, and NumOfProducts and Geography_Spain are also slightly negative. In these cases, model performance did not decline after permutation and appears marginally improved.

The permutation importance profile is concentrated in a small subset of predictors, with Geography_Germany, IsActiveMember, and Gender_Male contributing the largest measured performance impact. Balance provides a secondary contribution, while most other variables have minimal influence in this test. The presence of negative importances for CreditScore, NumOfProducts, and Geography_Spain indicates that these features did not contribute positively to performance under the reported permutation-based assessment.

Figures

▶ Test Result: SHAP Global Importance (validmind.model_validation.sklearn.SHAPGlobalImportance)

SHAP Global Importance

The SHAPGlobalImportance test evaluates global feature importance using SHAP values to quantify how strongly each feature contributes to model output across observations. The results are presented through a normalized feature importance bar chart and a SHAP summary plot. The bar chart ranks features by mean absolute SHAP value, while the summary plot shows the direction and spread of feature-level impacts, with color indicating relative feature value from low to high.

Key insights:

• Importance is concentrated in three features: IsActiveMember, Geography_Germany, and Gender_Male have the largest normalized SHAP importance values, at approximately 100, 87, and 66 respectively. These three features are materially higher than the remaining variables in the ranking.

• IsActiveMember is the dominant driver: IsActiveMember is the highest-ranked feature by a clear margin in the normalized importance plot. In the summary plot, observations split into two distinct effect levels, with low feature values aligned with negative SHAP contributions and high feature values aligned with strongly positive contributions.

• Geography_Germany shows strong directional separation: Geography_Germany is the second most important feature and shows two separated SHAP clusters in the summary plot. Low values are associated with negative SHAP values, while high values are associated with strongly positive SHAP values.

• Gender_Male has a sizable and directional effect: Gender_Male ranks third in importance and displays a similarly separated SHAP pattern. Higher feature values align with negative SHAP contributions, while lower values align with positive contributions.

• Balance is the leading continuous variable: Balance is the most important non-binary feature, with normalized importance around the mid-30s. Its summary pattern shows SHAP values extending from negative to positive, with higher feature values concentrated on the positive side of the distribution.

• Lower-ranked features have limited overall influence: CreditScore, HasCrCard, Tenure, and NumOfProducts contribute less than the top variables, and EstimatedSalary and Geography_Spain are near zero in the normalized importance chart. Their SHAP distributions are correspondingly concentrated closer to zero.

The SHAP results indicate that model behavior is driven primarily by a small set of features, led by IsActiveMember, Geography_Germany, and Gender_Male, with a pronounced drop in importance after these variables. The summary plot shows clear directional effects for several top-ranked binary features and a broader continuous impact pattern for Balance. Lower-ranked variables exhibit smaller SHAP magnitudes and narrower distributions around zero, indicating comparatively limited contribution to overall model output.

Parameters:

{
  "kernel_explainer_samples": 10
}
            

Figures

▶ Test Result: Weakspots Diagnosis (validmind.model_validation.sklearn.WeakspotsDiagnosis)

❌ Weakspots Diagnosis

The WeakspotsDiagnosis test evaluates model performance across binned regions of the feature space using accuracy, precision, recall, and F1 against thresholds of 0.75, 0.5, 0.5, and 0.7 respectively. The results are reported by feature and slice for both training and test datasets, with plots highlighting threshold breaches across CreditScore, Tenure, Balance, NumOfProducts, EstimatedSalary, Gender_Male, Geography_Germany, Geography_Spain, HasCrCard, and IsActiveMember. Across the reported slices, accuracy is below 0.75 in nearly all non-trivial bins, while precision, recall, and F1 vary more substantially by feature and segment. Several slices also have very small record counts, particularly at extreme Balance and CreditScore ranges.

Key insights:

• Accuracy is broadly below threshold: Test accuracy remains below the 0.75 threshold across nearly all reported slices. The only test slices above this level are Balance at (100359.236, 125449.045] with accuracy 0.6806 still below threshold, CreditScore at (349.5, 400.0] with accuracy 1.0 on 3 records, and Balance at (225808.281, 250898.09] with accuracy 1.0 on 1 record; most other slices range roughly from 0.20 to 0.71.

• CreditScore shows pronounced weak regions: CreditScore test results contain several of the weakest segments, including the (400.0, 450.0] bin with accuracy 0.20, precision 0.0, recall 0.0, and F1 0.0 on 10 records. Additional CreditScore bins remain below the F1 threshold of 0.7 in most cases, including (700.0, 750.0] with F1 0.4828 and recall 0.4565.

• Balance performance is uneven across slices: Balance test metrics vary materially by bin, from F1 0.4237 in the lowest balance slice and 0.4167 in (50179.618, 75269.427] to 0.7604 in (100359.236, 125449.045]. Lower-balance and several mid-to-upper-balance slices fall below threshold on both accuracy and F1, while isolated perfect values occur only in extremely small bins of 1 to 3 records.

• NumOfProducts has a persistent weak segment at two products: The NumOfProducts slice (1.9, 2.2] records test precision 0.3366 and F1 0.4387 on 234 records, making it one of the larger weak segments in the results. By comparison, the one-product slice has F1 0.6791, the three-product slice has F1 0.6429, and the four-product slice reaches F1 0.8 but on only 6 records.

• IsActiveMember separates into high- and low-recall regimes: For IsActiveMember = 0, the test slice shows recall 0.799 and F1 0.6824 on 364 records, while IsActiveMember = 1 shows recall 0.3945 and F1 0.4649 on 283 records. The same directional pattern is present in training data, where recall is 0.8220 for the 0 slice and 0.3648 for the 1 slice.

• Geography indicators show contrasting segment behavior: For Geography_Germany, the positive slice (0.9, 1.0] has stronger test performance with recall 0.9355 and F1 0.7708 than the complementary slice, which has recall 0.4674 and F1 0.4914. Geography_Spain shows the reverse type of split in recall and F1 magnitude, with the non-Spain slice at recall 0.6980 and F1 0.6369 versus the Spain slice at recall 0.4921 and F1 0.5439.

• EstimatedSalary and Tenure remain consistently below F1 threshold: All reported test slices for EstimatedSalary and Tenure have F1 below 0.7 except Tenure (4.0, 5.0] at 0.7342. EstimatedSalary test F1 ranges from 0.5070 to 0.7200, with several central salary bins clustered near 0.53 to 0.65; Tenure test F1 ranges from 0.5172 to 0.7342, with most bins between 0.55 and 0.67.

• Train-test gaps are generally modest in larger bins: In many higher-volume slices, train and test metrics are directionally similar, such as Geography_Germany, Geography_Spain, HasCrCard, Gender_Male, and several Tenure and EstimatedSalary bins. Larger divergences are concentrated in smaller slices, including extreme CreditScore and Balance bins where record counts are limited.

The weak spots results show that threshold breaches are widespread across the feature space, with accuracy below the configured cutoff in nearly all substantive segments and F1 frequently below 0.7. The most pronounced localized weaknesses appear in specific CreditScore, Balance, NumOfProducts, IsActiveMember, and geography-related slices, while EstimatedSalary and Tenure exhibit more uniformly moderate performance rather than isolated breakdowns. Train and test performance are broadly aligned in larger segments, whereas several extreme bins display unstable metrics alongside very small record counts.

Parameters:

{
  "thresholds": {
    "accuracy": 0.75,
    "precision": 0.5,
    "recall": 0.5,
    "f1": 0.7
  }
}
            

Tables

Slice	Number of Records	Feature	Accuracy	Precision	Recall	F1	Dataset
(-250.898, 25089.809]	200	Balance	0.6600	0.5319	0.3521	0.4237	test_dataset_final
(25089.809, 50179.618]	8	Balance	0.3750	1.0000	0.2857	0.4444	test_dataset_final
(50179.618, 75269.427]	24	Balance	0.4167	0.3846	0.4545	0.4167	test_dataset_final
(75269.427, 100359.236]	83	Balance	0.6145	0.5319	0.7143	0.6098	test_dataset_final
(100359.236, 125449.045]	144	Balance	0.6806	0.7087	0.8202	0.7604	test_dataset_final
(125449.045, 150538.854]	121	Balance	0.5868	0.5679	0.7541	0.6479	test_dataset_final
(150538.854, 175628.663]	56	Balance	0.5179	0.5000	0.7778	0.6087	test_dataset_final
(175628.663, 200718.472]	7	Balance	0.4286	0.4000	0.6667	0.5000	test_dataset_final
(200718.472, 225808.281]	3	Balance	0.6667	1.0000	0.6667	0.8000	test_dataset_final
(225808.281, 250898.09]	1	Balance	1.0000	1.0000	1.0000	1.0000	test_dataset_final
(-250.898, 25089.809]	857	Balance	0.6441	0.5613	0.3595	0.4383	train_dataset_final
(25089.809, 50179.618]	13	Balance	0.6154	0.7500	0.4286	0.5455	train_dataset_final
(50179.618, 75269.427]	88	Balance	0.5568	0.7333	0.4151	0.5301	train_dataset_final
(75269.427, 100359.236]	294	Balance	0.6122	0.6000	0.6575	0.6275	train_dataset_final
(100359.236, 125449.045]	575	Balance	0.7009	0.7320	0.8068	0.7676	train_dataset_final
(125449.045, 150538.854]	505	Balance	0.6337	0.6323	0.8107	0.7105	train_dataset_final
(150538.854, 175628.663]	183	Balance	0.5847	0.5794	0.7604	0.6577	train_dataset_final
(175628.663, 200718.472]	53	Balance	0.5283	0.5556	0.6897	0.6154	train_dataset_final
(200718.472, 225808.281]	16	Balance	0.5000	0.7273	0.6154	0.6667	train_dataset_final
(225808.281, 250898.09]	1	Balance	0.0000	0.0000	0.0000	0.0000	train_dataset_final
(349.5, 400.0]	3	CreditScore	1.0000	1.0000	1.0000	1.0000	test_dataset_final
(400.0, 450.0]	10	CreditScore	0.2000	0.0000	0.0000	0.0000	test_dataset_final
(450.0, 500.0]	25	CreditScore	0.5600	0.5238	0.9167	0.6667	test_dataset_final
(500.0, 550.0]	65	CreditScore	0.5846	0.5152	0.6071	0.5574	test_dataset_final
(550.0, 600.0]	101	CreditScore	0.6139	0.5333	0.7442	0.6214	test_dataset_final
(600.0, 650.0]	129	CreditScore	0.6357	0.5915	0.7000	0.6412	test_dataset_final
(650.0, 700.0]	123	CreditScore	0.7073	0.7333	0.6875	0.7097	test_dataset_final
(700.0, 750.0]	96	CreditScore	0.5312	0.5122	0.4565	0.4828	test_dataset_final
(750.0, 800.0]	51	CreditScore	0.6471	0.6452	0.7407	0.6897	test_dataset_final
(800.0, 850.0]	44	CreditScore	0.6364	0.7059	0.5217	0.6000	test_dataset_final
(349.5, 400.0]	11	CreditScore	0.6364	1.0000	0.6364	0.7778	train_dataset_final
(400.0, 450.0]	53	CreditScore	0.6415	0.6757	0.7812	0.7246	train_dataset_final
(450.0, 500.0]	120	CreditScore	0.6167	0.6081	0.7258	0.6618	train_dataset_final
(500.0, 550.0]	270	CreditScore	0.6370	0.6562	0.7095	0.6818	train_dataset_final
(550.0, 600.0]	353	CreditScore	0.6204	0.6453	0.6788	0.6616	train_dataset_final
(600.0, 650.0]	460	CreditScore	0.6196	0.6266	0.6398	0.6331	train_dataset_final
(650.0, 700.0]	498	CreditScore	0.6325	0.5992	0.6481	0.6227	train_dataset_final
(700.0, 750.0]	415	CreditScore	0.6747	0.6703	0.6193	0.6438	train_dataset_final
(750.0, 800.0]	239	CreditScore	0.6569	0.6696	0.6250	0.6466	train_dataset_final
(800.0, 850.0]	166	CreditScore	0.6747	0.6897	0.5263	0.5970	train_dataset_final
(-188.25, 19994.554]	70	EstimatedSalary	0.7000	0.6585	0.7941	0.7200	test_dataset_final
(19994.554, 39977.528]	70	EstimatedSalary	0.6571	0.7143	0.6410	0.6757	test_dataset_final
(39977.528, 59960.502]	59	EstimatedSalary	0.6949	0.5862	0.7391	0.6538	test_dataset_final
(59960.502, 79943.476]	75	EstimatedSalary	0.5333	0.5143	0.5000	0.5070	test_dataset_final
(79943.476, 99926.45]	68	EstimatedSalary	0.5588	0.4722	0.6071	0.5312	test_dataset_final
(99926.45, 119909.424]	63	EstimatedSalary	0.5714	0.5667	0.5484	0.5574	test_dataset_final
(119909.424, 139892.398]	65	EstimatedSalary	0.5846	0.5588	0.6129	0.5846	test_dataset_final
(139892.398, 159875.372]	53	EstimatedSalary	0.5660	0.5882	0.6897	0.6349	test_dataset_final
(159875.372, 179858.346]	71	EstimatedSalary	0.6620	0.5641	0.7586	0.6471	test_dataset_final
(179858.346, 199841.32]	53	EstimatedSalary	0.6604	0.6667	0.7143	0.6897	test_dataset_final
(-188.25, 19994.554]	253	EstimatedSalary	0.6245	0.6338	0.6767	0.6545	train_dataset_final
(19994.554, 39977.528]	233	EstimatedSalary	0.6438	0.6423	0.6695	0.6556	train_dataset_final
(39977.528, 59960.502]	241	EstimatedSalary	0.6680	0.6935	0.6719	0.6825	train_dataset_final
(59960.502, 79943.476]	269	EstimatedSalary	0.6431	0.6483	0.6763	0.6620	train_dataset_final
(79943.476, 99926.45]	256	EstimatedSalary	0.6484	0.6296	0.6800	0.6538	train_dataset_final
(99926.45, 119909.424]	285	EstimatedSalary	0.6596	0.6232	0.6565	0.6394	train_dataset_final
(119909.424, 139892.398]	266	EstimatedSalary	0.6429	0.6364	0.6412	0.6388	train_dataset_final
(139892.398, 159875.372]	246	EstimatedSalary	0.6423	0.6525	0.6210	0.6364	train_dataset_final
(159875.372, 179858.346]	272	EstimatedSalary	0.6250	0.6822	0.5906	0.6331	train_dataset_final
(179858.346, 199841.32]	264	EstimatedSalary	0.6061	0.5929	0.6385	0.6148	train_dataset_final
(-0.001, 0.1]	326	Gender_Male	0.6012	0.5905	0.7965	0.6782	test_dataset_final
(0.9, 1.0]	321	Gender_Male	0.6355	0.5856	0.4779	0.5263	test_dataset_final
(-0.001, 0.1]	1257	Gender_Male	0.6476	0.6628	0.7901	0.7209	train_dataset_final
(0.9, 1.0]	1328	Gender_Male	0.6333	0.6048	0.4795	0.5349	train_dataset_final
(-0.001, 0.1]	450	Geography_Germany	0.6044	0.5181	0.4674	0.4914	test_dataset_final
(0.9, 1.0]	197	Geography_Germany	0.6497	0.6554	0.9355	0.7708	test_dataset_final
(-0.001, 0.1]	1798	Geography_Germany	0.6263	0.5883	0.4537	0.5123	train_dataset_final
(0.9, 1.0]	787	Geography_Germany	0.6722	0.6873	0.9415	0.7946	train_dataset_final
(-0.001, 0.1]	500	Geography_Spain	0.6100	0.5856	0.6980	0.6369	test_dataset_final
(0.9, 1.0]	147	Geography_Spain	0.6463	0.6078	0.4921	0.5439	test_dataset_final
(-0.001, 0.1]	1973	Geography_Spain	0.6437	0.6512	0.6993	0.6744	train_dataset_final
(0.9, 1.0]	612	Geography_Spain	0.6291	0.5962	0.4644	0.5221	train_dataset_final
(-0.001, 0.1]	183	HasCrCard	0.5847	0.5727	0.6848	0.6238	test_dataset_final
(0.9, 1.0]	464	HasCrCard	0.6315	0.5966	0.6435	0.6192	test_dataset_final
(-0.001, 0.1]	767	HasCrCard	0.6089	0.6214	0.6493	0.6350	train_dataset_final
(0.9, 1.0]	1818	HasCrCard	0.6535	0.6523	0.6523	0.6523	train_dataset_final
(-0.001, 0.1]	364	IsActiveMember	0.5934	0.5955	0.7990	0.6824	test_dataset_final
(0.9, 1.0]	283	IsActiveMember	0.6502	0.5658	0.3945	0.4649	test_dataset_final
(-0.001, 0.1]	1379	IsActiveMember	0.6468	0.6640	0.8220	0.7346	train_dataset_final
(0.9, 1.0]	1206	IsActiveMember	0.6327	0.5723	0.3648	0.4456	train_dataset_final
(0.997, 1.3]	366	NumOfProducts	0.6230	0.6636	0.6952	0.6791	test_dataset_final
(1.9, 2.2]	234	NumOfProducts	0.6282	0.3366	0.6296	0.4387	test_dataset_final
(2.8, 3.1]	41	NumOfProducts	0.5122	1.0000	0.4737	0.6429	test_dataset_final
(3.7, 4.0]	6	NumOfProducts	0.6667	1.0000	0.6667	0.8000	test_dataset_final
(0.997, 1.3]	1526	NumOfProducts	0.6389	0.7059	0.6773	0.6913	train_dataset_final
(1.9, 2.2]	873	NumOfProducts	0.6552	0.3786	0.6037	0.4654	train_dataset_final
(2.8, 3.1]	148	NumOfProducts	0.6081	0.9773	0.6056	0.7478	train_dataset_final
(3.7, 4.0]	38	NumOfProducts	0.4737	1.0000	0.4737	0.6429	train_dataset_final
(-0.01, 1.0]	101	Tenure	0.6139	0.6393	0.6964	0.6667	test_dataset_final
(1.0, 2.0]	65	Tenure	0.6154	0.6486	0.6667	0.6575	test_dataset_final
(2.0, 3.0]	56	Tenure	0.5000	0.5000	0.5357	0.5172	test_dataset_final
(3.0, 4.0]	75	Tenure	0.6533	0.5135	0.7037	0.5938	test_dataset_final
(4.0, 5.0]	66	Tenure	0.6818	0.6905	0.7838	0.7342	test_dataset_final
(5.0, 6.0]	65	Tenure	0.5846	0.5806	0.5625	0.5714	test_dataset_final
(6.0, 7.0]	63	Tenure	0.6349	0.5000	0.6087	0.5490	test_dataset_final
(7.0, 8.0]	69	Tenure	0.6812	0.6129	0.6552	0.6333	test_dataset_final
(8.0, 9.0]	64	Tenure	0.5625	0.5556	0.6250	0.5882	test_dataset_final
(9.0, 10.0]	23	Tenure	0.6522	0.5000	0.6250	0.5556	test_dataset_final
(-0.01, 1.0]	363	Tenure	0.6143	0.6333	0.6786	0.6552	train_dataset_final
(1.0, 2.0]	247	Tenure	0.6275	0.6154	0.6557	0.6349	train_dataset_final
(2.0, 3.0]	281	Tenure	0.6512	0.6644	0.6644	0.6644	train_dataset_final
(3.0, 4.0]	271	Tenure	0.6494	0.6815	0.6389	0.6595	train_dataset_final
(4.0, 5.0]	243	Tenure	0.6502	0.6557	0.6504	0.6531	train_dataset_final
(5.0, 6.0]	253	Tenure	0.6166	0.5984	0.6230	0.6104	train_dataset_final
(6.0, 7.0]	256	Tenure	0.6289	0.5748	0.6404	0.6058	train_dataset_final
(7.0, 8.0]	269	Tenure	0.6543	0.6508	0.6260	0.6381	train_dataset_final
(8.0, 9.0]	270	Tenure	0.6704	0.6547	0.6894	0.6716	train_dataset_final
(9.0, 10.0]	132	Tenure	0.6515	0.7500	0.6154	0.6761	train_dataset_final

Figures

▶ Test Result: Overfit Diagnosis (validmind.model_validation.sklearn.OverfitDiagnosis)

Overfit Diagnosis

The Overfit Diagnosis test evaluates differences between training and test performance across feature-based segments to identify regions where the AUC gap exceeds the 0.04 threshold. The reported results list feature slices with training and test record counts, segment-level training AUC, test AUC, and the resulting gap. Overfit regions are observed across multiple continuous features, including CreditScore, Tenure, Balance, and EstimatedSalary, as well as for the Gender_Male indicator. The largest observed gaps are concentrated in specific slices rather than appearing uniformly across all segments.

Key insights:

• Largest gap in low CreditScore slice: The CreditScore slice (400.0, 450.0] shows the largest reported gap at 0.6622, with training AUC 0.6622 and test AUC 0.0, based on 53 training records and 10 test records.
• Tenure shows concentrated overfit regions: Several Tenure slices exceed the 0.04 threshold, with the largest gap in (2.0, 3.0] at 0.2206 (0.7040 training AUC vs 0.4834 test AUC). Additional elevated gaps appear in (8.0, 9.0] at 0.0862, (9.0, 10.0] at 0.1025, (5.0, 6.0] at 0.0594, and (1.0, 2.0] at 0.0513.
• Balance contains multiple high-gap bands: Balance has several slices above threshold, including (175628.663, 200718.472] with a gap of 0.2083, (50179.618, 75269.427] with 0.1653, (150538.854, 175628.663] with 0.1110, (200718.472, 225808.281] with 0.1026, and (125449.045, 150538.854] with 0.0775.
• EstimatedSalary gaps are repeated across adjacent ranges: EstimatedSalary exceeds the threshold in five reported slices, led by (59960.502, 79943.476] at 0.1387, (99926.45, 119909.424] at 0.1202, and (139892.398, 159875.372] at 0.1139, with smaller but still above-threshold gaps at (79943.476, 99926.45] and (119909.424, 139892.398].
• Binary feature signal is limited but present: For Gender_Male, the slice (-0.001, 0.1] shows a gap of 0.0552, with training AUC 0.6669 and test AUC 0.6117, based on 1,257 training records and 326 test records.

The test output identifies segment-level train-test AUC differences above the 0.04 threshold across several features, with the largest reported gaps occurring in isolated CreditScore, Tenure, and Balance slices. Tenure, Balance, and EstimatedSalary each show multiple above-threshold regions, indicating that the observed train-test divergence is distributed across several contiguous or repeated value ranges rather than limited to a single feature segment. The reported binary-feature result is more limited in scope, with one above-threshold Gender_Male slice and a smaller gap magnitude than the largest continuous-feature segments.

Tables

Overfit Diagnosis

Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
CreditScore	(400.0, 450.0]	53	10	0.6622	0.0000	0.6622
CreditScore	(500.0, 550.0]	270	65	0.6927	0.6342	0.0585
CreditScore	(700.0, 750.0]	415	96	0.7236	0.5874	0.1362
CreditScore	(750.0, 800.0]	239	51	0.6821	0.6343	0.0479
Tenure	(1.0, 2.0]	247	65	0.6662	0.6149	0.0513
Tenure	(2.0, 3.0]	281	56	0.7040	0.4834	0.2206
Tenure	(5.0, 6.0]	253	65	0.6854	0.6259	0.0594
Tenure	(8.0, 9.0]	270	64	0.7112	0.6250	0.0862
Tenure	(9.0, 10.0]	132	23	0.7025	0.6000	0.1025
Balance	(50179.618, 75269.427]	88	24	0.5919	0.4266	0.1653
Balance	(125449.045, 150538.854]	505	121	0.6860	0.6085	0.0775
Balance	(150538.854, 175628.663]	183	56	0.6537	0.5428	0.1110
Balance	(175628.663, 200718.472]	53	7	0.5417	0.3333	0.2083
Balance	(200718.472, 225808.281]	16	3	0.1026	0.0000	0.1026
EstimatedSalary	(59960.502, 79943.476]	269	75	0.6828	0.5442	0.1387
EstimatedSalary	(79943.476, 99926.45]	256	68	0.7132	0.6438	0.0694
EstimatedSalary	(99926.45, 119909.424]	285	63	0.7059	0.5857	0.1202
EstimatedSalary	(119909.424, 139892.398]	266	65	0.6839	0.6224	0.0615
EstimatedSalary	(139892.398, 159875.372]	246	53	0.7274	0.6135	0.1139
Gender_Male	(-0.001, 0.1]	1257	326	0.6669	0.6117	0.0552

Figures

▶ Test Result: Robustness Diagnosis (validmind.model_validation.sklearn.RobustnessDiagnosis)

✅ Robustness Diagnosis

The Robustness Diagnosis test evaluates model performance under increasing Gaussian noise applied to numeric input features. Results are reported as AUC and performance decay across perturbation sizes from 0.0 to 0.5 standard deviations for both train_dataset_final and test_dataset_final. Baseline AUC is 0.6898 on the training dataset and 0.6627 on the test dataset, and the table and plot show how these values change as perturbation size increases. All evaluated perturbation levels are marked as passed for both datasets.

Key insights:

• Performance is stable at low noise: From perturbation sizes 0.1 through 0.3, training AUC remains between 0.6872 and 0.6883 versus a 0.6898 baseline, with performance decay between 0.0015 and 0.0026. Over the same range, test AUC remains between 0.6582 and 0.6628 versus a 0.6627 baseline, with decay between -0.0 and 0.0046.

• Decay increases at higher perturbation levels: At perturbation size 0.4, AUC declines to 0.6734 on training and 0.6535 on test, corresponding to performance decay of 0.0164 and 0.0093. At 0.5, AUC further declines to 0.6699 on training and 0.6426 on test, with decay of 0.0199 and 0.0202.

• Test performance shows minor non-monotonic variation: The test dataset does not decline uniformly at lower perturbation levels, with AUC moving from 0.6582 at 0.1 to 0.6594 at 0.2 and 0.6628 at 0.3. This corresponds to performance decay improving from 0.0046 at 0.1 to 0.0034 at 0.2 and approximately 0.0 at 0.3 before larger declines appear at 0.4 and 0.5.

• All scenarios passed the configured criterion: Every evaluated row in the results table, including baseline and all perturbation sizes through 0.5, is marked Passed = true for both datasets.

The results show limited AUC deterioration through perturbation sizes up to 0.3, followed by more visible declines at 0.4 and 0.5 on both training and test datasets. The largest observed performance decay is 0.0199 on training and 0.0202 on test at the highest perturbation level. Across the full perturbation range, all scenarios remained within the configured pass condition documented by the test output.

Parameters:

{
  "scaling_factor_std_dev_list": [
    0.0,
    0.1,
    0.2,
    0.3,
    0.4,
    0.5
  ]
}
            

Tables

Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
Baseline (0.0)	train_dataset_final	2585	0.6898	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.6627	0.0000	True
Baseline (0.0)	train_dataset_final	2585	0.6898	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.6627	0.0000	True
0.1	train_dataset_final	2585	0.6882	0.0016	True
0.1	test_dataset_final	647	0.6582	0.0046	True
0.2	train_dataset_final	2585	0.6872	0.0026	True
0.2	test_dataset_final	647	0.6594	0.0034	True
0.3	train_dataset_final	2585	0.6883	0.0015	True
0.3	test_dataset_final	647	0.6628	-0.0000	True
0.4	train_dataset_final	2585	0.6734	0.0164	True
0.4	test_dataset_final	647	0.6535	0.0093	True
0.5	train_dataset_final	2585	0.6699	0.0199	True
0.5	test_dataset_final	647	0.6426	0.0202	True

Figures

In summary

In this final notebook, you learned how to:

• Refresh the connection from the ValidMind Library to the ValidMind Platform after you've inserted test-driven blocks to your documentation
• Include custom test results in your documentation
• View and configure the configuration for your documentation template

With our ValidMind for development series of notebooks, you learned how to document a record (model) end-to-end with the ValidMind Library by running through some common scenarios in a typical development setting:

• Running out-of-the-box tests
• Documenting your record (model) by adding evidence to documentation
• Extending the capabilities of the ValidMind Library by implementing custom tests
• Ensuring that the documentation is complete by running all tests in the documentation template

Next steps

Work with your documentation

Now that you've logged all your test results and generated a draft for your documentation, head to the ValidMind Platform to wrap up your documentation. Continue to work on your documentation by:

• Run and log more tests: Use the skills you learned in this series of notebooks to run and log more individual tests, including custom tests, then insert them into your documentation as supplementary evidence. (Learn more: validmind.tests)

• Inserting additional test results: Add Test-Driven Blocks under any relevant section of your documentation. (Learn more: Work with test results)

• Making qualitative edits to your test descriptions: Click on the description of any inserted test results to review and edit the ValidMind-generated test descriptions for quality and accuracy. (Learn more: Working with documentation)

• View guidelines: In any section of your documentation, click ValidMind Insights in the top right corner to reveal the Documentation Guidelines for each section to help guide the contents of your documentation. (Learn more: View development guidelines)

• Collaborate with other stakeholders: Use the ValidMind Platform's real-time collaborative features to work seamlessly together with the rest of your organization, including validators. Review suggested changes in your content blocks, work with versioned history, and use comments to discuss specific portions of your documentation. (Learn more: Collaborate with others)

When your documentation is complete and ready for review, submit it for approval from the same ValidMind Platform where you made your edits and collaborated with the rest of your organization, ensuring transparency and a thorough development history. (Learn more: Submit documents)

Learn more

Now that you're familiar with the basics, you can explore the following notebooks to get a deeper understanding on how the ValidMind Library allows you generate documentation for any use case:

Use cases

• Document an agentic AI system
• Document an Excel-based application scorecard model
• LLM model documentation demo

Discover more learning resources

Learn more about the ValidMind Library tools we used in this notebook:

• Explore tests
• Run dataset-based tests
• Implement custom tests
• Integrate external test providers
• Configure dataset features

We offer many interactive notebooks to help you automate testing, documenting, validating, and more:

• Run tests & test suites
• Use ValidMind Library features
• Code samples by use case

Or, visit our documentation to learn more about ValidMind.

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial