ValidMind for development 4 — Finalize testing and documentation

Learn how to use ValidMind for your end-to-end documentation process with our introductory notebook series. In this last notebook, finalize the testing and documentation of your model and have a fully documented sample model ready for review.

We'll first use run_documentation_tests() previously covered in 2 — Start the development process to ensure that your custom test results generated in 3 — Integrate custom tests are included in your documentation. Then, we'll view and update the configuration for the entire documentation template to suit your needs.

Learn by doing

Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Developer Fundamentals

Prerequisites

In order to finalize the testing and documentation for your sample model, you'll need to first have:

• Registered a model within the ValidMind Platform with a predefined documentation template
• Installed the ValidMind Library in your local environment, allowing you to access all its features
• Learned how to import and initialize datasets for use with ValidMind
• Learned how to run and log default and custom tests with ValidMind, including from external test providers
• Inserted test-driven blocks for the results of the following tests into your model's documentation:
  • HighPearsonCorrelation:balanced_raw_dataset
  • my_test_provider.ConfusionMatrix
  • my_custom_tests.ConfusionMatrix:test_dataset_normalized

Need help with the above steps?

Refer to the first three notebooks in this series:

• 1 — Set up the ValidMind Library
• 2 — Start the development process
• 3 — Integrate custom tests

Setting up

This section should be very familiar to you now — as we performed the same actions in the previous two notebooks in this series.

Initialize the ValidMind Library

As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Development from the Document drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="documentation",
)

Note: you may need to restart the kernel to use updated packages.

2026-07-14 05:30:05,683 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model development (ID: cmalgf3qi02ce199qm3rdkl46)
📁 Document Type: model_documentation

Import sample dataset

Next, we'll import the same public Bank Customer Churn Prediction dataset from Kaggle we used in the last notebooks so that we have something to work with:

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

We'll apply a simple rebalancing technique to the dataset before continuing:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

Remove highly correlated features

Let's also quickly remove highly correlated features from the dataset using the output from a ValidMind test.

As you learned previously, before we can run tests you'll need to initialize a ValidMind dataset object:

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:

# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships between features to identify highly correlated variable pairs that may indicate redundancy or multicollinearity. The results table reports the top 10 feature pairs ranked by Pearson correlation coefficient, alongside Pass/Fail status based on the configured absolute threshold of 0.3. Observed coefficients range from -0.1801 to 0.3356. Only one pair exceeds the threshold: (Age, Exited) with a coefficient of 0.3356, while the remaining reported pairs are below the threshold and marked as Pass.

Key insights:

• Single threshold breach observed: Only (Age, Exited) fails the test, with a Pearson correlation coefficient of 0.3356, which is above the configured threshold of 0.3.
• Most reported relationships are weak: The remaining nine reported feature pairs have coefficients between -0.1801 and 0.1464 in absolute terms, all below the threshold and classified as Pass.
• Top negative correlation remains limited: The largest negative relationship in the reported results is (IsActiveMember, Exited) at -0.1801, which remains well below the threshold.
• Correlation concentration is low: No other reported pair approaches the magnitude of the (Age, Exited) relationship; the next-largest absolute coefficient is 0.1801, indicating a clear separation between the single failing pair and the rest of the reported correlations.

The reported correlation structure is characterized by one feature pair exceeding the configured Pearson threshold and a broader set of low-magnitude linear relationships among the other top-ranked pairs. The most prominent observed relationship is the positive correlation between Age and Exited, while all other listed pairs remain below the threshold by a material margin. Overall, the test output shows limited concentration of high pairwise linear correlation within the reported top 10 relationships.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3356	Fail
(IsActiveMember, Exited)	-0.1801	Pass
(Balance, NumOfProducts)	-0.1715	Pass
(Balance, Exited)	0.1464	Pass
(NumOfProducts, Exited)	-0.0599	Pass
(NumOfProducts, IsActiveMember)	0.0477	Pass
(Age, NumOfProducts)	-0.0435	Pass
(Age, Balance)	0.0388	Pass
(Balance, HasCrCard)	-0.0380	Pass
(HasCrCard, IsActiveMember)	-0.0315	Pass

# From result object, extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3356	Fail
1	(IsActiveMember, Exited)	-0.1801	Pass
2	(Balance, NumOfProducts)	-0.1715	Pass
3	(Balance, Exited)	0.1464	Pass
4	(NumOfProducts, Exited)	-0.0599	Pass
5	(NumOfProducts, IsActiveMember)	0.0477	Pass
6	(Age, NumOfProducts)	-0.0435	Pass
7	(Age, Balance)	0.0388	Pass
8	(Balance, HasCrCard)	-0.0380	Pass
9	(HasCrCard, IsActiveMember)	-0.0315	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

# Extract feature names from the list of strings
high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

We can then re-initialize the dataset with a different input_id and the highly correlated features removed and re-run the test for confirmation:

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

# Re-run the test with the reduced feature set
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant variables or multicollinearity. The result table reports the top feature pairs ranked by Pearson correlation coefficient, along with Pass/Fail status based on the configured absolute threshold of 0.3. In this run, the reported coefficients range from -0.1801 to 0.1464, and all listed feature pairs are marked as Pass. The strongest relationships shown are between IsActiveMember and Exited, Balance and NumOfProducts, and Balance and Exited.

Key insights:

• No pair exceeds threshold: All reported absolute correlation coefficients are below the configured 0.3 threshold, and every listed pair receives a Pass result.
• Observed relationships are weak: The largest absolute coefficient is 0.1801 for (IsActiveMember, Exited), followed by 0.1715 for (Balance, NumOfProducts) and 0.1464 for (Balance, Exited), indicating limited linear association among the strongest reported pairs.
• Top correlations are concentrated in a few variables: Balance appears in four of the ten reported pairs, while Exited appears in five, indicating that these variables are involved in several of the strongest linear relationships shown in the output.
• Both positive and negative associations are present: The listed coefficients include negative values such as (IsActiveMember, Exited) at -0.1801 and (Balance, NumOfProducts) at -0.1715, as well as positive values such as (Balance, Exited) at 0.1464 and (Balance, EstimatedSalary) at 0.0275.

The reported correlation structure does not show any pairwise linear relationship exceeding the configured threshold. The strongest observed associations remain weak in magnitude, and the output reflects a mix of positive and negative relationships across the top-ranked pairs. Based on the reported results, the tested feature pairs do not exhibit high pairwise Pearson correlation under this test configuration.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1801	Pass
(Balance, NumOfProducts)	-0.1715	Pass
(Balance, Exited)	0.1464	Pass
(NumOfProducts, Exited)	-0.0599	Pass
(NumOfProducts, IsActiveMember)	0.0477	Pass
(Balance, HasCrCard)	-0.0380	Pass
(HasCrCard, IsActiveMember)	-0.0315	Pass
(CreditScore, Exited)	-0.0279	Pass
(Balance, EstimatedSalary)	0.0275	Pass
(Tenure, Exited)	-0.0249	Pass

Train the model

We'll then use ValidMind tests to train a simple logistic regression model on our prepared dataset:

# First encode the categorical features in our dataset with the highly correlated features removed
balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
5813	667	4	97585.32	2	0	0	57213.46	0	False	True	True
5032	618	5	133476.09	1	0	1	154843.40	0	False	True	True
1176	685	7	101868.51	1	0	1	113483.96	0	True	False	True
3596	497	8	111537.23	4	1	1	9497.99	1	True	False	False
6186	777	4	137851.31	1	1	0	5008.23	1	False	False	False

# Split the processed dataset into train and test
from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

from sklearn.linear_model import LogisticRegression

# Logistic Regression grid params
log_reg_params = {
    "penalty": ["l1", "l2"],
    "C": [0.001, 0.01, 0.1, 1, 10, 100, 1000],
    "solver": ["liblinear"],
}

# Grid search for Logistic Regression
from sklearn.model_selection import GridSearchCV

grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)
grid_log_reg.fit(X_train, y_train)

# Logistic Regression best estimator
log_reg = grid_log_reg.best_estimator_

/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(

Initialize the ValidMind objects

Let's initialize the ValidMind Dataset and Model objects in preparation for assigning model predictions to each dataset:

# Initialize the datasets into their own ValidMind dataset objects
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

# Initialize the ValidMind model object
vm_model = vm.init_model(log_reg, input_id="log_reg_model_v1")

Assign predictions

Once the model is registered, we'll assign predictions to the training and test datasets:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

2026-07-14 05:30:15,820 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-07-14 05:30:15,823 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-07-14 05:30:15,823 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-07-14 05:30:15,826 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-07-14 05:30:15,829 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-07-14 05:30:15,830 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-07-14 05:30:15,831 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-07-14 05:30:15,832 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Add custom tests

We'll also add the same custom tests we implemented in the previous notebook so that this session has access to the same custom inline test and local test provider.

Implement custom inline test

Let's set up a custom inline test that calculates the confusion matrix for a binary classification model:

# First create a confusion matrix plot
import matplotlib.pyplot as plt
from sklearn import metrics

# Get the predicted classes
y_pred = log_reg.predict(vm_test_ds.x)

confusion_matrix = metrics.confusion_matrix(y_test, y_pred)

cm_display = metrics.ConfusionMatrixDisplay(
    confusion_matrix=confusion_matrix, display_labels=[False, True]
)
cm_display.plot()

# Create the reusable ConfusionMatrix inline test with normalized matrix
@vm.test("my_custom_tests.ConfusionMatrix")
def confusion_matrix(dataset, model, normalize=False):
    """The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.

    The confusion matrix is a 2x2 table that contains 4 values:

    - True Positive (TP): the number of correct positive predictions
    - True Negative (TN): the number of correct negative predictions
    - False Positive (FP): the number of incorrect positive predictions
    - False Negative (FN): the number of incorrect negative predictions

    The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.
    """
    y_true = dataset.y
    y_pred = dataset.y_pred(model=model)

    if normalize:
        confusion_matrix = metrics.confusion_matrix(y_true, y_pred, normalize="all")
    else:
        confusion_matrix = metrics.confusion_matrix(y_true, y_pred)

    cm_display = metrics.ConfusionMatrixDisplay(
        confusion_matrix=confusion_matrix, display_labels=[False, True]
    )
    cm_display.plot()

    plt.close()  # close the plot to avoid displaying it

    return cm_display.figure_  # return the figure object itself

# Test dataset with normalize=True
result = vm.tests.run_test(
    "my_custom_tests.ConfusionMatrix:test_dataset_normalized",
    inputs={"model": vm_model, "dataset": vm_test_ds},
    params={"normalize": True},
)

Confusion Matrix Test Dataset Normalized

The ConfusionMatrix test evaluates classification outcomes by comparing true labels with predicted labels, and this result presents the normalized confusion matrix for the test dataset. The matrix is shown with true labels on the rows and predicted labels on the columns, so the four cells correspond to true negatives, false positives, false negatives, and true positives. The displayed normalized values are 0.32 for true negatives, 0.19 for false positives, 0.20 for false negatives, and 0.30 for true positives.

Key insights:

• Correct classifications exceed misclassifications: The diagonal cells sum to 0.62, combining 0.32 true negatives and 0.30 true positives, while the off-diagonal cells sum to 0.39, combining 0.19 false positives and 0.20 false negatives.
• Balanced performance across classes: The two correct classification cells are close in magnitude, with 0.32 for negatives classified correctly and 0.30 for positives classified correctly, indicating similar capture of both classes in the normalized result.
• Error types are nearly symmetric: False positives and false negatives are nearly equal at 0.19 and 0.20 respectively, indicating that misclassification is distributed similarly across the two error types.
• Slightly stronger identification of negatives: The true negative cell at 0.32 is the largest entry in the matrix, exceeding the true positive cell at 0.30 by 0.02.

The normalized confusion matrix shows that most observations fall on the diagonal, with correct classifications accounting for 0.62 of the total displayed mass. Correct identification of negative and positive cases is similar in magnitude, while false positive and false negative rates are also closely aligned. Overall, the result indicates a relatively balanced classification pattern with a small edge toward correct negative classification.

Parameters:

{
  "normalize": true
}
            

Figures

Add a local test provider

Finally, let's save our custom inline test to our local test provider:

# Create custom tests folder
tests_folder = "my_tests"

import os

# create tests folder
os.makedirs(tests_folder, exist_ok=True)

# remove existing tests
for f in os.listdir(tests_folder):
    # remove files and pycache
    if f.endswith(".py") or f == "__pycache__":
        os.system(f"rm -rf {tests_folder}/{f}")

# Save custom inline test to custom tests folder
confusion_matrix.save(
    tests_folder,
    imports=["import matplotlib.pyplot as plt", "from sklearn import metrics"],
)

2026-07-14 05:30:20,986 - INFO(validmind.tests.decorator): Saved to /home/runner/work/documentation/documentation/site/notebooks/EXECUTED/development/my_tests/ConfusionMatrix.py!Be sure to add any necessary imports to the top of the file.
2026-07-14 05:30:20,986 - INFO(validmind.tests.decorator): This metric can be run with the ID: <test_provider_namespace>.ConfusionMatrix

# Register local test provider
from validmind.tests import LocalTestProvider

# initialize the test provider with the tests folder we created earlier
my_test_provider = LocalTestProvider(tests_folder)

vm.tests.register_test_provider(
    namespace="my_test_provider",
    test_provider=my_test_provider,
)

Reconnect to ValidMind

After you insert test-driven blocks into your model documentation, changes should persist and become available every time you call vm.preview_template().

However, you'll need to reload the connection to the ValidMind Platform if you have added test-driven blocks when the connection was already established using reload():

vm.reload()

Now, when you run preview_template() again, the three test-driven blocks you added to your documentation in the last two notebooks in should show up in the template in sections 2.3 Correlations and Interactions and 3.2 Model Evaluation:

vm.preview_template()

▶ 1. Conceptual Soundness ('conceptual_soundness')

▶ 1.1. Model Overview ('model_overview')

Text Block: 'model_overview'

▶ 1.2. Intended Use and Business Use Case ('intended_use_business_use_case')

▶ 1.2.1. Intended Use ('intended_use')

Empty Section

▶ 1.2.2. Regulatory Requirements ('regulatory_requirements')

Empty Section

▶ 1.2.3. Model Limitations ('model_limitations')

Empty Section

▶ 1.3. Model Selection ('model_selection')

Empty Section

▶ 2. Data Preparation ('data_preparation')

▶ 2.1. Data description ('data_description')

Text Block: 'dataset_summary_text'

▶ Test: Dataset Description ('validmind.data_validation.DatasetDescription')

Dataset Description

Provides comprehensive analysis and statistical summaries of each column in a machine learning model's dataset.

Purpose

The test depicted in the script is meant to run a comprehensive analysis on a Machine Learning model's datasets. The test or metric is implemented to obtain a complete summary of the columns in the dataset, including vital statistics of each column such as count, distinct values, missing values, histograms for numerical, categorical, boolean, and text columns. This summary gives a comprehensive overview of the dataset to better understand the characteristics of the data that the model is trained on or evaluates.

Test Mechanism

The DatasetDescription class accomplishes the purpose as follows: firstly, the test method "run" infers the data type of each column in the dataset and stores the details (id, column type). For each column, the "describe_column" method is invoked to collect statistical information about the column, including count, missing value count and its proportion to the total, unique value count, and its proportion to the total. Depending on the data type of a column, histograms are generated that reflect the distribution of data within the column. Numerical columns use the "get_numerical_histograms" method to calculate histogram distribution, whereas for categorical, boolean and text columns, a histogram is computed with frequencies of each unique value in the datasets. For unsupported types, an error is raised. Lastly, a summary table is built to aggregate all the statistical insights and histograms of the columns in a dataset.

Signs of High Risk

• High ratio of missing values to total values in one or more columns which may impact the quality of the predictions.
• Unsupported data types in dataset columns.
• Large number of unique values in the dataset's columns which might make it harder for the model to establish patterns.
• Extreme skewness or irregular distribution of data as reflected in the histograms.

Strengths

• Provides a detailed analysis of the dataset with versatile summaries like count, unique values, histograms, etc.
• Flexibility in handling different types of data: numerical, categorical, boolean, and text.
• Useful in detecting problems in the dataset like missing values, unsupported data types, irregular data distribution, etc.
• The summary gives a comprehensive understanding of dataset features allowing developers to make informed decisions.

Limitations

• The computation can be expensive from a resource standpoint, particularly for large datasets with numerous columns.
• The histograms use an arbitrary number of bins which may not be the optimal number of bins for specific data distribution.
• Unsupported data types for columns will raise an error which may limit evaluating the dataset.
• Columns with all null or missing values are not included in histogram computation.
• This test only validates the quality of the dataset but doesn't address the model's performance directly.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DatasetDescription", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'data_quality_tests_text'

▶ Test: Class Imbalance ('validmind.data_validation.ClassImbalance')

Class Imbalance

Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model.

Purpose

The Class Imbalance test is designed to evaluate the distribution of target classes in a dataset that's utilized by a machine learning model. Specifically, it aims to ensure that the classes aren't overly skewed, which could lead to bias in the model's predictions. It's crucial to have a balanced training dataset to avoid creating a model that's biased with high accuracy for the majority class and low accuracy for the minority class.

Test Mechanism

This Class Imbalance test operates by calculating the frequency (expressed as a percentage) of each class in the target column of the dataset. It then checks whether each class appears in at least a set minimum percentage of the total records. This minimum percentage is a modifiable parameter, but the default value is set to 10%.

Signs of High Risk

• Any class that represents less than the pre-set minimum percentage threshold is marked as high risk, implying a potential class imbalance.
• The function provides a pass/fail outcome for each class based on this criterion.
• Fundamentally, if any class fails this test, it's highly likely that the dataset possesses imbalanced class distribution.

Strengths

• The test can spot under-represented classes that could affect the efficiency of a machine learning model.
• The calculation is straightforward and swift.
• The test is highly informative because it not only spots imbalance, but it also quantifies the degree of imbalance.
• The adjustable threshold enables flexibility and adaptation to differing use-cases or domain-specific needs.
• The test creates a visually insightful plot showing the classes and their corresponding proportions, enhancing interpretability and comprehension of the data.

Limitations

• The test might struggle to perform well or provide vital insights for datasets with a high number of classes. In such cases, the imbalance could be inevitable due to the inherent class distribution.
• Sensitivity to the threshold value might result in faulty detection of imbalance if the threshold is set excessively high.
• Regardless of the percentage threshold, it doesn't account for varying costs or impacts of misclassifying different classes, which might fluctuate based on specific applications or domains.
• While it can identify imbalances in class distribution, it doesn't provide direct methods to address or correct these imbalances.
• The test is only applicable for classification operations and unsuitable for regression or clustering tasks.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_percent_threshold	10

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_percent_threshold": "my_vm_min_percent_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.ClassImbalance", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Duplicates ('validmind.data_validation.Duplicates')

Duplicates

Tests dataset for duplicate entries, ensuring model reliability via data quality verification.

Purpose

The 'Duplicates' test is designed to check for duplicate rows within the dataset provided to the model. It serves as a measure of data quality, ensuring that the model isn't merely memorizing duplicate entries or being swayed by redundant information. This is an important step in the pre-processing of data for both classification and regression tasks.

Test Mechanism

This test operates by checking each row for duplicates in the dataset. If a text column is specified in the dataset, the test is conducted on this column; if not, the test is run on all feature columns. The number and percentage of duplicates are calculated and returned in a DataFrame. Additionally, a test is passed if the total count of duplicates falls below a specified minimum threshold.

Signs of High Risk

• A high number of duplicate rows in the dataset, which can lead to overfitting where the model performs well on the training data but poorly on unseen data.
• A high percentage of duplicate rows in the dataset, indicating potential problems with data collection or processing.

Strengths

• Assists in improving the reliability of the model's training process by ensuring the training data is not contaminated with duplicate entries, which can distort statistical analyses.
• Provides both absolute numbers and percentage values of duplicate rows, giving a thorough overview of data quality.
• Highly customizable as it allows for setting a user-defined minimum threshold to determine if the test has been passed.

Limitations

• Does not distinguish between benign duplicates (i.e., coincidental identical entries in different rows) and problematic duplicates originating from data collection or processing errors.
• The test becomes more computationally intensive as the size of the dataset increases, which might not be suitable for very large datasets.
• Can only check for exact duplicates and may miss semantically similar information packaged differently.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_threshold	1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.Duplicates", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: High Cardinality ('validmind.data_validation.HighCardinality')

High Cardinality

Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting.

Purpose

The “High Cardinality” test is used to evaluate the number of unique values present in the categorical columns of a dataset. In this context, high cardinality implies the presence of a large number of unique, non-repetitive values in the dataset.

Test Mechanism

The test first infers the dataset's type and then calculates an initial numeric threshold based on the test parameters. It only considers columns classified as "Categorical". For each of these columns, the number of distinct values (n_distinct) and the percentage of distinct values (p_distinct) are calculated. The test will pass if n_distinct is less than the calculated numeric threshold. Lastly, the results, which include details such as column name, number of distinct values, and pass/fail status, are compiled into a table.

Signs of High Risk

• A large number of distinct values (high cardinality) in one or more categorical columns implies a high risk.
• A column failing the test (n_distinct >= num_threshold) is another indicator of high risk.

Strengths

• The High Cardinality test is effective in early detection of potential overfitting and unwanted noise.
• It aids in identifying potential outliers and inconsistencies, thereby improving data quality.
• The test can be applied to both classification and regression task types, demonstrating its versatility.

Limitations

• The test is restricted to only "Categorical" data types and is thus not suitable for numerical or continuous features, limiting its scope.
• The test does not consider the relevance or importance of unique values in categorical features, potentially causing it to overlook critical data points.
• The threshold (both number and percent) used for the test is static and may not be optimal for diverse datasets and varied applications. Further mechanisms to adjust and refine this threshold could enhance its effectiveness.

Required Inputs: dataset

Parameters:

Parameter	Default Value
num_threshold	100
percent_threshold	0.1
threshold_type	'percent'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "num_threshold": "my_vm_num_threshold",
    "percent_threshold": "my_vm_percent_threshold",
    "threshold_type": "my_vm_threshold_type"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.HighCardinality", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Missing Values ('validmind.data_validation.MissingValues')

Missing Values

Evaluates dataset quality by ensuring missing value percentage across all features does not exceed a set threshold.

Purpose

The Missing Values test is designed to evaluate the quality of a dataset by measuring the number of missing values across all features. The objective is to ensure that the ratio of missing data to total data is less than a predefined threshold (as a percentage), defaulting to 1.0, in order to maintain the data quality necessary for reliable predictive strength in a machine learning model.

Test Mechanism

The mechanism for this test involves iterating through each column of the dataset, counting missing values (represented as NaNs), and calculating the percentage they represent against the total number of rows. The test then checks if the missing value percentage is less than or equal to the predefined min_percentage_threshold. The results are shown in a table summarizing each column, the number of missing values, the percentage of missing values in each column, and a Pass/Fail status based on the threshold comparison.

Signs of High Risk

• When the missing value percentage in any column exceeds the min_percentage_threshold value.
• Presence of missing values across many columns, leading to multiple instances of failing the threshold.

Strengths

• Quick and granular identification of missing data across each feature in the dataset.
• Provides an effective and straightforward means of maintaining data quality, essential for constructing efficient machine learning models.

Limitations

• Does not suggest the root causes of the missing values or recommend ways to impute or handle them.
• May overlook features with significant missing data but still less than the min_percentage_threshold, potentially impacting the model.
• Does not account for data encoded as values like "-999" or "None," which might not technically classify as missing but could bear similar implications.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_percentage_threshold	1.0

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_percentage_threshold": "my_vm_min_percentage_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.MissingValues", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Skewness ('validmind.data_validation.Skewness')

Skewness

Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data quality and optimize model performance.

Purpose

The purpose of the Skewness test is to measure the asymmetry in the distribution of data within a predictive machine learning model. Specifically, it evaluates the divergence of said distribution from a normal distribution. Understanding the level of skewness helps identify data quality issues, which are crucial for optimizing the performance of traditional machine learning models in both classification and regression settings.

Test Mechanism

This test calculates the skewness of numerical columns in the dataset, focusing specifically on numerical data types. The calculated skewness value is then compared against a predetermined maximum threshold, which is set by default to 1. If the skewness value is less than this maximum threshold, the test passes; otherwise, it fails. The test results, along with the skewness values and column names, are then recorded for further analysis.

Signs of High Risk

• Substantial skewness levels that significantly exceed the maximum threshold.
• Persistent skewness in the data, indicating potential issues with the foundational assumptions of the machine learning model.
• Subpar model performance, erroneous predictions, or biased inferences due to skewed data distributions.

Strengths

• Fast and efficient identification of unequal data distributions within a machine learning model.
• Adjustable maximum threshold parameter, allowing for customization based on user needs.
• Provides a clear quantitative measure to mitigate model risks related to data skewness.

Limitations

• Only evaluates numeric columns, potentially missing skewness or bias in non-numeric data.
• Assumes that data should follow a normal distribution, which may not always be applicable to real-world data.
• Subjective threshold for risk grading, requiring expert input and recurrent iterations for refinement.

Required Inputs: dataset

Parameters:

Parameter	Default Value
max_threshold	1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "max_threshold": "my_vm_max_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.Skewness", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Unique Rows ('validmind.data_validation.UniqueRows')

Unique Rows

Verifies the diversity of the dataset by ensuring that the count of unique rows exceeds a prescribed threshold.

Purpose

The UniqueRows test is designed to gauge the quality of the data supplied to the machine learning model by verifying that the count of distinct rows in the dataset exceeds a specific threshold, thereby ensuring a varied collection of data. Diversity in data is essential for training an unbiased and robust model that excels when faced with novel data.

Test Mechanism

The testing process starts with calculating the total number of rows in the dataset. Subsequently, the count of unique rows is determined for each column in the dataset. If the percentage of unique rows (calculated as the ratio of unique rows to the overall row count) is less than the prescribed minimum percentage threshold given as a function parameter, the test passes. The results are cached and a final pass or fail verdict is given based on whether all columns have successfully passed the test.

Signs of High Risk

• A lack of diversity in data columns, demonstrated by a count of unique rows that falls short of the preset minimum percentage threshold, is indicative of high risk.
• This lack of variety in the data signals potential issues with data quality, possibly leading to overfitting in the model and issues with generalization, thus posing a significant risk.

Strengths

• The UniqueRows test is efficient in evaluating the data's diversity across each information column in the dataset.
• This test provides a quick, systematic method to assess data quality based on uniqueness, which can be pivotal in developing effective and unbiased machine learning models.

Limitations

• A limitation of the UniqueRows test is its assumption that the data's quality is directly proportionate to its uniqueness, which may not always hold true. There might be contexts where certain non-unique rows are essential and should not be overlooked.
• The test does not consider the relative 'importance' of each column in predicting the output, treating all columns equally.
• This test may not be suitable or useful for categorical variables, where the count of unique categories is inherently limited.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_percent_threshold	1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_percent_threshold": "my_vm_min_percent_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.UniqueRows", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Too Many Zero Values ('validmind.data_validation.TooManyZeroValues')

Too Many Zero Values

Identifies numerical columns in a dataset that contain an excessive number of zero values, defined by a threshold percentage.

Purpose

The 'TooManyZeroValues' test is utilized to identify numerical columns in the dataset that may present a quantity of zero values considered excessive. The aim is to detect situations where these may implicate data sparsity or a lack of variation, limiting their effectiveness within a machine learning model. The definition of 'too many' is quantified as a percentage of total values, with a default set to 0.03%.

Test Mechanism

This test is conducted by looping through each column in the dataset and categorizing those that pertain to numerical data. On identifying a numerical column, the function computes the total quantity of zero values and their ratio to the total row count. Should the proportion exceed a pre-set threshold parameter, set by default at 0.03%, the column is considered to have failed the test. The results for each column are summarized and reported, indicating the count and percentage of zero values for each numerical column, alongside a status indicating whether the column has passed or failed the test.

Signs of High Risk

• Numerical columns showing a high ratio of zero values when compared to the total count of rows (exceeding the predetermined threshold).
• Columns characterized by zero values across the board suggest a complete lack of data variation, signifying high risk.

Strengths

• Assists in highlighting columns featuring an excess of zero values that could otherwise go unnoticed within a large dataset.
• Provides the flexibility to alter the threshold that determines when the quantity of zero values becomes 'too many', thus catering to specific needs of a particular analysis or model.
• Offers feedback in the form of both counts and percentages of zero values, which allows a closer inspection of the distribution and proportion of zeros within a column.
• Targets specifically numerical data, thereby avoiding inappropriate application to non-numerical columns and mitigating the risk of false test failures.

Limitations

• Is exclusively designed to check for zero values and doesn’t assess the potential impact of other values that could affect the dataset, such as extremely high or low figures, missing values, or outliers.
• Lacks the ability to detect a repetitive pattern of zeros, which could be significant in time-series or longitudinal data.
• Zero values can actually be meaningful in some contexts; therefore, tagging them as 'too many' could potentially misinterpret the data to some extent.
• This test does not take into consideration the context of the dataset, and fails to recognize that within certain columns, a high number of zero values could be quite normal and not necessarily an indicator of poor data quality.
• Cannot evaluate non-numerical or categorical columns, which might bring with them different types of concerns or issues.

Required Inputs: dataset

Parameters:

Parameter	Default Value
max_percent_threshold	0.03

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "max_percent_threshold": "my_vm_max_percent_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.TooManyZeroValues", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: IQR Outliers Table ('validmind.data_validation.IQROutliersTable')

IQR Outliers Table

Determines and summarizes outliers in numerical features using the Interquartile Range method.

Purpose

The "Interquartile Range Outliers Table" (IQROutliersTable) metric is designed to identify and summarize outliers within numerical features of a dataset using the Interquartile Range (IQR) method. This exercise is crucial in the pre-processing of data because outliers can substantially distort statistical analysis and impact the performance of machine learning models.

Test Mechanism

The IQR, which is the range separating the first quartile (25th percentile) from the third quartile (75th percentile), is calculated for each numerical feature within the dataset. An outlier is defined as a data point falling below the "Q1 - 1.5 * IQR" or above "Q3 + 1.5 * IQR" range. The test computes the number of outliers and their summary statistics (minimum, 25th percentile, median, 75th percentile, and maximum values) for each numerical feature. If no specific features are chosen, the test applies to all numerical features in the dataset. The default outlier threshold is set to 1.5 but can be customized by the user.

Signs of High Risk

• A large number of outliers in multiple features.
• Outliers significantly distanced from the mean value of variables.
• Extremely high or low outlier values indicative of data entry errors or other data quality issues.

Strengths

• Provides a comprehensive summary of outliers for each numerical feature, helping pinpoint features with potential quality issues.
• The IQR method is robust to extremely high or low outlier values as it is based on quartile calculations.
• Can be customized to work on selected features and set thresholds for outliers.

Limitations

• Might cause false positives if the variable deviates from a normal or near-normal distribution, especially for skewed distributions.
• Does not provide interpretation or recommendations for addressing outliers, relying on further analysis by users or data scientists.
• Only applicable to numerical features, not categorical data.
• Default thresholds may not be optimal for data with heavy pre-processing, manipulation, or inherently high kurtosis (heavy tails).

Required Inputs: dataset

Parameters:

Parameter	Default Value
threshold	1.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "threshold": "my_vm_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.IQROutliersTable", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: IQR Outliers Bar Plot ('validmind.data_validation.IQROutliersBarPlot')

IQR Outliers Bar Plot

Visualizes outlier distribution across percentiles in numerical data using the Interquartile Range (IQR) method.

Purpose

The InterQuartile Range Outliers Bar Plot (IQROutliersBarPlot) metric aims to visually analyze and evaluate the extent of outliers in numeric variables based on percentiles. Its primary purpose is to clarify the dataset's distribution, flag possible abnormalities in it, and gauge potential risks associated with processing potentially skewed data, which can affect the machine learning model's predictive prowess.

Test Mechanism

The examination invokes a series of steps:

1. For every numeric feature in the dataset, the 25th percentile (Q1) and 75th percentile (Q3) are calculated before deriving the Interquartile Range (IQR), the difference between Q1 and Q3.
2. Subsequently, the metric calculates the lower and upper thresholds by subtracting Q1 from the threshold times IQR and adding Q3 to threshold times IQR, respectively. The default threshold is set at 1.5.
3. Any value in the feature that falls below the lower threshold or exceeds the upper threshold is labeled as an outlier.
4. The number of outliers are tallied for different percentiles, such as [0-25], [25-50], [50-75], and [75-100].
5. These counts are employed to construct a bar plot for the feature, showcasing the distribution of outliers across different percentiles.

Signs of High Risk

• A prevalence of outliers in the data, potentially skewing its distribution.
• Outliers dominating higher percentiles (75-100) which implies the presence of extreme values, capable of severely influencing the model's performance.
• Certain features harboring most of their values as outliers, which signifies that these features might not contribute positively to the model's forecasting ability.

Strengths

• Effectively identifies outliers in the data through visual means, facilitating easier comprehension and offering insights into the outliers' possible impact on the model.
• Provides flexibility by accommodating all numeric features or a chosen subset.
• Task-agnostic in nature; it is viable for both classification and regression tasks.
• Can handle large datasets as its operation does not hinge on computationally heavy operations.

Limitations

• Its application is limited to numerical variables and does not extend to categorical ones.
• Only reveals the presence and distribution of outliers and does not provide insights into how these outliers might affect the model's predictive performance.
• The assumption that data is unimodal and symmetric may not always hold true. In cases with non-normal distributions, the results can be misleading.

Required Inputs: dataset

Parameters:

Parameter	Default Value
threshold	1.5
fig_width	800

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "threshold": "my_vm_threshold",
    "fig_width": "my_vm_fig_width"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.IQROutliersBarPlot", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 2.2. Descriptive Statistics ('descriptive_statistics')

▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Descriptive Statistics

Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's dataset.

Purpose

The purpose of the Descriptive Statistics metric is to provide a comprehensive summary of both numerical and categorical data within a dataset. This involves statistics such as count, mean, standard deviation, minimum and maximum values for numerical data. For categorical data, it calculates the count, number of unique values, most common value and its frequency, and the proportion of the most frequent value relative to the total. The goal is to visualize the overall distribution of the variables in the dataset, aiding in understanding the model's behavior and predicting its performance.

Test Mechanism

The testing mechanism utilizes two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function pulls out several summary statistics, while value_counts() accounts for unique values. The resulting data is formatted into two distinct tables, one for numerical and another for categorical variable summaries. These tables provide a clear summary of the main characteristics of the variables, which can be instrumental in assessing the model's performance.

Signs of High Risk

• Skewed data or significant outliers can represent high risk. For numerical data, this may be reflected via a significant difference between the mean and median (50% percentile).
• For categorical data, a lack of diversity (low count of unique values), or overdominance of a single category (high frequency of the top value) can indicate high risk.

Strengths

• Provides a comprehensive summary of the dataset, shedding light on the distribution and characteristics of the variables under consideration.
• It is a versatile and robust method, applicable to both numerical and categorical data.
• Helps highlight crucial anomalies such as outliers, extreme skewness, or lack of diversity, which are vital in understanding model behavior during testing and validation.

Limitations

• While this metric offers a high-level overview of the data, it may fail to detect subtle correlations or complex patterns.
• Does not offer any insights on the relationship between variables.
• Alone, descriptive statistics cannot be used to infer properties about future unseen data.
• Should be used in conjunction with other statistical tests to provide a comprehensive understanding of the model's data.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DescriptiveStatistics", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 2.3. Correlations and Interactions ('correlations')

▶ Test: Pearson Correlation Matrix ('validmind.data_validation.PearsonCorrelationMatrix')

Pearson Correlation Matrix

Evaluates linear dependency between numerical variables in a dataset via a Pearson Correlation coefficient heat map.

Purpose

This test is intended to evaluate the extent of linear dependency between all pairs of numerical variables in the given dataset. It provides the Pearson Correlation coefficient, which reveals any high correlations present. The purpose of doing this is to identify potential redundancy, as variables that are highly correlated can often be removed to reduce the dimensionality of the dataset without significantly impacting the model's performance.

Test Mechanism

This metric test generates a correlation matrix for all numerical variables in the dataset using the Pearson correlation formula. A heat map is subsequently created to visualize this matrix effectively. The color of each point on the heat map corresponds to the magnitude and direction (positive or negative) of the correlation, with a range from -1 (perfect negative correlation) to 1 (perfect positive correlation). Any correlation coefficients higher than 0.7 (in absolute terms) are indicated in white in the heat map, suggesting a high degree of correlation.

Signs of High Risk

• A large number of variables in the dataset showing a high degree of correlation (coefficients approaching ±1). This indicates redundancy within the dataset, suggesting that some variables may not be contributing new information to the model.
• Potential risk of overfitting.

Strengths

• Detects and quantifies the linearity of relationships between variables, aiding in identifying redundant variables to simplify models and potentially improve performance.
• The heatmap visualization provides an easy-to-understand overview of correlations, beneficial for users not comfortable with numerical matrices.

Limitations

• Limited to detecting linear relationships, potentially missing non-linear relationships which impede opportunities for dimensionality reduction.
• Measures only the degree of linear relationship, not the strength of one variable's effect on another.
• The 0.7 correlation threshold is arbitrary and might exclude valid dependencies with lower coefficients.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.PearsonCorrelationMatrix", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: High Pearson Correlation ('validmind.data_validation.HighPearsonCorrelation')

High Pearson Correlation

Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity.

Purpose

The High Pearson Correlation test measures the linear relationship between features in a dataset, with the main goal of identifying high correlations that might indicate feature redundancy or multicollinearity. Identification of such issues allows developers and risk management teams to properly deal with potential impacts on the machine learning model's performance and interpretability.

Test Mechanism

The test works by generating pairwise Pearson correlations for all features in the dataset, then sorting and eliminating duplicate and self-correlations. It assigns a Pass or Fail based on whether the absolute value of the correlation coefficient surpasses a pre-set threshold (defaulted at 0.3). It lastly returns the top n strongest correlations regardless of passing or failing status (where n is 10 by default but can be configured by passing the top_n_correlations parameter).

Signs of High Risk

• A high risk indication would be the presence of correlation coefficients exceeding the threshold.
• If the features share a strong linear relationship, this could lead to potential multicollinearity and model overfitting.
• Redundancy of variables can undermine the interpretability of the model due to uncertainty over the authenticity of individual variable's predictive power.

Strengths

• Provides a quick and simple means of identifying relationships between feature pairs.
• Generates a transparent output that displays pairs of correlated variables, the Pearson correlation coefficient, and a Pass or Fail status for each.
• Aids in early identification of potential multicollinearity issues that may disrupt model training.

Limitations

• Can only delineate linear relationships, failing to shed light on nonlinear relationships or dependencies.
• Sensitive to outliers where a few outliers could notably affect the correlation coefficient.
• Limited to identifying redundancy only within feature pairs; may fail to spot more complex relationships among three or more variables.

Required Inputs: dataset

Parameters:

Parameter	Default Value
max_threshold	0.3
top_n_correlations	10
feature_columns	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "max_threshold": "my_vm_max_threshold",
    "top_n_correlations": "my_vm_top_n_correlations",
    "feature_columns": "my_vm_feature_columns"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.HighPearsonCorrelation", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 2.4. Feature Selection and Engineering ('feature_selection')

Text Block: 'feature_selection'

▶ 3. Model Development ('model_development')

▶ 3.1. Model Training ('model_training')

▶ Test: Model Metadata ('validmind.model_validation.ModelMetadata')

Model Metadata

Compare metadata of different models and generate a summary table with the results.

Purpose: The purpose of this function is to compare the metadata of different models, including information about their architecture, framework, framework version, and programming language.

Test Mechanism: The function retrieves the metadata for each model using get_model_info, renames columns according to a predefined set of labels, and compiles this information into a summary table.

Signs of High Risk:

• Inconsistent or missing metadata across models can indicate potential issues in model documentation or management.
• Significant differences in framework versions or programming languages might pose challenges in model integration and deployment.

Strengths:

• Provides a clear comparison of essential model metadata.
• Standardizes metadata labels for easier interpretation and comparison.
• Helps identify potential compatibility or consistency issues across models.

Limitations:

• Assumes that the get_model_info function returns all necessary metadata fields.
• Relies on the correctness and completeness of the metadata provided by each model.
• Does not include detailed parameter information, focusing instead on high-level metadata.

Required Inputs: model

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.ModelMetadata", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Dataset Split ('validmind.data_validation.DatasetSplit')

Dataset Split

Evaluates and visualizes the distribution proportions among training, testing, and validation datasets of an ML model.

Purpose

The DatasetSplit test is designed to evaluate and visualize the distribution of data among training, testing, and validation datasets, if available, within a given machine learning model. The main purpose is to assess whether the model's datasets are split appropriately, as an imbalanced split might affect the model's ability to learn from the data and generalize to unseen data.

Test Mechanism

The DatasetSplit test first calculates the total size of all available datasets in the model. Then, for each individual dataset, the methodology involves determining the size of the dataset and its proportion relative to the total size. The results are then conveniently summarized in a table that shows dataset names, sizes, and proportions. Absolute size and proportion of the total dataset size are displayed for each individual dataset.

Signs of High Risk

• A very small training dataset, which may result in the model not learning enough from the data.
• A very large training dataset and a small test dataset, which may lead to model overfitting and poor generalization to unseen data.
• A small or non-existent validation dataset, which might complicate the model's performance assessment.

Strengths

• The DatasetSplit test provides a clear, understandable visualization of dataset split proportions, which can highlight any potential imbalance in dataset splits quickly.
• It covers a wide range of task types including classification, regression, and text-related tasks.
• The metric is not tied to any specific data type and is applicable to tabular data, time series data, or text data.

Limitations

• The DatasetSplit test does not provide any insight into the quality or diversity of the data within each split, just the size and proportion.
• The test does not give any recommendations or adjustments for imbalanced datasets.
• Potential lack of compatibility with more complex modes of data splitting (for example, stratified or time-based splits) could limit the applicability of this test.

Required Inputs: datasets

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DatasetSplit", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Population Stability Index ('validmind.model_validation.sklearn.PopulationStabilityIndex')

Population Stability Index

Assesses the Population Stability Index (PSI) to quantify the stability of an ML model's predictions across different datasets.

Purpose

The Population Stability Index (PSI) serves as a quantitative assessment for evaluating the stability of a machine learning model's output distributions when comparing two different datasets. Typically, these would be a development and a validation dataset or two datasets collected at different periods. The PSI provides a measurable indication of any significant shift in the model's performance over time or noticeable changes in the characteristics of the population the model is making predictions for.

Test Mechanism

The implementation of the PSI in this script involves calculating the PSI for each feature between the training and test datasets. Data from both datasets is sorted and placed into either a predetermined number of bins or quantiles. The boundaries for these bins are initially determined based on the distribution of the training data. The contents of each bin are calculated and their respective proportions determined. Subsequently, the PSI is derived for each bin through a logarithmic transformation of the ratio of the proportions of data for each feature in the training and test datasets. The PSI, along with the proportions of data in each bin for both datasets, are displayed in a summary table, a grouped bar chart, and a scatter plot.

Signs of High Risk

• A high PSI value is a clear indicator of high risk. Such a value suggests a significant shift in the model predictions or severe changes in the characteristics of the underlying population.
• This ultimately suggests that the model may not be performing as well as expected and that it may be less reliable for making future predictions.

Strengths

• The PSI provides a quantitative measure of the stability of a model over time or across different samples, making it an invaluable tool for evaluating changes in a model's performance.
• It allows for direct comparisons across different features based on the PSI value.
• The calculation and interpretation of the PSI are straightforward, facilitating its use in model risk management.
• The use of visual aids such as tables and charts further simplifies the comprehension and interpretation of the PSI.

Limitations

• The PSI test does not account for the interdependence between features: features that are dependent on one another may show similar shifts in their distributions, which in turn may result in similar PSI values.
• The PSI test does not inherently provide insights into why there are differences in distributions or why the PSI values may have changed.
• The test may not handle features with significant outliers adequately.
• Additionally, the PSI test is performed on model predictions, not on the underlying data distributions which can lead to misinterpretations. Any changes in PSI could be due to shifts in the model (model drift), changes in the relationships between features and the target variable (concept drift), or both. However, distinguishing between these causes is non-trivial.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
num_bins	10
mode	'fixed'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "num_bins": "my_vm_num_bins",
    "mode": "my_vm_mode"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.PopulationStabilityIndex", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 3.2. Model Evaluation ('model_evaluation')

▶ Test: Confusion Matrix ('validmind.model_validation.sklearn.ConfusionMatrix')

Confusion Matrix

Evaluates and visually represents the classification ML model's predictive performance using a Confusion Matrix heatmap.

Purpose

The Confusion Matrix tester is designed to assess the performance of a classification Machine Learning model. This performance is evaluated based on how well the model is able to correctly classify True Positives, True Negatives, False Positives, and False Negatives - fundamental aspects of model accuracy.

Test Mechanism

The mechanism used involves taking the predicted results (y_test_predict) from the classification model and comparing them against the actual values (y_test_true). A confusion matrix is built using the unique labels extracted from y_test_true, employing scikit-learn's metrics. The matrix is then visually rendered with the help of Plotly's create_annotated_heatmap function. A heatmap is created which provides a two-dimensional graphical representation of the model's performance, showcasing distributions of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).

Signs of High Risk

• High numbers of False Positives (FP) and False Negatives (FN), depicting that the model is not effectively classifying the values.
• Low numbers of True Positives (TP) and True Negatives (TN), implying that the model is struggling with correctly identifying class labels.

Strengths

• It provides a simplified yet comprehensive visual snapshot of the classification model's predictive performance.
• It distinctly brings out True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN), thus making it easier to focus on potential areas of improvement.
• The matrix is beneficial in dealing with multi-class classification problems as it can provide a simple view of complex model performances.
• It aids in understanding the different types of errors that the model could potentially make, as it provides in-depth insights into Type-I and Type-II errors.

Limitations

• In cases of unbalanced classes, the effectiveness of the confusion matrix might be lessened. It may wrongly interpret the accuracy of a model that is essentially just predicting the majority class.
• It does not provide a single unified statistic that could evaluate the overall performance of the model. Different aspects of the model's performance are evaluated separately instead.
• It mainly serves as a descriptive tool and does not offer the capability for statistical hypothesis testing.
• Risks of misinterpretation exist because the matrix doesn't directly provide precision, recall, or F1-score data. These metrics have to be computed separately.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
threshold	0.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "threshold": "my_vm_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ConfusionMatrix", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Classifier Performance In Sample ('validmind.model_validation.sklearn.ClassifierPerformance:in_sample')

Classifier Performance In Sample

Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy, and ROC AUC scores.

Purpose

The Classifier Performance test is designed to evaluate the performance of Machine Learning classification models. It accomplishes this by computing precision, recall, F1-Score, and accuracy, as well as the ROC AUC (Receiver operating characteristic - Area under the curve) scores, thereby providing a comprehensive analytic view of the models' performance. The test is adaptable, handling binary and multiclass models equally effectively.

Test Mechanism

The test produces a report that includes precision, recall, F1-Score, and accuracy, by leveraging the classification_report from scikit-learn's metrics module. For multiclass models, macro and weighted averages for these scores are also calculated. Additionally, the ROC AUC scores are calculated and included in the report using the multiclass_roc_auc_score function. The outcome of the test (report format) differs based on whether the model is binary or multiclass.

Signs of High Risk

• Low values for precision, recall, F1-Score, accuracy, and ROC AUC, indicating poor performance.
• Imbalance in precision and recall scores.
• A low ROC AUC score, especially scores close to 0.5 or lower, suggesting a failing model.

Strengths

• Versatile, capable of assessing both binary and multiclass models.
• Utilizes a variety of commonly employed performance metrics, offering a comprehensive view of model performance.
• The use of ROC-AUC as a metric is beneficial for evaluating unbalanced datasets.

Limitations

• Assumes correctly identified labels for binary classification models.
• Specifically designed for classification models and not suitable for regression models.
• May provide limited insights if the test dataset does not represent real-world scenarios adequately.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
average	'macro'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "average": "my_vm_average"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ClassifierPerformance:in_sample", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Classifier Performance Out Of Sample ('validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample')

Classifier Performance Out Of Sample

Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy, and ROC AUC scores.

Purpose

The Classifier Performance test is designed to evaluate the performance of Machine Learning classification models. It accomplishes this by computing precision, recall, F1-Score, and accuracy, as well as the ROC AUC (Receiver operating characteristic - Area under the curve) scores, thereby providing a comprehensive analytic view of the models' performance. The test is adaptable, handling binary and multiclass models equally effectively.

Test Mechanism

The test produces a report that includes precision, recall, F1-Score, and accuracy, by leveraging the classification_report from scikit-learn's metrics module. For multiclass models, macro and weighted averages for these scores are also calculated. Additionally, the ROC AUC scores are calculated and included in the report using the multiclass_roc_auc_score function. The outcome of the test (report format) differs based on whether the model is binary or multiclass.

Signs of High Risk

• Low values for precision, recall, F1-Score, accuracy, and ROC AUC, indicating poor performance.
• Imbalance in precision and recall scores.
• A low ROC AUC score, especially scores close to 0.5 or lower, suggesting a failing model.

Strengths

• Versatile, capable of assessing both binary and multiclass models.
• Utilizes a variety of commonly employed performance metrics, offering a comprehensive view of model performance.
• The use of ROC-AUC as a metric is beneficial for evaluating unbalanced datasets.

Limitations

• Assumes correctly identified labels for binary classification models.
• Specifically designed for classification models and not suitable for regression models.
• May provide limited insights if the test dataset does not represent real-world scenarios adequately.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
average	'macro'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "average": "my_vm_average"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Precision Recall Curve ('validmind.model_validation.sklearn.PrecisionRecallCurve')

Precision Recall Curve

Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve.

Purpose

The Precision Recall Curve metric is intended to evaluate the trade-off between precision and recall in classification models, particularly binary classification models. It assesses the model's capacity to produce accurate results (high precision), as well as its ability to capture a majority of all positive instances (high recall).

Test Mechanism

The test extracts ground truth labels and prediction probabilities from the model's test dataset. It applies the precision_recall_curve method from the sklearn metrics module to these extracted labels and predictions, which computes a precision-recall pair for each possible threshold. This calculation results in an array of precision and recall scores that can be plotted against each other to form the Precision-Recall Curve. This curve is then visually represented by using Plotly's scatter plot.

Signs of High Risk

• A lower area under the Precision-Recall Curve signifies high risk.
• This corresponds to a model yielding a high amount of false positives (low precision) and/or false negatives (low recall).
• If the curve is closer to the bottom left of the plot, rather than being closer to the top right corner, it can be a sign of high risk.

Strengths

• This metric aptly represents the balance between precision (minimizing false positives) and recall (minimizing false negatives), which is especially critical in scenarios where both values are significant.
• Through the graphic representation, it enables an intuitive understanding of the model's performance across different threshold levels.

Limitations

• This metric is only applicable to binary classification models - it raises errors for multiclass classification models or Foundation models.
• It may not fully represent the overall accuracy of the model if the cost of false positives and false negatives are extremely different, or if the dataset is heavily imbalanced.

Required Inputs: model, dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.PrecisionRecallCurve", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: ROC Curve ('validmind.model_validation.sklearn.ROCCurve')

ROC Curve

Evaluates binary classification model performance by generating and plotting the Receiver Operating Characteristic (ROC) curve and calculating the Area Under Curve (AUC) score.

Purpose

The Receiver Operating Characteristic (ROC) curve is designed to evaluate the performance of binary classification models. This curve illustrates the balance between the True Positive Rate (TPR) and False Positive Rate (FPR) across various threshold levels. In combination with the Area Under the Curve (AUC), the ROC curve aims to measure the model's discrimination ability between the two defined classes in a binary classification problem (e.g., default vs non-default). Ideally, a higher AUC score signifies superior model performance in accurately distinguishing between the positive and negative classes.

Test Mechanism

First, this script selects the target model and datasets that require binary classification. It then calculates the predicted probabilities for the test set, and uses this data, along with the true outcomes, to generate and plot the ROC curve. Additionally, it includes a line signifying randomness (AUC of 0.5). The AUC score for the model's ROC curve is also computed, presenting a numerical estimation of the model's performance. If any Infinite values are detected in the ROC threshold, these are effectively eliminated. The resulting ROC curve, AUC score, and thresholds are consequently saved for future reference.

Signs of High Risk

• A high risk is potentially linked to the model's performance if the AUC score drops below or nears 0.5.
• Another warning sign would be the ROC curve lying closer to the line of randomness, indicating no discriminative ability.
• For the model to be deemed competent at its classification tasks, it is crucial that the AUC score is significantly above 0.5.

Strengths

• The ROC Curve offers an inclusive visual depiction of a model's discriminative power throughout all conceivable classification thresholds, unlike other metrics that solely disclose model performance at one fixed threshold.
• Despite the proportions of the dataset, the AUC Score, which represents the entire ROC curve as a single data point, continues to be consistent, proving to be the ideal choice for such situations.

Limitations

• The primary limitation is that this test is exclusively structured for binary classification tasks, thus limiting its application towards other model types.
• Furthermore, its performance might be subpar with models that output probabilities highly skewed towards 0 or 1.
• At the extreme, the ROC curve could reflect high performance even when the majority of classifications are incorrect, provided that the model's ranking format is retained. This phenomenon is commonly termed the "Class Imbalance Problem".

Required Inputs: model, dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ROCCurve", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'model_validation_tests_text'

▶ Test: Training Test Degradation ('validmind.model_validation.sklearn.TrainingTestDegradation')

Training Test Degradation

Tests if model performance degradation between training and test datasets exceeds a predefined threshold.

Purpose

The TrainingTestDegradation class serves as a test to verify that the degradation in performance between the training and test datasets does not exceed a predefined threshold. This test measures the model's ability to generalize from its training data to unseen test data, assessing key classification metrics such as precision, recall, and f1 score to verify the model's robustness and reliability.

Test Mechanism

The code applies several predefined metrics, including precision, recall, and f1 scores, to the model's predictions for both the training and test datasets. It calculates the degradation as the difference between the training score and test score divided by the training score. The test is considered successful if the degradation for each metric is less than the preset maximum threshold (default: 0.10). The results are summarized in a table showing each metric's train score, test score, degradation percentage, and pass/fail status.

Signs of High Risk

• A degradation percentage that exceeds the maximum allowed threshold of 10% for any of the evaluated metrics.
• A high difference or gap between the metric scores on the training and the test datasets.
• The 'Pass/Fail' column displaying 'Fail' for any of the evaluated metrics.

Strengths

• Provides a quantitative measure of the model's ability to generalize to unseen data, which is key for predicting its practical real-world performance.
• By evaluating multiple metrics, it takes into account different facets of model performance and enables a more holistic evaluation.
• The use of a variable predefined threshold allows the flexibility to adjust the acceptability criteria for different scenarios.

Limitations

• The test compares raw performance on training and test data but does not factor in the nature of the data. Areas with less representation in the training set might still perform poorly on unseen data.
• It requires good coverage and balance in the test and training datasets to produce reliable results, which may not always be available.
• The test is currently only designed for classification tasks.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
max_threshold	0.1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "max_threshold": "my_vm_max_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.TrainingTestDegradation", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Minimum Accuracy ('validmind.model_validation.sklearn.MinimumAccuracy')

Minimum Accuracy

Checks if the model's prediction accuracy meets or surpasses a specified threshold.

Purpose

The Minimum Accuracy test’s objective is to verify whether the model's prediction accuracy on a specific dataset meets or surpasses a predetermined minimum threshold. Accuracy, which is simply the ratio of correct predictions to total predictions, is a key metric for evaluating the model's performance. Considering binary as well as multiclass classifications, accurate labeling becomes indispensable.

Test Mechanism

The test mechanism involves contrasting the model's accuracy score with a preset minimum threshold value, with the default being 0.7. The accuracy score is computed utilizing sklearn’s accuracy_score method, where the true labels y_true and predicted labels class_pred are compared. If the accuracy score is above the threshold, the test receives a passing mark. The test returns the result along with the accuracy score and threshold used for the test.

Signs of High Risk

• Model fails to achieve or surpass the predefined score threshold.
• Persistent scores below the threshold, indicating a high risk of inaccurate predictions.

Strengths

• Simplicity, presenting a straightforward measure of holistic model performance across all classes.
• Particularly advantageous when classes are balanced.
• Versatile, as it can be implemented on both binary and multiclass classification tasks.

Limitations

• Misleading accuracy scores when classes in the dataset are highly imbalanced.
• Favoritism towards the majority class, giving an inaccurate perception of model performance.
• Inability to measure the model's precision, recall, or capacity to manage false positives or false negatives.
• Focused on overall correctness and may not be sufficient for all types of model analytics.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
min_threshold	0.7

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.MinimumAccuracy", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Minimum F1 Score ('validmind.model_validation.sklearn.MinimumF1Score')

Minimum F1 Score

Assesses if the model's F1 score on the validation set meets a predefined minimum threshold, ensuring balanced performance between precision and recall.

Purpose

The main objective of this test is to ensure that the F1 score, a balanced measure of precision and recall, of the model meets or surpasses a predefined threshold on the validation dataset. The F1 score is highly useful for gauging model performance in classification tasks, especially in cases where the distribution of positive and negative classes is skewed.

Test Mechanism

The F1 score for the validation dataset is computed through scikit-learn's metrics in Python. The scoring mechanism differs based on the classification problem: for multi-class problems, macro averaging is used, and for binary classification, the built-in f1_score calculation is used. The obtained F1 score is then assessed against the predefined minimum F1 score that is expected from the model.

Signs of High Risk

• If a model returns an F1 score that is less than the established threshold, it is regarded as high risk.
• A low F1 score might suggest that the model is not finding an optimal balance between precision and recall, failing to effectively identify positive classes while minimizing false positives.

Strengths

• Provides a balanced measure of a model's performance by accounting for both false positives and false negatives.
• Particularly advantageous in scenarios with imbalanced class distribution, where accuracy can be misleading.
• Flexibility in setting the threshold value allows tailored minimum acceptable performance standards.

Limitations

• May not be suitable for all types of models and machine learning tasks.
• The F1 score assumes an equal cost for false positives and false negatives, which may not be true in some real-world scenarios.
• Practitioners might need to rely on other metrics such as precision, recall, or the ROC-AUC score that align more closely with specific requirements.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
min_threshold	0.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.MinimumF1Score", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Minimum ROCAUC Score ('validmind.model_validation.sklearn.MinimumROCAUCScore')

Minimum ROCAUC Score

Validates model by checking if the ROC AUC score meets or surpasses a specified threshold.

Purpose

The Minimum ROC AUC Score test is used to determine the model's performance by ensuring that the Receiver Operating Characteristic Area Under the Curve (ROC AUC) score on the validation dataset meets or exceeds a predefined threshold. The ROC AUC score indicates how well the model can distinguish between different classes, making it a crucial measure in binary and multiclass classification tasks.

Test Mechanism

This test implementation calculates the multiclass ROC AUC score on the true target values and the model's predictions. The test converts the multi-class target variables into binary format using LabelBinarizer before computing the score. If this ROC AUC score is higher than the predefined threshold (defaulted to 0.5), the test passes; otherwise, it fails. The results, including the ROC AUC score, the threshold, and whether the test passed or failed, are then stored in a ThresholdTestResult object.

Signs of High Risk

• A high risk or failure in the model's performance as related to this metric would be represented by a low ROC AUC score, specifically any score lower than the predefined minimum threshold. This suggests that the model is struggling to distinguish between different classes effectively.

Strengths

• The test considers both the true positive rate and false positive rate, providing a comprehensive performance measure.
• ROC AUC score is threshold-independent meaning it measures the model's quality across various classification thresholds.
• Works robustly with binary as well as multi-class classification problems.

Limitations

• ROC AUC may not be useful if the class distribution is highly imbalanced; it could perform well in terms of AUC but still fail to predict the minority class.
• The test does not provide insight into what specific aspects of the model are causing poor performance if the ROC AUC score is unsatisfactory.
• The use of macro average for multiclass ROC AUC score implies equal weightage to each class, which might not be appropriate if the classes are imbalanced.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
min_threshold	0.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.MinimumROCAUCScore", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 3.3. Model Explainability and Interpretability ('explainability')

▶ Test: Permutation Feature Importance ('validmind.model_validation.sklearn.PermutationFeatureImportance')

Permutation Feature Importance

Assesses the significance of each feature in a model by evaluating the impact on model performance when feature values are randomly rearranged.

Purpose

The Permutation Feature Importance (PFI) metric aims to assess the importance of each feature used by the Machine Learning model. The significance is measured by evaluating the decrease in the model's performance when the feature's values are randomly arranged.

Test Mechanism

PFI is calculated via the permutation_importance method from the sklearn.inspection module. This method shuffles the columns of the feature dataset and measures the impact on the model's performance. A significant decrease in performance after permutating a feature's values deems the feature as important. On the other hand, if performance remains the same, the feature is likely not important. The output of the PFI metric is a figure illustrating the importance of each feature.

Signs of High Risk

• The model heavily relies on a feature with highly variable or easily permutable values, indicating instability.
• A feature deemed unimportant by the model but expected to have a significant effect on the outcome based on domain knowledge is not influencing the model's predictions.

Strengths

• Provides insights into the importance of different features and may reveal underlying data structure.
• Can indicate overfitting if a particular feature or set of features overly impacts the model's predictions.
• Model-agnostic and can be used with any classifier that provides a measure of prediction accuracy before and after feature permutation.

Limitations

• Does not imply causality; it only presents the amount of information that a feature provides for the prediction task.
• Does not account for interactions between features. If features are correlated, the permutation importance may allocate importance to one and not the other.
• Cannot interact with certain libraries like statsmodels, pytorch, catboost, etc., thus limiting its applicability.

Required Inputs: model, dataset

Parameters:

Parameter	Default Value
fontsize	None
figure_height	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {
    "fontsize": "my_vm_fontsize",
    "figure_height": "my_vm_figure_height"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.PermutationFeatureImportance", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'validmind.model_validation.sklearn.SHAPGlobalImportance_global_importance_text'

▶ Test: SHAP Global Importance ('validmind.model_validation.sklearn.SHAPGlobalImportance')

SHAP Global Importance

Evaluates and visualizes global feature importance using SHAP values for model explanation and risk identification.

Purpose

The SHAP (SHapley Additive exPlanations) Global Importance metric aims to elucidate model outcomes by attributing them to the contributing features. It assigns a quantifiable global importance to each feature via their respective absolute Shapley values, thereby making it suitable for tasks like classification (both binary and multiclass). This metric forms an essential part of model risk management.

Test Mechanism

The exam begins with the selection of a suitable explainer which aligns with the model's type. For tree-based models like XGBClassifier, RandomForestClassifier, CatBoostClassifier, TreeExplainer is used whereas for linear models like LogisticRegression, XGBRegressor, LinearRegression, it is the LinearExplainer. Once the explainer calculates the Shapley values, these values are visualized using two specific graphical representations:

1. Mean Importance Plot: This graph portrays the significance of individual features based on their absolute Shapley values. It calculates the average of these absolute Shapley values across all instances to highlight the global importance of features.

2. Summary Plot: This visual tool combines the feature importance with their effects. Every dot on this chart represents a Shapley value for a certain feature in a specific case. The vertical axis is denoted by the feature whereas the horizontal one corresponds to the Shapley value. A color gradient indicates the value of the feature, gradually changing from low to high. Features are systematically organized in accordance with their importance.

Signs of High Risk

• Overemphasis on certain features in SHAP importance plots, thus hinting at the possibility of model overfitting
• Anomalies such as unexpected or illogical features showing high importance, which might suggest that the model's decisions are rooted in incorrect or undesirable reasoning
• A SHAP summary plot filled with high variability or scattered data points, indicating a cause for concern

Strengths

• SHAP does more than just illustrating global feature significance, it offers a detailed perspective on how different features shape the model's decision-making logic for each instance.
• It provides clear insights into model behavior.

Limitations

• High-dimensional data can convolute interpretations.
• Associating importance with tangible real-world impact still involves a certain degree of subjectivity.

Required Inputs: model, dataset

Parameters:

Parameter	Default Value
kernel_explainer_samples	10
tree_or_linear_explainer_samples	200
class_of_interest	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {
    "kernel_explainer_samples": "my_vm_kernel_explainer_samples",
    "tree_or_linear_explainer_samples": "my_vm_tree_or_linear_explainer_samples",
    "class_of_interest": "my_vm_class_of_interest"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.SHAPGlobalImportance", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 3.4. Model Diagnosis ('model_diagnosis')

Text Block: 'model_weak_spots_description'

▶ Test: Weakspots Diagnosis ('validmind.model_validation.sklearn.WeakspotsDiagnosis')

Weakspots Diagnosis

Identifies and visualizes weak spots in a machine learning model's performance across various sections of the feature space.

Purpose

The weak spots test is applied to evaluate the performance of a machine learning model within specific regions of its feature space. This test slices the feature space into various sections, evaluating the model's outputs within each section against specific performance metrics (e.g., accuracy, precision, recall, and F1 scores). The ultimate aim is to identify areas where the model's performance falls below the set thresholds, thereby exposing its possible weaknesses and limitations.

Test Mechanism

The test mechanism adopts an approach of dividing the feature space of the training dataset into numerous bins. The model's performance metrics (accuracy, precision, recall, F1 scores) are then computed for each bin on both the training and test datasets. A "weak spot" is identified if any of the performance metrics fall below a predetermined threshold for a particular bin on the test dataset. The test results are visually plotted as bar charts for each performance metric, indicating the bins which fail to meet the established threshold.

Signs of High Risk

• Any performance metric of the model dropping below the set thresholds.
• Significant disparity in performance between the training and test datasets within a bin could be an indication of overfitting.
• Regions or slices with consistently low performance metrics. Such instances could mean that the model struggles to handle specific types of input data adequately, resulting in potentially inaccurate predictions.

Strengths

• The test helps pinpoint precise regions of the feature space where the model's performance is below par, allowing for more targeted improvements to the model.
• The graphical presentation of the performance metrics offers an intuitive way to understand the model's performance across different feature areas.
• The test exhibits flexibility, letting users set different thresholds for various performance metrics according to the specific requirements of the application.

Limitations

• The binning system utilized for the feature space in the test could over-simplify the model's behavior within each bin. The granularity of this slicing depends on the chosen 'bins' parameter and can sometimes be arbitrary.
• The effectiveness of this test largely hinges on the selection of thresholds for the performance metrics, which may not hold universally applicable and could be subjected to the specifications of a particular model and application.
• The test is unable to handle datasets with a text column, limiting its application to numerical or categorical data types only.
• Despite its usefulness in highlighting problematic regions, the test does not offer direct suggestions for model improvement.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
features_columns	None
metrics	None
thresholds	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "features_columns": "my_vm_features_columns",
    "metrics": "my_vm_metrics",
    "thresholds": "my_vm_thresholds"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.WeakspotsDiagnosis", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'model_overfit_regions_description'

▶ Test: Overfit Diagnosis ('validmind.model_validation.sklearn.OverfitDiagnosis')

Overfit Diagnosis

Assesses potential overfitting in a model's predictions, identifying regions where performance between training and testing sets deviates significantly.

Purpose

The Overfit Diagnosis test aims to identify areas in a model's predictions where there is a significant difference in performance between the training and testing sets. This test helps to pinpoint specific regions or feature segments where the model may be overfitting.

Test Mechanism

This test compares the model's performance on training versus test data, grouped by feature columns. It calculates the difference between the training and test performance for each group and identifies regions where this difference exceeds a specified threshold:

• The test works for both classification and regression models.
• It defaults to using the AUC metric for classification models and the MSE metric for regression models.
• The threshold for identifying overfitting regions is set to 0.04 by default.
• The test calculates the performance metrics for each feature segment and plots regions where the performance gap exceeds the threshold.

Signs of High Risk

• Significant gaps between training and test performance metrics for specific feature segments.
• Multiple regions with performance gaps exceeding the defined threshold.
• Higher than expected differences in predicted versus actual values in the test set compared to the training set.

Strengths

• Identifies specific areas where overfitting occurs.
• Supports multiple performance metrics, providing flexibility.
• Applicable to both classification and regression models.
• Visualization of overfitting segments aids in better understanding and debugging.

Limitations

• The default threshold may not be suitable for all use cases and requires tuning.
• May not capture more subtle forms of overfitting that do not exceed the threshold.
• Assumes that the binning of features adequately represents the data segments.

Required Inputs: model, datasets

Parameters:

Parameter	Default Value
metric	None
cut_off_threshold	0.04

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "datasets": "my_vm_datasets"
}
params = {
    "metric": "my_vm_metric",
    "cut_off_threshold": "my_vm_cut_off_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.OverfitDiagnosis", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'model_robustness_description'

▶ Test: Robustness Diagnosis ('validmind.model_validation.sklearn.RobustnessDiagnosis')

Robustness Diagnosis

Assesses the robustness of a machine learning model by evaluating performance decay under noisy conditions.

Purpose

The Robustness Diagnosis test aims to evaluate the resilience of a machine learning model when subjected to perturbations or noise in its input data. This is essential for understanding the model's ability to handle real-world scenarios where data may be imperfect or corrupted.

Test Mechanism

This test introduces Gaussian noise to the numeric input features of the datasets at varying scales of standard deviation. The performance of the model is then measured using a specified metric. The process includes:

• Adding Gaussian noise to numerical input features based on scaling factors.
• Evaluating the model's performance on the perturbed data using metrics like AUC for classification tasks and MSE for regression tasks.
• Aggregating and plotting the results to visualize performance decay relative to perturbation size.

Signs of High Risk

• A significant drop in performance metrics with minimal noise.
• Performance decay values exceeding the specified threshold.
• Consistent failure to meet performance standards across multiple perturbation scales.

Strengths

• Provides insights into the model's robustness against noisy or corrupted data.
• Utilizes a variety of performance metrics suitable for both classification and regression tasks.
• Visualization helps in understanding the extent of performance degradation.

Limitations

• Gaussian noise might not adequately represent all types of real-world data perturbations.
• Performance thresholds are somewhat arbitrary and might need tuning.
• The test may not account for more complex or unstructured noise patterns that could affect model robustness.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
metric	None
scaling_factor_std_dev_list	[0.1, 0.2, 0.3, 0.4, 0.5]
performance_decay_threshold	0.05

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "metric": "my_vm_metric",
    "scaling_factor_std_dev_list": "my_vm_scaling_factor_std_dev_list",
    "performance_decay_threshold": "my_vm_performance_decay_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.RobustnessDiagnosis", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 4. Monitoring and Governance ('monitoring_governance')

▶ 4.1. Monitoring Plan ('monitoring_plan')

Text Block: 'monitoring_plan'

▶ 4.2. Monitoring Implementation ('monitoring_implementation')

Text Block: 'monitoring_implementation'

▶ 4.3. Governance Plan ('governance_plan')

Text Block: 'governance_plan'

Include custom test results

Since your custom test IDs are now part of your documentation template, you can now run tests for an entire section and all additional custom tests should be loaded without any issues.

Let's run all tests in the Model Evaluation section of the documentation. Note that we have been running the sample custom confusion matrix with normalize=True to demonstrate the ability to provide custom parameters.

In the Run the model evaluation tests section of 2 — Start the development process, you learned how to assign inputs to individual tests with run_documentation_tests(). Assigning parameters is similar, you only need to provide assign a params dictionary to a given test ID, my_test_provider.ConfusionMatrix in this case.

test_config = {
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "dataset": vm_train_ds,
            "model": vm_model,
        },
    },
    "my_test_provider.ConfusionMatrix": {
        "params": {"normalize": True},
        "inputs": {"dataset": vm_test_ds, "model": vm_model},
    },
}
results = vm.run_documentation_tests(
    section=["model_evaluation"],
    inputs={
        "dataset": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds
        "model": vm_model,
        "datasets": (
            vm_train_ds,
            vm_test_ds,
        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds
    },
    config=test_config,
)

2026-07-14 05:30:21,442 - WARNING(validmind.vm_models.test_suite.runner): Config key 'my_test_provider.ConfusionMatrix' does not match a test_id in the template.
    Ensure you registered a content block with the correct content_id in the template
    The configuration for this test will be ignored.

Test suite complete!

18/18 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Evaluation

Model Evaluation

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

The Confusion Matrix test evaluates classification performance by comparing predicted labels with observed labels and displaying the counts of correct and incorrect classifications by class. The result shows four outcome cells: 204 true negatives, 193 true positives, 122 false positives, and 128 false negatives. These counts summarize how observations in classes 0 and 1 were assigned across predicted labels 0 and 1. The heatmap provides a direct view of the balance between correct classifications and misclassifications for both classes.

Key insights:

• Correct classifications exceed errors: The matrix contains 204 true negatives and 193 true positives, compared with 122 false positives and 128 false negatives. Correct predictions total 397, while misclassifications total 250.

• Error counts are relatively balanced: False negatives (128) and false positives (122) are close in magnitude. This indicates that misclassification is not concentrated in only one error type.

• Negative class is identified slightly more often correctly: True negatives total 204, compared with 193 true positives. The difference is modest, indicating similar but not identical classification performance across the two classes.

• Observed class counts are similar: Based on the matrix totals, the true class 1 count is 321 (128 + 193) and the true class 0 count is 326 (204 + 122). This reflects a near-balanced distribution of observed classes in the evaluated sample.

The confusion matrix shows that the model produced more correct classifications than incorrect ones, with similar volumes of true negatives and true positives. Misclassification is split fairly evenly between false positives and false negatives, rather than being dominated by one error direction. The evaluated sample also appears nearly balanced across the two observed classes, which supports direct comparison of classification outcomes between class 0 and class 1.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

The Classifier Performance test evaluates classification performance using precision, recall, F1-score, accuracy, and ROC AUC. For this in-sample binary classification result, the output includes class-level precision, recall, and F1 values for classes 0 and 1, together with macro and weighted averages, plus overall accuracy and ROC AUC. The reported class metrics are closely aligned across the two classes, and the summary metrics provide an aggregate view of overall in-sample discrimination and classification accuracy.

Key insights:

• Balanced class-level performance: Precision, recall, and F1 are similar across classes 0 and 1. Class 0 records precision 0.6348, recall 0.6442, and F1 0.6395, while class 1 records precision 0.6403, recall 0.6309, and F1 0.6356.
• Aggregate metrics are tightly aligned: Weighted-average and macro-average precision, recall, and F1 are all 0.6375-0.6376. This indicates little divergence between class-specific performance and overall summarized performance.
• Accuracy matches average recall and F1: Overall accuracy is 0.6375, which is numerically consistent with the reported weighted and macro average recall and F1 values of 0.6375.
• ROC AUC exceeds classification rate metrics: ROC AUC is 0.6819, compared with accuracy of 0.6375. This shows the ranking-based discrimination measure is higher than the threshold-based classification metrics reported in the table.

Overall, the in-sample results show closely matched performance across both classes, with no material separation between class-specific precision, recall, and F1 values. The macro and weighted averages are effectively identical, indicating consistent aggregate performance across the reported classes. Accuracy is 0.6375, while ROC AUC is 0.6819, showing stronger discrimination on the ranking metric than on the reported threshold-based classification outcomes.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6348	0.6442	0.6395
1	0.6403	0.6309	0.6356
Weighted Average	0.6376	0.6375	0.6375
Macro Average	0.6376	0.6375	0.6375

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6375
ROC AUC	0.6819

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

The Classifier Performance test evaluates classification performance using precision, recall, F1-score, accuracy, and ROC AUC. In this out-of-sample result, the report presents class-level precision, recall, and F1 values for classes 0 and 1, along with macro and weighted averages. A separate summary table reports overall accuracy of 0.6136 and ROC AUC of 0.6627. The class-specific metrics are closely aligned across the two classes, with all reported precision, recall, and F1 values falling near 0.60 to 0.63.

Key insights:

• Balanced class-level performance: Class 0 and class 1 show similar precision, recall, and F1 values. Precision is 0.6145 for class 0 and 0.6127 for class 1, while F1 is 0.6201 and 0.6069 respectively, indicating limited divergence in performance across classes.
• Slightly stronger detection of class 0: Recall for class 0 is 0.6258 versus 0.6012 for class 1. This indicates higher observed capture of class 0 than class 1 in the out-of-sample evaluation.
• Aggregate metrics are internally consistent: Weighted-average and macro-average precision, recall, and F1 are all 0.6135 to 0.6136. Accuracy is also 0.6136, showing close alignment between class-level averages and overall classification accuracy.
• ROC AUC exceeds accuracy level: ROC AUC is 0.6627, compared with accuracy of 0.6136. This indicates the rank-order discrimination measure is higher than the threshold-dependent accuracy reported in the same test.

Overall, the out-of-sample results show a tightly clustered performance profile across both classes and across aggregate summary metrics. The model records accuracy of 0.6136 and ROC AUC of 0.6627, with macro and weighted averages nearly identical, indicating limited class-level imbalance in the reported metrics. The largest class-level difference appears in recall, where class 0 performs modestly above class 1.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6145	0.6258	0.6201
1	0.6127	0.6012	0.6069
Weighted Average	0.6136	0.6136	0.6135
Macro Average	0.6136	0.6135	0.6135

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6136
ROC AUC	0.6627

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

The PrecisionRecallCurve test evaluates the trade-off between precision and recall across classification thresholds for a binary model. The result is presented as a precision-recall curve with recall on the x-axis and precision on the y-axis, showing model behavior from very low recall through near-complete recall coverage. The curve begins with very high precision at minimal recall, then settles into a lower band as recall increases, with precision gradually declining toward the right side of the plot.

Key insights:

• Precision declines as recall increases: Precision is highest at the extreme low-recall region, reaching approximately 1.0 at the left edge of the curve, and decreases progressively as recall approaches 1.0, ending near 0.50.

• Mid-range recall remains moderately stable: Across roughly the 0.2 to 0.5 recall range, precision stays within a relatively narrow band around 0.65 to 0.70, indicating a comparatively stable trade-off in the middle portion of the curve.

• Early-threshold region is volatile: At very low recall levels, the curve shows sharp movement, with precision dropping quickly from near 1.0 into the low-to-mid 0.60 range before partially recovering.

• High recall corresponds to lower precision: Beyond approximately 0.6 recall, the curve trends downward more steadily, with precision moving from around 0.62 toward roughly 0.50 as recall nears the upper end of the scale.

The precision-recall profile shows strong precision only at very low recall, followed by a moderate and comparatively stable mid-range region, and then a continued decline in precision at higher recall levels. Overall, the curve indicates that broader positive-class coverage is associated with increasing loss of precision, with the most balanced portion of the trade-off appearing in the middle recall range.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

The ROCCurve test evaluates binary classification performance by plotting the receiver operating characteristic and calculating the area under the curve (AUC). The figure shows the ROC curve for log_reg_model_v1 on test_dataset_final, with the model curve plotted against the random-classification reference line. The reported AUC is 0.66, and the curve remains above the diagonal baseline across most of the false positive rate range.

Key insights:

• AUC measured at 0.66: The model’s reported area under the ROC curve is 0.66 on the test dataset, summarizing discrimination performance across all classification thresholds.

• Performance exceeds random baseline: The ROC curve lies above the random reference line (AUC = 0.5) for most of the plotted range, indicating stronger ranking performance than a random classifier.

• Gradual gain in true positive rate: The curve shows a steady increase in true positive rate as false positive rate rises, rather than a sharp early lift concentrated at very low false positive rates.

The ROC results show that log_reg_model_v1 distinguishes between the two classes better than the random benchmark, with an AUC of 0.66 on test_dataset_final. The curve’s position above the diagonal confirms positive discriminative ability, while its gradual shape indicates moderate separation rather than strong concentration of true positives at low false positive rates.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

The TrainingTestDegradation test evaluates whether model performance degradation from the training dataset to the test dataset remains within the predefined threshold across classification metrics. The results table reports precision, recall, and F1-score for classes 1 and 0 on both datasets, together with the calculated degradation percentages and pass/fail outcomes. Observed degradation percentages range from 2.8593% to 4.6984%, and all six metric evaluations are marked as passing against the configured threshold.

Key insights:

• All metrics passed threshold: Every evaluated metric for both class 1 and class 0 is marked as Pass, with no degradation result exceeding the maximum threshold referenced in the test description.
• Class 1 shows higher degradation: For class 1, degradation ranges from 4.3081% for precision to 4.6984% for recall, with F1-score degradation at 4.5051%. These values are consistently higher than the corresponding degradation values for class 0.
• Class 0 remains comparatively stable: For class 0, degradation ranges from 2.8593% for recall to 3.21% for precision, with F1-score degradation at 3.0362%, indicating a narrower train-to-test performance gap than class 1.
• Recall has the largest class-specific decline: The highest degradation observed in the table is class 1 recall at 4.6984%, based on a decrease from 0.6309 on the training dataset to 0.6012 on the test dataset.

The test results show that train-to-test performance declines are limited across all evaluated classification metrics and remain within the defined acceptance threshold. Degradation is consistently higher for class 1 than for class 0, with the largest decline observed in class 1 recall. Overall, the reported results indicate a moderate and bounded reduction in performance between the training and test datasets for both classes.

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
1	Precision	0.6403	0.6127	4.3081	Pass
1	Recall	0.6309	0.6012	4.6984	Pass
1	F1-Score	0.6356	0.6069	4.5051	Pass
0	Precision	0.6348	0.6145	3.2100	Pass
0	Recall	0.6442	0.6258	2.8593	Pass
0	F1-Score	0.6395	0.6201	3.0362	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

The Minimum Accuracy test evaluates whether the model’s prediction accuracy on the evaluated dataset meets or exceeds a specified minimum threshold. The result table reports an accuracy score of 0.6136 against a threshold of 0.7, along with the corresponding pass/fail outcome. In this run, the recorded outcome is Fail because the observed score is below the defined cutoff.

Key insights:

• Accuracy below threshold: The observed accuracy score is 0.6136, which is lower than the configured minimum threshold of 0.7.
• Test outcome is fail: The reported pass/fail status for this evaluation is Fail, reflecting that the minimum accuracy condition was not met.
• Shortfall is measurable: The difference between the observed accuracy and the threshold is 0.0864.

The result indicates that, on the evaluated dataset, overall classification accuracy did not reach the minimum level defined by the test configuration. The recorded score of 0.6136 remained below the 0.7 threshold, and the test therefore returned a failing outcome.

Tables

Score	Threshold	Pass/Fail
0.6136	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

The MinimumF1Score test evaluates whether the model’s F1 score on the validation set meets a predefined minimum threshold. The result table reports a validation F1 score of 0.6069 alongside a threshold of 0.5000, with the outcome recorded as Pass. These values show the comparison used to assess balanced classification performance based on precision and recall.

Key insights:

• F1 score exceeds threshold: The observed validation F1 score is 0.6069 versus a minimum threshold of 0.5000, placing the result above the required cutoff.
• Test outcome is pass: The result is explicitly labeled Pass in the test output, indicating that the threshold condition defined for this test was satisfied.
• Positive margin over minimum: The observed score exceeds the threshold by 0.1069, showing a measurable gap between the achieved validation performance and the minimum benchmark used in the test.

The test result shows that the model’s validation F1 score met the predefined minimum standard used in this assessment. The recorded score of 0.6069 was above the 0.5000 threshold, and the test output classified the result as Pass. Taken together, the result indicates that the model satisfied the configured minimum criterion for balanced precision-recall performance on the validation dataset.

Tables

Score	Threshold	Pass/Fail
0.6069	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

The Minimum ROC AUC Score test evaluates whether the model’s ROC AUC on the validation dataset meets or exceeds a predefined minimum threshold. The result table reports a ROC AUC score of 0.6627 against a threshold of 0.5000, along with the corresponding pass/fail outcome. The recorded outcome for this test is Pass.

Key insights:

• Threshold exceeded: The ROC AUC score of 0.6627 is above the minimum threshold of 0.5000 used in this test.
• Test status is pass: The reported pass/fail result is Pass, consistent with the score exceeding the defined cutoff.
• Positive margin to threshold: The observed score exceeds the threshold by 0.1627, indicating performance above the minimum acceptance level defined for this test.

The test result shows that the model satisfied the minimum ROC AUC requirement on the validation dataset. The reported score of 0.6627 exceeded the configured threshold of 0.5000, and the test outcome was recorded as Pass.

Tables

Score	Threshold	Pass/Fail
0.6627	0.5	Pass

Configuring documentation template tests

Let's call the utility function vm.get_test_suite().get_default_config() which will return the default configuration for the entire documentation template as a dictionary:

• This configuration will contain all the test IDs and their default parameters.
• You can then modify this configuration as needed and pass it to run_documentation_tests() to run all tests in the documentation template if needed.
• You still have the option to continue running tests for one section at a time; get_default_config() simply provides a useful reference for providing default parameters to every test.

import json

model_test_suite = vm.get_test_suite()
config = model_test_suite.get_default_config()
print("Suite Config: \n", json.dumps(config, indent=2))

Suite Config: 
 {
  "validmind.data_validation.DatasetDescription": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.data_validation.ClassImbalance": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_percent_threshold": 10
    }
  },
  "validmind.data_validation.Duplicates": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_threshold": 1
    }
  },
  "validmind.data_validation.HighCardinality": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "num_threshold": 100,
      "percent_threshold": 0.1,
      "threshold_type": "percent"
    }
  },
  "validmind.data_validation.MissingValues": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_percentage_threshold": 1.0
    }
  },
  "validmind.data_validation.Skewness": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "max_threshold": 1
    }
  },
  "validmind.data_validation.UniqueRows": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_percent_threshold": 1
    }
  },
  "validmind.data_validation.TooManyZeroValues": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "max_percent_threshold": 0.03
    }
  },
  "validmind.data_validation.IQROutliersTable": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "threshold": 1.5
    }
  },
  "validmind.data_validation.IQROutliersBarPlot": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "threshold": 1.5,
      "fig_width": 800
    }
  },
  "validmind.data_validation.DescriptiveStatistics": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.data_validation.PearsonCorrelationMatrix": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.data_validation.HighPearsonCorrelation": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "max_threshold": 0.3,
      "top_n_correlations": 10,
      "feature_columns": null
    }
  },
  "validmind.model_validation.ModelMetadata": {
    "inputs": {
      "model": "model"
    },
    "params": {}
  },
  "validmind.data_validation.DatasetSplit": {
    "inputs": {
      "datasets": "datasets"
    },
    "params": {}
  },
  "validmind.model_validation.sklearn.PopulationStabilityIndex": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "num_bins": 10,
      "mode": "fixed"
    }
  },
  "validmind.model_validation.sklearn.ConfusionMatrix": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "threshold": 0.5
    }
  },
  "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "average": "macro"
    }
  },
  "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "average": "macro"
    }
  },
  "validmind.model_validation.sklearn.PrecisionRecallCurve": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.model_validation.sklearn.ROCCurve": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.model_validation.sklearn.TrainingTestDegradation": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "max_threshold": 0.1
    }
  },
  "validmind.model_validation.sklearn.MinimumAccuracy": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "min_threshold": 0.7
    }
  },
  "validmind.model_validation.sklearn.MinimumF1Score": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "min_threshold": 0.5
    }
  },
  "validmind.model_validation.sklearn.MinimumROCAUCScore": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "min_threshold": 0.5
    }
  },
  "validmind.model_validation.sklearn.PermutationFeatureImportance": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {
      "fontsize": null,
      "figure_height": null
    }
  },
  "validmind.model_validation.sklearn.SHAPGlobalImportance": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {
      "kernel_explainer_samples": 10,
      "tree_or_linear_explainer_samples": 200,
      "class_of_interest": null
    }
  },
  "validmind.model_validation.sklearn.WeakspotsDiagnosis": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "features_columns": null,
      "metrics": null,
      "thresholds": null
    }
  },
  "validmind.model_validation.sklearn.OverfitDiagnosis": {
    "inputs": {
      "model": "model",
      "datasets": "datasets"
    },
    "params": {
      "metric": null,
      "cut_off_threshold": 0.04
    }
  },
  "validmind.model_validation.sklearn.RobustnessDiagnosis": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "metric": null,
      "scaling_factor_std_dev_list": [
        0.1,
        0.2,
        0.3,
        0.4,
        0.5
      ],
      "performance_decay_threshold": 0.05
    }
  }
}

Preview test configuration

The default config does not assign any inputs to a test, but you can assign inputs to individual tests as needed depending on the datasets and records (models) you want to pass to individual tests.

For this particular documentation template (binary classification), the ValidMind Library provides a sample configuration that can be used to populate the entire documentation using the following inputs as placeholders:

• A raw_dataset raw dataset
• A train_dataset training dataset
• A test_dataset test dataset
• A trained model instance

As part of updating the config you will need to ensure the correct input_ids are used in the final config passed to run_documentation_tests().

from validmind.datasets.classification import customer_churn
from validmind.utils import preview_test_config

test_config = customer_churn.get_demo_test_config()
preview_test_config(test_config)

{
    "validmind.data_validation.DatasetDescription": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {}
    },
    "validmind.data_validation.ClassImbalance": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_percent_threshold": 10
        }
    },
    "validmind.data_validation.Duplicates": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_threshold": 1
        }
    },
    "validmind.data_validation.HighCardinality": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "num_threshold": 100,
            "percent_threshold": 0.1,
            "threshold_type": "percent"
        }
    },
    "validmind.data_validation.MissingValues": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_percentage_threshold": 1.0
        }
    },
    "validmind.data_validation.Skewness": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "max_threshold": 1
        }
    },
    "validmind.data_validation.UniqueRows": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_percent_threshold": 1
        }
    },
    "validmind.data_validation.TooManyZeroValues": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "max_percent_threshold": 0.03
        }
    },
    "validmind.data_validation.IQROutliersTable": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "threshold": 1.5
        }
    },
    "validmind.data_validation.IQROutliersBarPlot": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "threshold": 1.5,
            "fig_width": 800
        }
    },
    "validmind.data_validation.DescriptiveStatistics": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {}
    },
    "validmind.data_validation.PearsonCorrelationMatrix": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {}
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "max_threshold": 0.3,
            "top_n_correlations": 10,
            "feature_columns": null
        }
    },
    "validmind.model_validation.ModelMetadata": {
        "inputs": {
            "model": "model"
        },
        "params": {}
    },
    "validmind.data_validation.DatasetSplit": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ]
        },
        "params": {}
    },
    "validmind.model_validation.sklearn.PopulationStabilityIndex": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "num_bins": 10,
            "mode": "fixed"
        }
    },
    "validmind.model_validation.sklearn.ConfusionMatrix": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "threshold": 0.5
        }
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "model": "model",
            "dataset": "train_dataset"
        }
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        }
    },
    "validmind.model_validation.sklearn.PrecisionRecallCurve": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {}
    },
    "validmind.model_validation.sklearn.ROCCurve": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {}
    },
    "validmind.model_validation.sklearn.TrainingTestDegradation": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "max_threshold": 0.1
        }
    },
    "validmind.model_validation.sklearn.MinimumAccuracy": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "min_threshold": 0.7
        }
    },
    "validmind.model_validation.sklearn.MinimumF1Score": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "min_threshold": 0.5
        }
    },
    "validmind.model_validation.sklearn.MinimumROCAUCScore": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "min_threshold": 0.5
        }
    },
    "validmind.model_validation.sklearn.PermutationFeatureImportance": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {
            "fontsize": null,
            "figure_height": null
        }
    },
    "validmind.model_validation.sklearn.SHAPGlobalImportance": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {
            "kernel_explainer_samples": 10,
            "tree_or_linear_explainer_samples": 200,
            "class_of_interest": null
        }
    },
    "validmind.model_validation.sklearn.WeakspotsDiagnosis": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "features_columns": null,
            "metrics": null,
            "thresholds": null
        }
    },
    "validmind.model_validation.sklearn.OverfitDiagnosis": {
        "inputs": {
            "model": "model",
            "datasets": [
                "train_dataset",
                "test_dataset"
            ]
        },
        "params": {
            "metric": null,
            "cut_off_threshold": 0.04
        }
    },
    "validmind.model_validation.sklearn.RobustnessDiagnosis": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "metric": null,
            "scaling_factor_std_dev_list": [
                0.1,
                0.2,
                0.3,
                0.4,
                0.5
            ],
            "performance_decay_threshold": 0.05
        }
    }
}

Run updated documentation section tests

Using this sample configuration, let's finish populating documentation by running all tests for the Model Development section of the documentation.

Recall that the training and test datasets in our exercise have the following input_id values:

• train_dataset_final for the training dataset
• test_dataset_final for the test dataset

config = {
    "validmind.model_validation.ModelMetadata": {
        "inputs": {"model": "log_reg_model_v1"},
    },
    "validmind.data_validation.DatasetSplit": {
        "inputs": {"datasets": ["train_dataset_final", "test_dataset_final"]},
    },
    "validmind.model_validation.sklearn.PopulationStabilityIndex": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {"num_bins": 10, "mode": "fixed"},
    },
    "validmind.model_validation.sklearn.ConfusionMatrix": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "my_test_provider.ConfusionMatrix": {
        "inputs": {"dataset": "test_dataset_final", "model": "log_reg_model_v1"},
    },
    "my_custom_tests.ConfusionMatrix:test_dataset_normalized": {
        "inputs": {"dataset": "test_dataset_final", "model": "log_reg_model_v1"},
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "train_dataset_final"}
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"}
    },
    "validmind.model_validation.sklearn.PrecisionRecallCurve": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "validmind.model_validation.sklearn.ROCCurve": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "validmind.model_validation.sklearn.TrainingTestDegradation": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {
            "metrics": ["accuracy", "precision", "recall", "f1"],
            "max_threshold": 0.1,
        },
    },
    "validmind.model_validation.sklearn.MinimumAccuracy": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"min_threshold": 0.7},
    },
    "validmind.model_validation.sklearn.MinimumF1Score": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"min_threshold": 0.5},
    },
    "validmind.model_validation.sklearn.MinimumROCAUCScore": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"min_threshold": 0.5},
    },
    "validmind.model_validation.sklearn.PermutationFeatureImportance": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "validmind.model_validation.sklearn.SHAPGlobalImportance": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"kernel_explainer_samples": 10},
    },
    "validmind.model_validation.sklearn.WeakspotsDiagnosis": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {
            "thresholds": {"accuracy": 0.75, "precision": 0.5, "recall": 0.5, "f1": 0.7}
        },
    },
    "validmind.model_validation.sklearn.OverfitDiagnosis": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {"cut_off_percentage": 4},
    },
    "validmind.model_validation.sklearn.RobustnessDiagnosis": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {
            "scaling_factor_std_dev_list": [0.0, 0.1, 0.2, 0.3, 0.4, 0.5],
            "accuracy_decay_threshold": 4,
        },
    },
}


full_suite = vm.run_documentation_tests(
    section="model_development",
    config=config,
)

2026-07-14 05:30:43,878 - WARNING(validmind.vm_models.test_suite.runner): Config key 'my_test_provider.ConfusionMatrix' does not match a test_id in the template.
    Ensure you registered a content block with the correct content_id in the template
    The configuration for this test will be ignored.
2026-07-14 05:30:43,879 - WARNING(validmind.vm_models.test_suite.runner): Config key 'my_custom_tests.ConfusionMatrix:test_dataset_normalized' does not match a test_id in the template.
    Ensure you registered a content block with the correct content_id in the template
    The configuration for this test will be ignored.

Test suite complete!

34/34 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Development

Model Development

▶ Test Result: Model Metadata (validmind.model_validation.ModelMetadata)

Model Metadata

The ModelMetadata test compares core metadata fields across models to summarize implementation characteristics, including modeling technique, framework, framework version, and programming language. The result is presented as a single-row summary table for the documented model. The table reports the modeling technique as SKlearnModel, the modeling framework as sklearn, the framework version as 1.9.0, and the programming language as Python.

Key insights:

• Metadata fields are populated: The summary table contains values for all reported metadata fields: modeling technique, modeling framework, framework version, and programming language.
• Single implementation stack identified: The model is documented as SKlearnModel using the sklearn framework.
• Framework and language are explicitly versioned: The table records framework version 1.9.0 and identifies Python as the programming language.

The metadata summary documents a single, fully specified implementation profile for the model. The reported configuration is internally clear at the level of modeling technique, framework, version, and programming language, with no missing fields shown in the result.

Tables

Modeling Technique	Modeling Framework	Framework Version	Programming Language
SKlearnModel	sklearn	1.9.0	Python

▶ Test Result: Dataset Split (validmind.data_validation.DatasetSplit)

Dataset Split

The DatasetSplit test evaluates the distribution of records across the available model datasets. The results summarize the absolute size and share of the total population for each split, showing 2,585 records in train_dataset_final and 647 records in test_dataset_final, for a combined total of 3,232 records. The corresponding proportions are 79.98% for the training dataset and 20.02% for the test dataset.

Key insights:

• Training data comprises most records: train_dataset_final contains 2,585 of 3,232 total records, representing 79.98% of the full dataset.
• Test data accounts for one-fifth: test_dataset_final contains 647 records, corresponding to 20.02% of the total dataset.
• Two-way split is documented: The reported split includes training and test datasets only, with no validation dataset shown in the test output.

The dataset split is concentrated in the training dataset, with approximately four-fifths of records allocated to training and one-fifth allocated to testing. The documented result reflects a two-way partition across 3,232 total observations, and the test output does not include a separate validation split.

Tables

Dataset	Size	Proportion
train_dataset_final	2585	79.98%
test_dataset_final	647	20.02%
Total	3232	100%

▶ Test Result: Population Stability Index (validmind.model_validation.sklearn.PopulationStabilityIndex)

Population Stability Index

The Population Stability Index (PSI) test evaluates the stability of the distribution being compared between the initial and new datasets. The results are presented across 10 fixed bins, showing counts, population shares, and bin-level PSI contributions for train_dataset_final versus test_dataset_final. The table and plot show that population proportions are broadly similar across bins, with total PSI reported as 0.0099 and individual bin contributions ranging from 0.0001 to 0.0028.

Key insights:

• Overall PSI is low: The total PSI is 0.0099, indicating that the aggregate difference between the initial and new distributions is small in magnitude based on the reported calculation.
• Differences are concentrated in a few bins: The largest bin-level PSI contributions occur in Bin 7 (0.0028) and Bin 9 (0.0027), making these the main contributors to the total PSI.
• Most bins show minimal contribution: Seven of the ten bins have PSI contributions of 0.0013 or less, with the smallest contribution in Bin 2 at 0.0001.
• Largest share changes occur in upper bins: Bin 7 decreases from 12.9594% to 11.1283%, and Bin 9 decreases from 7.0406% to 5.7187%, while Bin 3 increases from 13.3462% to 14.6832% and Bin 6 increases from 12.1857% to 13.1376%.

The PSI result shows a high degree of similarity between the initial and new distributions at the aggregate level, with only small bin-level deviations observed. The measured shift is not uniform across bins, as the total PSI is driven primarily by changes in Bins 7 and 9, while the remaining bins contribute only marginally. Overall, the distributional comparison indicates limited movement between train_dataset_final and test_dataset_final within the reported 10-bin fixed-bin framework.

Parameters:

{
  "num_bins": 10,
  "mode": "fixed"
}
            

Tables

Population Stability Index for train_dataset_final and test_dataset_final Datasets

Bin	Count Initial	Percent Initial (%)	Count New	Percent New (%)	PSI
0	99	3.8298	27	4.1731	0.0003
1	189	7.3114	44	6.8006	0.0004
2	308	11.9149	79	12.2102	0.0001
3	345	13.3462	95	14.6832	0.0013
4	372	14.3907	89	13.7558	0.0003
5	355	13.7331	94	14.5286	0.0004
6	315	12.1857	85	13.1376	0.0007
7	335	12.9594	72	11.1283	0.0028
8	85	3.2882	25	3.8640	0.0009
9	182	7.0406	37	5.7187	0.0027
Total	2585	100.0000	647	100.0000	0.0099

Figures

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

The Confusion Matrix test evaluates classification performance by comparing predicted labels with observed labels and summarizing the results across true positives, true negatives, false positives, and false negatives. The heatmap shows four observed outcome counts: 193 true positives, 204 true negatives, 122 false positives, and 128 false negatives. These values describe how predictions were distributed across the two classes and provide a direct view of correct and incorrect classifications for both positive and negative cases.

Key insights:

• Correct classifications exceed errors: The matrix contains 193 true positives and 204 true negatives, compared with 122 false positives and 128 false negatives. In aggregate, correct predictions are more frequent than misclassifications.
• Balanced performance across classes: True positives (193) and true negatives (204) are close in magnitude, indicating similar volumes of correct classification for the positive and negative classes.
• Error types are comparable: False positives and false negatives are also close in count, at 122 and 128 respectively. This indicates that misclassification is distributed relatively evenly between over-predicting and under-predicting the positive class.
• Negative class is classified slightly better: The true negative count exceeds the true positive count by 11 observations, while false positives are 6 fewer than false negatives. This reflects slightly stronger classification results for the negative class than for the positive class.

The confusion matrix indicates that the model produces more correct than incorrect classifications, with comparable performance across both classes. Correct predictions are slightly higher for the negative class, and the two error types occur at similar levels. Overall, the result reflects a relatively balanced classification pattern rather than one dominated by a single type of prediction outcome.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

The Classifier Performance test evaluates classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The in-sample results are reported for both classes and as macro and weighted averages. Class-level precision, recall, and F1 values are shown for classes 0 and 1, alongside overall accuracy of 0.6375 and ROC AUC of 0.6819. The reported macro and weighted averages are identical at 0.6376 for precision and 0.6375 for recall and F1.

Key insights:

• Balanced class-level performance: Precision, recall, and F1 are closely aligned across the two classes. Class 0 records precision/recall/F1 of 0.6348/0.6442/0.6395, while class 1 records 0.6403/0.6309/0.6356.
• Similar aggregate averages: Macro average and weighted average are identical across all reported metrics, with precision of 0.6376 and recall and F1 of 0.6375.
• Accuracy matches overall F1 scale: Overall accuracy is 0.6375, which is effectively the same as the reported weighted and macro F1 values of 0.6375.
• ROC AUC exceeds accuracy level: ROC AUC is reported at 0.6819, above the observed accuracy of 0.6375.

The in-sample performance results show a consistent metric profile across both classes and across aggregate summary measures. Class-specific precision, recall, and F1 values are closely grouped, with no large divergence between classes 0 and 1. Overall accuracy, macro F1, and weighted F1 are numerically aligned at approximately 0.64, while ROC AUC is higher at 0.6819.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6348	0.6442	0.6395
1	0.6403	0.6309	0.6356
Weighted Average	0.6376	0.6375	0.6375
Macro Average	0.6376	0.6375	0.6375

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6375
ROC AUC	0.6819

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

The Classifier Performance test evaluates classification model performance using precision, recall, F1-score, accuracy, and ROC AUC. The out-of-sample results are presented by class for precision, recall, and F1, along with macro and weighted averages, and separate summary values for accuracy and ROC AUC. Class-level precision ranges from 0.6127 to 0.6145, recall ranges from 0.6012 to 0.6258, and F1 ranges from 0.6069 to 0.6201. Overall, the weighted and macro averages are both approximately 0.6135 to 0.6136, with accuracy of 0.6136 and ROC AUC of 0.6627.

Key insights:

• Balanced class-level precision: Precision is nearly identical across the two classes, at 0.6145 for Class 0 and 0.6127 for Class 1, indicating limited disparity in positive prediction accuracy across classes.
• Higher recall for Class 0: Recall is 0.6258 for Class 0 versus 0.6012 for Class 1, a gap of 0.0246, showing that Class 0 is identified more frequently than Class 1 in the out-of-sample results.
• Closely aligned aggregate averages: Macro-average and weighted-average precision, recall, and F1 are all 0.6135 to 0.6136, indicating little difference between unweighted and support-weighted summary performance.
• ROC AUC exceeds accuracy: ROC AUC is 0.6627 compared with accuracy of 0.6136, showing stronger rank-order discrimination than the single-threshold classification summary.

The out-of-sample performance results show a tightly clustered metric profile across both classes and across aggregate summaries. Precision, recall, and F1 are all near 0.61, with only modest class-level variation driven primarily by lower recall for Class 1. The near equality of macro and weighted averages indicates consistent aggregate performance across classes, while the ROC AUC of 0.6627 reflects discrimination that is stronger than the reported accuracy at the applied classification threshold.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6145	0.6258	0.6201
1	0.6127	0.6012	0.6069
Weighted Average	0.6136	0.6136	0.6135
Macro Average	0.6136	0.6135	0.6135

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6136
ROC AUC	0.6627

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

The PrecisionRecallCurve test evaluates the trade-off between precision and recall for this binary classification model across classification thresholds. The result is presented as a precision-recall curve with recall on the x-axis and precision on the y-axis. The curve begins with very high precision at near-zero recall, then moves through a fluctuating early segment, reaches a local high around moderate-low recall, and gradually declines as recall increases toward 1.0.

Key insights:

• Precision declines as recall rises: Precision is highest at very low recall and trends downward overall as recall increases, ending near 0.50 at recall close to 1.0.
• Early-threshold region is volatile: In the low-recall range, the curve shows sharp movement, with precision dropping quickly from 1.0 and fluctuating in the low-to-mid 0.60s before stabilizing.
• Best observed mid-range precision occurs early: The curve reaches a local peak around recall near 0.18–0.20, where precision is approximately 0.75–0.76.
• Mid-recall performance is comparatively stable: From roughly 0.30 to 0.55 recall, precision remains within a relatively narrow band around 0.65–0.67 before declining further beyond that range.

The curve shows a clear trade-off between capturing more positive cases and maintaining prediction precision. Performance is strongest at very low recall and at an early local peak near 0.20 recall, while the middle portion of the curve remains comparatively stable in the mid-0.60 precision range. At higher recall levels, precision declines progressively, with the curve approaching 0.50 near full recall.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

The ROCCurve test evaluates binary classification performance by plotting the tradeoff between true positive rate and false positive rate across classification thresholds and by calculating the area under the ROC curve (AUC). The figure shows the ROC curve for log_reg_model_v1 on test_dataset_final together with the random-classification reference line. The reported AUC is 0.66, and the ROC curve remains above the diagonal baseline across most of the false positive rate range.

Key insights:

• AUC indicates above-random discrimination: The ROC curve reports an AUC of 0.66, which is higher than the random benchmark of 0.5 shown in the figure.
• ROC curve stays above baseline: The plotted curve lies above the dashed random reference line for most threshold levels, indicating stronger ranking performance than random classification across the evaluated range.
• Discrimination varies across thresholds: The curve rises gradually rather than sharply, showing that gains in true positive rate are accompanied by increasing false positive rates across much of the threshold range.

The ROC results show that log_reg_model_v1 demonstrates measurable discriminative ability on test_dataset_final, with performance above the random baseline as reflected by the AUC of 0.66 and the ROC curve’s position above the diagonal reference. At the same time, the gradual shape of the curve indicates that class separation is not uniform across thresholds, with improvements in sensitivity occurring alongside higher false positive rates.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

The TrainingTestDegradation test evaluates whether performance degradation between the training and test datasets remains below the defined threshold for key classification metrics. The results table compares training and test precision, recall, and F1-score for classes 1 and 0, and reports the corresponding degradation percentages against a maximum threshold of 10%. Across all six metric evaluations, degradation values fall between 2.8593% and 4.6984%, with each result marked as Pass.

Key insights:

• All metrics passed threshold: Every evaluated metric for both classes is below the 10% maximum degradation threshold. No Pass/Fail entry is marked as Fail.

• Higher degradation for class 1: Class 1 shows degradation values of 4.3081% for precision, 4.6984% for recall, and 4.5051% for F1-score. These values are consistently higher than the corresponding class 0 degradations.

• Class 0 shows smaller train-test gaps: For class 0, degradation is 3.21% for precision, 2.8593% for recall, and 3.0362% for F1-score. This indicates narrower differences between training and test performance for this class.

• Recall has the largest class-level degradation: The largest degradation reported is class 1 recall at 4.6984%. The smallest degradation is class 0 recall at 2.8593%.

The test results show limited performance decline from training to test data across all evaluated classification metrics. Degradation remains below the 10% threshold in every case, with somewhat larger reductions observed for class 1 than for class 0. Overall, the train-test performance differences are modest and consistently within the configured acceptance limit.

Parameters:

{
  "max_threshold": 0.1
}
            

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
1	Precision	0.6403	0.6127	4.3081	Pass
1	Recall	0.6309	0.6012	4.6984	Pass
1	F1-Score	0.6356	0.6069	4.5051	Pass
0	Precision	0.6348	0.6145	3.2100	Pass
0	Recall	0.6442	0.6258	2.8593	Pass
0	F1-Score	0.6395	0.6201	3.0362	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

The Minimum Accuracy test evaluates whether the model’s classification accuracy meets or exceeds a specified minimum threshold. The result table reports the observed accuracy score, the threshold applied in the test, and the resulting pass/fail outcome. In this test run, the model achieved an accuracy score of 0.6136 against a threshold of 0.7000, and the test result is recorded as Fail.

Key insights:

• Accuracy is below threshold: The observed accuracy score is 0.6136, which is lower than the configured minimum threshold of 0.7000.
• Test outcome is fail: Because the reported score does not meet the threshold, the test result is marked as Fail.
• Shortfall is measurable: The difference between the observed score and the threshold is 0.0864.

The result shows that the model did not satisfy the minimum accuracy criterion defined for this test run. The reported failure is directly attributable to the observed accuracy score of 0.6136 falling below the 0.7000 threshold, with a shortfall of 0.0864.

Parameters:

{
  "min_threshold": 0.7
}
            

Tables

Score	Threshold	Pass/Fail
0.6136	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

The MinimumF1Score test evaluates whether the model’s F1 score on the validation dataset meets a predefined minimum threshold. The result table reports a validation F1 score of 0.6069 alongside a minimum threshold of 0.5, with an overall test outcome of Pass. These values show the measured score and the benchmark used for assessment in this test.

Key insights:

• Threshold exceeded on validation: The observed validation F1 score is 0.6069, which is above the configured minimum threshold of 0.5.
• Positive test outcome recorded: The result is marked as Pass, consistent with the score exceeding the defined threshold.
• Observed margin above threshold: The validation F1 score exceeds the threshold by 0.1069, indicating separation between the measured result and the minimum acceptance level.

The test result shows that the model’s validation F1 score met the predefined minimum criterion. The recorded score of 0.6069 was above the 0.5 threshold, and the test outcome was therefore reported as Pass.

Parameters:

{
  "min_threshold": 0.5
}
            

Tables

Score	Threshold	Pass/Fail
0.6069	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

The Minimum ROC AUC Score test evaluates whether the model’s ROC AUC on the validation dataset meets or exceeds the specified minimum threshold. The result table reports a ROC AUC score of 0.6627, alongside the configured threshold of 0.5 and the corresponding pass/fail status. These values summarize the outcome of the threshold-based validation check for this performance metric.

Key insights:

• Threshold exceeded: The observed ROC AUC score is 0.6627 compared with a minimum threshold of 0.5, placing the result 0.1627 above the required cutoff.
• Test passed: The recorded outcome is "Pass," indicating that the validation result satisfied the threshold condition defined for this test.
• Moderate discriminative score observed: The reported ROC AUC value of 0.6627 indicates the level of class discrimination captured by this test result on the validation dataset.

The test result shows that the model met the minimum ROC AUC requirement under the configured threshold test. The observed score of 0.6627 exceeded the 0.5 cutoff, and the test was recorded as a pass. Collectively, these results indicate that the model satisfied this specific validation criterion based on ROC AUC.

Parameters:

{
  "min_threshold": 0.5
}
            

Tables

Score	Threshold	Pass/Fail
0.6627	0.5	Pass

▶ Test Result: Permutation Feature Importance (validmind.model_validation.sklearn.PermutationFeatureImportance)

Permutation Feature Importance

The Permutation Feature Importance test evaluates the significance of each input feature by measuring the change in model performance after that feature is randomly permuted. The results are presented as a horizontal bar chart of permutation importance values for the model inputs. The chart shows a small set of features with clearly positive importance values, several features with values near zero, and multiple features with negative importance values. The largest positive values are observed for IsActiveMember, Geography_Germany, and Gender_Male.

Key insights:

• IsActiveMember has the highest importance: IsActiveMember shows the largest positive permutation importance in the chart, at slightly above 0.04, indicating the greatest measured performance decrease when this feature is permuted.

• Two additional features contribute materially: Geography_Germany and Gender_Male also display strong positive importance values, both in the mid-0.03 range and below IsActiveMember.

• Geography_Spain has limited positive effect: Geography_Spain has a small positive importance value close to zero relative to the leading features, indicating a much smaller measured contribution.

• Several features are near zero or negative: Tenure is effectively at zero, while Balance, EstimatedSalary, HasCrCard, NumOfProducts, and CreditScore show negative permutation importance values, with CreditScore appearing to be the most negative.

The permutation importance results show that model performance is most sensitive to IsActiveMember, with Geography_Germany and Gender_Male also contributing materially. Geography_Spain contributes only marginally, while Tenure has negligible measured impact in this test. The remaining features exhibit negative importance values, indicating that permuting them did not reduce performance in the same way as the leading variables and, in several cases, was associated with slightly improved measured performance.

Figures

▶ Test Result: SHAP Global Importance (validmind.model_validation.sklearn.SHAPGlobalImportance)

SHAP Global Importance

The SHAPGlobalImportance test evaluates global feature importance by measuring the magnitude and direction of feature contributions to model output using SHAP values. The results include a normalized feature importance bar chart and a SHAP summary plot, which together show the relative ranking of input variables and the distribution of their instance-level impacts. The bar chart ranks features from highest to lowest normalized mean absolute SHAP value, while the summary plot shows whether higher or lower feature values are associated with positive or negative contributions to the model output.

Key insights:

• Importance is concentrated in three features: IsActiveMember, Geography_Germany, and Gender_Male have the highest normalized SHAP importance, at approximately 100, 92, and 78 respectively, and are separated from the remaining features by a visible drop in importance.
• Balance is the leading continuous driver: Balance is the next most important feature at approximately 42 normalized importance, notably above the rest of the lower-ranked variables such as NumOfProducts, HasCrCard, Tenure, Geography_Spain, CreditScore, and EstimatedSalary.
• Binary features show directional separation: In the summary plot, IsActiveMember, Geography_Germany, and Gender_Male each display two clearly separated SHAP clusters, indicating distinct contribution patterns associated with their lower and higher feature values.
• Higher balance aligns with positive SHAP values: For Balance, higher feature values are concentrated on the positive side of the SHAP axis, while lower values appear more often on the negative side, indicating a directional relationship between larger balances and higher model output.
• Lower-ranked features have limited contribution range: CreditScore and EstimatedSalary show SHAP values tightly concentrated around zero, and their normalized importance is among the smallest in the ranking, indicating comparatively limited influence on model output.
• Tenure effects remain near neutral: Tenure exhibits a relatively narrow spread of SHAP values around zero compared with the top-ranked variables, indicating modest instance-level effect size despite some variation across feature values.

The SHAP results show a model whose global behavior is driven primarily by a small set of features, with IsActiveMember, Geography_Germany, and Gender_Male accounting for the largest contributions and Balance standing out as the most influential continuous variable. The summary plot indicates clear directional patterns for several top-ranked features, particularly the positive association of higher Balance values with model output. The remaining variables contribute over narrower SHAP ranges and at materially lower normalized importance levels, with EstimatedSalary and CreditScore having the smallest observed influence.

Parameters:

{
  "kernel_explainer_samples": 10
}
            

Figures

▶ Test Result: Weakspots Diagnosis (validmind.model_validation.sklearn.WeakspotsDiagnosis)

❌ Weakspots Diagnosis

The WeakspotsDiagnosis test evaluates model performance across slices of the feature space using threshold-based comparisons for accuracy, precision, recall, and F1. The results are reported for training and test datasets across bins of Balance, CreditScore, EstimatedSalary, Tenure, and several binary indicators, using thresholds of 0.75 for accuracy, 0.5 for precision and recall, and 0.7 for F1. The tables and plots show which feature slices fall above or below each threshold and where train and test performance differ within the same slice. Several slices also have very small record counts, including bins with 0, 4, 6, 8, and 16 test records.

Key insights:

• Accuracy is below threshold throughout: Test accuracy does not reach the 0.75 threshold in any reported slice. The highest observed test accuracy values are 0.7213 for EstimatedSalary (139982.992, 159978.908], 0.7164 for Tenure (3.0, 4.0], and 0.6857 for CreditScore (800.0, 850.0].

• Low-balance slices are a major weak region: For Balance, the lowest three test slices show weak performance across multiple metrics: F1 values of 0.3971, 0.0, and 0.2222; recall values of 0.3103, 0.0, and 0.2; and precision values of 0.551, 0.0, and 0.25. The slice (23838.756, 47677.512] contains 4 test records and records 0.0 for all four metrics.

• NumOfProducts has the most pronounced slice variation: Test results differ materially by product count. For NumOfProducts, the slice (1.9, 2.2] has precision 0.3014, recall 0.4231, and F1 0.352, while the slice (3.7, 4.0] has accuracy 0.25, recall 0.25, and F1 0.4 on 8 test records. By contrast, the slice (2.8, 3.1] records test precision 0.96 and F1 0.7385.

• Inactive-member status separates recall performance: For IsActiveMember, the test slice (-0.001, 0.1] has recall 0.78 and F1 0.6933, while the slice (0.9, 1.0] has recall 0.3058 and F1 0.3978. This creates a large difference across the two slices despite similar accuracy values of 0.6012 and 0.6279.

• Germany slice shows asymmetric performance: For Geography_Germany, the non-Germany slice (-0.001, 0.1] has test recall 0.4184 and F1 0.4767, below the corresponding thresholds, while the Germany slice (0.9, 1.0] has recall 0.888 and F1 0.7603. This indicates substantially different performance across the two geographic slices.

• CreditScore weakness is concentrated in specific bands: CreditScore performance is uneven across bins. The test slice (450.0, 500.0] has accuracy 0.4815, precision 0.3846, recall 0.4545, and F1 0.4167, while the slice (700.0, 750.0] has precision 0.4773 and F1 0.4941. Only the test slices (500.0, 550.0] and (550.0, 600.0] meet or exceed the F1 threshold at 0.7013 and 0.5814 respectively, with only the former exceeding 0.7.

• EstimatedSalary has mixed but not monotonic performance: Across EstimatedSalary bins, test F1 ranges from 0.5075 to 0.7213 and recall ranges from 0.4359 to 0.7619. The strongest test F1 is 0.7213 in (139982.992, 159978.908], while weaker slices include (159978.908, 179974.824] with recall 0.4359 and F1 0.5075, and (79995.244, 99991.16] with accuracy 0.5185 and F1 0.5517.

• Several weak spots coincide with sparse test support: Multiple extreme or weak slices have limited test observations, including Balance bins with 0, 4, 6, and 16 records; NumOfProducts (3.7, 4.0] with 8 records; and CreditScore (349.5, 400.0] with 4 records. These slices show some of the most extreme metric values, including perfect or zero-valued metrics in isolated cases.

The weak spot analysis shows that performance shortfalls are not uniform across the feature space. Accuracy remains below the configured threshold in all reported slices, while the most pronounced weaknesses appear in low-balance segments, selected CreditScore bands, specific NumOfProducts groups, the active-member slice, and the non-Germany geography slice. The results also show that some of the largest metric swings occur in sparsely populated bins, whereas several broader slices with larger record counts still remain below threshold across F1, recall, or accuracy.

Parameters:

{
  "thresholds": {
    "accuracy": 0.75,
    "precision": 0.5,
    "recall": 0.5,
    "f1": 0.7
  }
}
            

Tables

Slice	Number of Records	Feature	Accuracy	Precision	Recall	F1	Dataset
(-238.388, 23838.756]	210	Balance	0.6095	0.5510	0.3103	0.3971	test_dataset_final
(23838.756, 47677.512]	4	Balance	0.0000	0.0000	0.0000	0.0000	test_dataset_final
(47677.512, 71516.268]	16	Balance	0.5625	0.2500	0.2000	0.2222	test_dataset_final
(71516.268, 95355.024]	54	Balance	0.5926	0.6400	0.5517	0.5926	test_dataset_final
(95355.024, 119193.78]	130	Balance	0.6308	0.6364	0.7778	0.7000	test_dataset_final
(119193.78, 143032.536]	147	Balance	0.6190	0.6154	0.7273	0.6667	test_dataset_final
(143032.536, 166871.292]	63	Balance	0.6667	0.6429	0.8182	0.7200	test_dataset_final
(166871.292, 190710.048]	16	Balance	0.6250	0.5833	0.8750	0.7000	test_dataset_final
(190710.048, 214548.804]	6	Balance	0.5000	0.7500	0.6000	0.6667	test_dataset_final
(214548.804, 238387.56]	0	Balance	0.0000	0.0000	0.0000	0.0000	test_dataset_final
(-238.388, 23838.756]	824	Balance	0.6517	0.5692	0.3535	0.4361	train_dataset_final
(23838.756, 47677.512]	20	Balance	0.6000	0.7143	0.4545	0.5556	train_dataset_final
(47677.512, 71516.268]	75	Balance	0.4933	0.5882	0.4545	0.5128	train_dataset_final
(71516.268, 95355.024]	228	Balance	0.5614	0.5667	0.5862	0.5763	train_dataset_final
(95355.024, 119193.78]	543	Balance	0.6980	0.7207	0.7717	0.7453	train_dataset_final
(119193.78, 143032.536]	510	Balance	0.6431	0.6657	0.7611	0.7102	train_dataset_final
(143032.536, 166871.292]	273	Balance	0.6117	0.5896	0.7445	0.6581	train_dataset_final
(166871.292, 190710.048]	91	Balance	0.5275	0.5625	0.7059	0.6261	train_dataset_final
(190710.048, 214548.804]	18	Balance	0.5000	0.7500	0.6000	0.6667	train_dataset_final
(214548.804, 238387.56]	3	Balance	1.0000	1.0000	1.0000	1.0000	train_dataset_final
(349.5, 400.0]	4	CreditScore	0.5000	1.0000	0.5000	0.6667	test_dataset_final
(400.0, 450.0]	16	CreditScore	0.6250	0.8571	0.5455	0.6667	test_dataset_final
(450.0, 500.0]	27	CreditScore	0.4815	0.3846	0.4545	0.4167	test_dataset_final
(500.0, 550.0]	73	CreditScore	0.6849	0.7500	0.6585	0.7013	test_dataset_final
(550.0, 600.0]	86	CreditScore	0.5814	0.5952	0.5682	0.5814	test_dataset_final
(600.0, 650.0]	132	CreditScore	0.6212	0.5938	0.6129	0.6032	test_dataset_final
(650.0, 700.0]	127	CreditScore	0.6299	0.6406	0.6308	0.6357	test_dataset_final
(700.0, 750.0]	97	CreditScore	0.5567	0.4773	0.5122	0.4941	test_dataset_final
(750.0, 800.0]	50	CreditScore	0.6400	0.5714	0.7273	0.6400	test_dataset_final
(800.0, 850.0]	35	CreditScore	0.6857	0.8000	0.6000	0.6857	test_dataset_final
(349.5, 400.0]	10	CreditScore	0.7000	1.0000	0.7000	0.8235	train_dataset_final
(400.0, 450.0]	48	CreditScore	0.5833	0.5556	0.6522	0.6000	train_dataset_final
(450.0, 500.0]	118	CreditScore	0.6949	0.7015	0.7460	0.7231	train_dataset_final
(500.0, 550.0]	261	CreditScore	0.6552	0.6692	0.6593	0.6642	train_dataset_final
(550.0, 600.0]	371	CreditScore	0.6334	0.6400	0.6667	0.6531	train_dataset_final
(600.0, 650.0]	466	CreditScore	0.6094	0.6130	0.6026	0.6078	train_dataset_final
(650.0, 700.0]	504	CreditScore	0.6270	0.5909	0.6164	0.6034	train_dataset_final
(700.0, 750.0]	396	CreditScore	0.6490	0.6740	0.6040	0.6371	train_dataset_final
(750.0, 800.0]	256	CreditScore	0.6602	0.6508	0.6560	0.6534	train_dataset_final
(800.0, 850.0]	155	CreditScore	0.6387	0.6825	0.5443	0.6056	train_dataset_final
(-188.379, 20007.496]	61	EstimatedSalary	0.5902	0.6452	0.5882	0.6154	test_dataset_final
(20007.496, 40003.412]	52	EstimatedSalary	0.6538	0.5517	0.7619	0.6400	test_dataset_final
(40003.412, 59999.328]	58	EstimatedSalary	0.6379	0.5652	0.5417	0.5532	test_dataset_final
(59999.328, 79995.244]	86	EstimatedSalary	0.6395	0.6591	0.6444	0.6517	test_dataset_final
(79995.244, 99991.16]	54	EstimatedSalary	0.5185	0.5517	0.5517	0.5517	test_dataset_final
(99991.16, 119987.076]	72	EstimatedSalary	0.5972	0.5610	0.6765	0.6133	test_dataset_final
(119987.076, 139982.992]	58	EstimatedSalary	0.6379	0.6957	0.5333	0.6038	test_dataset_final
(139982.992, 159978.908]	61	EstimatedSalary	0.7213	0.7333	0.7097	0.7213	test_dataset_final
(159978.908, 179974.824]	71	EstimatedSalary	0.5352	0.6071	0.4359	0.5075	test_dataset_final
(179974.824, 199970.74]	74	EstimatedSalary	0.6081	0.5676	0.6176	0.5915	test_dataset_final
(-188.379, 20007.496]	259	EstimatedSalary	0.6718	0.6818	0.6767	0.6792	train_dataset_final
(20007.496, 40003.412]	259	EstimatedSalary	0.6255	0.6585	0.5956	0.6255	train_dataset_final
(40003.412, 59999.328]	275	EstimatedSalary	0.6764	0.6532	0.6378	0.6454	train_dataset_final
(59999.328, 79995.244]	271	EstimatedSalary	0.6236	0.6129	0.5846	0.5984	train_dataset_final
(79995.244, 99991.16]	255	EstimatedSalary	0.6392	0.6212	0.6613	0.6406	train_dataset_final
(99991.16, 119987.076]	265	EstimatedSalary	0.6075	0.6034	0.5469	0.5738	train_dataset_final
(119987.076, 139982.992]	246	EstimatedSalary	0.6220	0.6429	0.6767	0.6593	train_dataset_final
(139982.992, 159978.908]	253	EstimatedSalary	0.5968	0.5714	0.6281	0.5984	train_dataset_final
(159978.908, 179974.824]	259	EstimatedSalary	0.6795	0.7244	0.6571	0.6891	train_dataset_final
(179974.824, 199970.74]	243	EstimatedSalary	0.6296	0.6320	0.6423	0.6371	train_dataset_final
(-0.001, 0.1]	299	Gender_Male	0.5953	0.6048	0.7697	0.6773	test_dataset_final
(0.9, 1.0]	348	Gender_Male	0.6293	0.6286	0.4231	0.5057	test_dataset_final
(-0.001, 0.1]	1275	Gender_Male	0.6329	0.6473	0.7907	0.7118	train_dataset_final
(0.9, 1.0]	1310	Gender_Male	0.6420	0.6240	0.4238	0.5048	train_dataset_final
(-0.001, 0.1]	451	Geography_Germany	0.6009	0.5541	0.4184	0.4767	test_dataset_final
(0.9, 1.0]	196	Geography_Germany	0.6429	0.6647	0.8880	0.7603	test_dataset_final
(-0.001, 0.1]	1769	Geography_Germany	0.6190	0.5793	0.4386	0.4993	train_dataset_final
(0.9, 1.0]	816	Geography_Germany	0.6777	0.6911	0.9093	0.7853	train_dataset_final
(-0.001, 0.1]	504	Geography_Spain	0.6190	0.6178	0.6324	0.6250	test_dataset_final
(0.9, 1.0]	143	Geography_Spain	0.5944	0.5893	0.4853	0.5323	test_dataset_final
(-0.001, 0.1]	2001	Geography_Spain	0.6432	0.6506	0.6670	0.6587	train_dataset_final
(0.9, 1.0]	584	Geography_Spain	0.6182	0.5899	0.4885	0.5344	train_dataset_final
(-0.001, 0.1]	225	HasCrCard	0.5956	0.5299	0.6327	0.5767	test_dataset_final
(0.9, 1.0]	422	HasCrCard	0.6232	0.6616	0.5874	0.6223	test_dataset_final
(-0.001, 0.1]	763	HasCrCard	0.6042	0.6152	0.6338	0.6244	train_dataset_final
(0.9, 1.0]	1822	HasCrCard	0.6515	0.6521	0.6296	0.6406	train_dataset_final
(-0.001, 0.1]	346	IsActiveMember	0.6012	0.6240	0.7800	0.6933	test_dataset_final
(0.9, 1.0]	301	IsActiveMember	0.6279	0.5692	0.3058	0.3978	test_dataset_final
(-0.001, 0.1]	1402	IsActiveMember	0.6255	0.6450	0.7985	0.7136	train_dataset_final
(0.9, 1.0]	1183	IsActiveMember	0.6517	0.6221	0.3424	0.4417	train_dataset_final
(0.997, 1.3]	372	NumOfProducts	0.6075	0.6744	0.6561	0.6651	test_dataset_final
(1.9, 2.2]	225	NumOfProducts	0.6400	0.3014	0.4231	0.3520	test_dataset_final
(2.8, 3.1]	42	NumOfProducts	0.5952	0.9600	0.6000	0.7385	test_dataset_final
(3.7, 4.0]	8	NumOfProducts	0.2500	1.0000	0.2500	0.4000	test_dataset_final
(0.997, 1.3]	1486	NumOfProducts	0.6258	0.7009	0.6667	0.6834	train_dataset_final
(1.9, 2.2]	911	NumOfProducts	0.6828	0.3952	0.6027	0.4774	train_dataset_final
(2.8, 3.1]	152	NumOfProducts	0.5263	0.9857	0.4929	0.6571	train_dataset_final
(3.7, 4.0]	36	NumOfProducts	0.4444	1.0000	0.4444	0.6154	train_dataset_final
(-0.01, 1.0]	102	Tenure	0.6176	0.6739	0.5636	0.6139	test_dataset_final
(1.0, 2.0]	57	Tenure	0.5965	0.6071	0.5862	0.5965	test_dataset_final
(2.0, 3.0]	74	Tenure	0.5541	0.5814	0.6250	0.6024	test_dataset_final
(3.0, 4.0]	67	Tenure	0.7164	0.7188	0.6970	0.7077	test_dataset_final
(4.0, 5.0]	58	Tenure	0.6897	0.5833	0.6364	0.6087	test_dataset_final
(5.0, 6.0]	60	Tenure	0.5333	0.5714	0.5000	0.5333	test_dataset_final
(6.0, 7.0]	61	Tenure	0.5902	0.5556	0.5357	0.5455	test_dataset_final
(7.0, 8.0]	74	Tenure	0.6216	0.5833	0.6176	0.6000	test_dataset_final
(8.0, 9.0]	73	Tenure	0.6027	0.5750	0.6571	0.6133	test_dataset_final
(9.0, 10.0]	21	Tenure	0.6190	0.7273	0.6154	0.6667	test_dataset_final
(-0.01, 1.0]	376	Tenure	0.6436	0.6465	0.7056	0.6748	train_dataset_final
(1.0, 2.0]	257	Tenure	0.6187	0.6183	0.6279	0.6231	train_dataset_final
(2.0, 3.0]	259	Tenure	0.6023	0.6183	0.6045	0.6113	train_dataset_final
(3.0, 4.0]	264	Tenure	0.6023	0.6320	0.5725	0.6008	train_dataset_final
(4.0, 5.0]	261	Tenure	0.6475	0.6643	0.6739	0.6691	train_dataset_final
(5.0, 6.0]	260	Tenure	0.6308	0.6032	0.6230	0.6129	train_dataset_final
(6.0, 7.0]	248	Tenure	0.6452	0.5963	0.5963	0.5963	train_dataset_final
(7.0, 8.0]	262	Tenure	0.6794	0.6909	0.6032	0.6441	train_dataset_final
(8.0, 9.0]	269	Tenure	0.6691	0.6562	0.6512	0.6537	train_dataset_final
(9.0, 10.0]	129	Tenure	0.6279	0.7049	0.5890	0.6418	train_dataset_final

Figures

▶ Test Result: Overfit Diagnosis (validmind.model_validation.sklearn.OverfitDiagnosis)

Overfit Diagnosis

The Overfit Diagnosis test evaluates differences between training and test discrimination across feature-based slices to identify regions with elevated train-test AUC gaps. Results are reported by feature, slice, sample counts, training AUC, test AUC, and the resulting gap, with a cutoff threshold of 0.04 shown in the plots. The table identifies flagged slices across CreditScore, Tenure, Balance, NumOfProducts, EstimatedSalary, and Gender_Male, with gap magnitudes ranging from 0.0443 to 0.6465. The accompanying plots show that some features contain multiple above-threshold slices, while others show isolated gaps.

Key insights:

• Largest gap occurs in Balance: The highest reported gap is for Balance in the slice (23838.756, 47677.512], where training AUC is 0.6465 and test AUC is 0.0, producing a gap of 0.6465. This slice is based on 20 training records and 4 test records.

• Tenure shows repeated above-threshold slices: Tenure has six flagged slices: (-0.01, 1.0] (0.0481), (2.0, 3.0] (0.1111), (4.0, 5.0] (0.0589), (5.0, 6.0] (0.0934), (8.0, 9.0] (0.0546), and (9.0, 10.0] (0.0458). The largest Tenure gap is in (2.0, 3.0], with training AUC 0.6405 and test AUC 0.5294.

• CreditScore gaps are concentrated in specific bands: Three CreditScore slices exceed the threshold: (450.0, 500.0] with gap 0.2149, (550.0, 600.0] with gap 0.0443, and (700.0, 750.0] with gap 0.0850. The (450.0, 500.0] band has the largest CreditScore gap, with training AUC 0.7149 versus test AUC 0.5.

• EstimatedSalary contains several flagged regions: Five EstimatedSalary slices exceed the threshold: (40003.412, 59999.328] (0.0519), (79995.244, 99991.16] (0.0681), (99991.16, 119987.076] (0.0475), (159978.908, 179974.824] (0.1164), and (179974.824, 199970.74] (0.0473). The largest gap appears in (159978.908, 179974.824], where training AUC is 0.7262 and test AUC is 0.6098.

• NumOfProducts shows elevated gaps in two segments: NumOfProducts has two flagged slices: (1.9, 2.2] with gap 0.0859 and (2.8, 3.1] with gap 0.1083. The latter has the larger gap, with training AUC 0.7333 and test AUC 0.625.

• A broad binary segment is also flagged: Gender_Male includes one above-threshold slice, (-0.001, 0.1], with 1,275 training records and 299 test records. In this slice, training AUC is 0.6585 and test AUC is 0.6001, for a gap of 0.0584.

The results show that train-test AUC gaps above the 0.04 threshold are concentrated in a subset of feature regions rather than uniformly across all segments. The most pronounced gap is observed in a narrow Balance slice with very small sample counts, while repeated above-threshold gaps appear across multiple slices of Tenure, EstimatedSalary, and CreditScore. NumOfProducts and Gender_Male also contain flagged regions, but with fewer affected slices. Overall, the diagnosis indicates localized differences in discrimination between training and test data across several feature-defined segments.

Tables

Overfit Diagnosis

Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
CreditScore	(450.0, 500.0]	118	27	0.7149	0.5000	0.2149
CreditScore	(550.0, 600.0]	371	86	0.6752	0.6310	0.0443
CreditScore	(700.0, 750.0]	396	97	0.6991	0.6141	0.0850
Tenure	(-0.01, 1.0]	376	102	0.6987	0.6507	0.0481
Tenure	(2.0, 3.0]	259	74	0.6405	0.5294	0.1111
Tenure	(4.0, 5.0]	261	58	0.7015	0.6427	0.0589
Tenure	(5.0, 6.0]	260	60	0.6838	0.5904	0.0934
Tenure	(8.0, 9.0]	269	73	0.7028	0.6481	0.0546
Tenure	(9.0, 10.0]	129	21	0.6901	0.6442	0.0458
Balance	(23838.756, 47677.512]	20	4	0.6465	0.0000	0.6465
Balance	(95355.024, 119193.78]	543	130	0.7282	0.6722	0.0560
NumOfProducts	(1.9, 2.2]	911	225	0.7091	0.6232	0.0859
NumOfProducts	(2.8, 3.1]	152	42	0.7333	0.6250	0.1083
EstimatedSalary	(40003.412, 59999.328]	275	58	0.7075	0.6556	0.0519
EstimatedSalary	(79995.244, 99991.16]	255	54	0.6777	0.6097	0.0681
EstimatedSalary	(99991.16, 119987.076]	265	72	0.6504	0.6029	0.0475
EstimatedSalary	(159978.908, 179974.824]	259	71	0.7262	0.6098	0.1164
EstimatedSalary	(179974.824, 199970.74]	243	74	0.6848	0.6375	0.0473
Gender_Male	(-0.001, 0.1]	1275	299	0.6585	0.6001	0.0584

Figures

▶ Test Result: Robustness Diagnosis (validmind.model_validation.sklearn.RobustnessDiagnosis)

✅ Robustness Diagnosis

The Robustness Diagnosis test evaluates model robustness by measuring AUC under increasing Gaussian noise applied to numeric input features. Results are reported for train_dataset_final and test_dataset_final across perturbation sizes from 0.0 to 0.5 standard deviations, with corresponding performance decay values relative to baseline. Baseline AUC is 0.6819 on the training dataset and 0.6627 on the test dataset, and the table and plot show how these values change as perturbation size increases. All evaluated perturbation levels are marked as passed for both datasets.

Key insights:

• Training AUC declines gradually: Training AUC decreases from 0.6819 at baseline to 0.6636 at perturbation size 0.5, corresponding to a maximum recorded performance decay of 0.0183. The decline is monotonic across the tested perturbation levels.

• Test performance is non-monotonic: Test AUC changes from 0.6627 at baseline to 0.6616 at perturbation size 0.5, with intermediate values of 0.6617, 0.6580, 0.6631, and 0.6419. This includes a small improvement at perturbation size 0.3 relative to baseline, reflected by a performance decay of -0.0004.

• Largest test degradation occurs at 0.4: The lowest observed test AUC is 0.6419 at perturbation size 0.4, which corresponds to the largest test performance decay of 0.0208. This is the most pronounced reduction observed across either dataset.

• All perturbation scenarios passed: Every reported train and test evaluation, including baseline and all noise levels from 0.1 to 0.5, is marked as passed in the result table.

The robustness results show limited AUC movement under the tested Gaussian perturbations, with a steady decline on the training dataset and more variable behavior on the test dataset. The largest observed degradation occurs on the test dataset at perturbation size 0.4, while the test AUC at 0.3 slightly exceeds its baseline value. Across all evaluated perturbation sizes, both datasets remain within the test’s passing criteria.

Parameters:

{
  "scaling_factor_std_dev_list": [
    0.0,
    0.1,
    0.2,
    0.3,
    0.4,
    0.5
  ]
}
            

Tables

Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
Baseline (0.0)	train_dataset_final	2585	0.6819	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.6627	0.0000	True
Baseline (0.0)	train_dataset_final	2585	0.6819	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.6627	0.0000	True
0.1	train_dataset_final	2585	0.6811	0.0008	True
0.1	test_dataset_final	647	0.6617	0.0010	True
0.2	train_dataset_final	2585	0.6786	0.0032	True
0.2	test_dataset_final	647	0.6580	0.0047	True
0.3	train_dataset_final	2585	0.6726	0.0093	True
0.3	test_dataset_final	647	0.6631	-0.0004	True
0.4	train_dataset_final	2585	0.6670	0.0149	True
0.4	test_dataset_final	647	0.6419	0.0208	True
0.5	train_dataset_final	2585	0.6636	0.0183	True
0.5	test_dataset_final	647	0.6616	0.0011	True

Figures

In summary

In this final notebook, you learned how to:

• Refresh the connection from the ValidMind Library to the ValidMind Platform after you've inserted test-driven blocks to your documentation
• Include custom test results in your documentation
• View and configure the configuration for your documentation template

With our ValidMind for development series of notebooks, you learned how to document a record (model) end-to-end with the ValidMind Library by running through some common scenarios in a typical development setting:

• Running out-of-the-box tests
• Documenting your record (model) by adding evidence to documentation
• Extending the capabilities of the ValidMind Library by implementing custom tests
• Ensuring that the documentation is complete by running all tests in the documentation template

Next steps

Work with your documentation

Now that you've logged all your test results and generated a draft for your documentation, head to the ValidMind Platform to wrap up your documentation. Continue to work on your documentation by:

• Run and log more tests: Use the skills you learned in this series of notebooks to run and log more individual tests, including custom tests, then insert them into your documentation as supplementary evidence. (Learn more: validmind.tests)

• Inserting additional test results: Add Test-Driven Blocks under any relevant section of your documentation. (Learn more: Work with test results)

• Making qualitative edits to your test descriptions: Click on the description of any inserted test results to review and edit the ValidMind-generated test descriptions for quality and accuracy. (Learn more: Working with documentation)

• View guidelines: In any section of your documentation, click ValidMind Insights in the top right corner to reveal the Documentation Guidelines for each section to help guide the contents of your documentation. (Learn more: View development guidelines)

• Collaborate with other stakeholders: Use the ValidMind Platform's real-time collaborative features to work seamlessly together with the rest of your organization, including validators. Review suggested changes in your content blocks, work with versioned history, and use comments to discuss specific portions of your documentation. (Learn more: Collaborate with others)

When your documentation is complete and ready for review, submit it for approval from the same ValidMind Platform where you made your edits and collaborated with the rest of your organization, ensuring transparency and a thorough development history. (Learn more: Submit documents)

Learn more

Now that you're familiar with the basics, you can explore the following notebooks to get a deeper understanding on how the ValidMind Library allows you generate documentation for any use case:

Use cases

• Document an agentic AI system
• Document an Excel-based application scorecard model
• LLM model documentation demo

Discover more learning resources

Learn more about the ValidMind Library tools we used in this notebook:

• Explore tests
• Run dataset-based tests
• Implement custom tests
• Integrate external test providers
• Configure dataset features

We offer many interactive notebooks to help you automate testing, documenting, validating, and more:

• Run tests & test suites
• Use ValidMind Library features
• Code samples by use case

Or, visit our documentation to learn more about ValidMind.

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial