ValidMind for development 4 — Finalize testing and documentation

Learn how to use ValidMind for your end-to-end documentation process with our introductory notebook series. In this last notebook, finalize the testing and documentation of your model and have a fully documented sample model ready for review.

We'll first use run_documentation_tests() previously covered in 2 — Start the development process to ensure that your custom test results generated in 3 — Integrate custom tests are included in your documentation. Then, we'll view and update the configuration for the entire documentation template to suit your needs.

Learn by doing

Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Developer Fundamentals

Prerequisites

In order to finalize the testing and documentation for your sample model, you'll need to first have:

• Registered a model within the ValidMind Platform with a predefined documentation template
• Installed the ValidMind Library in your local environment, allowing you to access all its features
• Learned how to import and initialize datasets for use with ValidMind
• Learned how to run and log default and custom tests with ValidMind, including from external test providers
• Inserted test-driven blocks for the results of the following tests into your model's documentation:
  • HighPearsonCorrelation:balanced_raw_dataset
  • my_test_provider.ConfusionMatrix
  • my_custom_tests.ConfusionMatrix:test_dataset_normalized

Need help with the above steps?

Refer to the first three notebooks in this series:

• 1 — Set up the ValidMind Library
• 2 — Start the development process
• 3 — Integrate custom tests

Setting up

This section should be very familiar to you now — as we performed the same actions in the previous two notebooks in this series.

Initialize the ValidMind Library

As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Development from the Document drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="documentation",
)

Note: you may need to restart the kernel to use updated packages.

2026-07-31 16:43:24,992 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model development (ID: cmalgf3qi02ce199qm3rdkl46)
📁 Document Type: model_documentation

Import sample dataset

Next, we'll import the same public Bank Customer Churn Prediction dataset from Kaggle we used in the last notebooks so that we have something to work with:

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

We'll apply a simple rebalancing technique to the dataset before continuing:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

Remove highly correlated features

Let's also quickly remove highly correlated features from the dataset using the output from a ValidMind test.

As you learned previously, before we can run tests you'll need to initialize a ValidMind dataset object:

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:

# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant variables or concentrations of multicollinearity. The results table reports the top correlations by absolute magnitude, listing each feature pair, its Pearson correlation coefficient, and a Pass/Fail outcome relative to the configured threshold of 0.3. Across the ten reported pairs, coefficients range from -0.1947 to 0.3497. One pair exceeds the threshold and is marked Fail, while the remaining reported pairs are marked Pass.

Key insights:

• Single threshold breach observed: The pair (Age, Exited) records the largest absolute correlation at 0.3497, exceeding the 0.3 threshold and receiving a Fail result.
• Remaining reported correlations are modest: All other listed feature pairs have absolute correlations below 0.2, including (IsActiveMember, Exited) at -0.1947 and (Balance, NumOfProducts) at -0.1691.
• Correlations are concentrated near zero: Several reported pairs show very small linear relationships, including (HasCrCard, IsActiveMember) at -0.0475, (NumOfProducts, IsActiveMember) at 0.0408, (Tenure, EstimatedSalary) at 0.0378, (Age, HasCrCard) at -0.0371, and (Age, NumOfProducts) at -0.0367.
• Reported results include target relationships: Multiple entries involve Exited, with coefficients of 0.3497 for Age, -0.1947 for IsActiveMember, 0.1389 for Balance, and -0.0616 for NumOfProducts.

The reported correlation structure is dominated by low-magnitude pairwise linear relationships, with only one listed pair crossing the configured threshold. The largest observed association is between Age and Exited, while all other reported coefficients remain materially smaller in absolute value. Overall, the result indicates a limited concentration of higher linear correlation within the top reported pairs.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3497	Fail
(IsActiveMember, Exited)	-0.1947	Pass
(Balance, NumOfProducts)	-0.1691	Pass
(Balance, Exited)	0.1389	Pass
(NumOfProducts, Exited)	-0.0616	Pass
(HasCrCard, IsActiveMember)	-0.0475	Pass
(NumOfProducts, IsActiveMember)	0.0408	Pass
(Tenure, EstimatedSalary)	0.0378	Pass
(Age, HasCrCard)	-0.0371	Pass
(Age, NumOfProducts)	-0.0367	Pass

# From result object, extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3497	Fail
1	(IsActiveMember, Exited)	-0.1947	Pass
2	(Balance, NumOfProducts)	-0.1691	Pass
3	(Balance, Exited)	0.1389	Pass
4	(NumOfProducts, Exited)	-0.0616	Pass
5	(HasCrCard, IsActiveMember)	-0.0475	Pass
6	(NumOfProducts, IsActiveMember)	0.0408	Pass
7	(Tenure, EstimatedSalary)	0.0378	Pass
8	(Age, HasCrCard)	-0.0371	Pass
9	(Age, NumOfProducts)	-0.0367	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

# Extract feature names from the list of strings
high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

We can then re-initialize the dataset with a different input_id and the highly correlated features removed and re-run the test for confirmation:

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

# Re-run the test with the reduced feature set
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates linear relationships between feature pairs to identify potentially redundant variables or multicollinearity. The results table reports the top 10 pairwise Pearson correlation coefficients using a threshold of 0.3 for pass/fail classification. All reported coefficients are below the threshold in absolute value, and each listed feature pair is marked as Pass. The observed coefficients range from -0.1947 to 0.1389, with both positive and negative relationships represented.

Key insights:

• No pair exceeds threshold: All 10 reported feature pairs have absolute correlation values below 0.3 and are classified as Pass under the configured threshold.
• Strongest relationship is modest: The largest absolute correlation is between IsActiveMember and Exited at -0.1947, indicating the most pronounced linear relationship in the reported set remains limited in magnitude.
• Top positive correlation remains low: The highest positive coefficient is 0.1389 for Balance and Exited, which is materially below the 0.3 threshold.
• Correlation levels are uniformly small: The remaining reported coefficients cluster close to zero, including values such as -0.1691 for Balance and NumOfProducts, -0.0616 for NumOfProducts and Exited, and several coefficients below 0.05 in absolute value.

The reported correlation structure shows no high linear dependence among the top-ranked feature pairs under the applied 0.3 threshold. The strongest observed associations are modest, and most reported relationships are weak and close to zero. Based on the reported output, the test does not identify feature pairs with high Pearson correlation.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1947	Pass
(Balance, NumOfProducts)	-0.1691	Pass
(Balance, Exited)	0.1389	Pass
(NumOfProducts, Exited)	-0.0616	Pass
(HasCrCard, IsActiveMember)	-0.0475	Pass
(NumOfProducts, IsActiveMember)	0.0408	Pass
(Tenure, EstimatedSalary)	0.0378	Pass
(Tenure, HasCrCard)	0.0359	Pass
(CreditScore, IsActiveMember)	0.0338	Pass
(Tenure, IsActiveMember)	-0.0264	Pass

Train the model

We'll then use ValidMind tests to train a simple logistic regression model on our prepared dataset:

# First encode the categorical features in our dataset with the highly correlated features removed
balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
7409	642	4	125476.31	1	1	1	91775.51	0	False	False	True
2408	678	4	174852.89	1	1	1	28149.06	0	False	False	False
818	640	8	81677.22	2	0	0	34925.56	0	True	False	True
1915	556	9	0.00	1	1	0	175149.20	1	False	False	False
1723	634	5	123642.36	1	1	1	49725.16	1	True	False	False

# Split the processed dataset into train and test
from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

from sklearn.linear_model import LogisticRegression

# Logistic Regression grid params
log_reg_params = {
    "penalty": ["l1", "l2"],
    "C": [0.001, 0.01, 0.1, 1, 10, 100, 1000],
    "solver": ["liblinear"],
}

# Grid search for Logistic Regression
from sklearn.model_selection import GridSearchCV

grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)
grid_log_reg.fit(X_train, y_train)

# Logistic Regression best estimator
log_reg = grid_log_reg.best_estimator_

/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1403: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1429: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(

Initialize the ValidMind objects

Let's initialize the ValidMind Dataset and Model objects in preparation for assigning model predictions to each dataset:

# Initialize the datasets into their own ValidMind dataset objects
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

# Initialize the ValidMind model object
vm_model = vm.init_model(log_reg, input_id="log_reg_model_v1")

Assign predictions

Once the model is registered, we'll assign predictions to the training and test datasets:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

2026-07-31 16:43:37,704 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-07-31 16:43:37,706 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-07-31 16:43:37,706 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-07-31 16:43:37,708 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-07-31 16:43:37,711 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-07-31 16:43:37,712 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-07-31 16:43:37,713 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-07-31 16:43:37,715 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Add custom tests

We'll also add the same custom tests we implemented in the previous notebook so that this session has access to the same custom inline test and local test provider.

Implement custom inline test

Let's set up a custom inline test that calculates the confusion matrix for a binary classification model:

# First create a confusion matrix plot
import matplotlib.pyplot as plt
from sklearn import metrics

# Get the predicted classes
y_pred = log_reg.predict(vm_test_ds.x)

confusion_matrix = metrics.confusion_matrix(y_test, y_pred)

cm_display = metrics.ConfusionMatrixDisplay(
    confusion_matrix=confusion_matrix, display_labels=[False, True]
)
cm_display.plot()

# Create the reusable ConfusionMatrix inline test with normalized matrix
@vm.test("my_custom_tests.ConfusionMatrix")
def confusion_matrix(dataset, model, normalize=False):
    """The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known.

    The confusion matrix is a 2x2 table that contains 4 values:

    - True Positive (TP): the number of correct positive predictions
    - True Negative (TN): the number of correct negative predictions
    - False Positive (FP): the number of incorrect positive predictions
    - False Negative (FN): the number of incorrect negative predictions

    The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure.
    """
    y_true = dataset.y
    y_pred = dataset.y_pred(model=model)

    if normalize:
        confusion_matrix = metrics.confusion_matrix(y_true, y_pred, normalize="all")
    else:
        confusion_matrix = metrics.confusion_matrix(y_true, y_pred)

    cm_display = metrics.ConfusionMatrixDisplay(
        confusion_matrix=confusion_matrix, display_labels=[False, True]
    )
    cm_display.plot()

    plt.close()  # close the plot to avoid displaying it

    return cm_display.figure_  # return the figure object itself

# Test dataset with normalize=True
result = vm.tests.run_test(
    "my_custom_tests.ConfusionMatrix:test_dataset_normalized",
    inputs={"model": vm_model, "dataset": vm_test_ds},
    params={"normalize": True},
)

Confusion Matrix Test Dataset Normalized

The Confusion Matrix test evaluates classification performance by comparing predicted labels with true labels on the test dataset. In this result, the matrix is shown in normalized form, so each cell represents a proportion rather than a raw count. The four cells correspond to true negatives, false positives, false negatives, and true positives, with observed values of 0.34, 0.17, 0.17, and 0.32 respectively. This layout provides a direct view of how predictions are distributed across correct and incorrect classifications for both classes.

Key insights:

• Correct classifications dominate: The diagonal cells sum to 0.66, comprising 0.34 true negatives and 0.32 true positives. This indicates that correct predictions account for approximately two-thirds of the normalized outcomes.

• Error types are balanced: The off-diagonal cells are equal at 0.17 for both false positives and false negatives. The model therefore shows symmetrical misclassification rates across the two error categories.

• Class-level performance is similar: For the negative class, the normalized split is 0.34 correctly predicted as False and 0.17 incorrectly predicted as True. For the positive class, the normalized split is 0.32 correctly predicted as True and 0.17 incorrectly predicted as False, indicating similar classification behavior across classes.

The normalized confusion matrix shows that most observations fall on the diagonal, with 0.66 of outcomes classified correctly and 0.34 misclassified. Misclassification is evenly divided between false positives and false negatives, while the correctly classified proportions for the two classes are close at 0.34 and 0.32. Collectively, these results indicate balanced prediction behavior across the two classes without a visible concentration of error in one error type.

Parameters:

{
  "normalize": true
}
            

Figures

Add a local test provider

Finally, let's save our custom inline test to our local test provider:

# Create custom tests folder
tests_folder = "my_tests"

import os

# create tests folder
os.makedirs(tests_folder, exist_ok=True)

# remove existing tests
for f in os.listdir(tests_folder):
    # remove files and pycache
    if f.endswith(".py") or f == "__pycache__":
        os.system(f"rm -rf {tests_folder}/{f}")

# Save custom inline test to custom tests folder
confusion_matrix.save(
    tests_folder,
    imports=["import matplotlib.pyplot as plt", "from sklearn import metrics"],
)

2026-07-31 16:43:43,649 - INFO(validmind.tests.decorator): Saved to /home/runner/work/documentation/documentation/site/notebooks/EXECUTED/development/my_tests/ConfusionMatrix.py!Be sure to add any necessary imports to the top of the file.
2026-07-31 16:43:43,649 - INFO(validmind.tests.decorator): This metric can be run with the ID: <test_provider_namespace>.ConfusionMatrix

# Register local test provider
from validmind.tests import LocalTestProvider

# initialize the test provider with the tests folder we created earlier
my_test_provider = LocalTestProvider(tests_folder)

vm.tests.register_test_provider(
    namespace="my_test_provider",
    test_provider=my_test_provider,
)

Reconnect to ValidMind

After you insert test-driven blocks into your model documentation, changes should persist and become available every time you call vm.preview_template().

However, you'll need to reload the connection to the ValidMind Platform if you have added test-driven blocks when the connection was already established using reload():

vm.reload()

Now, when you run preview_template() again, the three test-driven blocks you added to your documentation in the last two notebooks in should show up in the template in sections 2.3 Correlations and Interactions and 3.2 Model Evaluation:

vm.preview_template()

▶ 1. Conceptual Soundness ('conceptual_soundness')

▶ 1.1. Model Overview ('model_overview')

Text Block: 'model_overview'

▶ 1.2. Intended Use and Business Use Case ('intended_use_business_use_case')

▶ 1.2.1. Intended Use ('intended_use')

Empty Section

▶ 1.2.2. Regulatory Requirements ('regulatory_requirements')

Empty Section

▶ 1.2.3. Model Limitations ('model_limitations')

Empty Section

▶ 1.3. Model Selection ('model_selection')

Empty Section

▶ 2. Data Preparation ('data_preparation')

▶ 2.1. Data description ('data_description')

Text Block: 'dataset_summary_text'

▶ Test: Dataset Description ('validmind.data_validation.DatasetDescription')

Dataset Description

Provides comprehensive analysis and statistical summaries of each column in a machine learning model's dataset.

Purpose

The test depicted in the script is meant to run a comprehensive analysis on a Machine Learning model's datasets. The test or metric is implemented to obtain a complete summary of the columns in the dataset, including vital statistics of each column such as count, distinct values, missing values, histograms for numerical, categorical, boolean, and text columns. This summary gives a comprehensive overview of the dataset to better understand the characteristics of the data that the model is trained on or evaluates.

Test Mechanism

The DatasetDescription class accomplishes the purpose as follows: firstly, the test method "run" infers the data type of each column in the dataset and stores the details (id, column type). For each column, the "describe_column" method is invoked to collect statistical information about the column, including count, missing value count and its proportion to the total, unique value count, and its proportion to the total. Depending on the data type of a column, histograms are generated that reflect the distribution of data within the column. Numerical columns use the "get_numerical_histograms" method to calculate histogram distribution, whereas for categorical, boolean and text columns, a histogram is computed with frequencies of each unique value in the datasets. For unsupported types, an error is raised. Lastly, a summary table is built to aggregate all the statistical insights and histograms of the columns in a dataset.

Signs of High Risk

• High ratio of missing values to total values in one or more columns which may impact the quality of the predictions.
• Unsupported data types in dataset columns.
• Large number of unique values in the dataset's columns which might make it harder for the model to establish patterns.
• Extreme skewness or irregular distribution of data as reflected in the histograms.

Strengths

• Provides a detailed analysis of the dataset with versatile summaries like count, unique values, histograms, etc.
• Flexibility in handling different types of data: numerical, categorical, boolean, and text.
• Useful in detecting problems in the dataset like missing values, unsupported data types, irregular data distribution, etc.
• The summary gives a comprehensive understanding of dataset features allowing developers to make informed decisions.

Limitations

• The computation can be expensive from a resource standpoint, particularly for large datasets with numerous columns.
• The histograms use an arbitrary number of bins which may not be the optimal number of bins for specific data distribution.
• Unsupported data types for columns will raise an error which may limit evaluating the dataset.
• Columns with all null or missing values are not included in histogram computation.
• This test only validates the quality of the dataset but doesn't address the model's performance directly.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DatasetDescription", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'data_quality_tests_text'

▶ Test: Class Imbalance ('validmind.data_validation.ClassImbalance')

Class Imbalance

Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model.

Purpose

The Class Imbalance test is designed to evaluate the distribution of target classes in a dataset that's utilized by a machine learning model. Specifically, it aims to ensure that the classes aren't overly skewed, which could lead to bias in the model's predictions. It's crucial to have a balanced training dataset to avoid creating a model that's biased with high accuracy for the majority class and low accuracy for the minority class.

Test Mechanism

This Class Imbalance test operates by calculating the frequency (expressed as a percentage) of each class in the target column of the dataset. It then checks whether each class appears in at least a set minimum percentage of the total records. This minimum percentage is a modifiable parameter, but the default value is set to 10%.

Signs of High Risk

• Any class that represents less than the pre-set minimum percentage threshold is marked as high risk, implying a potential class imbalance.
• The function provides a pass/fail outcome for each class based on this criterion.
• Fundamentally, if any class fails this test, it's highly likely that the dataset possesses imbalanced class distribution.

Strengths

• The test can spot under-represented classes that could affect the efficiency of a machine learning model.
• The calculation is straightforward and swift.
• The test is highly informative because it not only spots imbalance, but it also quantifies the degree of imbalance.
• The adjustable threshold enables flexibility and adaptation to differing use-cases or domain-specific needs.
• The test creates a visually insightful plot showing the classes and their corresponding proportions, enhancing interpretability and comprehension of the data.

Limitations

• The test might struggle to perform well or provide vital insights for datasets with a high number of classes. In such cases, the imbalance could be inevitable due to the inherent class distribution.
• Sensitivity to the threshold value might result in faulty detection of imbalance if the threshold is set excessively high.
• Regardless of the percentage threshold, it doesn't account for varying costs or impacts of misclassifying different classes, which might fluctuate based on specific applications or domains.
• While it can identify imbalances in class distribution, it doesn't provide direct methods to address or correct these imbalances.
• The test is only applicable for classification operations and unsuitable for regression or clustering tasks.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_percent_threshold	10

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_percent_threshold": "my_vm_min_percent_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.ClassImbalance", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Duplicates ('validmind.data_validation.Duplicates')

Duplicates

Tests dataset for duplicate entries, ensuring model reliability via data quality verification.

Purpose

The 'Duplicates' test is designed to check for duplicate rows within the dataset provided to the model. It serves as a measure of data quality, ensuring that the model isn't merely memorizing duplicate entries or being swayed by redundant information. This is an important step in the pre-processing of data for both classification and regression tasks.

Test Mechanism

This test operates by checking each row for duplicates in the dataset. If a text column is specified in the dataset, the test is conducted on this column; if not, the test is run on all feature columns. The number and percentage of duplicates are calculated and returned in a DataFrame. Additionally, a test is passed if the total count of duplicates falls below a specified minimum threshold.

Signs of High Risk

• A high number of duplicate rows in the dataset, which can lead to overfitting where the model performs well on the training data but poorly on unseen data.
• A high percentage of duplicate rows in the dataset, indicating potential problems with data collection or processing.

Strengths

• Assists in improving the reliability of the model's training process by ensuring the training data is not contaminated with duplicate entries, which can distort statistical analyses.
• Provides both absolute numbers and percentage values of duplicate rows, giving a thorough overview of data quality.
• Highly customizable as it allows for setting a user-defined minimum threshold to determine if the test has been passed.

Limitations

• Does not distinguish between benign duplicates (i.e., coincidental identical entries in different rows) and problematic duplicates originating from data collection or processing errors.
• The test becomes more computationally intensive as the size of the dataset increases, which might not be suitable for very large datasets.
• Can only check for exact duplicates and may miss semantically similar information packaged differently.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_threshold	1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.Duplicates", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: High Cardinality ('validmind.data_validation.HighCardinality')

High Cardinality

Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting.

Purpose

The “High Cardinality” test is used to evaluate the number of unique values present in the categorical columns of a dataset. In this context, high cardinality implies the presence of a large number of unique, non-repetitive values in the dataset.

Test Mechanism

The test first infers the dataset's type and then calculates an initial numeric threshold based on the test parameters. It only considers columns classified as "Categorical". For each of these columns, the number of distinct values (n_distinct) and the percentage of distinct values (p_distinct) are calculated. The test will pass if n_distinct is less than the calculated numeric threshold. Lastly, the results, which include details such as column name, number of distinct values, and pass/fail status, are compiled into a table.

Signs of High Risk

• A large number of distinct values (high cardinality) in one or more categorical columns implies a high risk.
• A column failing the test (n_distinct >= num_threshold) is another indicator of high risk.

Strengths

• The High Cardinality test is effective in early detection of potential overfitting and unwanted noise.
• It aids in identifying potential outliers and inconsistencies, thereby improving data quality.
• The test can be applied to both classification and regression task types, demonstrating its versatility.

Limitations

• The test is restricted to only "Categorical" data types and is thus not suitable for numerical or continuous features, limiting its scope.
• The test does not consider the relevance or importance of unique values in categorical features, potentially causing it to overlook critical data points.
• The threshold (both number and percent) used for the test is static and may not be optimal for diverse datasets and varied applications. Further mechanisms to adjust and refine this threshold could enhance its effectiveness.

Required Inputs: dataset

Parameters:

Parameter	Default Value
num_threshold	100
percent_threshold	0.1
threshold_type	'percent'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "num_threshold": "my_vm_num_threshold",
    "percent_threshold": "my_vm_percent_threshold",
    "threshold_type": "my_vm_threshold_type"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.HighCardinality", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Missing Values ('validmind.data_validation.MissingValues')

Missing Values

Evaluates dataset quality by ensuring missing value percentage across all features does not exceed a set threshold.

Purpose

The Missing Values test is designed to evaluate the quality of a dataset by measuring the number of missing values across all features. The objective is to ensure that the ratio of missing data to total data is less than a predefined threshold (as a percentage), defaulting to 1.0, in order to maintain the data quality necessary for reliable predictive strength in a machine learning model.

Test Mechanism

The mechanism for this test involves iterating through each column of the dataset, counting missing values (represented as NaNs), and calculating the percentage they represent against the total number of rows. The test then checks if the missing value percentage is less than or equal to the predefined min_percentage_threshold. The results are shown in a table summarizing each column, the number of missing values, the percentage of missing values in each column, and a Pass/Fail status based on the threshold comparison.

Signs of High Risk

• When the missing value percentage in any column exceeds the min_percentage_threshold value.
• Presence of missing values across many columns, leading to multiple instances of failing the threshold.

Strengths

• Quick and granular identification of missing data across each feature in the dataset.
• Provides an effective and straightforward means of maintaining data quality, essential for constructing efficient machine learning models.

Limitations

• Does not suggest the root causes of the missing values or recommend ways to impute or handle them.
• May overlook features with significant missing data but still less than the min_percentage_threshold, potentially impacting the model.
• Does not account for data encoded as values like "-999" or "None," which might not technically classify as missing but could bear similar implications.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_percentage_threshold	1.0

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_percentage_threshold": "my_vm_min_percentage_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.MissingValues", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Skewness ('validmind.data_validation.Skewness')

Skewness

Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data quality and optimize model performance.

Purpose

The purpose of the Skewness test is to measure the asymmetry in the distribution of data within a predictive machine learning model. Specifically, it evaluates the divergence of said distribution from a normal distribution. Understanding the level of skewness helps identify data quality issues, which are crucial for optimizing the performance of traditional machine learning models in both classification and regression settings.

Test Mechanism

This test calculates the skewness of numerical columns in the dataset, focusing specifically on numerical data types. The calculated skewness value is then compared against a predetermined maximum threshold, which is set by default to 1. If the skewness value is less than this maximum threshold, the test passes; otherwise, it fails. The test results, along with the skewness values and column names, are then recorded for further analysis.

Signs of High Risk

• Substantial skewness levels that significantly exceed the maximum threshold.
• Persistent skewness in the data, indicating potential issues with the foundational assumptions of the machine learning model.
• Subpar model performance, erroneous predictions, or biased inferences due to skewed data distributions.

Strengths

• Fast and efficient identification of unequal data distributions within a machine learning model.
• Adjustable maximum threshold parameter, allowing for customization based on user needs.
• Provides a clear quantitative measure to mitigate model risks related to data skewness.

Limitations

• Only evaluates numeric columns, potentially missing skewness or bias in non-numeric data.
• Assumes that data should follow a normal distribution, which may not always be applicable to real-world data.
• Subjective threshold for risk grading, requiring expert input and recurrent iterations for refinement.

Required Inputs: dataset

Parameters:

Parameter	Default Value
max_threshold	1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "max_threshold": "my_vm_max_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.Skewness", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Unique Rows ('validmind.data_validation.UniqueRows')

Unique Rows

Verifies the diversity of the dataset by ensuring that the count of unique rows exceeds a prescribed threshold.

Purpose

The UniqueRows test is designed to gauge the quality of the data supplied to the machine learning model by verifying that the count of distinct rows in the dataset exceeds a specific threshold, thereby ensuring a varied collection of data. Diversity in data is essential for training an unbiased and robust model that excels when faced with novel data.

Test Mechanism

The testing process starts with calculating the total number of rows in the dataset. Subsequently, the count of unique rows is determined for each column in the dataset. If the percentage of unique rows (calculated as the ratio of unique rows to the overall row count) is less than the prescribed minimum percentage threshold given as a function parameter, the test passes. The results are cached and a final pass or fail verdict is given based on whether all columns have successfully passed the test.

Signs of High Risk

• A lack of diversity in data columns, demonstrated by a count of unique rows that falls short of the preset minimum percentage threshold, is indicative of high risk.
• This lack of variety in the data signals potential issues with data quality, possibly leading to overfitting in the model and issues with generalization, thus posing a significant risk.

Strengths

• The UniqueRows test is efficient in evaluating the data's diversity across each information column in the dataset.
• This test provides a quick, systematic method to assess data quality based on uniqueness, which can be pivotal in developing effective and unbiased machine learning models.

Limitations

• A limitation of the UniqueRows test is its assumption that the data's quality is directly proportionate to its uniqueness, which may not always hold true. There might be contexts where certain non-unique rows are essential and should not be overlooked.
• The test does not consider the relative 'importance' of each column in predicting the output, treating all columns equally.
• This test may not be suitable or useful for categorical variables, where the count of unique categories is inherently limited.

Required Inputs: dataset

Parameters:

Parameter	Default Value
min_percent_threshold	1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "min_percent_threshold": "my_vm_min_percent_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.UniqueRows", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Too Many Zero Values ('validmind.data_validation.TooManyZeroValues')

Too Many Zero Values

Identifies numerical columns in a dataset that contain an excessive number of zero values, defined by a threshold percentage.

Purpose

The 'TooManyZeroValues' test is utilized to identify numerical columns in the dataset that may present a quantity of zero values considered excessive. The aim is to detect situations where these may implicate data sparsity or a lack of variation, limiting their effectiveness within a machine learning model. The definition of 'too many' is quantified as a percentage of total values, with a default set to 0.03%.

Test Mechanism

This test is conducted by looping through each column in the dataset and categorizing those that pertain to numerical data. On identifying a numerical column, the function computes the total quantity of zero values and their ratio to the total row count. Should the proportion exceed a pre-set threshold parameter, set by default at 0.03%, the column is considered to have failed the test. The results for each column are summarized and reported, indicating the count and percentage of zero values for each numerical column, alongside a status indicating whether the column has passed or failed the test.

Signs of High Risk

• Numerical columns showing a high ratio of zero values when compared to the total count of rows (exceeding the predetermined threshold).
• Columns characterized by zero values across the board suggest a complete lack of data variation, signifying high risk.

Strengths

• Assists in highlighting columns featuring an excess of zero values that could otherwise go unnoticed within a large dataset.
• Provides the flexibility to alter the threshold that determines when the quantity of zero values becomes 'too many', thus catering to specific needs of a particular analysis or model.
• Offers feedback in the form of both counts and percentages of zero values, which allows a closer inspection of the distribution and proportion of zeros within a column.
• Targets specifically numerical data, thereby avoiding inappropriate application to non-numerical columns and mitigating the risk of false test failures.

Limitations

• Is exclusively designed to check for zero values and doesn’t assess the potential impact of other values that could affect the dataset, such as extremely high or low figures, missing values, or outliers.
• Lacks the ability to detect a repetitive pattern of zeros, which could be significant in time-series or longitudinal data.
• Zero values can actually be meaningful in some contexts; therefore, tagging them as 'too many' could potentially misinterpret the data to some extent.
• This test does not take into consideration the context of the dataset, and fails to recognize that within certain columns, a high number of zero values could be quite normal and not necessarily an indicator of poor data quality.
• Cannot evaluate non-numerical or categorical columns, which might bring with them different types of concerns or issues.

Required Inputs: dataset

Parameters:

Parameter	Default Value
max_percent_threshold	0.03

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "max_percent_threshold": "my_vm_max_percent_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.TooManyZeroValues", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: IQR Outliers Table ('validmind.data_validation.IQROutliersTable')

IQR Outliers Table

Determines and summarizes outliers in numerical features using the Interquartile Range method.

Purpose

The "Interquartile Range Outliers Table" (IQROutliersTable) metric is designed to identify and summarize outliers within numerical features of a dataset using the Interquartile Range (IQR) method. This exercise is crucial in the pre-processing of data because outliers can substantially distort statistical analysis and impact the performance of machine learning models.

Test Mechanism

The IQR, which is the range separating the first quartile (25th percentile) from the third quartile (75th percentile), is calculated for each numerical feature within the dataset. An outlier is defined as a data point falling below the "Q1 - 1.5 * IQR" or above "Q3 + 1.5 * IQR" range. The test computes the number of outliers and their summary statistics (minimum, 25th percentile, median, 75th percentile, and maximum values) for each numerical feature. If no specific features are chosen, the test applies to all numerical features in the dataset. The default outlier threshold is set to 1.5 but can be customized by the user.

Signs of High Risk

• A large number of outliers in multiple features.
• Outliers significantly distanced from the mean value of variables.
• Extremely high or low outlier values indicative of data entry errors or other data quality issues.

Strengths

• Provides a comprehensive summary of outliers for each numerical feature, helping pinpoint features with potential quality issues.
• The IQR method is robust to extremely high or low outlier values as it is based on quartile calculations.
• Can be customized to work on selected features and set thresholds for outliers.

Limitations

• Might cause false positives if the variable deviates from a normal or near-normal distribution, especially for skewed distributions.
• Does not provide interpretation or recommendations for addressing outliers, relying on further analysis by users or data scientists.
• Only applicable to numerical features, not categorical data.
• Default thresholds may not be optimal for data with heavy pre-processing, manipulation, or inherently high kurtosis (heavy tails).

Required Inputs: dataset

Parameters:

Parameter	Default Value
threshold	1.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "threshold": "my_vm_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.IQROutliersTable", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: IQR Outliers Bar Plot ('validmind.data_validation.IQROutliersBarPlot')

IQR Outliers Bar Plot

Visualizes outlier distribution across percentiles in numerical data using the Interquartile Range (IQR) method.

Purpose

The InterQuartile Range Outliers Bar Plot (IQROutliersBarPlot) metric aims to visually analyze and evaluate the extent of outliers in numeric variables based on percentiles. Its primary purpose is to clarify the dataset's distribution, flag possible abnormalities in it, and gauge potential risks associated with processing potentially skewed data, which can affect the machine learning model's predictive prowess.

Test Mechanism

The examination invokes a series of steps:

1. For every numeric feature in the dataset, the 25th percentile (Q1) and 75th percentile (Q3) are calculated before deriving the Interquartile Range (IQR), the difference between Q1 and Q3.
2. Subsequently, the metric calculates the lower and upper thresholds by subtracting Q1 from the threshold times IQR and adding Q3 to threshold times IQR, respectively. The default threshold is set at 1.5.
3. Any value in the feature that falls below the lower threshold or exceeds the upper threshold is labeled as an outlier.
4. The number of outliers are tallied for different percentiles, such as [0-25], [25-50], [50-75], and [75-100].
5. These counts are employed to construct a bar plot for the feature, showcasing the distribution of outliers across different percentiles.

Signs of High Risk

• A prevalence of outliers in the data, potentially skewing its distribution.
• Outliers dominating higher percentiles (75-100) which implies the presence of extreme values, capable of severely influencing the model's performance.
• Certain features harboring most of their values as outliers, which signifies that these features might not contribute positively to the model's forecasting ability.

Strengths

• Effectively identifies outliers in the data through visual means, facilitating easier comprehension and offering insights into the outliers' possible impact on the model.
• Provides flexibility by accommodating all numeric features or a chosen subset.
• Task-agnostic in nature; it is viable for both classification and regression tasks.
• Can handle large datasets as its operation does not hinge on computationally heavy operations.

Limitations

• Its application is limited to numerical variables and does not extend to categorical ones.
• Only reveals the presence and distribution of outliers and does not provide insights into how these outliers might affect the model's predictive performance.
• The assumption that data is unimodal and symmetric may not always hold true. In cases with non-normal distributions, the results can be misleading.

Required Inputs: dataset

Parameters:

Parameter	Default Value
threshold	1.5
fig_width	800

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "threshold": "my_vm_threshold",
    "fig_width": "my_vm_fig_width"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.IQROutliersBarPlot", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 2.2. Descriptive Statistics ('descriptive_statistics')

▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Descriptive Statistics

Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's dataset.

Purpose

The purpose of the Descriptive Statistics metric is to provide a comprehensive summary of both numerical and categorical data within a dataset. This involves statistics such as count, mean, standard deviation, minimum and maximum values for numerical data. For categorical data, it calculates the count, number of unique values, most common value and its frequency, and the proportion of the most frequent value relative to the total. The goal is to visualize the overall distribution of the variables in the dataset, aiding in understanding the model's behavior and predicting its performance.

Test Mechanism

The testing mechanism utilizes two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function pulls out several summary statistics, while value_counts() accounts for unique values. The resulting data is formatted into two distinct tables, one for numerical and another for categorical variable summaries. These tables provide a clear summary of the main characteristics of the variables, which can be instrumental in assessing the model's performance.

Signs of High Risk

• Skewed data or significant outliers can represent high risk. For numerical data, this may be reflected via a significant difference between the mean and median (50% percentile).
• For categorical data, a lack of diversity (low count of unique values), or overdominance of a single category (high frequency of the top value) can indicate high risk.

Strengths

• Provides a comprehensive summary of the dataset, shedding light on the distribution and characteristics of the variables under consideration.
• It is a versatile and robust method, applicable to both numerical and categorical data.
• Helps highlight crucial anomalies such as outliers, extreme skewness, or lack of diversity, which are vital in understanding model behavior during testing and validation.

Limitations

• While this metric offers a high-level overview of the data, it may fail to detect subtle correlations or complex patterns.
• Does not offer any insights on the relationship between variables.
• Alone, descriptive statistics cannot be used to infer properties about future unseen data.
• Should be used in conjunction with other statistical tests to provide a comprehensive understanding of the model's data.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DescriptiveStatistics", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 2.3. Correlations and Interactions ('correlations')

▶ Test: Pearson Correlation Matrix ('validmind.data_validation.PearsonCorrelationMatrix')

Pearson Correlation Matrix

Evaluates linear dependency between numerical variables in a dataset via a Pearson Correlation coefficient heat map.

Purpose

This test is intended to evaluate the extent of linear dependency between all pairs of numerical variables in the given dataset. It provides the Pearson Correlation coefficient, which reveals any high correlations present. The purpose of doing this is to identify potential redundancy, as variables that are highly correlated can often be removed to reduce the dimensionality of the dataset without significantly impacting the model's performance.

Test Mechanism

This metric test generates a correlation matrix for all numerical variables in the dataset using the Pearson correlation formula. A heat map is subsequently created to visualize this matrix effectively. The color of each point on the heat map corresponds to the magnitude and direction (positive or negative) of the correlation, with a range from -1 (perfect negative correlation) to 1 (perfect positive correlation). Any correlation coefficients higher than 0.7 (in absolute terms) are indicated in white in the heat map, suggesting a high degree of correlation.

Signs of High Risk

• A large number of variables in the dataset showing a high degree of correlation (coefficients approaching ±1). This indicates redundancy within the dataset, suggesting that some variables may not be contributing new information to the model.
• Potential risk of overfitting.

Strengths

• Detects and quantifies the linearity of relationships between variables, aiding in identifying redundant variables to simplify models and potentially improve performance.
• The heatmap visualization provides an easy-to-understand overview of correlations, beneficial for users not comfortable with numerical matrices.

Limitations

• Limited to detecting linear relationships, potentially missing non-linear relationships which impede opportunities for dimensionality reduction.
• Measures only the degree of linear relationship, not the strength of one variable's effect on another.
• The 0.7 correlation threshold is arbitrary and might exclude valid dependencies with lower coefficients.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.PearsonCorrelationMatrix", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: High Pearson Correlation ('validmind.data_validation.HighPearsonCorrelation')

High Pearson Correlation

Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity.

Purpose

The High Pearson Correlation test measures the linear relationship between features in a dataset, with the main goal of identifying high correlations that might indicate feature redundancy or multicollinearity. Identification of such issues allows developers and risk management teams to properly deal with potential impacts on the machine learning model's performance and interpretability.

Test Mechanism

The test works by generating pairwise Pearson correlations for all features in the dataset, then sorting and eliminating duplicate and self-correlations. It assigns a Pass or Fail based on whether the absolute value of the correlation coefficient surpasses a pre-set threshold (defaulted at 0.3). It lastly returns the top n strongest correlations regardless of passing or failing status (where n is 10 by default but can be configured by passing the top_n_correlations parameter).

Signs of High Risk

• A high risk indication would be the presence of correlation coefficients exceeding the threshold.
• If the features share a strong linear relationship, this could lead to potential multicollinearity and model overfitting.
• Redundancy of variables can undermine the interpretability of the model due to uncertainty over the authenticity of individual variable's predictive power.

Strengths

• Provides a quick and simple means of identifying relationships between feature pairs.
• Generates a transparent output that displays pairs of correlated variables, the Pearson correlation coefficient, and a Pass or Fail status for each.
• Aids in early identification of potential multicollinearity issues that may disrupt model training.

Limitations

• Can only delineate linear relationships, failing to shed light on nonlinear relationships or dependencies.
• Sensitive to outliers where a few outliers could notably affect the correlation coefficient.
• Limited to identifying redundancy only within feature pairs; may fail to spot more complex relationships among three or more variables.

Required Inputs: dataset

Parameters:

Parameter	Default Value
max_threshold	0.3
top_n_correlations	10
feature_columns	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {
    "max_threshold": "my_vm_max_threshold",
    "top_n_correlations": "my_vm_top_n_correlations",
    "feature_columns": "my_vm_feature_columns"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.HighPearsonCorrelation", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 2.4. Feature Selection and Engineering ('feature_selection')

Text Block: 'feature_selection'

▶ 3. Model Development ('model_development')

▶ 3.1. Model Training ('model_training')

▶ Test: Model Metadata ('validmind.model_validation.ModelMetadata')

Model Metadata

Compare metadata of different models and generate a summary table with the results.

Purpose: The purpose of this function is to compare the metadata of different models, including information about their architecture, framework, framework version, and programming language.

Test Mechanism: The function retrieves the metadata for each model using get_model_info, renames columns according to a predefined set of labels, and compiles this information into a summary table.

Signs of High Risk:

• Inconsistent or missing metadata across models can indicate potential issues in model documentation or management.
• Significant differences in framework versions or programming languages might pose challenges in model integration and deployment.

Strengths:

• Provides a clear comparison of essential model metadata.
• Standardizes metadata labels for easier interpretation and comparison.
• Helps identify potential compatibility or consistency issues across models.

Limitations:

• Assumes that the get_model_info function returns all necessary metadata fields.
• Relies on the correctness and completeness of the metadata provided by each model.
• Does not include detailed parameter information, focusing instead on high-level metadata.

Required Inputs: model

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.ModelMetadata", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Dataset Split ('validmind.data_validation.DatasetSplit')

Dataset Split

Evaluates and visualizes the distribution proportions among training, testing, and validation datasets of an ML model.

Purpose

The DatasetSplit test is designed to evaluate and visualize the distribution of data among training, testing, and validation datasets, if available, within a given machine learning model. The main purpose is to assess whether the model's datasets are split appropriately, as an imbalanced split might affect the model's ability to learn from the data and generalize to unseen data.

Test Mechanism

The DatasetSplit test first calculates the total size of all available datasets in the model. Then, for each individual dataset, the methodology involves determining the size of the dataset and its proportion relative to the total size. The results are then conveniently summarized in a table that shows dataset names, sizes, and proportions. Absolute size and proportion of the total dataset size are displayed for each individual dataset.

Signs of High Risk

• A very small training dataset, which may result in the model not learning enough from the data.
• A very large training dataset and a small test dataset, which may lead to model overfitting and poor generalization to unseen data.
• A small or non-existent validation dataset, which might complicate the model's performance assessment.

Strengths

• The DatasetSplit test provides a clear, understandable visualization of dataset split proportions, which can highlight any potential imbalance in dataset splits quickly.
• It covers a wide range of task types including classification, regression, and text-related tasks.
• The metric is not tied to any specific data type and is applicable to tabular data, time series data, or text data.

Limitations

• The DatasetSplit test does not provide any insight into the quality or diversity of the data within each split, just the size and proportion.
• The test does not give any recommendations or adjustments for imbalanced datasets.
• Potential lack of compatibility with more complex modes of data splitting (for example, stratified or time-based splits) could limit the applicability of this test.

Required Inputs: datasets

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DatasetSplit", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Population Stability Index ('validmind.model_validation.sklearn.PopulationStabilityIndex')

Population Stability Index

Assesses the Population Stability Index (PSI) to quantify the stability of an ML model's predictions across different datasets.

Purpose

The Population Stability Index (PSI) serves as a quantitative assessment for evaluating the stability of a machine learning model's output distributions when comparing two different datasets. Typically, these would be a development and a validation dataset or two datasets collected at different periods. The PSI provides a measurable indication of any significant shift in the model's performance over time or noticeable changes in the characteristics of the population the model is making predictions for.

Test Mechanism

The implementation of the PSI in this script involves calculating the PSI for each feature between the training and test datasets. Data from both datasets is sorted and placed into either a predetermined number of bins or quantiles. The boundaries for these bins are initially determined based on the distribution of the training data. The contents of each bin are calculated and their respective proportions determined. Subsequently, the PSI is derived for each bin through a logarithmic transformation of the ratio of the proportions of data for each feature in the training and test datasets. The PSI, along with the proportions of data in each bin for both datasets, are displayed in a summary table, a grouped bar chart, and a scatter plot.

Signs of High Risk

• A high PSI value is a clear indicator of high risk. Such a value suggests a significant shift in the model predictions or severe changes in the characteristics of the underlying population.
• This ultimately suggests that the model may not be performing as well as expected and that it may be less reliable for making future predictions.

Strengths

• The PSI provides a quantitative measure of the stability of a model over time or across different samples, making it an invaluable tool for evaluating changes in a model's performance.
• It allows for direct comparisons across different features based on the PSI value.
• The calculation and interpretation of the PSI are straightforward, facilitating its use in model risk management.
• The use of visual aids such as tables and charts further simplifies the comprehension and interpretation of the PSI.

Limitations

• The PSI test does not account for the interdependence between features: features that are dependent on one another may show similar shifts in their distributions, which in turn may result in similar PSI values.
• The PSI test does not inherently provide insights into why there are differences in distributions or why the PSI values may have changed.
• The test may not handle features with significant outliers adequately.
• Additionally, the PSI test is performed on model predictions, not on the underlying data distributions which can lead to misinterpretations. Any changes in PSI could be due to shifts in the model (model drift), changes in the relationships between features and the target variable (concept drift), or both. However, distinguishing between these causes is non-trivial.
• For multiclass models the PSI is computed one-vs-rest (one table/plot per class), which requires per-class probabilities from the model's predict_proba. Models that cannot produce a full per-class probability matrix (e.g. metadata-only models, or predictions supplied as a single precomputed probability column) are skipped for the multiclass case.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
num_bins	10
mode	'fixed'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "num_bins": "my_vm_num_bins",
    "mode": "my_vm_mode"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.PopulationStabilityIndex", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 3.2. Model Evaluation ('model_evaluation')

▶ Test: Confusion Matrix ('validmind.model_validation.sklearn.ConfusionMatrix')

Confusion Matrix

Evaluates and visually represents the classification ML model's predictive performance using a Confusion Matrix heatmap.

Purpose

The Confusion Matrix tester is designed to assess the performance of a classification Machine Learning model. This performance is evaluated based on how well the model is able to correctly classify True Positives, True Negatives, False Positives, and False Negatives - fundamental aspects of model accuracy.

Test Mechanism

The mechanism used involves taking the predicted results (y_test_predict) from the classification model and comparing them against the actual values (y_test_true). A confusion matrix is built using the unique labels extracted from y_test_true, employing scikit-learn's metrics. The matrix is then visually rendered with the help of Plotly's create_annotated_heatmap function. A heatmap is created which provides a two-dimensional graphical representation of the model's performance, showcasing distributions of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).

Signs of High Risk

• High numbers of False Positives (FP) and False Negatives (FN), depicting that the model is not effectively classifying the values.
• Low numbers of True Positives (TP) and True Negatives (TN), implying that the model is struggling with correctly identifying class labels.

Strengths

• It provides a simplified yet comprehensive visual snapshot of the classification model's predictive performance.
• It distinctly brings out True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN), thus making it easier to focus on potential areas of improvement.
• The matrix is beneficial in dealing with multi-class classification problems as it can provide a simple view of complex model performances.
• It aids in understanding the different types of errors that the model could potentially make, as it provides in-depth insights into Type-I and Type-II errors.

Limitations

• In cases of unbalanced classes, the effectiveness of the confusion matrix might be lessened. It may wrongly interpret the accuracy of a model that is essentially just predicting the majority class.
• It does not provide a single unified statistic that could evaluate the overall performance of the model. Different aspects of the model's performance are evaluated separately instead.
• It mainly serves as a descriptive tool and does not offer the capability for statistical hypothesis testing.
• Risks of misinterpretation exist because the matrix doesn't directly provide precision, recall, or F1-score data. These metrics have to be computed separately.
• The threshold parameter only applies to binary classification (it splits a single positive-class probability into two classes). For multiclass targets the model's argmax class predictions are used and threshold is ignored.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
threshold	0.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "threshold": "my_vm_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ConfusionMatrix", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Classifier Performance In Sample ('validmind.model_validation.sklearn.ClassifierPerformance:in_sample')

Classifier Performance In Sample

Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy, and ROC AUC scores.

Purpose

The Classifier Performance test is designed to evaluate the performance of Machine Learning classification models. It accomplishes this by computing precision, recall, F1-Score, and accuracy, as well as the ROC AUC (Receiver operating characteristic - Area under the curve) scores, thereby providing a comprehensive analytic view of the models' performance. The test is adaptable, handling binary and multiclass models equally effectively.

Test Mechanism

The test produces a report that includes precision, recall, F1-Score, and accuracy, by leveraging the classification_report from scikit-learn's metrics module. For multiclass models, macro and weighted averages for these scores are also calculated. Additionally, the ROC AUC scores are calculated and included in the report using the multiclass_roc_auc_score function. The outcome of the test (report format) differs based on whether the model is binary or multiclass.

Signs of High Risk

• Low values for precision, recall, F1-Score, accuracy, and ROC AUC, indicating poor performance.
• Imbalance in precision and recall scores.
• A low ROC AUC score, especially scores close to 0.5 or lower, suggesting a failing model.

Strengths

• Versatile, capable of assessing both binary and multiclass models.
• Utilizes a variety of commonly employed performance metrics, offering a comprehensive view of model performance.
• The use of ROC-AUC as a metric is beneficial for evaluating unbalanced datasets.

Limitations

• Assumes correctly identified labels for binary classification models.
• Specifically designed for classification models and not suitable for regression models.
• May provide limited insights if the test dataset does not represent real-world scenarios adequately.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
average	'macro'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "average": "my_vm_average"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ClassifierPerformance:in_sample", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Classifier Performance Out Of Sample ('validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample')

Classifier Performance Out Of Sample

Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy, and ROC AUC scores.

Purpose

The Classifier Performance test is designed to evaluate the performance of Machine Learning classification models. It accomplishes this by computing precision, recall, F1-Score, and accuracy, as well as the ROC AUC (Receiver operating characteristic - Area under the curve) scores, thereby providing a comprehensive analytic view of the models' performance. The test is adaptable, handling binary and multiclass models equally effectively.

Test Mechanism

The test produces a report that includes precision, recall, F1-Score, and accuracy, by leveraging the classification_report from scikit-learn's metrics module. For multiclass models, macro and weighted averages for these scores are also calculated. Additionally, the ROC AUC scores are calculated and included in the report using the multiclass_roc_auc_score function. The outcome of the test (report format) differs based on whether the model is binary or multiclass.

Signs of High Risk

• Low values for precision, recall, F1-Score, accuracy, and ROC AUC, indicating poor performance.
• Imbalance in precision and recall scores.
• A low ROC AUC score, especially scores close to 0.5 or lower, suggesting a failing model.

Strengths

• Versatile, capable of assessing both binary and multiclass models.
• Utilizes a variety of commonly employed performance metrics, offering a comprehensive view of model performance.
• The use of ROC-AUC as a metric is beneficial for evaluating unbalanced datasets.

Limitations

• Assumes correctly identified labels for binary classification models.
• Specifically designed for classification models and not suitable for regression models.
• May provide limited insights if the test dataset does not represent real-world scenarios adequately.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
average	'macro'

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "average": "my_vm_average"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Precision Recall Curve ('validmind.model_validation.sklearn.PrecisionRecallCurve')

Precision Recall Curve

Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve.

Purpose

The Precision Recall Curve metric is intended to evaluate the trade-off between precision and recall in classification models, particularly binary classification models. It assesses the model's capacity to produce accurate results (high precision), as well as its ability to capture a majority of all positive instances (high recall).

Test Mechanism

The test extracts ground truth labels and prediction probabilities from the model's test dataset. It applies the precision_recall_curve method from the sklearn metrics module to these extracted labels and predictions, which computes a precision-recall pair for each possible threshold. This calculation results in an array of precision and recall scores that can be plotted against each other to form the Precision-Recall Curve. This curve is then visually represented by using Plotly's scatter plot.

Signs of High Risk

• A lower area under the Precision-Recall Curve signifies high risk.
• This corresponds to a model yielding a high amount of false positives (low precision) and/or false negatives (low recall).
• If the curve is closer to the bottom left of the plot, rather than being closer to the top right corner, it can be a sign of high risk.

Strengths

• This metric aptly represents the balance between precision (minimizing false positives) and recall (minimizing false negatives), which is especially critical in scenarios where both values are significant.
• Through the graphic representation, it enables an intuitive understanding of the model's performance across different threshold levels.

Limitations

• For multiclass models the curve is computed one-vs-rest (one curve per class plus a micro-average), which requires per-class probabilities from the model's predict_proba. Models that cannot produce a full per-class probability matrix (e.g. Foundation/metadata-only models, or predictions supplied as a single precomputed probability column) are skipped for the multiclass case.
• It may not fully represent the overall accuracy of the model if the cost of false positives and false negatives are extremely different, or if the dataset is heavily imbalanced.

Required Inputs: model, dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.PrecisionRecallCurve", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: ROC Curve ('validmind.model_validation.sklearn.ROCCurve')

ROC Curve

Evaluates classification model performance by generating and plotting the Receiver Operating Characteristic (ROC) curve and calculating the Area Under Curve (AUC) score, for both binary and multiclass models.

Purpose

The Receiver Operating Characteristic (ROC) curve evaluates the performance of classification models. This curve illustrates the balance between the True Positive Rate (TPR) and False Positive Rate (FPR) across various threshold levels. In combination with the Area Under the Curve (AUC), the ROC curve measures the model's discrimination ability between classes. For binary problems (e.g., default vs non-default) a single curve is drawn for the positive class. For multiclass problems the curve is computed one-vs-rest — one curve and AUC per class, plus a micro-average across all classes — so the model's discrimination ability can be assessed for every class. Ideally, a higher AUC score signifies superior model performance in accurately distinguishing between classes.

Test Mechanism

This test selects the target model and dataset and determines the number of classes from the true labels. For binary targets it computes the predicted probabilities for the positive class and, together with the true outcomes, generates and plots a single ROC curve. For multiclass targets it obtains the full per-class probability matrix from the model and plots a one-vs-rest curve for each class along with a micro-average curve. In both cases a line signifying randomness (AUC of 0.5) is included, and the AUC score(s) are computed as a numerical estimation of performance. If any Infinite values are detected in the ROC threshold, these are effectively eliminated. The resulting ROC curves, AUC scores, and thresholds are consequently saved for future reference.

Signs of High Risk

• A high risk is potentially linked to the model's performance if the AUC score drops below or nears 0.5.
• Another warning sign would be the ROC curve lying closer to the line of randomness, indicating no discriminative ability.
• For the model to be deemed competent at its classification tasks, it is crucial that the AUC score is significantly above 0.5.

Strengths

• The ROC Curve offers an inclusive visual depiction of a model's discriminative power throughout all conceivable classification thresholds, unlike other metrics that solely disclose model performance at one fixed threshold.
• Despite the proportions of the dataset, the AUC Score, which represents the entire ROC curve as a single data point, continues to be consistent, proving to be the ideal choice for such situations.

Limitations

• For multiclass models the curve is computed one-vs-rest (one curve per class plus a micro-average), which requires per-class probabilities from the model's predict_proba. Models that cannot produce a full per-class probability matrix (e.g. metadata-only models, or predictions supplied as a single precomputed probability column) are skipped for the multiclass case.
• Furthermore, its performance might be subpar with models that output probabilities highly skewed towards 0 or 1.
• At the extreme, the ROC curve could reflect high performance even when the majority of classifications are incorrect, provided that the model's ranking format is retained. This phenomenon is commonly termed the "Class Imbalance Problem".

Required Inputs: model, dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.ROCCurve", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'model_validation_tests_text'

▶ Test: Training Test Degradation ('validmind.model_validation.sklearn.TrainingTestDegradation')

Training Test Degradation

Tests if model performance degradation between training and test datasets exceeds a predefined threshold.

Purpose

The TrainingTestDegradation class serves as a test to verify that the degradation in performance between the training and test datasets does not exceed a predefined threshold. This test measures the model's ability to generalize from its training data to unseen test data, assessing key classification metrics such as precision, recall, and f1 score to verify the model's robustness and reliability.

Test Mechanism

The code applies several predefined metrics, including precision, recall, and f1 scores, to the model's predictions for both the training and test datasets. It calculates the degradation as the difference between the training score and test score divided by the training score. The test is considered successful if the degradation for each metric is less than the preset maximum threshold (default: 0.10). The results are summarized in a table showing each metric's train score, test score, degradation percentage, and pass/fail status.

Signs of High Risk

• A degradation percentage that exceeds the maximum allowed threshold of 10% for any of the evaluated metrics.
• A high difference or gap between the metric scores on the training and the test datasets.
• The 'Pass/Fail' column displaying 'Fail' for any of the evaluated metrics.

Strengths

• Provides a quantitative measure of the model's ability to generalize to unseen data, which is key for predicting its practical real-world performance.
• By evaluating multiple metrics, it takes into account different facets of model performance and enables a more holistic evaluation.
• The use of a variable predefined threshold allows the flexibility to adjust the acceptability criteria for different scenarios.

Limitations

• The test compares raw performance on training and test data but does not factor in the nature of the data. Areas with less representation in the training set might still perform poorly on unseen data.
• It requires good coverage and balance in the test and training datasets to produce reliable results, which may not always be available.
• The test is currently only designed for classification tasks.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
max_threshold	0.1

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "max_threshold": "my_vm_max_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.TrainingTestDegradation", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Minimum Accuracy ('validmind.model_validation.sklearn.MinimumAccuracy')

Minimum Accuracy

Checks if the model's prediction accuracy meets or surpasses a specified threshold.

Purpose

The Minimum Accuracy test’s objective is to verify whether the model's prediction accuracy on a specific dataset meets or surpasses a predetermined minimum threshold. Accuracy, which is simply the ratio of correct predictions to total predictions, is a key metric for evaluating the model's performance. Considering binary as well as multiclass classifications, accurate labeling becomes indispensable.

Test Mechanism

The test mechanism involves contrasting the model's accuracy score with a preset minimum threshold value, with the default being 0.7. The accuracy score is computed utilizing sklearn’s accuracy_score method, where the true labels y_true and predicted labels class_pred are compared. If the accuracy score is above the threshold, the test receives a passing mark. The test returns the result along with the accuracy score and threshold used for the test.

Signs of High Risk

• Model fails to achieve or surpass the predefined score threshold.
• Persistent scores below the threshold, indicating a high risk of inaccurate predictions.

Strengths

• Simplicity, presenting a straightforward measure of holistic model performance across all classes.
• Particularly advantageous when classes are balanced.
• Versatile, as it can be implemented on both binary and multiclass classification tasks.

Limitations

• Misleading accuracy scores when classes in the dataset are highly imbalanced.
• Favoritism towards the majority class, giving an inaccurate perception of model performance.
• Inability to measure the model's precision, recall, or capacity to manage false positives or false negatives.
• Focused on overall correctness and may not be sufficient for all types of model analytics.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
min_threshold	0.7

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.MinimumAccuracy", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Minimum F1 Score ('validmind.model_validation.sklearn.MinimumF1Score')

Minimum F1 Score

Assesses if the model's F1 score on the validation set meets a predefined minimum threshold, ensuring balanced performance between precision and recall.

Purpose

The main objective of this test is to ensure that the F1 score, a balanced measure of precision and recall, of the model meets or surpasses a predefined threshold on the validation dataset. The F1 score is highly useful for gauging model performance in classification tasks, especially in cases where the distribution of positive and negative classes is skewed.

Test Mechanism

The F1 score for the validation dataset is computed through scikit-learn's metrics in Python. The scoring mechanism differs based on the classification problem: for multi-class problems, macro averaging is used, and for binary classification, the built-in f1_score calculation is used. The obtained F1 score is then assessed against the predefined minimum F1 score that is expected from the model.

Signs of High Risk

• If a model returns an F1 score that is less than the established threshold, it is regarded as high risk.
• A low F1 score might suggest that the model is not finding an optimal balance between precision and recall, failing to effectively identify positive classes while minimizing false positives.

Strengths

• Provides a balanced measure of a model's performance by accounting for both false positives and false negatives.
• Particularly advantageous in scenarios with imbalanced class distribution, where accuracy can be misleading.
• Flexibility in setting the threshold value allows tailored minimum acceptable performance standards.

Limitations

• May not be suitable for all types of models and machine learning tasks.
• The F1 score assumes an equal cost for false positives and false negatives, which may not be true in some real-world scenarios.
• Practitioners might need to rely on other metrics such as precision, recall, or the ROC-AUC score that align more closely with specific requirements.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
min_threshold	0.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.MinimumF1Score", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ Test: Minimum ROCAUC Score ('validmind.model_validation.sklearn.MinimumROCAUCScore')

Minimum ROCAUC Score

Validates model by checking if the ROC AUC score meets or surpasses a specified threshold.

Purpose

The Minimum ROC AUC Score test is used to determine the model's performance by ensuring that the Receiver Operating Characteristic Area Under the Curve (ROC AUC) score on the validation dataset meets or exceeds a predefined threshold. The ROC AUC score indicates how well the model can distinguish between different classes, making it a crucial measure in binary and multiclass classification tasks.

Test Mechanism

This test implementation calculates the multiclass ROC AUC score on the true target values and the model's predictions. The test converts the multi-class target variables into binary format using LabelBinarizer before computing the score. If this ROC AUC score is higher than the predefined threshold (defaulted to 0.5), the test passes; otherwise, it fails. The results, including the ROC AUC score, the threshold, and whether the test passed or failed, are then stored in a ThresholdTestResult object.

Signs of High Risk

• A high risk or failure in the model's performance as related to this metric would be represented by a low ROC AUC score, specifically any score lower than the predefined minimum threshold. This suggests that the model is struggling to distinguish between different classes effectively.

Strengths

• The test considers both the true positive rate and false positive rate, providing a comprehensive performance measure.
• ROC AUC score is threshold-independent meaning it measures the model's quality across various classification thresholds.
• Works robustly with binary as well as multi-class classification problems.

Limitations

• ROC AUC may not be useful if the class distribution is highly imbalanced; it could perform well in terms of AUC but still fail to predict the minority class.
• The test does not provide insight into what specific aspects of the model are causing poor performance if the ROC AUC score is unsatisfactory.
• The use of macro average for multiclass ROC AUC score implies equal weightage to each class, which might not be appropriate if the classes are imbalanced.

Required Inputs: dataset, model

Parameters:

Parameter	Default Value
min_threshold	0.5

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset",
    "model": "my_vm_model"
}
params = {
    "min_threshold": "my_vm_min_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.MinimumROCAUCScore", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 3.3. Model Explainability and Interpretability ('explainability')

▶ Test: Permutation Feature Importance ('validmind.model_validation.sklearn.PermutationFeatureImportance')

Permutation Feature Importance

Assesses the significance of each feature in a model by evaluating the impact on model performance when feature values are randomly rearranged.

Purpose

The Permutation Feature Importance (PFI) metric aims to assess the importance of each feature used by the Machine Learning model. The significance is measured by evaluating the decrease in the model's performance when the feature's values are randomly arranged.

Test Mechanism

PFI is calculated via the permutation_importance method from the sklearn.inspection module. This method shuffles the columns of the feature dataset and measures the impact on the model's performance. A significant decrease in performance after permutating a feature's values deems the feature as important. On the other hand, if performance remains the same, the feature is likely not important. The output of the PFI metric is a figure illustrating the importance of each feature.

Signs of High Risk

• The model heavily relies on a feature with highly variable or easily permutable values, indicating instability.
• A feature deemed unimportant by the model but expected to have a significant effect on the outcome based on domain knowledge is not influencing the model's predictions.

Strengths

• Provides insights into the importance of different features and may reveal underlying data structure.
• Can indicate overfitting if a particular feature or set of features overly impacts the model's predictions.
• Model-agnostic and can be used with any classifier that provides a measure of prediction accuracy before and after feature permutation.

Limitations

• Does not imply causality; it only presents the amount of information that a feature provides for the prediction task.
• Does not account for interactions between features. If features are correlated, the permutation importance may allocate importance to one and not the other.
• Cannot interact with certain libraries like statsmodels, pytorch, catboost, etc., thus limiting its applicability.

Required Inputs: model, dataset

Parameters:

Parameter	Default Value
fontsize	None
figure_height	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {
    "fontsize": "my_vm_fontsize",
    "figure_height": "my_vm_figure_height"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.PermutationFeatureImportance", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'validmind.model_validation.sklearn.SHAPGlobalImportance_global_importance_text'

▶ Test: SHAP Global Importance ('validmind.model_validation.sklearn.SHAPGlobalImportance')

SHAP Global Importance

Evaluates and visualizes global feature importance using SHAP values for model explanation and risk identification.

Purpose

The SHAP (SHapley Additive exPlanations) Global Importance metric aims to elucidate model outcomes by attributing them to the contributing features. It assigns a quantifiable global importance to each feature via their respective absolute Shapley values, thereby making it suitable for tasks like classification (both binary and multiclass). This metric forms an essential part of model risk management.

Test Mechanism

The exam begins with the selection of a suitable explainer which aligns with the model's type. For tree-based models like XGBClassifier, RandomForestClassifier, CatBoostClassifier, TreeExplainer is used whereas for linear models like LogisticRegression, XGBRegressor, LinearRegression, it is the LinearExplainer. Once the explainer calculates the Shapley values, these values are visualized using two specific graphical representations:

1. Mean Importance Plot: This graph portrays the significance of individual features based on their absolute Shapley values. It calculates the average of these absolute Shapley values across all instances to highlight the global importance of features.

2. Summary Plot: This visual tool combines the feature importance with their effects. Every dot on this chart represents a Shapley value for a certain feature in a specific case. The vertical axis is denoted by the feature whereas the horizontal one corresponds to the Shapley value. A color gradient indicates the value of the feature, gradually changing from low to high. Features are systematically organized in accordance with their importance.

Signs of High Risk

• Overemphasis on certain features in SHAP importance plots, thus hinting at the possibility of model overfitting
• Anomalies such as unexpected or illogical features showing high importance, which might suggest that the model's decisions are rooted in incorrect or undesirable reasoning
• A SHAP summary plot filled with high variability or scattered data points, indicating a cause for concern

Strengths

• SHAP does more than just illustrating global feature significance, it offers a detailed perspective on how different features shape the model's decision-making logic for each instance.
• It provides clear insights into model behavior.

Limitations

• High-dimensional data can convolute interpretations.
• Associating importance with tangible real-world impact still involves a certain degree of subjectivity.

Required Inputs: model, dataset

Parameters:

Parameter	Default Value
kernel_explainer_samples	10
tree_or_linear_explainer_samples	200
class_of_interest	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "dataset": "my_vm_dataset"
}
params = {
    "kernel_explainer_samples": "my_vm_kernel_explainer_samples",
    "tree_or_linear_explainer_samples": "my_vm_tree_or_linear_explainer_samples",
    "class_of_interest": "my_vm_class_of_interest"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.SHAPGlobalImportance", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 3.4. Model Diagnosis ('model_diagnosis')

Text Block: 'model_weak_spots_description'

▶ Test: Weakspots Diagnosis ('validmind.model_validation.sklearn.WeakspotsDiagnosis')

Weakspots Diagnosis

Identifies and visualizes weak spots in a machine learning model's performance across various sections of the feature space.

Purpose

The weak spots test is applied to evaluate the performance of a machine learning model within specific regions of its feature space. This test slices the feature space into various sections, evaluating the model's outputs within each section against specific performance metrics (e.g., accuracy, precision, recall, and F1 scores). The ultimate aim is to identify areas where the model's performance falls below the set thresholds, thereby exposing its possible weaknesses and limitations.

Test Mechanism

The test mechanism adopts an approach of dividing the feature space of the training dataset into numerous bins. The model's performance metrics (accuracy, precision, recall, F1 scores) are then computed for each bin on both the training and test datasets. A "weak spot" is identified if any of the performance metrics fall below a predetermined threshold for a particular bin on the test dataset. The test results are visually plotted as bar charts for each performance metric, indicating the bins which fail to meet the established threshold.

Signs of High Risk

• Any performance metric of the model dropping below the set thresholds.
• Significant disparity in performance between the training and test datasets within a bin could be an indication of overfitting.
• Regions or slices with consistently low performance metrics. Such instances could mean that the model struggles to handle specific types of input data adequately, resulting in potentially inaccurate predictions.

Strengths

• The test helps pinpoint precise regions of the feature space where the model's performance is below par, allowing for more targeted improvements to the model.
• The graphical presentation of the performance metrics offers an intuitive way to understand the model's performance across different feature areas.
• The test exhibits flexibility, letting users set different thresholds for various performance metrics according to the specific requirements of the application.

Limitations

• The binning system utilized for the feature space in the test could over-simplify the model's behavior within each bin. The granularity of this slicing depends on the chosen 'bins' parameter and can sometimes be arbitrary.
• The effectiveness of this test largely hinges on the selection of thresholds for the performance metrics, which may not hold universally applicable and could be subjected to the specifications of a particular model and application.
• The test is unable to handle datasets with a text column, limiting its application to numerical or categorical data types only.
• Despite its usefulness in highlighting problematic regions, the test does not offer direct suggestions for model improvement.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
features_columns	None
metrics	None
thresholds	None

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "features_columns": "my_vm_features_columns",
    "metrics": "my_vm_metrics",
    "thresholds": "my_vm_thresholds"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.WeakspotsDiagnosis", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'model_overfit_regions_description'

▶ Test: Overfit Diagnosis ('validmind.model_validation.sklearn.OverfitDiagnosis')

Overfit Diagnosis

Assesses potential overfitting in a model's predictions, identifying regions where performance between training and testing sets deviates significantly.

Purpose

The Overfit Diagnosis test aims to identify areas in a model's predictions where there is a significant difference in performance between the training and testing sets. This test helps to pinpoint specific regions or feature segments where the model may be overfitting.

Test Mechanism

This test compares the model's performance on training versus test data, grouped by feature columns. It calculates the difference between the training and test performance for each group and identifies regions where this difference exceeds a specified threshold:

• The test works for both classification and regression models.
• It defaults to using the AUC metric for classification models and the MSE metric for regression models.
• The threshold for identifying overfitting regions is set to 0.04 by default.
• The test calculates the performance metrics for each feature segment and plots regions where the performance gap exceeds the threshold.

Signs of High Risk

• Significant gaps between training and test performance metrics for specific feature segments.
• Multiple regions with performance gaps exceeding the defined threshold.
• Higher than expected differences in predicted versus actual values in the test set compared to the training set.

Strengths

• Identifies specific areas where overfitting occurs.
• Supports multiple performance metrics, providing flexibility.
• Applicable to both classification and regression models.
• Visualization of overfitting segments aids in better understanding and debugging.

Limitations

• The default threshold may not be suitable for all use cases and requires tuning.
• May not capture more subtle forms of overfitting that do not exceed the threshold.
• Assumes that the binning of features adequately represents the data segments.

Required Inputs: model, datasets

Parameters:

Parameter	Default Value
metric	None
cut_off_threshold	0.04

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "model": "my_vm_model",
    "datasets": "my_vm_datasets"
}
params = {
    "metric": "my_vm_metric",
    "cut_off_threshold": "my_vm_cut_off_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.OverfitDiagnosis", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Text Block: 'model_robustness_description'

▶ Test: Robustness Diagnosis ('validmind.model_validation.sklearn.RobustnessDiagnosis')

Robustness Diagnosis

Assesses the robustness of a machine learning model by evaluating performance decay under noisy conditions.

Purpose

The Robustness Diagnosis test aims to evaluate the resilience of a machine learning model when subjected to perturbations or noise in its input data. This is essential for understanding the model's ability to handle real-world scenarios where data may be imperfect or corrupted.

Test Mechanism

This test introduces Gaussian noise to the numeric input features of the datasets at varying scales of standard deviation. The performance of the model is then measured using a specified metric. The process includes:

• Adding Gaussian noise to numerical input features based on scaling factors.
• Evaluating the model's performance on the perturbed data using metrics like AUC for classification tasks and MSE for regression tasks.
• Aggregating and plotting the results to visualize performance decay relative to perturbation size.

Signs of High Risk

• A significant drop in performance metrics with minimal noise.
• Performance decay values exceeding the specified threshold.
• Consistent failure to meet performance standards across multiple perturbation scales.

Strengths

• Provides insights into the model's robustness against noisy or corrupted data.
• Utilizes a variety of performance metrics suitable for both classification and regression tasks.
• Visualization helps in understanding the extent of performance degradation.

Limitations

• Gaussian noise might not adequately represent all types of real-world data perturbations.
• Performance thresholds are somewhat arbitrary and might need tuning.
• The test may not account for more complex or unstructured noise patterns that could affect model robustness.

Required Inputs: datasets, model

Parameters:

Parameter	Default Value
metric	None
scaling_factor_std_dev_list	[0.1, 0.2, 0.3, 0.4, 0.5]
performance_decay_threshold	0.05

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "datasets": "my_vm_datasets",
    "model": "my_vm_model"
}
params = {
    "metric": "my_vm_metric",
    "scaling_factor_std_dev_list": "my_vm_scaling_factor_std_dev_list",
    "performance_decay_threshold": "my_vm_performance_decay_threshold"
}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.model_validation.sklearn.RobustnessDiagnosis", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

▶ 4. Monitoring and Governance ('monitoring_governance')

▶ 4.1. Monitoring Plan ('monitoring_plan')

Text Block: 'monitoring_plan'

▶ 4.2. Monitoring Implementation ('monitoring_implementation')

Text Block: 'monitoring_implementation'

▶ 4.3. Governance Plan ('governance_plan')

Text Block: 'governance_plan'

Include custom test results

Since your custom test IDs are now part of your documentation template, you can now run tests for an entire section and all additional custom tests should be loaded without any issues.

Let's run all tests in the Model Evaluation section of the documentation. Note that we have been running the sample custom confusion matrix with normalize=True to demonstrate the ability to provide custom parameters.

In the Run the model evaluation tests section of 2 — Start the development process, you learned how to assign inputs to individual tests with run_documentation_tests(). Assigning parameters is similar, you only need to provide assign a params dictionary to a given test ID, my_test_provider.ConfusionMatrix in this case.

test_config = {
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "dataset": vm_train_ds,
            "model": vm_model,
        },
    },
    "my_test_provider.ConfusionMatrix": {
        "params": {"normalize": True},
        "inputs": {"dataset": vm_test_ds, "model": vm_model},
    },
}
results = vm.run_documentation_tests(
    section=["model_evaluation"],
    inputs={
        "dataset": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds
        "model": vm_model,
        "datasets": (
            vm_train_ds,
            vm_test_ds,
        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds
    },
    config=test_config,
)

2026-07-31 16:43:44,244 - WARNING(validmind.vm_models.test_suite.runner): Config key 'my_test_provider.ConfusionMatrix' does not match a test_id in the template.
    Ensure you registered a content block with the correct content_id in the template
    The configuration for this test will be ignored.

Test suite complete!

18/18 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Evaluation

Model Evaluation

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

The Confusion Matrix test evaluates classification performance by comparing predicted labels against observed labels and summarizing the results as true positives, true negatives, false positives, and false negatives. The heatmap shows counts for each outcome across the two classes, with 221 true negatives, 110 false positives, 111 false negatives, and 205 true positives. These counts provide a direct view of how predictions are distributed between correct classifications and the two types of misclassification.

Key insights:

• Correct classifications exceed errors: The model records 221 true negatives and 205 true positives, for 426 correct classifications in total, compared with 110 false positives and 111 false negatives, for 221 misclassifications.
• Misclassification types are nearly balanced: False positives and false negatives are almost equal in magnitude, with counts of 110 and 111 respectively. This indicates that the observed error distribution is not concentrated in only one error type.
• Negative-class identification is slightly stronger: True negatives (221) are higher than true positives (205). This indicates slightly more correct classifications for class 0 than for class 1 in the observed sample.
• Observed classes are relatively balanced: For true class 0, the matrix shows 221 negatives and 110 false positives, totaling 331 observations. For true class 1, the matrix shows 205 positives and 111 false negatives, totaling 316 observations, indicating similar representation across the two classes.

The confusion matrix shows that correct predictions outnumber misclassifications, with both classes contributing materially to the correctly classified outcomes. Error counts are closely balanced between false positives and false negatives, and the class distribution in the evaluated sample is relatively even. Overall, the result reflects a model whose observed classification performance is distributed across both classes without a pronounced concentration of error in a single class or error type.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

The Classifier Performance test evaluates classification model performance using precision, recall, F1-score, accuracy, and ROC AUC. The in-sample results are presented for both classes and as macro and weighted averages. Class-level precision ranges from 0.6243 to 0.6346, recall ranges from 0.6200 to 0.6389, and F1 ranges from 0.6272 to 0.6315. Aggregate metrics show weighted-average and macro-average F1 of 0.6294, overall accuracy of 0.6294, and ROC AUC of 0.6774.

Key insights:

• Balanced class-level performance: The two classes show similar results across all reported metrics. Precision differs by 0.0103 between classes, recall differs by 0.0189, and F1 differs by 0.0043.
• Aggregate metrics are closely aligned: Weighted-average precision, recall, and F1 are 0.6295, 0.6294, and 0.6294, respectively, while macro-average values are also 0.6295, 0.6295, and 0.6294. This indicates little variation between class-balanced and support-weighted summaries in the reported results.
• Class 0 has slightly higher recall: Recall for class 0 is 0.6389 compared with 0.6200 for class 1, while class 1 has slightly higher precision at 0.6346 versus 0.6243 for class 0.
• ROC AUC exceeds accuracy and F1: ROC AUC is 0.6774, compared with accuracy of 0.6294 and average F1 of 0.6294. This shows stronger rank-order discrimination than the threshold-based classification metrics.

The in-sample results show a broadly even performance profile across the two classes, with only small differences between class-specific precision, recall, and F1 values. Aggregate macro and weighted averages are effectively identical, indicating that the summarized performance is stable across the reported averaging approaches. Overall accuracy and F1 are 0.6294, and ROC AUC of 0.6774 indicates somewhat stronger separability than is reflected in the threshold-based classification outcomes.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6243	0.6389	0.6315
1	0.6346	0.6200	0.6272
Weighted Average	0.6295	0.6294	0.6294
Macro Average	0.6295	0.6295	0.6294

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6294
ROC AUC	0.6774

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

The Classifier Performance test evaluates out-of-sample classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The results are reported for both classes, along with macro and weighted averages across classes. Class-level precision, recall, and F1 values are shown for classes 0 and 1, while overall performance is summarized by accuracy of 0.6584 and ROC AUC of 0.7058. The reported macro and weighted averages are both approximately 0.658 across precision, recall, and F1.

Key insights:

• Class performance is balanced: Class 0 and class 1 show similar precision, recall, and F1 values. F1 is 0.6667 for class 0 and 0.6498 for class 1, indicating limited dispersion in class-level performance.
• Aggregate metrics are closely aligned: Weighted-average precision, recall, and F1 are each 0.6584, and macro-average values are 0.6582 across the same metrics. The close alignment between macro and weighted averages indicates little difference between the aggregated scoring views reported here.
• ROC AUC exceeds accuracy: ROC AUC is 0.7058 compared with accuracy of 0.6584. This shows stronger ranking performance than point-classification accuracy in the reported out-of-sample evaluation.
• Class 0 is marginally stronger: Class 0 records slightly higher precision (0.6657 vs. 0.6508), recall (0.6677 vs. 0.6487), and F1 (0.6667 vs. 0.6498) than class 1.

The out-of-sample results show moderate and internally consistent classification performance across the reported metrics. Performance is relatively even across the two classes, with class 0 modestly outperforming class 1 on all class-level measures. The near-equality of macro and weighted averages indicates that the aggregate performance summaries are stable across averaging methods, while the ROC AUC of 0.7058 is higher than the observed accuracy of 0.6584.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6657	0.6677	0.6667
1	0.6508	0.6487	0.6498
Weighted Average	0.6584	0.6584	0.6584
Macro Average	0.6582	0.6582	0.6582

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6584
ROC AUC	0.7058

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

The PrecisionRecallCurve test evaluates the trade-off between precision and recall across classification thresholds for a binary model. The result is presented as a precision-recall curve with recall on the horizontal axis and precision on the vertical axis, showing how precision changes as recall increases. In this output, the curve begins with very high precision at very low recall, rises into the upper-0.8 range at low recall levels, and then declines gradually as recall extends toward 1.0. By high recall levels, precision approaches approximately 0.50.

Key insights:

• High precision at low recall: At very low recall, precision reaches approximately 1.0 and remains above 0.80 through the early portion of the curve, indicating stronger positive identification at more selective thresholds.

• Gradual precision decay: After the initial peak, precision declines progressively as recall increases, moving from the upper-0.7 to low-0.6 range through the mid-to-late sections of the curve.

• Mid-range operating region remains elevated: Across roughly the middle of the recall range, precision is generally maintained around the high-0.6 to high-0.7 range, showing a relatively smooth trade-off rather than an abrupt deterioration.

• Precision converges near 0.50 at full recall: As recall approaches 1.0, precision falls to about 0.50, reflecting weaker class selectivity when the threshold is loosened to capture nearly all positive cases.

The curve shows a clear inverse relationship between recall and precision, with the strongest precision concentrated at low recall levels and a steady reduction as recall increases. The decline is gradual through most of the curve, with mid-range precision remaining materially above 0.60 before falling further at high recall. Overall, the result indicates that model discrimination is strongest under more selective thresholds and becomes less precise as coverage of positive cases expands.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

The ROCCurve test evaluates classification performance by plotting the Receiver Operating Characteristic curve and calculating the Area Under the Curve (AUC) to measure class discrimination. The result for log_reg_model_v1 on test_dataset_final shows a single ROC curve for the binary classification setting alongside the random-classifier reference line. The plotted curve remains above the diagonal baseline across the full false positive rate range, and the legend reports an AUC value of 0.71.

Key insights:

• AUC indicates positive discrimination: The reported AUC of 0.71 shows that the model separates the two classes better than the random baseline shown at AUC = 0.5.

• ROC curve stays above randomness: The ROC curve is consistently positioned above the diagonal reference line, indicating higher true positive rates than a random classifier across thresholds.

• Performance varies across thresholds: The curve rises more gradually through the mid-range of false positive rates and approaches a true positive rate near 1.0 only at high false positive rates, reflecting threshold-dependent trade-offs between sensitivity and false alarms.

The ROC result indicates that log_reg_model_v1 demonstrates measurable classification discrimination on test_dataset_final, with performance above random ranking as reflected by the AUC of 0.71. The curve’s position above the baseline throughout the range supports this result, while its gradual shape through intermediate thresholds shows that gains in true positive rate are accompanied by increasing false positive rate rather than near-perfect separation.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

The TrainingTestDegradation test evaluates whether model performance degradation from the training dataset to the test dataset remains within a predefined threshold across classification metrics. The results table compares training and test scores for precision, recall, and F1-score for both Class 1 and Class 0, and reports degradation as the relative change from training to test performance. In this run, all reported degradation values are negative, indicating higher metric values on the test dataset than on the training dataset. The table also records a pass result for every metric evaluated.

Key insights:

• All metrics passed the threshold: Every evaluated combination of class and metric is marked as Pass. No reported degradation exceeds the stated maximum threshold.
• Test scores exceed training scores throughout: All six degradation values are negative, reflecting higher performance on the test dataset than on the training dataset for both classes across precision, recall, and F1-score.
• Largest improvement appears in Class 0 precision: Class 0 precision increased from 0.6243 on the training dataset to 0.6657 on the test dataset, corresponding to a degradation value of -6.6195%, which is the largest magnitude change in the table.
• Class 1 metrics also improved consistently: For Class 1, precision increased from 0.6346 to 0.6508, recall from 0.6200 to 0.6487, and F1-score from 0.6272 to 0.6498, with degradation values ranging from -2.5444% to -4.6345%.
• Performance gains are observed across both classes: Class 0 shows improvements in precision, recall, and F1-score from 0.6243, 0.6389, and 0.6315 on training to 0.6657, 0.6677, and 0.6667 on test, while Class 1 shows the same directional pattern across all three metrics.

The results indicate that the measured training-to-test degradation criterion was satisfied for all evaluated metrics and classes. Observed metric changes are uniformly favorable on the test dataset, with negative degradation values ranging from -2.5444% to -6.6195%. Across both Class 1 and Class 0, precision, recall, and F1-score all improved from training to test, and no metric shows evidence of performance decline under this test.

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
1	Precision	0.6346	0.6508	-2.5444	Pass
1	Recall	0.6200	0.6487	-4.6345	Pass
1	F1-Score	0.6272	0.6498	-3.5911	Pass
0	Precision	0.6243	0.6657	-6.6195	Pass
0	Recall	0.6389	0.6677	-4.5019	Pass
0	F1-Score	0.6315	0.6667	-5.5623	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

The Minimum Accuracy test evaluates whether the model’s prediction accuracy meets or exceeds a predefined threshold. The result table reports an accuracy score of 0.6584 against a threshold of 0.7000, along with the associated pass/fail outcome. The recorded outcome for this test run is Fail.

Key insights:

• Accuracy is below threshold: The observed accuracy score is 0.6584, which is lower than the configured minimum threshold of 0.7000.
• Test outcome is fail: The result is explicitly marked as Fail, reflecting that the measured accuracy did not meet the required cutoff.
• Shortfall is measurable: The difference between the observed score and threshold is 0.0416, indicating the extent to which the result falls below the minimum requirement.

This test result shows that the model did not satisfy the minimum accuracy criterion in this run. The measured accuracy remained below the specified threshold, and the fail status is consistent with that gap.

Tables

Score	Threshold	Pass/Fail
0.6584	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

The MinimumF1Score test evaluates whether the model’s F1 score on the validation dataset meets a predefined minimum threshold. The result table reports a validation F1 score of 0.6498, a threshold of 0.5, and an overall test outcome of Pass. These values indicate the comparison used in the test and the resulting status for this validation run.

Key insights:

• F1 score exceeds threshold: The observed validation F1 score is 0.6498 versus a minimum threshold of 0.5, placing the score 0.1498 above the configured cutoff.
• Test outcome is pass: The recorded Pass/Fail status is Pass, consistent with the reported score-threshold comparison.

The test result shows that the model’s validation F1 score exceeded the predefined minimum threshold in this run. The reported pass status aligns directly with the numerical result, with the observed score remaining above the configured acceptance level.

Tables

Score	Threshold	Pass/Fail
0.6498	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

The Minimum ROC AUC Score test evaluates whether the model’s ROC AUC score on the validation dataset meets or exceeds the defined minimum threshold. The reported result includes the observed ROC AUC score, the comparison threshold, and the pass/fail outcome for the test. In this run, the model produced a ROC AUC score of 0.7058 against a threshold of 0.5, and the test result is recorded as Pass.

Key insights:

• Score exceeds threshold: The observed ROC AUC score of 0.7058 is above the minimum threshold of 0.5 by 0.2058.
• Test status is pass: The result is recorded as Pass based on the score exceeding the defined threshold.
• Single threshold comparison reported: The output provides one validation score and one threshold comparison, with no additional segment-level or class-level breakdown included in the result.

The result shows that the model satisfied the Minimum ROC AUC Score test under the configured validation criterion. The observed ROC AUC score exceeded the specified threshold, resulting in a passing outcome. The reported evidence is limited to the aggregate score-to-threshold comparison documented in the test output.

Tables

Score	Threshold	Pass/Fail
0.7058	0.5	Pass

Configuring documentation template tests

Let's call the utility function vm.get_test_suite().get_default_config() which will return the default configuration for the entire documentation template as a dictionary:

• This configuration will contain all the test IDs and their default parameters.
• You can then modify this configuration as needed and pass it to run_documentation_tests() to run all tests in the documentation template if needed.
• You still have the option to continue running tests for one section at a time; get_default_config() simply provides a useful reference for providing default parameters to every test.

import json

model_test_suite = vm.get_test_suite()
config = model_test_suite.get_default_config()
print("Suite Config: \n", json.dumps(config, indent=2))

Suite Config: 
 {
  "validmind.data_validation.DatasetDescription": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.data_validation.ClassImbalance": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_percent_threshold": 10
    }
  },
  "validmind.data_validation.Duplicates": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_threshold": 1
    }
  },
  "validmind.data_validation.HighCardinality": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "num_threshold": 100,
      "percent_threshold": 0.1,
      "threshold_type": "percent"
    }
  },
  "validmind.data_validation.MissingValues": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_percentage_threshold": 1.0
    }
  },
  "validmind.data_validation.Skewness": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "max_threshold": 1
    }
  },
  "validmind.data_validation.UniqueRows": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "min_percent_threshold": 1
    }
  },
  "validmind.data_validation.TooManyZeroValues": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "max_percent_threshold": 0.03
    }
  },
  "validmind.data_validation.IQROutliersTable": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "threshold": 1.5
    }
  },
  "validmind.data_validation.IQROutliersBarPlot": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "threshold": 1.5,
      "fig_width": 800
    }
  },
  "validmind.data_validation.DescriptiveStatistics": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.data_validation.PearsonCorrelationMatrix": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.data_validation.HighPearsonCorrelation": {
    "inputs": {
      "dataset": "dataset"
    },
    "params": {
      "max_threshold": 0.3,
      "top_n_correlations": 10,
      "feature_columns": null
    }
  },
  "validmind.model_validation.ModelMetadata": {
    "inputs": {
      "model": "model"
    },
    "params": {}
  },
  "validmind.data_validation.DatasetSplit": {
    "inputs": {
      "datasets": "datasets"
    },
    "params": {}
  },
  "validmind.model_validation.sklearn.PopulationStabilityIndex": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "num_bins": 10,
      "mode": "fixed"
    }
  },
  "validmind.model_validation.sklearn.ConfusionMatrix": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "threshold": 0.5
    }
  },
  "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "average": "macro"
    }
  },
  "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "average": "macro"
    }
  },
  "validmind.model_validation.sklearn.PrecisionRecallCurve": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.model_validation.sklearn.ROCCurve": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {}
  },
  "validmind.model_validation.sklearn.TrainingTestDegradation": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "max_threshold": 0.1
    }
  },
  "validmind.model_validation.sklearn.MinimumAccuracy": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "min_threshold": 0.7
    }
  },
  "validmind.model_validation.sklearn.MinimumF1Score": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "min_threshold": 0.5
    }
  },
  "validmind.model_validation.sklearn.MinimumROCAUCScore": {
    "inputs": {
      "dataset": "dataset",
      "model": "model"
    },
    "params": {
      "min_threshold": 0.5
    }
  },
  "validmind.model_validation.sklearn.PermutationFeatureImportance": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {
      "fontsize": null,
      "figure_height": null
    }
  },
  "validmind.model_validation.sklearn.SHAPGlobalImportance": {
    "inputs": {
      "model": "model",
      "dataset": "dataset"
    },
    "params": {
      "kernel_explainer_samples": 10,
      "tree_or_linear_explainer_samples": 200,
      "class_of_interest": null
    }
  },
  "validmind.model_validation.sklearn.WeakspotsDiagnosis": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "features_columns": null,
      "metrics": null,
      "thresholds": null
    }
  },
  "validmind.model_validation.sklearn.OverfitDiagnosis": {
    "inputs": {
      "model": "model",
      "datasets": "datasets"
    },
    "params": {
      "metric": null,
      "cut_off_threshold": 0.04
    }
  },
  "validmind.model_validation.sklearn.RobustnessDiagnosis": {
    "inputs": {
      "datasets": "datasets",
      "model": "model"
    },
    "params": {
      "metric": null,
      "scaling_factor_std_dev_list": [
        0.1,
        0.2,
        0.3,
        0.4,
        0.5
      ],
      "performance_decay_threshold": 0.05
    }
  }
}

Preview test configuration

The default config does not assign any inputs to a test, but you can assign inputs to individual tests as needed depending on the datasets and records (models) you want to pass to individual tests.

For this particular documentation template (binary classification), the ValidMind Library provides a sample configuration that can be used to populate the entire documentation using the following inputs as placeholders:

• A raw_dataset raw dataset
• A train_dataset training dataset
• A test_dataset test dataset
• A trained model instance

As part of updating the config you will need to ensure the correct input_ids are used in the final config passed to run_documentation_tests().

from validmind.datasets.classification import customer_churn
from validmind.utils import preview_test_config

test_config = customer_churn.get_demo_test_config()
preview_test_config(test_config)

{
    "validmind.data_validation.DatasetDescription": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {}
    },
    "validmind.data_validation.ClassImbalance": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_percent_threshold": 10
        }
    },
    "validmind.data_validation.Duplicates": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_threshold": 1
        }
    },
    "validmind.data_validation.HighCardinality": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "num_threshold": 100,
            "percent_threshold": 0.1,
            "threshold_type": "percent"
        }
    },
    "validmind.data_validation.MissingValues": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_percentage_threshold": 1.0
        }
    },
    "validmind.data_validation.Skewness": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "max_threshold": 1
        }
    },
    "validmind.data_validation.UniqueRows": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "min_percent_threshold": 1
        }
    },
    "validmind.data_validation.TooManyZeroValues": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "max_percent_threshold": 0.03
        }
    },
    "validmind.data_validation.IQROutliersTable": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "threshold": 1.5
        }
    },
    "validmind.data_validation.IQROutliersBarPlot": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "threshold": 1.5,
            "fig_width": 800
        }
    },
    "validmind.data_validation.DescriptiveStatistics": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {}
    },
    "validmind.data_validation.PearsonCorrelationMatrix": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {}
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "inputs": {
            "dataset": "raw_dataset"
        },
        "params": {
            "max_threshold": 0.3,
            "top_n_correlations": 10,
            "feature_columns": null
        }
    },
    "validmind.model_validation.ModelMetadata": {
        "inputs": {
            "model": "model"
        },
        "params": {}
    },
    "validmind.data_validation.DatasetSplit": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ]
        },
        "params": {}
    },
    "validmind.model_validation.sklearn.PopulationStabilityIndex": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "num_bins": 10,
            "mode": "fixed"
        }
    },
    "validmind.model_validation.sklearn.ConfusionMatrix": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "threshold": 0.5
        }
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "model": "model",
            "dataset": "train_dataset"
        }
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        }
    },
    "validmind.model_validation.sklearn.PrecisionRecallCurve": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {}
    },
    "validmind.model_validation.sklearn.ROCCurve": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {}
    },
    "validmind.model_validation.sklearn.TrainingTestDegradation": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "max_threshold": 0.1
        }
    },
    "validmind.model_validation.sklearn.MinimumAccuracy": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "min_threshold": 0.7
        }
    },
    "validmind.model_validation.sklearn.MinimumF1Score": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "min_threshold": 0.5
        }
    },
    "validmind.model_validation.sklearn.MinimumROCAUCScore": {
        "inputs": {
            "dataset": "test_dataset",
            "model": "model"
        },
        "params": {
            "min_threshold": 0.5
        }
    },
    "validmind.model_validation.sklearn.PermutationFeatureImportance": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {
            "fontsize": null,
            "figure_height": null
        }
    },
    "validmind.model_validation.sklearn.SHAPGlobalImportance": {
        "inputs": {
            "model": "model",
            "dataset": "test_dataset"
        },
        "params": {
            "kernel_explainer_samples": 10,
            "tree_or_linear_explainer_samples": 200,
            "class_of_interest": null
        }
    },
    "validmind.model_validation.sklearn.WeakspotsDiagnosis": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "features_columns": null,
            "metrics": null,
            "thresholds": null
        }
    },
    "validmind.model_validation.sklearn.OverfitDiagnosis": {
        "inputs": {
            "model": "model",
            "datasets": [
                "train_dataset",
                "test_dataset"
            ]
        },
        "params": {
            "metric": null,
            "cut_off_threshold": 0.04
        }
    },
    "validmind.model_validation.sklearn.RobustnessDiagnosis": {
        "inputs": {
            "datasets": [
                "train_dataset",
                "test_dataset"
            ],
            "model": "model"
        },
        "params": {
            "metric": null,
            "scaling_factor_std_dev_list": [
                0.1,
                0.2,
                0.3,
                0.4,
                0.5
            ],
            "performance_decay_threshold": 0.05
        }
    }
}

Run updated documentation section tests

Using this sample configuration, let's finish populating documentation by running all tests for the Model Development section of the documentation.

Recall that the training and test datasets in our exercise have the following input_id values:

• train_dataset_final for the training dataset
• test_dataset_final for the test dataset

config = {
    "validmind.model_validation.ModelMetadata": {
        "inputs": {"model": "log_reg_model_v1"},
    },
    "validmind.data_validation.DatasetSplit": {
        "inputs": {"datasets": ["train_dataset_final", "test_dataset_final"]},
    },
    "validmind.model_validation.sklearn.PopulationStabilityIndex": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {"num_bins": 10, "mode": "fixed"},
    },
    "validmind.model_validation.sklearn.ConfusionMatrix": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "my_test_provider.ConfusionMatrix": {
        "inputs": {"dataset": "test_dataset_final", "model": "log_reg_model_v1"},
    },
    "my_custom_tests.ConfusionMatrix:test_dataset_normalized": {
        "inputs": {"dataset": "test_dataset_final", "model": "log_reg_model_v1"},
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "train_dataset_final"}
    },
    "validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"}
    },
    "validmind.model_validation.sklearn.PrecisionRecallCurve": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "validmind.model_validation.sklearn.ROCCurve": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "validmind.model_validation.sklearn.TrainingTestDegradation": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {
            "metrics": ["accuracy", "precision", "recall", "f1"],
            "max_threshold": 0.1,
        },
    },
    "validmind.model_validation.sklearn.MinimumAccuracy": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"min_threshold": 0.7},
    },
    "validmind.model_validation.sklearn.MinimumF1Score": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"min_threshold": 0.5},
    },
    "validmind.model_validation.sklearn.MinimumROCAUCScore": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"min_threshold": 0.5},
    },
    "validmind.model_validation.sklearn.PermutationFeatureImportance": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
    },
    "validmind.model_validation.sklearn.SHAPGlobalImportance": {
        "inputs": {"model": "log_reg_model_v1", "dataset": "test_dataset_final"},
        "params": {"kernel_explainer_samples": 10},
    },
    "validmind.model_validation.sklearn.WeakspotsDiagnosis": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {
            "thresholds": {"accuracy": 0.75, "precision": 0.5, "recall": 0.5, "f1": 0.7}
        },
    },
    "validmind.model_validation.sklearn.OverfitDiagnosis": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {"cut_off_percentage": 4},
    },
    "validmind.model_validation.sklearn.RobustnessDiagnosis": {
        "inputs": {
            "model": "log_reg_model_v1",
            "datasets": ["train_dataset_final", "test_dataset_final"],
        },
        "params": {
            "scaling_factor_std_dev_list": [0.0, 0.1, 0.2, 0.3, 0.4, 0.5],
            "accuracy_decay_threshold": 4,
        },
    },
}


full_suite = vm.run_documentation_tests(
    section="model_development",
    config=config,
)

2026-07-31 16:44:07,893 - WARNING(validmind.vm_models.test_suite.runner): Config key 'my_test_provider.ConfusionMatrix' does not match a test_id in the template.
    Ensure you registered a content block with the correct content_id in the template
    The configuration for this test will be ignored.
2026-07-31 16:44:07,894 - WARNING(validmind.vm_models.test_suite.runner): Config key 'my_custom_tests.ConfusionMatrix:test_dataset_normalized' does not match a test_id in the template.
    Ensure you registered a content block with the correct content_id in the template
    The configuration for this test will be ignored.

Test suite complete!

34/34 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Development

Model Development

▶ Test Result: Model Metadata (validmind.model_validation.ModelMetadata)

Model Metadata

The ModelMetadata test compares model metadata fields to provide a standardized summary of model implementation characteristics. The result table presents the modeling technique, modeling framework, framework version, and programming language for the documented model. In this output, the table contains a single metadata record covering all four reported fields.

Key insights:

• Single model metadata record: The result consists of one row, indicating that the comparison output contains metadata for one model instance only.
• SKlearnModel technique reported: The modeling technique is listed as SKlearnModel, identifying the model class recorded in the metadata summary.
• sklearn framework version documented: The modeling framework is reported as sklearn with framework version 1.9.0, providing a specific implementation reference in the metadata.
• Python language identified: The programming language field is populated as Python, completing the set of reported high-level implementation attributes.

The metadata summary documents a single model implementation using the SKlearnModel technique within the sklearn framework, version 1.9.0, and implemented in Python. All fields displayed in the summary table are populated for this record, and the result serves as a concise inventory of the model’s reported technical metadata.

Tables

Modeling Technique	Modeling Framework	Framework Version	Programming Language
SKlearnModel	sklearn	1.9.0	Python

▶ Test Result: Dataset Split (validmind.data_validation.DatasetSplit)

Dataset Split

The DatasetSplit test evaluates the distribution of observations across the available model datasets. The result table reports the absolute size and share of total records for each dataset included in the split. In this run, the documented datasets are train_dataset_final and test_dataset_final, along with the total combined sample size. The table shows 2,585 records in the training dataset, 647 records in the test dataset, and 3,232 records overall.

Key insights:

• Training set contains most records: train_dataset_final includes 2,585 observations, representing 79.98% of the total dataset.
• Test set accounts for one-fifth: test_dataset_final includes 647 observations, representing 20.02% of the total dataset.
• Two-way split is documented: The reported split includes training and testing datasets only, with no validation dataset shown in the result table.

The dataset split result shows a total sample of 3,232 records allocated across two documented subsets. Most observations are assigned to the training dataset, while the remainder are assigned to the test dataset in an approximately 80/20 distribution. The reported output reflects a train-test split without a separate validation dataset included in this test result.

Tables

Dataset	Size	Proportion
train_dataset_final	2585	79.98%
test_dataset_final	647	20.02%
Total	3232	100%

▶ Test Result: Population Stability Index (validmind.model_validation.sklearn.PopulationStabilityIndex)

Population Stability Index

The Population Stability Index (PSI) test evaluates the stability of the model output distribution by comparing the binned distributions of the initial and new datasets. The results are presented across 10 fixed bins, showing counts, population shares, and bin-level PSI contributions for train_dataset_final and test_dataset_final. The initial dataset contains 2,585 observations and the new dataset contains 647 observations, with bin population shares and PSI contributions reported for each segment. The total PSI across all bins is 0.0198.

Key insights:

• Low overall PSI: The total PSI is 0.0198, indicating that the aggregate difference between the initial and new distributions is small based on the reported bin-level contributions.

• Drift is concentrated in few bins: The largest PSI contributions occur in Bin 7 (0.0056), Bin 2 (0.0042), and Bin 8 (0.0041), while several bins contribute minimally, including Bin 5 (0.0001), Bin 9 (0.0002), and Bin 6 (0.0003).

• Most pronounced share decrease in Bin 7: Bin 7 declines from 10.2901% in the initial dataset to 8.0371% in the new dataset, producing the largest single-bin PSI contribution.

• Largest share increase appears in Bin 4: Bin 4 increases from 15.2031% to 17.0015%, representing the largest upward change in population share among the reported bins, with a PSI contribution of 0.0020.

• Mid-range bins remain close: Bins 3, 5, and 6 show relatively similar population shares between datasets, with differences of approximately 1.25 percentage points or less and PSI contributions at or below 0.0012 for Bins 3 and 6 and 0.0001 for Bin 5.

The PSI results show a high degree of similarity between the initial and new distributions at the aggregate level, as reflected in the total PSI of 0.0198. Observed distributional differences are localized rather than broad-based, with the largest changes concentrated in Bins 2, 7, and 8. Across the remaining bins, population shares are comparatively close and contribute only modestly to the total PSI.

Parameters:

{
  "num_bins": 10,
  "mode": "fixed"
}
            

Tables

Population Stability Index for train_dataset_final and test_dataset_final Datasets

Bin	Count Initial	Percent Initial (%)	Count New	Percent New (%)	PSI
0	130	5.0290	38	5.8733	0.0013
1	217	8.3946	60	9.2736	0.0009
2	289	11.1799	59	9.1190	0.0042
3	360	13.9265	82	12.6739	0.0012
4	393	15.2031	110	17.0015	0.0020
5	335	12.9594	86	13.2921	0.0001
6	324	12.5338	85	13.1376	0.0003
7	266	10.2901	52	8.0371	0.0056
8	118	4.5648	39	6.0278	0.0041
9	153	5.9188	36	5.5641	0.0002
Total	2585	100.0000	647	100.0000	0.0198

Figures

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

The Confusion Matrix test evaluates classification performance by comparing predicted class labels with observed class labels and summarizing results as true positives, true negatives, false positives, and false negatives. The heatmap shows counts for each outcome across the two classes. In this result, the matrix contains 221 true negatives, 205 true positives, 110 false positives, and 111 false negatives. The displayed counts provide a direct view of how predictions are distributed between correct and incorrect classifications for both classes.

Key insights:

• Correct classifications exceed errors: The model records 221 true negatives and 205 true positives, compared with 110 false positives and 111 false negatives. Correct predictions total 426, while misclassifications total 221.
• Error types are nearly balanced: False positives and false negatives are almost identical in magnitude, at 110 and 111 respectively. This indicates that misclassification is distributed similarly across the two error types rather than being concentrated in one direction.
• Class-level correct predictions are comparable: True negatives (221) and true positives (205) are of similar scale. This indicates that the model identifies both class 0 and class 1 correctly at comparable counts in this test sample.
• Observed class counts are similar: The matrix implies 331 observations with true class 0 and 316 observations with true class 1. This reflects a relatively balanced observed class distribution in the evaluated sample.

The confusion matrix shows that correct classifications are more frequent than misclassifications, with similar volumes of true negatives and true positives. Error counts are also closely matched between false positives and false negatives, indicating no strong asymmetry in the observed misclassification pattern. Overall, the result reflects a relatively balanced classification profile across the two classes within the evaluated sample.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

The Classifier Performance test evaluates binary classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The results are presented by class for precision, recall, and F1, along with macro and weighted averages, and separate summary metrics for overall accuracy and ROC AUC. Class-level metrics are reported for classes 0 and 1, with values concentrated in a narrow range, while overall accuracy is 0.6294 and ROC AUC is 0.6774.

Key insights:

• Balanced class-level performance: Precision, recall, and F1 are closely aligned across classes 0 and 1. Class 0 records precision 0.6243, recall 0.6389, and F1 0.6315, while class 1 records precision 0.6346, recall 0.6200, and F1 0.6272.
• Aggregate metrics are highly consistent: Weighted average and macro average metrics are effectively identical, with precision at 0.6295, recall at 0.6294-0.6295, and F1 at 0.6294. This indicates limited dispersion between class-specific and aggregate performance.
• Accuracy matches aggregate recall and F1: Overall accuracy is 0.6294, matching the reported weighted-average recall and F1 to four decimal places. The aggregate classification metrics therefore present a consistent summary of in-sample performance.
• ROC AUC exceeds accuracy level: ROC AUC is 0.6774, which is higher than the reported accuracy of 0.6294. This indicates that ranking performance, as measured by ROC AUC, is stronger than the single-threshold classification result reflected in accuracy.

The in-sample results show a tightly grouped performance profile across both classes and across aggregate metrics. Class-specific precision, recall, and F1 values are all near 0.63, with only small differences between classes, and the macro and weighted averages are nearly identical. Overall accuracy is 0.6294, while ROC AUC is 0.6774, indicating stronger discrimination performance than the threshold-based classification metrics alone.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6243	0.6389	0.6315
1	0.6346	0.6200	0.6272
Weighted Average	0.6295	0.6294	0.6294
Macro Average	0.6295	0.6295	0.6294

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6294
ROC AUC	0.6774

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

The Classifier Performance test evaluates binary classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The results are presented by class for precision, recall, and F1, along with macro and weighted averages that summarize performance across both classes. For Class 0, precision, recall, and F1 are 0.6657, 0.6677, and 0.6667, while for Class 1 they are 0.6508, 0.6487, and 0.6498. Overall accuracy is 0.6584 and ROC AUC is 0.7058.

Key insights:

• Class-level performance is closely aligned: Precision, recall, and F1 are similar across the two classes, with Class 0 showing F1 = 0.6667 and Class 1 showing F1 = 0.6498. The spread between class-level precision and recall values is limited, indicating comparable treatment of both classes in the out-of-sample results.
• Aggregate averages are consistent: Weighted-average and macro-average precision, recall, and F1 all cluster near 0.658, with weighted averages at 0.6584 and macro averages at 0.6582. This narrow difference indicates that the aggregate summary metrics are stable across averaging methods.
• ROC AUC exceeds accuracy and F1 levels: ROC AUC is 0.7058, compared with accuracy of 0.6584 and average F1 values near 0.658. This indicates stronger rank-order separation than the threshold-dependent classification metrics alone.

The out-of-sample results show broadly even classification performance across both classes, with precision, recall, and F1 concentrated in a narrow range around 0.65 to 0.67. Aggregate metrics are internally consistent, as reflected in the close alignment between macro and weighted averages and the matching weighted-average F1 and accuracy value of 0.6584. ROC AUC of 0.7058 is higher than the threshold-based summary metrics, indicating that discrimination as measured by score ranking is stronger than the reported point-classification performance.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6657	0.6677	0.6667
1	0.6508	0.6487	0.6498
Weighted Average	0.6584	0.6584	0.6584
Macro Average	0.6582	0.6582	0.6582

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6584
ROC AUC	0.7058

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

The PrecisionRecallCurve test evaluates the trade-off between precision and recall across classification thresholds for the binary model. The result is presented as a precision-recall curve with recall on the horizontal axis and precision on the vertical axis. The plotted curve begins at precision 1.0 at near-zero recall, rises to a local peak just below 0.9 at low recall, and then declines gradually as recall increases toward 1.0. Across the full recall range, precision remains above 0.5 and trends downward more noticeably beyond approximately mid-range recall levels.

Key insights:

• High precision at low recall: At very low recall, precision reaches 1.0 and then remains in the upper 0.8 range through the early portion of the curve, indicating stronger positive class accuracy at stricter thresholds.
• Gradual precision decline: Precision decreases progressively as recall increases, moving from roughly the high 0.7 to low 0.8 range in the middle of the curve to approximately 0.5 near full recall.
• Mid-range trade-off visible: Around the central recall range, the curve sits near approximately 0.67 to 0.78 precision, showing a moderate reduction in precision as additional positive cases are captured.
• No abrupt collapse in performance: The curve descends in a mostly continuous manner rather than dropping sharply at a specific recall level, indicating a steady precision-recall trade-off across thresholds.

The precision-recall result shows a clear inverse relationship between recall and precision across threshold settings. Precision is strongest at low recall levels, remains in a moderate range through the middle of the curve, and declines toward 0.5 as recall approaches 1.0. Overall, the curve reflects a continuous trade-off structure without a single abrupt deterioration point.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

The ROCCurve test evaluates classification model discrimination by plotting the Receiver Operating Characteristic curve and calculating the Area Under the Curve across classification thresholds. For log_reg_model_v1 on test_dataset_final, the result shows a single binary ROC curve for the positive class together with the random-classification reference line. The plotted curve remains above the diagonal reference throughout the displayed range, and the legend reports an AUC value of 0.71 versus the 0.50 random baseline.

Key insights:

• AUC exceeds random baseline: The reported AUC is 0.71, compared with the reference AUC of 0.50 for random classification. This indicates the model ranks positive cases above negative cases more effectively than chance on the test dataset.
• ROC curve stays above diagonal: The ROC curve lies above the random reference line across the plotted threshold range. This reflects higher true positive rates than false positive rates at corresponding operating points.
• Performance is threshold-dependent: The curve rises progressively rather than approaching the upper-left corner early. The plotted shape shows that true positive rate improves as false positive rate increases across thresholds, without indicating near-perfect separation.

The ROC result shows that log_reg_model_v1 has measurable discriminative ability on test_dataset_final, with an AUC of 0.71 and a curve consistently above the random baseline. The observed curve shape indicates positive-class separation that is stronger than chance but not close to perfect classification. Collectively, the result documents a moderate level of ranking performance across thresholds for this binary classification task.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

The TrainingTestDegradation test evaluates whether model performance degradation from the training dataset to the test dataset remains within the configured threshold. The results compare precision, recall, and F1-score between the training and test datasets for both class 1 and class 0, using a maximum degradation threshold of 10%. For all six metric-class combinations, the reported degradation values are negative, reflecting higher scores on the test dataset than on the training dataset. The table also shows a Pass result for every evaluated metric.

Key insights:

• No threshold breaches observed: All evaluated metrics passed the test, with no degradation value exceeding the configured 10% threshold.
• Test performance exceeds training performance: Every metric shows a negative degradation percentage, indicating that test scores are higher than training scores across both classes and all three metrics.
• Largest improvement in class 0 precision: Class 0 precision increased from 0.6243 on the training dataset to 0.6657 on the test dataset, corresponding to a degradation value of -6.6195%, which is the largest magnitude change in the table.
• Class 1 metrics improve consistently: For class 1, precision, recall, and F1-score increase from 0.6346 to 0.6508, 0.6200 to 0.6487, and 0.6272 to 0.6498, respectively, with degradation values ranging from -2.5444% to -4.6345%.
• Class 0 metrics improve across all measures: For class 0, precision, recall, and F1-score increase from 0.6243 to 0.6657, 0.6389 to 0.6677, and 0.6315 to 0.6667, respectively, with degradation values ranging from -4.5019% to -6.6195%.

The test results indicate that performance did not deteriorate from training to test for any evaluated metric. Instead, all reported precision, recall, and F1-score values are higher on the test dataset for both class 1 and class 0, producing negative degradation percentages throughout. Collectively, the results show that all evaluated train-test performance comparisons remained within the configured threshold and passed the test.

Parameters:

{
  "max_threshold": 0.1
}
            

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
1	Precision	0.6346	0.6508	-2.5444	Pass
1	Recall	0.6200	0.6487	-4.6345	Pass
1	F1-Score	0.6272	0.6498	-3.5911	Pass
0	Precision	0.6243	0.6657	-6.6195	Pass
0	Recall	0.6389	0.6677	-4.5019	Pass
0	F1-Score	0.6315	0.6667	-5.5623	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

The Minimum Accuracy test evaluates whether the model’s prediction accuracy meets or exceeds a defined minimum threshold. The result table reports a single accuracy score alongside the configured threshold and the associated pass/fail outcome. In this run, the model accuracy is 0.6584, the threshold is 0.7000, and the recorded test result is Fail.

Key insights:

• Accuracy below threshold: The observed accuracy score of 0.6584 is lower than the minimum threshold of 0.7000 by 0.0416, which is reflected in the failed test outcome.
• Failed threshold assessment: The test result is explicitly marked as Fail, indicating that the model did not satisfy the minimum accuracy criterion under the specified parameter setting.

The result shows that the model’s measured accuracy did not reach the configured minimum threshold in this evaluation. The shortfall between the observed score and the threshold is 0.0416, and the test outcome is therefore recorded as failed.

Parameters:

{
  "min_threshold": 0.7
}
            

Tables

Score	Threshold	Pass/Fail
0.6584	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

The MinimumF1Score test evaluates whether the model’s F1 score on the validation dataset meets the predefined minimum threshold. The result table reports a validation F1 score of 0.6498 alongside a minimum threshold of 0.5, with the outcome recorded as Pass. These values provide the basis for assessing whether the model achieved the required minimum balance between precision and recall under this test.

Key insights:

• Threshold exceeded on validation: The recorded F1 score is 0.6498 versus a minimum threshold of 0.5, placing the observed result above the required cutoff.
• Test outcome is pass: The result is explicitly marked as Pass in the test output, reflecting that the observed F1 score satisfied the configured minimum criterion.
• Positive margin to threshold: The observed F1 score exceeds the threshold by 0.1498, indicating separation between the measured validation performance and the minimum requirement used in the test.

The result shows that the model met the minimum F1 score criterion on the validation dataset. The observed score of 0.6498 was above the configured threshold of 0.5, and the test output classified the result as a pass. Collectively, these results indicate that the model satisfied this specific validation performance check based on the F1 metric.

Parameters:

{
  "min_threshold": 0.5
}
            

Tables

Score	Threshold	Pass/Fail
0.6498	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

The Minimum ROC AUC Score test evaluates whether the model’s ROC AUC score meets or exceeds a defined minimum threshold on the validation dataset. The result table reports a ROC AUC score of 0.7058, alongside the configured threshold of 0.5 and the corresponding pass/fail outcome. These results present the measured discrimination score and the threshold comparison used in the test.

Key insights:

• Threshold exceeded: The recorded ROC AUC score is 0.7058 versus a minimum threshold of 0.5, placing the observed value 0.2058 above the test cutoff.
• Test passed: The reported outcome is Pass, reflecting that the observed ROC AUC score satisfied the threshold condition defined for this test.
• Single validation metric reported: The result set contains one summarized discrimination measure, consisting of the ROC AUC score, the threshold, and the binary pass/fail status.

The test result shows that the model achieved a ROC AUC score of 0.7058 on the evaluated dataset and met the configured minimum threshold of 0.5. The documented outcome is a pass, based on the direct comparison between the observed score and the threshold. The result provides a concise view of model discrimination under this specific validation criterion.

Parameters:

{
  "min_threshold": 0.5
}
            

Tables

Score	Threshold	Pass/Fail
0.7058	0.5	Pass

▶ Test Result: Permutation Feature Importance (validmind.model_validation.sklearn.PermutationFeatureImportance)

Permutation Feature Importance

The Permutation Feature Importance test evaluates the relative significance of model inputs by measuring the change in model performance after each feature is randomly permuted. The result is presented as a ranked bar chart of permutation importances, with larger positive values indicating a greater reduction in performance when the feature is shuffled. The chart shows a concentrated importance profile led by Geography_Germany and IsActiveMember, followed by a smaller contribution from Gender_Male and a limited set of lower-impact variables. It also includes features with near-zero importance and one feature with negative importance.

Key insights:

• Importance is concentrated in two features: Geography_Germany and IsActiveMember have the largest permutation importances, both near 0.06 or higher, and are materially above the rest of the feature set.
• Gender_Male is a secondary driver: Gender_Male is the next most important feature at approximately 0.03, which is notably below the top two features but clearly above the remaining variables.
• Most remaining features have limited impact: Balance and NumOfProducts show modest positive importance around 0.01, while CreditScore contributes only a small positive value and Geography_Spain, Tenure, and EstimatedSalary are close to zero.
• One feature shows negative importance: HasCrCard has a negative permutation importance of roughly -0.01, indicating model performance does not decline when this feature is permuted and is slightly higher in the permuted condition shown by this test.

The permutation importance profile indicates that model performance is driven primarily by a small subset of inputs, with the strongest dependence concentrated in Geography_Germany and IsActiveMember. A second tier of contribution is visible for Gender_Male, while most other variables have limited or negligible measured effect on performance in this test. The presence of a negative importance for HasCrCard distinguishes it from the rest of the feature set and indicates that its measured contribution is not positive under the reported permutation analysis.

Figures

▶ Test Result: SHAP Global Importance (validmind.model_validation.sklearn.SHAPGlobalImportance)

SHAP Global Importance

The SHAPGlobalImportance test evaluates global feature importance using SHAP values to quantify how strongly each feature contributes to model output. The results include a normalized feature-importance bar chart and a SHAP summary plot showing the direction and spread of feature effects across observations. The normalized importance plot ranks features from highest to lowest average absolute SHAP contribution, while the summary plot shows whether lower or higher feature values are associated with negative or positive SHAP impacts. Together, these visuals provide a view of both the relative importance of each feature and the distribution of their contribution patterns.

Key insights:

• Importance is concentrated in three features: IsActiveMember has the highest normalized SHAP importance at 100, followed by Geography_Germany at approximately 88 and Gender_Male at approximately 65. These three features are visibly separated from the remainder of the feature set in the importance ranking.

• Secondary importance drops materially after the top tier: Balance is the next most important feature at roughly 28, followed by NumOfProducts and HasCrCard near 18–20, CreditScore near 13, and Geography_Spain near 12. This indicates a clear decline in average absolute SHAP contribution after the top three features.

• Tenure and EstimatedSalary contribute minimally: Tenure and EstimatedSalary appear near zero in the normalized importance plot, and their SHAP values in the summary plot are tightly clustered around zero. Their observed contribution to model output is negligible relative to the other features shown.

• Binary features show directional separation: In the summary plot, IsActiveMember, Geography_Germany, and Gender_Male each show distinct SHAP clusters on opposite sides of zero for low versus high feature values. This indicates a consistent directional effect for these encoded binary features across observations.

• Balance and CreditScore show continuous spread: Balance and CreditScore exhibit a broader horizontal dispersion of SHAP values than most other non-top-ranked features. Balance is mostly associated with positive SHAP values extending to roughly 0.25, while CreditScore spans both negative and positive contributions, with most points concentrated close to zero.

The SHAP results show that model behavior is driven primarily by IsActiveMember, Geography_Germany, and Gender_Male, with a substantial reduction in contribution magnitude for all remaining features. Balance, NumOfProducts, HasCrCard, CreditScore, and Geography_Spain contribute at a secondary level, while Tenure and EstimatedSalary have little observable effect on model output in this test. The summary plot further shows that the most important binary features have clear directional separation, whereas Balance and CreditScore display more distributed contribution patterns across observations.

Parameters:

{
  "kernel_explainer_samples": 10
}
            

Figures

▶ Test Result: Weakspots Diagnosis (validmind.model_validation.sklearn.WeakspotsDiagnosis)

❌ Weakspots Diagnosis

The WeakspotsDiagnosis test evaluates model performance across slices of the feature space by comparing accuracy, precision, recall, and F1 in each bin against predefined thresholds. The reported results cover the features Balance, CreditScore, EstimatedSalary, Gender_Male, Geography_Germany, Geography_Spain, HasCrCard, IsActiveMember, NumOfProducts, and Tenure on both training and test datasets. Across these slices, the tables and plots show multiple bins with test-set metrics below the thresholds of 0.75 for accuracy, 0.5 for precision and recall, and 0.7 for F1, alongside the corresponding training-set values and bin record counts. Several features also include sparse extreme bins with very small sample sizes, where metric values are either zero or perfect.

Key insights:

• Weak spots are widespread across features: Test-set accuracy is below the 0.75 threshold in most reported slices across Balance, CreditScore, EstimatedSalary, HasCrCard, IsActiveMember, NumOfProducts, Tenure, Geography_Germany, Geography_Spain, and Gender_Male. F1 is also below the 0.7 threshold in many of these same regions.

• Balance shows the most severe localized weakness: Multiple Balance test bins fall materially below threshold, including slices with F1 of 0.4228, 0.0, 0.1818, 0.6769, 0.5714, and 0.6667. The weakest results occur in sparse bins such as (23838.756, 47677.512] with 1 test record and (47677.512, 71516.268] with 16 test records.

• NumOfProducts degrades sharply at higher values: For NumOfProducts, the test bin (1.9, 2.2] has precision 0.3488 and F1 0.4317, while (3.7, 4.0] has accuracy 0.25, recall 0.25, and F1 0.4. The higher-product-count bins also have relatively small test counts of 33 and 8 records for the last two bins.

• IsActiveMember has a pronounced low-recall segment: In the test slice (0.9, 1.0] for IsActiveMember, recall is 0.3186 and F1 is 0.4138 despite precision of 0.5902 and accuracy of 0.6611. The corresponding training slice shows a nearly identical pattern with recall 0.3099 and F1 0.4149.

• Gender_Male and Geography_Germany show stable segment asymmetry: For Gender_Male = 1, the test metrics are weaker than for Gender_Male = 0, with recall 0.4429 and F1 0.5082 versus recall 0.8125 and F1 0.7390. For Geography_Germany = 0, both training and test F1 remain near 0.50, while Geography_Germany = 1 shows materially stronger performance with test F1 0.8013.

• CreditScore remains below the accuracy threshold throughout: All CreditScore test bins have accuracy below 0.75 except the lowest band, which is exactly 0.75 with only 4 test records. F1 falls below 0.7 in most CreditScore bins, with stronger test performance concentrated in the 450–500 and 750–800 ranges.

• EstimatedSalary is mixed, with one distinct weak bin: Most EstimatedSalary test precision and recall values remain above 0.5, but F1 is below 0.7 in most salary bins. The slice (159994.942, 179982.841] is the clearest weak spot, with accuracy 0.5806, recall 0.4412, and F1 0.5357.

• Tenure weaknesses are concentrated in selected bins: Tenure test F1 exceeds 0.7 only in bins (2.0, 3.0] and (5.0, 6.0], while bins such as (6.0, 7.0] and (9.0, 10.0] remain notably weaker with F1 of 0.5085 and 0.5641. Precision in the (6.0, 7.0] test bin is also below threshold at 0.4688.

The weak spot analysis shows that sub-threshold performance is not isolated to a single feature or region, but appears across a broad set of feature slices. The most pronounced weaknesses are concentrated in specific Balance and NumOfProducts bins, as well as the IsActiveMember = 1 segment, where low recall and low F1 are persistent across both training and test datasets. Several categorical segment differences, including Gender_Male and Geography_Germany, are consistent between training and test, while some extreme numerical bins are based on very small record counts and therefore present highly variable metric values.

Parameters:

{
  "thresholds": {
    "accuracy": 0.75,
    "precision": 0.5,
    "recall": 0.5,
    "f1": 0.7
  }
}
            

Tables

Slice	Number of Records	Feature	Accuracy	Precision	Recall	F1	Dataset
(-238.388, 23838.756]	203	Balance	0.6502	0.5778	0.3333	0.4228	test_dataset_final
(23838.756, 47677.512]	1	Balance	0.0000	0.0000	0.0000	0.0000	test_dataset_final
(47677.512, 71516.268]	16	Balance	0.4375	0.1667	0.2000	0.1818	test_dataset_final
(71516.268, 95355.024]	56	Balance	0.6250	0.6471	0.7097	0.6769	test_dataset_final
(95355.024, 119193.78]	134	Balance	0.7313	0.7294	0.8267	0.7750	test_dataset_final
(119193.78, 143032.536]	143	Balance	0.6643	0.6591	0.7632	0.7073	test_dataset_final
(143032.536, 166871.292]	67	Balance	0.6567	0.6429	0.7714	0.7013	test_dataset_final
(166871.292, 190710.048]	20	Balance	0.5500	0.5000	0.6667	0.5714	test_dataset_final
(190710.048, 214548.804]	5	Balance	0.6000	1.0000	0.5000	0.6667	test_dataset_final
(214548.804, 238387.56]	1	Balance	1.0000	1.0000	1.0000	1.0000	test_dataset_final
(-238.388, 23838.756]	833	Balance	0.6435	0.5613	0.3684	0.4449	train_dataset_final
(23838.756, 47677.512]	25	Balance	0.5200	0.6250	0.3571	0.4545	train_dataset_final
(47677.512, 71516.268]	76	Balance	0.5658	0.6667	0.5000	0.5714	train_dataset_final
(71516.268, 95355.024]	216	Balance	0.5972	0.6239	0.5965	0.6099	train_dataset_final
(95355.024, 119193.78]	512	Balance	0.6699	0.7100	0.7630	0.7355	train_dataset_final
(119193.78, 143032.536]	517	Balance	0.6228	0.6562	0.7075	0.6809	train_dataset_final
(143032.536, 166871.292]	292	Balance	0.6062	0.5538	0.7630	0.6417	train_dataset_final
(166871.292, 190710.048]	91	Balance	0.6044	0.6167	0.7400	0.6727	train_dataset_final
(190710.048, 214548.804]	21	Balance	0.3333	0.5833	0.4375	0.5000	train_dataset_final
(214548.804, 238387.56]	2	Balance	1.0000	1.0000	1.0000	1.0000	train_dataset_final
(349.5, 400.0]	4	CreditScore	0.7500	1.0000	0.7500	0.8571	test_dataset_final
(400.0, 450.0]	21	CreditScore	0.6667	0.5000	0.7143	0.5882	test_dataset_final
(450.0, 500.0]	33	CreditScore	0.7273	0.6667	0.8000	0.7273	test_dataset_final
(500.0, 550.0]	79	CreditScore	0.6203	0.5750	0.6389	0.6053	test_dataset_final
(550.0, 600.0]	68	CreditScore	0.6471	0.7027	0.6667	0.6842	test_dataset_final
(600.0, 650.0]	117	CreditScore	0.6581	0.6441	0.6667	0.6552	test_dataset_final
(650.0, 700.0]	137	CreditScore	0.6642	0.6000	0.6207	0.6102	test_dataset_final
(700.0, 750.0]	101	CreditScore	0.6040	0.6596	0.5636	0.6078	test_dataset_final
(750.0, 800.0]	61	CreditScore	0.7377	0.7742	0.7273	0.7500	test_dataset_final
(800.0, 850.0]	26	CreditScore	0.6923	0.7000	0.5833	0.6364	test_dataset_final
(349.5, 400.0]	10	CreditScore	0.5000	1.0000	0.5000	0.6667	train_dataset_final
(400.0, 450.0]	47	CreditScore	0.6383	0.7083	0.6296	0.6667	train_dataset_final
(450.0, 500.0]	119	CreditScore	0.5966	0.5714	0.7458	0.6471	train_dataset_final
(500.0, 550.0]	267	CreditScore	0.6554	0.6690	0.6786	0.6738	train_dataset_final
(550.0, 600.0]	383	CreditScore	0.6240	0.6280	0.6599	0.6436	train_dataset_final
(600.0, 650.0]	481	CreditScore	0.6029	0.6034	0.5858	0.5945	train_dataset_final
(650.0, 700.0]	484	CreditScore	0.6384	0.6429	0.6025	0.6220	train_dataset_final
(700.0, 750.0]	370	CreditScore	0.6351	0.6587	0.5851	0.6197	train_dataset_final
(750.0, 800.0]	250	CreditScore	0.6680	0.6348	0.6404	0.6376	train_dataset_final
(800.0, 850.0]	174	CreditScore	0.6092	0.6234	0.5517	0.5854	train_dataset_final
(-108.129, 20079.649]	65	EstimatedSalary	0.6154	0.6000	0.6562	0.6269	test_dataset_final
(20079.649, 40067.548]	57	EstimatedSalary	0.6667	0.6897	0.6667	0.6780	test_dataset_final
(40067.548, 60055.447]	64	EstimatedSalary	0.7344	0.7692	0.6452	0.7018	test_dataset_final
(60055.447, 80043.346]	68	EstimatedSalary	0.6176	0.6667	0.6316	0.6486	test_dataset_final
(80043.346, 100031.245]	73	EstimatedSalary	0.6027	0.5476	0.6970	0.6133	test_dataset_final
(100031.245, 120019.144]	53	EstimatedSalary	0.6226	0.6000	0.6000	0.6000	test_dataset_final
(120019.144, 140007.043]	74	EstimatedSalary	0.6757	0.6750	0.7105	0.6923	test_dataset_final
(140007.043, 159994.942]	75	EstimatedSalary	0.7467	0.6970	0.7188	0.7077	test_dataset_final
(159994.942, 179982.841]	62	EstimatedSalary	0.5806	0.6818	0.4412	0.5357	test_dataset_final
(179982.841, 199970.74]	56	EstimatedSalary	0.7143	0.6296	0.7391	0.6800	test_dataset_final
(-108.129, 20079.649]	260	EstimatedSalary	0.6769	0.7000	0.6691	0.6842	train_dataset_final
(20079.649, 40067.548]	242	EstimatedSalary	0.6240	0.6446	0.6190	0.6316	train_dataset_final
(40067.548, 60055.447]	259	EstimatedSalary	0.6409	0.6260	0.6210	0.6235	train_dataset_final
(60055.447, 80043.346]	276	EstimatedSalary	0.6413	0.6371	0.5940	0.6148	train_dataset_final
(80043.346, 100031.245]	256	EstimatedSalary	0.6172	0.5935	0.6033	0.5984	train_dataset_final
(100031.245, 120019.144]	258	EstimatedSalary	0.5775	0.6031	0.5809	0.5918	train_dataset_final
(120019.144, 140007.043]	252	EstimatedSalary	0.6230	0.6290	0.6142	0.6215	train_dataset_final
(140007.043, 159994.942]	249	EstimatedSalary	0.6104	0.5814	0.6356	0.6073	train_dataset_final
(159994.942, 179982.841]	269	EstimatedSalary	0.6468	0.6835	0.6507	0.6667	train_dataset_final
(179982.841, 199970.74]	264	EstimatedSalary	0.6326	0.6429	0.6090	0.6255	train_dataset_final
(-0.001, 0.1]	301	Gender_Male	0.6645	0.6777	0.8125	0.7390	test_dataset_final
(0.9, 1.0]	346	Gender_Male	0.6532	0.5962	0.4429	0.5082	test_dataset_final
(-0.001, 0.1]	1268	Gender_Male	0.6285	0.6429	0.7778	0.7040	train_dataset_final
(0.9, 1.0]	1317	Gender_Male	0.6302	0.6165	0.4241	0.5026	train_dataset_final
(-0.001, 0.1]	445	Geography_Germany	0.6382	0.5676	0.4641	0.5106	test_dataset_final
(0.9, 1.0]	202	Geography_Germany	0.7030	0.7246	0.8963	0.8013	test_dataset_final
(-0.001, 0.1]	1780	Geography_Germany	0.6107	0.5733	0.4405	0.4982	train_dataset_final
(0.9, 1.0]	805	Geography_Germany	0.6708	0.6896	0.8902	0.7771	train_dataset_final
(-0.001, 0.1]	492	Geography_Spain	0.6667	0.6732	0.6786	0.6759	test_dataset_final
(0.9, 1.0]	155	Geography_Spain	0.6323	0.5574	0.5312	0.5440	test_dataset_final
(-0.001, 0.1]	2002	Geography_Spain	0.6364	0.6466	0.6528	0.6497	train_dataset_final
(0.9, 1.0]	583	Geography_Spain	0.6055	0.5796	0.4925	0.5325	train_dataset_final
(-0.001, 0.1]	175	HasCrCard	0.5771	0.5306	0.6500	0.5843	test_dataset_final
(0.9, 1.0]	472	HasCrCard	0.6886	0.7051	0.6483	0.6755	test_dataset_final
(-0.001, 0.1]	783	HasCrCard	0.6271	0.6393	0.6763	0.6573	train_dataset_final
(0.9, 1.0]	1802	HasCrCard	0.6304	0.6322	0.5937	0.6123	train_dataset_final
(-0.001, 0.1]	346	IsActiveMember	0.6561	0.6654	0.8325	0.7396	test_dataset_final
(0.9, 1.0]	301	IsActiveMember	0.6611	0.5902	0.3186	0.4138	test_dataset_final
(-0.001, 0.1]	1378	IsActiveMember	0.6118	0.6363	0.8039	0.7103	train_dataset_final
(0.9, 1.0]	1207	IsActiveMember	0.6495	0.6276	0.3099	0.4149	train_dataset_final
(0.997, 1.3]	364	NumOfProducts	0.6731	0.7441	0.7072	0.7252	test_dataset_final
(1.9, 2.2]	242	NumOfProducts	0.6736	0.3488	0.5660	0.4317	test_dataset_final
(2.8, 3.1]	33	NumOfProducts	0.4848	1.0000	0.4848	0.6531	test_dataset_final
(3.7, 4.0]	8	NumOfProducts	0.2500	1.0000	0.2500	0.4000	test_dataset_final
(0.997, 1.3]	1483	NumOfProducts	0.6278	0.7134	0.6452	0.6776	train_dataset_final
(1.9, 2.2]	913	NumOfProducts	0.6506	0.3593	0.5917	0.4471	train_dataset_final
(2.8, 3.1]	153	NumOfProducts	0.5621	0.9878	0.5510	0.7074	train_dataset_final
(3.7, 4.0]	36	NumOfProducts	0.4444	1.0000	0.4444	0.6154	train_dataset_final
(-0.01, 1.0]	91	Tenure	0.6703	0.7073	0.6170	0.6591	test_dataset_final
(1.0, 2.0]	69	Tenure	0.6812	0.6333	0.6333	0.6333	test_dataset_final
(2.0, 3.0]	62	Tenure	0.6613	0.7812	0.6410	0.7042	test_dataset_final
(3.0, 4.0]	69	Tenure	0.5942	0.6500	0.6500	0.6500	test_dataset_final
(4.0, 5.0]	65	Tenure	0.6769	0.5758	0.7308	0.6441	test_dataset_final
(5.0, 6.0]	54	Tenure	0.7407	0.7407	0.7407	0.7407	test_dataset_final
(6.0, 7.0]	73	Tenure	0.6027	0.4688	0.5556	0.5085	test_dataset_final
(7.0, 8.0]	54	Tenure	0.6852	0.7500	0.5556	0.6383	test_dataset_final
(8.0, 9.0]	71	Tenure	0.6901	0.6341	0.7879	0.7027	test_dataset_final
(9.0, 10.0]	39	Tenure	0.5641	0.5789	0.5500	0.5641	test_dataset_final
(-0.01, 1.0]	391	Tenure	0.6292	0.6531	0.6244	0.6384	train_dataset_final
(1.0, 2.0]	281	Tenure	0.5623	0.5182	0.5547	0.5358	train_dataset_final
(2.0, 3.0]	260	Tenure	0.6154	0.6400	0.5926	0.6154	train_dataset_final
(3.0, 4.0]	257	Tenure	0.6304	0.6579	0.5725	0.6122	train_dataset_final
(4.0, 5.0]	265	Tenure	0.6679	0.6691	0.6791	0.6741	train_dataset_final
(5.0, 6.0]	253	Tenure	0.6324	0.6466	0.5906	0.6173	train_dataset_final
(6.0, 7.0]	238	Tenure	0.6429	0.6106	0.6273	0.6188	train_dataset_final
(7.0, 8.0]	268	Tenure	0.6903	0.6866	0.6917	0.6891	train_dataset_final
(8.0, 9.0]	256	Tenure	0.6133	0.6159	0.6489	0.6320	train_dataset_final
(9.0, 10.0]	116	Tenure	0.5948	0.6557	0.6061	0.6299	train_dataset_final

Figures

▶ Test Result: Overfit Diagnosis (validmind.model_validation.sklearn.OverfitDiagnosis)

Overfit Diagnosis

The Overfit Diagnosis test evaluates differences in model discrimination between training and test data across feature-based slices using AUC and a cutoff threshold of 0.04. The reported output identifies feature segments where the train-test AUC gap exceeds this threshold and provides the corresponding sample counts and AUC values for each flagged slice. Flagged regions are present for CreditScore, Tenure, Balance, NumOfProducts, and EstimatedSalary, with gaps ranging from 0.0420 to 0.8027 in the tabulated results. The accompanying plots also show threshold-based gap patterns across additional features, including both positive and negative AUC gaps by slice.

Key insights:

• Largest gap in NumOfProducts: The NumOfProducts slice (2.8, 3.1] shows the largest tabulated train-test difference, with Training AUC = 0.8027, Test AUC = 0.0000, and Gap = 0.8027 across 153 training records and 33 test records.

• Balance includes extreme low-volume slice: The Balance slice (23838.756, 47677.512] shows a Gap = 0.5455, with Training AUC = 0.5455 and Test AUC = 0.0000, but this result is based on 25 training records and 1 test record. A second Balance slice, (47677.512, 71516.268], is also flagged with Gap = 0.0714.

• Multiple Tenure regions are flagged: Three Tenure slices exceed the 0.04 threshold: (4.0, 5.0] with Gap = 0.0420, (6.0, 7.0] with Gap = 0.0920, and (9.0, 10.0] with Gap = 0.0550. Among these, (6.0, 7.0] has the largest Tenure gap, with Training AUC = 0.7200 and Test AUC = 0.6280.

• EstimatedSalary shows a concentrated low-range gap: The EstimatedSalary slice (-108.129, 20079.649] is flagged with Training AUC = 0.7287, Test AUC = 0.6439, and Gap = 0.0848, based on 260 training records and 65 test records.

• CreditScore has a localized flagged segment: For CreditScore, the slice (400.0, 450.0] exceeds the threshold with Training AUC = 0.7333, Test AUC = 0.6531, and Gap = 0.0803 across 47 training records and 21 test records.

The test output identifies several localized segments where training and test AUC differ by more than the 0.04 threshold. The most pronounced tabulated gaps occur in NumOfProducts and Balance, while Tenure contains multiple flagged slices and CreditScore and EstimatedSalary each contain one flagged region. The results indicate that the observed train-test performance differences are concentrated in specific feature intervals rather than uniformly distributed across all reported features.

Tables

Overfit Diagnosis

Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
CreditScore	(400.0, 450.0]	47	21	0.7333	0.6531	0.0803
Tenure	(4.0, 5.0]	265	65	0.7196	0.6775	0.0420
Tenure	(6.0, 7.0]	238	73	0.7200	0.6280	0.0920
Tenure	(9.0, 10.0]	116	39	0.6603	0.6053	0.0550
Balance	(23838.756, 47677.512]	25	1	0.5455	0.0000	0.5455
Balance	(47677.512, 71516.268]	76	16	0.5987	0.5273	0.0714
NumOfProducts	(2.8, 3.1]	153	33	0.8027	0.0000	0.8027
EstimatedSalary	(-108.129, 20079.649]	260	65	0.7287	0.6439	0.0848

Figures

▶ Test Result: Robustness Diagnosis (validmind.model_validation.sklearn.RobustnessDiagnosis)

✅ Robustness Diagnosis

The Robustness Diagnosis test evaluates model performance under progressively higher levels of Gaussian noise applied to input features. The results report AUC on the training and test datasets across perturbation sizes from 0.0 to 0.5 standard deviations, together with performance decay relative to the baseline. Baseline AUC is 0.6774 on train_dataset_final and 0.7058 on test_dataset_final, and the table and plot show how these values change as perturbation magnitude increases. All evaluated perturbation settings are marked as passed for both datasets.

Key insights:

• Performance declines gradually with noise: AUC decreases from 0.6774 to 0.6605 on train_dataset_final and from 0.7058 to 0.6845 on test_dataset_final between baseline and perturbation size 0.5. The corresponding performance decay reaches 0.0169 on train and 0.0212 on test at the highest perturbation level.

• Test dataset shows larger maximum decay: At perturbation size 0.5, the test dataset records the largest observed decay at 0.0212 versus 0.0169 for the training dataset. A similar pattern appears at perturbation size 0.4, where decay is 0.0178 on test and 0.0170 on train.

• Intermediate perturbations are not strictly monotonic: Between perturbation sizes 0.2 and 0.3, training AUC moves from 0.6708 to 0.6723 and test AUC from 0.7010 to 0.7021. This produces a small reduction in reported decay at 0.3 relative to 0.2 for both datasets.

• All scenarios passed the test: Every baseline and perturbed evaluation in the result table is flagged as passed. This includes all tested perturbation sizes from 0.1 through 0.5 for both train_dataset_final and test_dataset_final.

The robustness results show that model discrimination decreases as Gaussian perturbation strength increases, with the largest observed reductions appearing at the highest noise levels tested. The test dataset exhibits slightly greater performance decay than the training dataset at the upper perturbation settings, while both datasets retain AUC values above their respective plotted threshold lines throughout the evaluation. Taken together, the results indicate measurable but limited degradation across the tested noise range, with all evaluated conditions recorded as passing.

Parameters:

{
  "scaling_factor_std_dev_list": [
    0.0,
    0.1,
    0.2,
    0.3,
    0.4,
    0.5
  ]
}
            

Tables

Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
Baseline (0.0)	train_dataset_final	2585	0.6774	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.7058	0.0000	True
Baseline (0.0)	train_dataset_final	2585	0.6774	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.7058	0.0000	True
0.1	train_dataset_final	2585	0.6750	0.0024	True
0.1	test_dataset_final	647	0.7031	0.0027	True
0.2	train_dataset_final	2585	0.6708	0.0066	True
0.2	test_dataset_final	647	0.7010	0.0048	True
0.3	train_dataset_final	2585	0.6723	0.0051	True
0.3	test_dataset_final	647	0.7021	0.0037	True
0.4	train_dataset_final	2585	0.6604	0.0170	True
0.4	test_dataset_final	647	0.6879	0.0178	True
0.5	train_dataset_final	2585	0.6605	0.0169	True
0.5	test_dataset_final	647	0.6845	0.0212	True

Figures

In summary

In this final notebook, you learned how to:

• Refresh the connection from the ValidMind Library to the ValidMind Platform after you've inserted test-driven blocks to your documentation
• Include custom test results in your documentation
• View and configure the configuration for your documentation template

With our ValidMind for development series of notebooks, you learned how to document a record (model) end-to-end with the ValidMind Library by running through some common scenarios in a typical development setting:

• Running out-of-the-box tests
• Documenting your record (model) by adding evidence to documentation
• Extending the capabilities of the ValidMind Library by implementing custom tests
• Ensuring that the documentation is complete by running all tests in the documentation template

Next steps

Work with your documentation

Now that you've logged all your test results and generated a draft for your documentation, head to the ValidMind Platform to wrap up your documentation. Continue to work on your documentation by:

• Run and log more tests: Use the skills you learned in this series of notebooks to run and log more individual tests, including custom tests, then insert them into your documentation as supplementary evidence. (Learn more: validmind.tests)

• Inserting additional test results: Add Test-Driven Blocks under any relevant section of your documentation. (Learn more: Work with test results)

• Making qualitative edits to your test descriptions: Click on the description of any inserted test results to review and edit the ValidMind-generated test descriptions for quality and accuracy. (Learn more: Working with documentation)

• View guidelines: In any section of your documentation, click ValidMind Insights in the top right corner to reveal the Documentation Guidelines for each section to help guide the contents of your documentation. (Learn more: View development guidelines)

• Collaborate with other stakeholders: Use the ValidMind Platform's real-time collaborative features to work seamlessly together with the rest of your organization, including validators. Review suggested changes in your content blocks, work with versioned history, and use comments to discuss specific portions of your documentation. (Learn more: Collaborate with others)

When your documentation is complete and ready for review, submit it for approval from the same ValidMind Platform where you made your edits and collaborated with the rest of your organization, ensuring transparency and a thorough development history. (Learn more: Submit documents)

Learn more

Now that you're familiar with the basics, you can explore the following notebooks to get a deeper understanding on how the ValidMind Library allows you generate documentation for any use case:

Use cases

• Document an agentic AI system
• Document an Excel-based application scorecard model
• LLM model documentation demo

Discover more learning resources

Learn more about the ValidMind Library tools we used in this notebook:

• Explore tests
• Run dataset-based tests
• Implement custom tests
• Integrate external test providers
• Configure dataset features

We offer many interactive notebooks to help you automate testing, documenting, validating, and more:

• Run tests & test suites
• Use ValidMind Library features
• Code samples by use case

Or, visit our documentation to learn more about ValidMind.

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial