ValidMind for validation 3 — Developing a potential challenger

Learn how to use ValidMind for your end-to-end validation process with our series of four introductory notebooks. In this third notebook, develop a potential challenger and then pass your challenger and its predictions to ValidMind.

A challenger is an alternate record (model) that attempts to outperform the champion, ensuring that the best performing fit-for-purpose record is always considered for deployment. Challengers also help avoid over-reliance on a single record, and allow testing of new features, algorithms, or data sources without disrupting the production lifecycle.

Learn by doing

Our course tailor-made for validators new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Validator Fundamentals

Prerequisites

In order to develop potential challengers with this notebook, you'll need to first have:

• Registered a model within the ValidMind Platform and granted yourself access to the model as a validator
• Installed the ValidMind Library in your local environment, allowing you to access all its features
• Learned how to import and initialize datasets for use with ValidMind
• Understood the basics of how to run and log tests with ValidMind
• Run data quality tests on the datasets used to train the champion, and logged the results of those tests to ValidMind
• Inserted your logged test results into your validation report

Need help with the above steps?

Refer to the first two notebooks in this series:

• 1 — Set up the ValidMind Library for validation
• 2 — Start the validation process

Setting up

This section should be quite familiar to you — as we performed the same actions in the previous notebook, 2 — Start the validation process.

Initialize the ValidMind Library

As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Validation from the Document drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="validation-report",
)

Note: you may need to restart the kernel to use updated packages.

2026-10-30 20:59:31,056 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model validation (ID: cmalguc9y02ok199q2db381ib)
📁 Document Type: validation_report

Import the sample dataset

Next, we'll load in the sample Bank Customer Churn Prediction dataset used to develop the champion that we will independently preprocess:

# Load the sample dataset
from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

Preprocess the dataset

We’ll apply a simple rebalancing technique to the dataset before continuing:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

Let’s also quickly remove highly correlated features from the dataset using the output from a ValidMind test.

As you know, before we can run tests you’ll need to initialize a ValidMind dataset object with the init_dataset function:

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:

# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant or highly collinear variables. The results table lists the top correlations by coefficient magnitude, along with a Pass/Fail designation based on the configured absolute correlation threshold of 0.3. Among the reported pairs, coefficients range from -0.1984 to 0.3602, and only one pair exceeds the threshold. The strongest observed relationship is between Age and Exited, with a coefficient of 0.3602 and a Fail status.

Key insights:

• Single threshold breach identified: Only the pair (Age, Exited) exceeds the 0.3 threshold, with a Pearson correlation coefficient of 0.3602, and is the only result marked Fail in the reported output.
• Remaining reported correlations are modest: All other listed feature pairs have absolute correlation values below 0.20, including (IsActiveMember, Exited) at -0.1984 and (Balance, NumOfProducts) at -0.1898, and are marked Pass.
• Observed relationships are mostly weak: Beyond the Age–Exited pair, the remaining coefficients cluster close to zero, with several values between approximately -0.05 and 0.15, such as (Balance, Exited) at 0.1462 and (CreditScore, Exited) at -0.0322.

The reported correlation structure is characterized by one identified high-correlation pair and a set of otherwise low-magnitude linear relationships. Age and Exited represent the only pair crossing the configured threshold, while the rest of the reported feature pairs remain below the test cutoff and are concentrated in a weak-correlation range. Overall, the result indicates limited evidence of high pairwise linear association within the reported top correlations.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3602	Fail
(IsActiveMember, Exited)	-0.1984	Pass
(Balance, NumOfProducts)	-0.1898	Pass
(Balance, Exited)	0.1462	Pass
(NumOfProducts, IsActiveMember)	0.0538	Pass
(NumOfProducts, Exited)	-0.0521	Pass
(Age, Balance)	0.0486	Pass
(Age, NumOfProducts)	-0.0411	Pass
(Tenure, IsActiveMember)	-0.0353	Pass
(CreditScore, Exited)	-0.0322	Pass

# From result object, extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3602	Fail
1	(IsActiveMember, Exited)	-0.1984	Pass
2	(Balance, NumOfProducts)	-0.1898	Pass
3	(Balance, Exited)	0.1462	Pass
4	(NumOfProducts, IsActiveMember)	0.0538	Pass
5	(NumOfProducts, Exited)	-0.0521	Pass
6	(Age, Balance)	0.0486	Pass
7	(Age, NumOfProducts)	-0.0411	Pass
8	(Tenure, IsActiveMember)	-0.0353	Pass
9	(CreditScore, Exited)	-0.0322	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

# Extract feature names from the list of strings
high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

We can then re-initialize the dataset with a different input_id and the highly correlated features removed and re-run the test for confirmation:

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

# Re-run the test with the reduced feature set
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among dataset features to identify potentially redundant variables or signs of multicollinearity. The result table reports the top feature pairs ranked by absolute Pearson correlation coefficient together with a pass/fail assessment using a threshold of 0.3. In this run, the reported coefficients range from -0.1984 to 0.1462, and all listed pairs are classified as Pass. The strongest observed relationships are shown for (IsActiveMember, Exited), (Balance, NumOfProducts), and (Balance, Exited).

Key insights:

• No correlations exceed threshold: All reported feature pairs pass the test against the 0.3 threshold, with the largest absolute coefficient equal to 0.1984.
• Strongest relationship remains modest: The highest absolute correlation is between IsActiveMember and Exited at -0.1984, indicating the most pronounced linear relationship in the reported set is still limited in magnitude.
• Balance appears in multiple top pairs: Balance is present in three of the ten reported pairs, with correlations of -0.1898 with NumOfProducts, 0.1462 with Exited, and -0.0322 with HasCrCard.
• Reported relationships are mostly weak: Aside from the top three pairs, the remaining coefficients are close to zero, ranging from -0.0521 to 0.0538 in magnitude, with Tenure and EstimatedSalary showing 0.0313.

The reported correlation structure is uniformly low across the top-ranked feature pairs included in the test output. No listed pair reaches the configured threshold, and the strongest observed associations remain below 0.2 in absolute value. Collectively, the results indicate limited linear dependence among the reported pairs, with the most visible concentration of relationships involving Balance but at modest magnitudes.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1984	Pass
(Balance, NumOfProducts)	-0.1898	Pass
(Balance, Exited)	0.1462	Pass
(NumOfProducts, IsActiveMember)	0.0538	Pass
(NumOfProducts, Exited)	-0.0521	Pass
(Tenure, IsActiveMember)	-0.0353	Pass
(CreditScore, Exited)	-0.0322	Pass
(Balance, HasCrCard)	-0.0322	Pass
(HasCrCard, IsActiveMember)	-0.0321	Pass
(Tenure, EstimatedSalary)	0.0313	Pass

Split the preprocessed dataset

With our raw dataset rebalanced with highly correlated features removed, let's now spilt our dataset into train and test in preparation for model evaluation testing:

# Encode categorical features in the dataset
balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
3069	679	0	88335.05	1	0	0	159584.81	0	True	False	False
6598	571	5	0.00	2	0	0	72849.29	0	False	False	True
7011	701	3	105588.66	1	0	1	74694.41	0	False	False	False
5318	517	9	142147.32	1	0	0	39488.04	1	False	False	True
1515	658	5	126397.66	1	0	0	156361.58	1	False	False	False

from sklearn.model_selection import train_test_split

# Split the dataset into train and test
train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

# Initialize the split datasets
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

Import the champion model

With our raw dataset assessed and preprocessed, let's go ahead and import the champion submitted by the development team in the format of a .pkl file: lr_model_champion.pkl

# Import the champion model
import pickle as pkl

with open("lr_model_champion.pkl", "rb") as f:
    log_reg = pkl.load(f)

/opt/hostedtoolcache/Python/3.11.17/x64/lib/python3.11/site-packages/sklearn/base.py:442: InconsistentVersionWarning: Trying to unpickle estimator LogisticRegression from version 1.3.2 when using version 1.7.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(

Training a potential challenger model

We're curious how an alternate model compares to our champion, so let's train a challenger as a basis for our testing.

Our champion logistic regression model is a simpler, parametric model that assumes a linear relationship between the independent variables and the log-odds of the outcome. While logistic regression may not capture complex patterns as effectively, it offers a high degree of interpretability and is easier to explain to stakeholders. However, risk is not calculated in isolation from a single factor, but rather in consideration with trade-offs in predictive performance, ease of interpretability, and overall alignment with business objectives.

Random forest classification model

A random forest classification model is an ensemble machine learning algorithm that uses multiple decision trees to classify data. In ensemble learning, multiple models are combined to improve prediction accuracy and robustness.

Random forest classification models generally have higher accuracy because they capture complex, non-linear relationships, but as a result they lack transparency in their predictions.

# Import the Random Forest Classification model
from sklearn.ensemble import RandomForestClassifier

# Create the model instance with 50 decision trees
rf_model = RandomForestClassifier(
    n_estimators=50,
    random_state=42,
)

# Train the model
rf_model.fit(X_train, y_train)

RandomForestClassifier(n_estimators=50, random_state=42)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Initialize the ValidMind models

In addition to the initialized datasets, you'll also need to initialize a ValidMind model object (vm_model) that can be passed to other functions for analysis and tests on the data for each of our two models.

• Despite the naming convention, ValidMind model objects can be any type of record you want to test, document, validate, or monitor with the ValidMind Library.
• From classical statistical and machine learning models, to generative and agentic AI systems and more, the ValidMind model object provides a consistent wrapper around your record so it can be passed as a unified input to any ValidMind test or test suite, with results sent directly to the ValidMind Platform.

Initialize your model objects with vm.init_model():

# Initialize the champion logistic regression model
vm_log_model = vm.init_model(
    log_reg,
    input_id="log_model_champion",
)

# Initialize the challenger random forest classification model
vm_rf_model = vm.init_model(
    rf_model,
    input_id="rf_model",
)

Assign predictions

With our models registered, we'll move on to assigning both the predictive probabilities coming directly from each model's predictions, and the binary prediction after applying the cutoff threshold described in the Compute binary predictions step above.

• The assign_predictions() method from the Dataset object can link existing predictions to any number of models.
• This method links the model's class prediction values and probabilities to our vm_train_ds and vm_test_ds datasets.

If no prediction values are passed, the method will compute predictions automatically:

# Champion — Logistic regression model
vm_train_ds.assign_predictions(model=vm_log_model)
vm_test_ds.assign_predictions(model=vm_log_model)

# Challenger — Random forest classification model
vm_train_ds.assign_predictions(model=vm_rf_model)
vm_test_ds.assign_predictions(model=vm_rf_model)

2026-10-30 20:59:46,764 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-10-30 20:59:46,766 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-10-30 20:59:46,767 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-10-30 20:59:46,768 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-10-30 20:59:46,769 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-10-30 20:59:46,770 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-10-30 20:59:46,770 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-10-30 20:59:46,771 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-10-30 20:59:46,772 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-10-30 20:59:46,786 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-10-30 20:59:46,786 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-10-30 20:59:46,800 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-10-30 20:59:46,802 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-10-30 20:59:46,807 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-10-30 20:59:46,807 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-10-30 20:59:46,812 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Running model evaluation tests

With our setup complete, let's run the rest of our validation tests. Since we have already verified the data quality of the dataset used to train our champion, we will now focus on comprehensive performance evaluations of both the champion and challenger models.

Run model performance tests

Let's run some performance tests, beginning with independent testing of our champion logistic regression model, then moving on to our potential challenger model.

Use vm.tests.list_tests() to identify all the model performance tests for classification:

vm.tests.list_tests(tags=["model_performance"], task="classification")

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.model_validation.sklearn.CalibrationCurve	Calibration Curve	Evaluates the calibration of probability estimates by comparing predicted probabilities against observed...	True	False	['model', 'dataset']	{'n_bins': {'type': 'int', 'default': 10}}	['sklearn', 'model_performance', 'classification']	['classification']
validmind.model_validation.sklearn.ClassifierPerformance	Classifier Performance	Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy,...	False	True	['dataset', 'model']	{'average': {'type': 'str', 'default': 'macro'}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.ConfusionMatrix	Confusion Matrix	Evaluates and visually represents the classification ML model's predictive performance using a Confusion Matrix...	True	False	['dataset', 'model']	{'threshold': {'type': 'float', 'default': 0.5}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.sklearn.HyperParametersTuning	Hyper Parameters Tuning	Performs exhaustive grid search over specified parameter ranges to find optimal model configurations...	False	True	['model', 'dataset']	{'param_grid': {'type': 'dict', 'default': None}, 'scoring': {'type': 'Union', 'default': None}, 'thresholds': {'type': 'Union', 'default': None}, 'fit_params': {'type': 'dict', 'default': None}}	['sklearn', 'model_performance']	['clustering', 'classification']
validmind.model_validation.sklearn.MinimumAccuracy	Minimum Accuracy	Checks if the model's prediction accuracy meets or surpasses a specified threshold....	False	True	['dataset', 'model']	{'min_threshold': {'type': 'float', 'default': 0.7}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.MinimumF1Score	Minimum F1 Score	Assesses if the model's F1 score on the validation set meets a predefined minimum threshold, ensuring balanced...	False	True	['dataset', 'model']	{'min_threshold': {'type': 'float', 'default': 0.5}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.MinimumROCAUCScore	Minimum ROCAUC Score	Validates model by checking if the ROC AUC score meets or surpasses a specified threshold....	False	True	['dataset', 'model']	{'min_threshold': {'type': 'float', 'default': 0.5}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.ModelsPerformanceComparison	Models Performance Comparison	Evaluates and compares the performance of multiple Machine Learning models using various metrics like accuracy,...	False	True	['dataset', 'models']	{}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'model_comparison']	['classification', 'text_classification']
validmind.model_validation.sklearn.PopulationStabilityIndex	Population Stability Index	Assesses the Population Stability Index (PSI) to quantify the stability of an ML model's predictions across...	True	True	['datasets', 'model']	{'num_bins': {'type': 'int', 'default': 10}, 'mode': {'type': 'str', 'default': 'fixed'}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.PrecisionRecallCurve	Precision Recall Curve	Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve....	True	False	['model', 'dataset']	{}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.sklearn.ROCCurve	ROC Curve	Evaluates classification model performance by generating and plotting the Receiver Operating Characteristic...	True	False	['model', 'dataset']	{}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.sklearn.RegressionErrors	Regression Errors	Assesses the performance and error distribution of a regression model using various error metrics....	False	True	['model', 'dataset']	{}	['sklearn', 'model_performance']	['regression', 'classification']
validmind.model_validation.sklearn.TrainingTestDegradation	Training Test Degradation	Tests if model performance degradation between training and test datasets exceeds a predefined threshold....	False	True	['datasets', 'model']	{'max_threshold': {'type': 'float', 'default': 0.1}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.statsmodels.GINITable	GINI Table	Evaluates classification model performance using AUC, GINI, and KS metrics for training and test datasets....	False	True	['dataset', 'model']	{}	['model_performance']	['classification']
validmind.ongoing_monitoring.CalibrationCurveDrift	Calibration Curve Drift	Evaluates changes in probability calibration between reference and monitoring datasets....	True	True	['datasets', 'model']	{'n_bins': {'type': 'int', 'default': 10}, 'drift_pct_threshold': {'type': 'float', 'default': 20}}	['sklearn', 'binary_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.ongoing_monitoring.ClassDiscriminationDrift	Class Discrimination Drift	Compares classification discrimination metrics between reference and monitoring datasets....	False	True	['datasets', 'model']	{'drift_pct_threshold': {'type': '_empty', 'default': 20}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.ongoing_monitoring.ClassificationAccuracyDrift	Classification Accuracy Drift	Compares classification accuracy metrics between reference and monitoring datasets....	False	True	['datasets', 'model']	{'drift_pct_threshold': {'type': '_empty', 'default': 20}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.ongoing_monitoring.ConfusionMatrixDrift	Confusion Matrix Drift	Compares confusion matrix metrics between reference and monitoring datasets....	False	True	['datasets', 'model']	{'drift_pct_threshold': {'type': '_empty', 'default': 20}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.ongoing_monitoring.ROCCurveDrift	ROC Curve Drift	Compares ROC curves between reference and monitoring datasets....	True	False	['datasets', 'model']	{}	['sklearn', 'binary_classification', 'model_performance', 'visualization']	['classification', 'text_classification']

We'll isolate the specific tests we want to run in mpt:

• model_validation.sklearn.ClassifierPerformance
• model_validation.sklearn.ConfusionMatrix
• model_validation.sklearn.MinimumAccuracy
• model_validation.sklearn.MinimumF1Score
• model_validation.sklearn.ROCCurve

As we learned in the previous notebook 2 — Start the model validation process, you can use a custom result_id to tag the individual result with a unique identifier by appending this result_id to the test_id with a : separator. We'll append an identifier for our champion model here:

mpt = [
    "validmind.model_validation.sklearn.ClassifierPerformance:logreg_champion",
    "validmind.model_validation.sklearn.ConfusionMatrix:logreg_champion",
    "validmind.model_validation.sklearn.MinimumAccuracy:logreg_champion",
    "validmind.model_validation.sklearn.MinimumF1Score:logreg_champion",
    "validmind.model_validation.sklearn.ROCCurve:logreg_champion"
]

Evaluate performance of the champion model

Now, let's run and log our batch of model performance tests using our testing dataset (vm_test_ds) for our champion model:

• The test set serves as a proxy for real-world data, providing an unbiased estimate of model performance since it was not used during training or tuning.
• The test set also acts as protection against selection bias and model tweaking, giving a final, more unbiased checkpoint.

for test in mpt:
    vm.tests.run_test(
        test,
        inputs={
            "dataset": vm_test_ds, "model" : vm_log_model,
        },
    ).log()

Classifier Performance Logreg Champion

The Classifier Performance test evaluates classification effectiveness using precision, recall, F1-score, accuracy, and ROC AUC. For logreg_champion, the results are reported for both classes individually and as macro and weighted averages. Class-level precision ranges from 0.6353 to 0.6384, recall ranges from 0.6125 to 0.6606, and F1-scores range from 0.6252 to 0.6477. Overall accuracy is 0.6368 and ROC AUC is 0.6866.

Key insights:

• Balanced class-level precision: Precision is closely aligned across the two classes, at 0.6384 for class 0 and 0.6353 for class 1, indicating similar positive predictive performance across class assignments.
• Recall differs by class: Recall is 0.6125 for class 0 and 0.6606 for class 1, showing stronger class identification for class 1 than for class 0.
• Class 1 has higher F1-score: The F1-score for class 1 is 0.6477 versus 0.6252 for class 0, reflecting the higher combined precision-recall balance achieved for class 1.
• Aggregate metrics are internally consistent: Weighted-average and macro-average metrics are nearly identical, with weighted F1 at 0.6366 and macro F1 at 0.6364, indicating limited divergence between class-specific and overall summarized performance.
• ROC AUC exceeds accuracy: ROC AUC is 0.6866 compared with accuracy of 0.6368, showing stronger ranking performance than threshold-based classification accuracy.

The results show a broadly even precision profile across classes, with a more noticeable difference in recall and F1-score favoring class 1. Aggregate metrics remain tightly clustered around 0.636, indicating consistent overall performance across summary views. The ROC AUC of 0.6866 is higher than the observed accuracy, distinguishing the model’s ranking capability from its realized classification performance at the evaluated threshold.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6384	0.6125	0.6252
1	0.6353	0.6606	0.6477
Weighted Average	0.6368	0.6368	0.6366
Macro Average	0.6369	0.6365	0.6364

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6368
ROC AUC	0.6866

2026-10-30 20:59:57,584 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ClassifierPerformance:logreg_champion does not exist in model's document

Confusion Matrix Logreg Champion

The Confusion Matrix test evaluates the classification model’s predictive performance by comparing predicted labels against observed labels and displaying the counts of true positives, true negatives, false positives, and false negatives. In this result, the matrix reports 216 true positives, 196 true negatives, 124 false positives, and 111 false negatives. The plot therefore shows how predictions are distributed across correct and incorrect classifications for both class 0 and class 1 outcomes.

Key insights:

• Correct classifications exceed errors: The model records 216 true positives and 196 true negatives, compared with 124 false positives and 111 false negatives. This indicates that correct classifications are more frequent than misclassifications in both classes combined.
• Positive-class detection is stronger: True positives (216) exceed false negatives (111), showing that observed class 1 cases are more often identified correctly than missed.
• False positives remain material: The model produces 124 false positives versus 196 true negatives. This indicates a notable number of class 0 observations are assigned to class 1.
• Error types are relatively balanced: False positives (124) and false negatives (111) are close in magnitude, indicating that misclassification is not concentrated in only one error type.

Overall, the confusion matrix shows that the model produces more correct than incorrect classifications, with the largest count in the true positive cell and a slightly smaller true negative count. Misclassifications are present in both directions, with false positives modestly exceeding false negatives. The result indicates a reasonably balanced error pattern rather than one dominated by only missed positives or incorrect positive assignments.

Figures

2026-10-30 21:00:06,367 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ConfusionMatrix:logreg_champion does not exist in model's document

❌ Minimum Accuracy Logreg Champion

The Minimum Accuracy test evaluates whether the model’s prediction accuracy meets or exceeds a predefined threshold. The result table reports the observed accuracy score alongside the threshold used for assessment and the resulting pass/fail outcome. For this run, the model recorded an accuracy score of 0.6368 against a threshold of 0.7, and the test result is marked as Fail.

Key insights:

• Accuracy below threshold: The observed accuracy score is 0.6368, which is lower than the minimum threshold of 0.7 used in this test.
• Test outcome is fail: Because the reported score does not meet the stated threshold, the test output is classified as Fail.
• Shortfall is measurable: The difference between the observed score and the threshold is 0.0632, indicating the extent to which the result falls below the minimum criterion.

The result shows that the model did not satisfy the minimum accuracy criterion defined for this test run. The measured accuracy of 0.6368 remained below the 0.7 threshold, producing a failing outcome. Taken together, the test indicates that overall classification correctness, as measured by accuracy in this evaluation, did not reach the specified minimum level.

Tables

Score	Threshold	Pass/Fail
0.6368	0.7	Fail

2026-10-30 21:00:12,043 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumAccuracy:logreg_champion does not exist in model's document

✅ Minimum F1 Score Logreg Champion

The MinimumF1Score test evaluates whether the model’s F1 score on the validation dataset meets a predefined minimum threshold. The result table reports a validation F1 score of 0.6477 alongside a threshold of 0.5, with the outcome recorded as Pass. These values present the measured score, the comparison benchmark, and the resulting test status for the evaluated model.

Key insights:

• F1 score exceeds threshold: The observed validation F1 score is 0.6477, which is above the minimum threshold of 0.5 used in this test.
• Positive test outcome recorded: The test result is explicitly marked as Pass, reflecting the comparison between the observed F1 score and the defined threshold.
• Clear margin above benchmark: The score exceeds the threshold by 0.1477, indicating separation between the measured validation performance and the minimum acceptance level defined for this test.

The result shows that the model’s validation F1 score met the predefined minimum criterion used in this assessment. The recorded score of 0.6477 was higher than the threshold of 0.5, and the test outcome was therefore classified as Pass. Collectively, the result and supporting values indicate that the model satisfied this specific validation-set performance check based on F1 score.

Tables

Score	Threshold	Pass/Fail
0.6477	0.5	Pass

2026-10-30 21:00:17,682 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumF1Score:logreg_champion does not exist in model's document

ROC Curve Logreg Champion

The ROCCurve test evaluates the classification model’s ability to distinguish between classes by plotting the Receiver Operating Characteristic curve and calculating the Area Under the Curve. For logreg_champion on test_dataset_final, the result is a single binary-class ROC curve with an AUC of 0.69. The plot shows the relationship between true positive rate and false positive rate across classification thresholds, alongside the diagonal random-classification benchmark with AUC = 0.5.

Key insights:

• AUC exceeds random benchmark: The reported AUC is 0.69, which is above the random baseline of 0.5 shown in the figure. This indicates the model ranks positive cases better than random classification on the tested dataset.
• ROC curve remains above baseline: The ROC curve lies above the diagonal benchmark across most of the false positive rate range. This reflects higher true positive rates than the random reference at comparable false positive rates.
• Discrimination is moderate: The curve shows a gradual upward trajectory rather than a sharp rise near the y-axis. This pattern is consistent with moderate class separation rather than near-perfect discrimination.

The ROC result shows that logreg_champion demonstrates measurable discriminatory ability on test_dataset_final, with an AUC of 0.69 and a curve that remains above the random baseline. The observed shape of the curve indicates performance that is directionally positive but not strong enough to reflect highly concentrated true positive capture at low false positive rates. Overall, the test result reflects moderate binary classification discrimination in this evaluation sample.

Figures

2026-10-30 21:00:27,649 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ROCCurve:logreg_champion does not exist in model's document

Note the output returned indicating that a test-driven block doesn't currently exist in your documentation for some test IDs.

That's expected, as when we run validations tests the results logged need to be manually added to your report as part of your compliance assessment process within the ValidMind Platform.

Log an artifact

As we can observe from the output above, our champion doesn't pass the MinimumAccuracy based on the default thresholds of the out-of-the-box test, so let's log an artifact (finding) in the ValidMind Platform (Learn more: Add and manage artifacts):

1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

2. In the left sidebar that appears for your model, click Validation under Documents.

3. Click on 2.2.2. Model Performance to expand that section.

4. Under the Model Performance Metrics guideline, click to expand the Artifacts panel.

5. Click Link Artifact and select Validation Issue as the type of artifact.

6. Click + Add Validation Issue and enter in the details for your validation issue, for example:

   • Title — Champion Logistic Regression Model Fails Minimum Accuracy Threshold
   • Risk Area — Model Performance
   • Documentation Section — 3.2. Model Evaluation
   • Description — The logistic regression champion model was subjected to a Minimum Accuracy test to determine whether its predictive accuracy meets the predefined performance threshold of 0.7. The model achieved an accuracy score of 0.6136, which falls below the required minimum. As a result, the test produced a Fail outcome.

7. Click Add Validation Issue to submit the validation issue.

8. Select the validation issue you just added to link to your validation report.

9. Click Update Linked Artifacts to insert your validation issue.

10. Confirm that the validation issue you inserted has been correctly inserted into section 2.2.2. Model Performance of the report.

11. Click on the validation issue to expand the issue, where you can adjust details such as severity, owner, due date, status, etc. as well as include proposed remediation plans or supporting documentation as attachments.

Evaluate performance of challenger model

We've now conducted similar tests as the development team for our champion, with the aim of verifying their test results.

Next, let's see how our challengers compare. We'll use the same batch of tests here as we did in mpt, but append a different result_id to indicate that these results should be associated with our challenger:

mpt_chall = [
    "validmind.model_validation.sklearn.ClassifierPerformance:champion_vs_challenger",
    "validmind.model_validation.sklearn.ConfusionMatrix:champion_vs_challenger",
    "validmind.model_validation.sklearn.MinimumAccuracy:champion_vs_challenger",
    "validmind.model_validation.sklearn.MinimumF1Score:champion_vs_challenger",
    "validmind.model_validation.sklearn.ROCCurve:champion_vs_challenger"
]

We'll run each test once for each model with the same vm_test_ds dataset to compare them:

for test in mpt_chall:
    vm.tests.run_test(
        test,
        input_grid={
            "dataset": [vm_test_ds], "model" : [vm_log_model,vm_rf_model]
        }
    ).log()

Classifier Performance Champion Vs Challenger

The Classifier Performance test evaluates classification model performance using precision, recall, F1-score, accuracy, and ROC AUC. The results compare the champion model (log_model_champion) with the challenger (rf_model) across both classes and aggregate averages. For log_model_champion, weighted-average precision, recall, and F1 are 0.6368, 0.6368, and 0.6366, with accuracy of 0.6368 and ROC AUC of 0.6866. For rf_model, weighted-average precision, recall, and F1 are all 0.6971, with accuracy of 0.6971 and ROC AUC of 0.753.

Key insights:

• Challenger outperforms champion overall: rf_model exceeds log_model_champion on every reported aggregate metric. Accuracy increases from 0.6368 to 0.6971, weighted-average F1 from 0.6366 to 0.6971, and ROC AUC from 0.6866 to 0.753.

• Higher class-level performance for both classes: rf_model shows stronger class-level results than log_model_champion for both Class 0 and Class 1. For Class 0, F1 rises from 0.6252 to 0.6957; for Class 1, F1 rises from 0.6477 to 0.6985.

• More balanced precision and recall profile: rf_model has closely aligned precision and recall within each class, with Class 0 at 0.6914 precision and 0.7000 recall, and Class 1 at 0.7028 precision and 0.6942 recall. log_model_champion also shows similar precision-recall balance, but at lower levels for both classes.

• Aggregate averages are internally consistent: For both models, weighted-average and macro-average metrics are nearly identical. In rf_model, these averages are exactly 0.6971 across precision, recall, and F1, while in log_model_champion they differ only marginally around 0.6365-0.6369.

The results show that rf_model delivers higher performance than log_model_champion across class-level and aggregate classification metrics. The improvement is consistent across precision, recall, F1-score, accuracy, and ROC AUC rather than concentrated in a single measure. Both models display relatively balanced precision and recall between classes, with the primary distinction being the uniformly higher metric levels observed for rf_model.

Tables

model	Class	Precision	Recall	F1
log_model_champion	0	0.6384	0.6125	0.6252
log_model_champion	1	0.6353	0.6606	0.6477
log_model_champion	Weighted Average	0.6368	0.6368	0.6366
log_model_champion	Macro Average	0.6369	0.6365	0.6364
rf_model	0	0.6914	0.7000	0.6957
rf_model	1	0.7028	0.6942	0.6985
rf_model	Weighted Average	0.6971	0.6971	0.6971
rf_model	Macro Average	0.6971	0.6971	0.6971

model	Metric	Value
log_model_champion	Accuracy	0.6368
log_model_champion	ROC AUC	0.6866
rf_model	Accuracy	0.6971
rf_model	ROC AUC	0.7530

2026-10-30 21:00:36,730 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ClassifierPerformance:champion_vs_challenger does not exist in model's document

Confusion Matrix Champion Vs Challenger

The ConfusionMatrix:champion_vs_challenger test evaluates classification performance by comparing predicted labels with observed labels and summarizing the results as true positives, true negatives, false positives, and false negatives. The output presents separate confusion matrices for the champion model (log_model_champion) and the challenger model (rf_model). For log_model_champion, the matrix shows 216 true positives, 196 true negatives, 124 false positives, and 111 false negatives. For rf_model, the matrix shows 227 true positives, 224 true negatives, 96 false positives, and 100 false negatives.

Key insights:

• Challenger records more correct classifications: rf_model produces 227 true positives and 224 true negatives, compared with 216 true positives and 196 true negatives for log_model_champion, indicating higher correct classification counts in both classes.

• Challenger reduces both error types: rf_model shows 96 false positives and 100 false negatives, versus 124 false positives and 111 false negatives for log_model_champion. This reflects fewer incorrect positive and incorrect negative classifications in the challenger model.

• False positives exceed false negatives for champion: In log_model_champion, false positives (124) are higher than false negatives (111). In rf_model, false positives (96) are slightly lower than false negatives (100), indicating a different error balance between the two models.

• Positive-class detection is higher in challenger: The challenger identifies more positive cases correctly, with true positives increasing from 216 to 227 while false negatives decrease from 111 to 100.

The confusion matrix results show that rf_model outperforms log_model_champion on the observed test sample across all four confusion matrix components. Correct classifications increase for both the positive and negative classes, while both false positives and false negatives decline. The comparison also indicates a shift in error composition, with the champion exhibiting more false positives than false negatives, whereas the challenger shows a nearly balanced distribution between the two error types.

Figures

2026-10-30 21:00:48,034 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ConfusionMatrix:champion_vs_challenger does not exist in model's document

❌ Minimum Accuracy Champion Vs Challenger

The Minimum Accuracy test evaluates whether each model’s prediction accuracy meets or exceeds a predefined threshold. The results table reports accuracy scores for the champion model and the challenger model against a threshold of 0.7, using a pass/fail indicator to show whether the threshold was met. In this run, both evaluated models are shown with scores below the threshold, alongside their corresponding fail outcomes.

Key insights:

• Both models failed threshold: The champion model (log_model_champion) and the challenger model (rf_model) both received a fail result against the 0.7 minimum accuracy threshold.
• Challenger scored higher: rf_model recorded an accuracy score of 0.6971, which is higher than log_model_champion at 0.6368.
• Challenger was near cutoff: The challenger model’s score of 0.6971 was 0.0029 below the 0.7 threshold, while the champion model’s score of 0.6368 was 0.0632 below threshold.
• Performance gap is measurable: The difference in accuracy between the challenger and champion models is 0.0603, with the challenger showing the stronger result within this test.

The test results show that neither evaluated model satisfied the minimum accuracy requirement of 0.7 on the tested dataset. Within that outcome, the challenger model demonstrated stronger observed accuracy than the champion model and came close to the cutoff, while the champion model remained further below the threshold. Collectively, the results indicate sub-threshold accuracy for both models in this comparison, with relative performance favoring the challenger.

Tables

model	Score	Threshold	Pass/Fail
log_model_champion	0.6368	0.7	Fail
rf_model	0.6971	0.7	Fail

2026-10-30 21:00:55,807 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumAccuracy:champion_vs_challenger does not exist in model's document

✅ Minimum F1 Score Champion Vs Challenger

The MinimumF1Score test evaluates whether each model’s validation-set F1 score meets a predefined minimum threshold. The result table reports the F1 score, threshold, and pass/fail status for two models: log_model_champion and rf_model. Both models are evaluated against the same threshold of 0.5, with observed scores of 0.6477 and 0.6985, respectively. The table therefore provides a direct comparison of threshold attainment and relative F1 performance across the two models.

Key insights:

• Both models passed the threshold: log_model_champion recorded an F1 score of 0.6477 and rf_model recorded 0.6985, with both exceeding the minimum threshold of 0.5 and receiving a pass result.

• Random forest achieved the higher F1 score: rf_model outperformed log_model_champion on this test, with an F1 score higher by 0.0508 (0.6985 versus 0.6477).

• Positive margin above threshold for both models: log_model_champion exceeded the threshold by 0.1477, while rf_model exceeded it by 0.1985, indicating both models cleared the minimum requirement by a measurable margin.

The results show that both evaluated models satisfied the minimum F1 score criterion on the validation dataset. Among the two, rf_model produced the stronger F1 result and the larger margin above the threshold. Overall, the test outcome indicates that the compared models met the defined minimum standard for balanced precision-recall performance in this evaluation.

Tables

model	Score	Threshold	Pass/Fail
log_model_champion	0.6477	0.5	Pass
rf_model	0.6985	0.5	Pass

2026-10-30 21:01:02,482 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumF1Score:champion_vs_challenger does not exist in model's document

ROC Curve Champion Vs Challenger

The ROCCurve:champion_vs_challenger test evaluates classification performance by plotting the ROC curve and calculating the AUC for each model. The results show ROC curves for the champion logistic model and the challenger random forest model on the same test dataset, each compared against the random-classification reference line. The logistic model is reported with an AUC of 0.69, while the random forest model is reported with an AUC of 0.75. Both curves remain above the diagonal random baseline across most of the false positive rate range.

Key insights:

• Challenger shows higher AUC: The random forest challenger records an AUC of 0.75 versus 0.69 for the logistic champion, indicating stronger ranking performance on the evaluated test dataset.

• Both models exceed random baseline: Each ROC curve lies above the 0.5 AUC reference, and both reported AUC values are above 0.5, showing discrimination beyond random classification.

• Random forest curve rises more quickly: The random forest ROC curve achieves higher true positive rates at comparable false positive rates through much of the curve, reflecting better separation between classes than the logistic model in the plotted results.

The ROC results indicate that both evaluated models demonstrate discriminatory ability on the test dataset, with the challenger random forest outperforming the champion logistic model on AUC. The difference between AUC values, 0.75 versus 0.69, shows stronger class separation for the challenger in this comparison. Overall, the plotted ROC profiles and reported AUC values rank the random forest model ahead of the logistic model under this test.

Figures

2026-10-30 21:01:13,436 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ROCCurve:champion_vs_challenger does not exist in model's document

Based on the performance metrics, our challenger random forest classification model passes the MinimumAccuracy where our champion did not.

In your validation report, support your recommendation in your validation issue's Proposed Remediation Plan to investigate the usage of our challenger by inserting the performance tests we logged with this notebook into the appropriate section.

Run diagnostic tests

Next, we want to inspect the robustness and stability testing comparison between our champion and challenger.

Use list_tests() to list all available diagnosis tests applicable to classification tasks:

vm.tests.list_tests(tags=["model_diagnosis"], task="classification")

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.model_validation.sklearn.OverfitDiagnosis	Overfit Diagnosis	Assesses potential overfitting in a model's predictions, identifying regions where performance between training and...	True	True	['model', 'datasets']	{'metric': {'type': 'str', 'default': None}, 'cut_off_threshold': {'type': 'float', 'default': 0.04}}	['sklearn', 'binary_classification', 'multiclass_classification', 'linear_regression', 'model_diagnosis']	['classification', 'regression']
validmind.model_validation.sklearn.RobustnessDiagnosis	Robustness Diagnosis	Assesses the robustness of a machine learning model by evaluating performance decay under noisy conditions....	True	True	['datasets', 'model']	{'metric': {'type': 'str', 'default': None}, 'scaling_factor_std_dev_list': {'type': 'List', 'default': [0.1, 0.2, 0.3, 0.4, 0.5]}, 'performance_decay_threshold': {'type': 'float', 'default': 0.05}}	['sklearn', 'model_diagnosis', 'visualization']	['classification', 'regression']
validmind.model_validation.sklearn.WeakspotsDiagnosis	Weakspots Diagnosis	Identifies and visualizes weak spots in a machine learning model's performance across various sections of the...	True	True	['datasets', 'model']	{'features_columns': {'type': 'Optional', 'default': None}, 'metrics': {'type': 'Optional', 'default': None}, 'thresholds': {'type': 'Optional', 'default': None}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_diagnosis', 'visualization']	['classification', 'text_classification']

Let’s now assess the models for potential signs of overfitting and identify any sub-segments where performance may inconsistent with the model_validation.sklearn.OverfitDiagnosis test.

Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:

• Since the training dataset (vm_train_ds) was used to fit the model, we use this set to establish a baseline performance for how well the model performs on data it has already seen.
• The testing dataset (vm_test_ds) was never seen during training, and here simulates real-world generalization, or how well the model performs on new, unseen data.

vm.tests.run_test(
    test_id="validmind.model_validation.sklearn.OverfitDiagnosis:champion_vs_challenger",
    input_grid={
        "datasets": [[vm_train_ds,vm_test_ds]],
        "model" : [vm_log_model,vm_rf_model]
    }
).log()

Overfit Diagnosis Champion Vs Challenger

The Overfit Diagnosis test evaluates differences between training and test set performance across feature slices to identify localized overfitting regions. The results compare AUC by feature segment for the champion logistic model (log_model_champion) and the challenger random forest model (rf_model) using a gap threshold of 0.04. For the logistic model, only selected slices exceed the threshold and several plotted segments remain near or below the cutoff. For the random forest model, the training AUC is approximately 1.0 across reported slices and the train-test AUC gap exceeds the threshold across all reported features and segments.

Key insights:

• Random forest gaps are pervasive: Every reported rf_model segment exceeds the 0.04 threshold, with training AUC at 1.0 or 0.9999 throughout the table and test AUC materially lower in each slice.
• Champion model overfit regions are limited: log_model_champion shows threshold breaches in a smaller set of slices, including CreditScore (450, 500] with gap 0.218, Tenure slices (1, 2], (3, 4], and (4, 5] with gaps from 0.0621 to 0.1151, Balance slices with gaps up to 0.5121, IsActiveMember = 0 with gap 0.0407, and EstimatedSalary (179834.694, 199805.63] with gap 0.1168.
• Largest overall gap occurs in rf_model Balance: The maximum reported gap is 1.0 for rf_model in Balance slice (200040.867, 222267.63], where training AUC is 1.0 and test AUC is 0.0, based on 14 training and 4 test records.
• Champion model Balance slices are the most uneven: Within log_model_champion, the largest gaps occur in Balance, notably (44453.526, 66680.289] with gap 0.5121 and (177814.104, 200040.867] with gap 0.4226. Another Balance slice, (200040.867, 222267.63], has gap 0.25 with 14 training and 4 test records.
• rf_model CreditScore shows consistently high separation: Across CreditScore slices, rf_model gaps range from 0.098 to 0.3534, with the largest gap at (550, 600] and elevated gaps across nearly the full score range.
• rf_model overfitting extends across categorical indicators: Binary segments also show substantial gaps for rf_model, including HasCrCard (0.2504, 0.2447), IsActiveMember (0.2679, 0.2736), Geography_Germany (0.2378, 0.2930), Geography_Spain (0.2624, 0.1854), and Gender_Male (0.2745, 0.2331).

The results show a clear contrast between the two models. The champion logistic model exhibits localized train-test AUC gaps concentrated in a limited number of slices, with the largest discrepancies observed in specific Balance ranges and selected Tenure, CreditScore, and EstimatedSalary segments. In contrast, the challenger random forest model displays uniformly elevated gaps across all reported features, with near-perfect training AUC and consistently lower test AUC across both continuous and binary feature slices.

Tables

model	Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
log_model_champion	CreditScore	(450.0, 500.0]	127	25	0.6855	0.4675	0.2180
log_model_champion	Tenure	(1.0, 2.0]	263	62	0.6329	0.5708	0.0621
log_model_champion	Tenure	(3.0, 4.0]	262	54	0.6918	0.5767	0.1151
log_model_champion	Tenure	(4.0, 5.0]	272	62	0.7161	0.6400	0.0761
log_model_champion	Balance	(44453.526, 66680.289]	57	11	0.5788	0.0667	0.5121
log_model_champion	Balance	(111133.815, 133360.578]	543	136	0.7465	0.6207	0.1258
log_model_champion	Balance	(177814.104, 200040.867]	37	17	0.5893	0.1667	0.4226
log_model_champion	Balance	(200040.867, 222267.63]	14	4	0.2500	0.0000	0.2500
log_model_champion	IsActiveMember	(-0.001, 0.1]	1354	364	0.6543	0.6136	0.0407
log_model_champion	EstimatedSalary	(179834.694, 199805.63]	262	66	0.7067	0.5899	0.1168
rf_model	CreditScore	(400.0, 450.0]	45	11	1.0000	0.6667	0.3333
rf_model	CreditScore	(450.0, 500.0]	127	25	1.0000	0.6591	0.3409
rf_model	CreditScore	(500.0, 550.0]	255	75	1.0000	0.7473	0.2527
rf_model	CreditScore	(550.0, 600.0]	381	101	1.0000	0.6466	0.3534
rf_model	CreditScore	(600.0, 650.0]	456	116	1.0000	0.7261	0.2739
rf_model	CreditScore	(650.0, 700.0]	508	119	1.0000	0.8210	0.1790
rf_model	CreditScore	(700.0, 750.0]	394	104	1.0000	0.7545	0.2455
rf_model	CreditScore	(750.0, 800.0]	243	57	1.0000	0.7303	0.2697
rf_model	CreditScore	(800.0, 850.0]	166	35	1.0000	0.9020	0.0980
rf_model	Tenure	(-0.01, 1.0]	353	119	1.0000	0.7376	0.2624
rf_model	Tenure	(1.0, 2.0]	263	62	1.0000	0.7104	0.2896
rf_model	Tenure	(2.0, 3.0]	275	70	1.0000	0.7887	0.2113
rf_model	Tenure	(3.0, 4.0]	262	54	1.0000	0.7223	0.2777
rf_model	Tenure	(4.0, 5.0]	272	62	1.0000	0.7660	0.2340
rf_model	Tenure	(5.0, 6.0]	251	60	0.9999	0.7783	0.2216
rf_model	Tenure	(6.0, 7.0]	242	67	1.0000	0.7723	0.2277
rf_model	Tenure	(7.0, 8.0]	255	69	1.0000	0.7517	0.2483
rf_model	Tenure	(8.0, 9.0]	275	64	1.0000	0.7441	0.2559
rf_model	Tenure	(9.0, 10.0]	137	20	1.0000	0.7172	0.2828
rf_model	Balance	(-222.268, 22226.763]	837	194	1.0000	0.7907	0.2093
rf_model	Balance	(22226.763, 44453.526]	13	3	1.0000	0.5000	0.5000
rf_model	Balance	(44453.526, 66680.289]	57	11	1.0000	0.7833	0.2167
rf_model	Balance	(66680.289, 88907.052]	169	51	1.0000	0.7769	0.2231
rf_model	Balance	(88907.052, 111133.815]	420	108	1.0000	0.7735	0.2265
rf_model	Balance	(111133.815, 133360.578]	543	136	1.0000	0.6400	0.3600
rf_model	Balance	(133360.578, 155587.341]	365	84	1.0000	0.6699	0.3301
rf_model	Balance	(155587.341, 177814.104]	130	37	1.0000	0.7368	0.2632
rf_model	Balance	(177814.104, 200040.867]	37	17	1.0000	0.7986	0.2014
rf_model	Balance	(200040.867, 222267.63]	14	4	1.0000	0.0000	1.0000
rf_model	NumOfProducts	(0.997, 1.3]	1496	377	1.0000	0.6660	0.3340
rf_model	NumOfProducts	(1.9, 2.2]	898	225	1.0000	0.6577	0.3423
rf_model	NumOfProducts	(2.8, 3.1]	159	33	1.0000	0.5111	0.4889
rf_model	HasCrCard	(-0.001, 0.1]	770	204	1.0000	0.7496	0.2504
rf_model	HasCrCard	(0.9, 1.0]	1815	443	1.0000	0.7553	0.2447
rf_model	IsActiveMember	(-0.001, 0.1]	1354	364	1.0000	0.7321	0.2679
rf_model	IsActiveMember	(0.9, 1.0]	1231	283	1.0000	0.7264	0.2736
rf_model	EstimatedSalary	(-103.439, 20067.206]	258	51	1.0000	0.7672	0.2328
rf_model	EstimatedSalary	(20067.206, 40038.142]	242	69	1.0000	0.8471	0.1529
rf_model	EstimatedSalary	(40038.142, 60009.078]	249	59	1.0000	0.7264	0.2736
rf_model	EstimatedSalary	(60009.078, 79980.014]	294	71	1.0000	0.8389	0.1611
rf_model	EstimatedSalary	(79980.014, 99950.95]	247	66	1.0000	0.6296	0.3704
rf_model	EstimatedSalary	(99950.95, 119921.886]	259	64	1.0000	0.7613	0.2387
rf_model	EstimatedSalary	(119921.886, 139892.822]	262	63	1.0000	0.7289	0.2711
rf_model	EstimatedSalary	(139892.822, 159863.758]	247	68	1.0000	0.7195	0.2805
rf_model	EstimatedSalary	(159863.758, 179834.694]	265	68	0.9999	0.7079	0.2920
rf_model	EstimatedSalary	(179834.694, 199805.63]	262	66	1.0000	0.8304	0.1696
rf_model	Geography_Germany	(-0.001, 0.1]	1778	445	1.0000	0.7622	0.2378
rf_model	Geography_Germany	(0.9, 1.0]	807	202	1.0000	0.7070	0.2930
rf_model	Geography_Spain	(-0.001, 0.1]	2006	493	1.0000	0.7376	0.2624
rf_model	Geography_Spain	(0.9, 1.0]	579	154	1.0000	0.8146	0.1854
rf_model	Gender_Male	(-0.001, 0.1]	1244	302	1.0000	0.7255	0.2745
rf_model	Gender_Male	(0.9, 1.0]	1341	345	1.0000	0.7669	0.2331

Figures

2026-10-30 21:01:29,989 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.OverfitDiagnosis:champion_vs_challenger does not exist in model's document

Let's also conduct robustness and stability testing of the two models with the model_validation.sklearn.RobustnessDiagnosis test.

Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets.

Again, we'll use both the training and testing datasets to establish baseline performance and to simulate real-world generalization:

vm.tests.run_test(
    test_id="validmind.model_validation.sklearn.RobustnessDiagnosis:Champion_vs_LogRegression",
    input_grid={
        "datasets": [[vm_train_ds,vm_test_ds]],
        "model" : [vm_log_model,vm_rf_model]
    },
).log()

❌ Robustness Diagnosis Champion Vs Log Regression

The Robustness Diagnosis test evaluates model resilience by measuring AUC under increasing Gaussian noise applied to numeric input features. Results are reported for log_model_champion and rf_model on both train_dataset_final and test_dataset_final across perturbation sizes from baseline through 0.5 standard deviations. The tables and plots show the progression of AUC and performance decay for each model-dataset combination, together with pass/fail outcomes at each perturbation level.

Key insights:

• Logistic model degrades gradually: log_model_champion shows limited AUC decline across all perturbation levels. Train AUC moves from 0.6855 at baseline to 0.6717 at 0.5 noise, while test AUC moves from 0.6866 to 0.6629, with maximum performance decay of 0.0138 on train and 0.0237 on test.

• Logistic model passes all tests: All log_model_champion observations are marked as passed on both training and test datasets from baseline through perturbation size 0.5.

• Random forest shows sharper train sensitivity: rf_model train AUC declines from 1.0000 at baseline to 0.7969 at perturbation size 0.5, with performance decay increasing from 0.0192 at 0.1 noise to 0.2031 at 0.5 noise. Failures begin on the training dataset at perturbation size 0.2 and continue through 0.5.

• Random forest test performance remains stronger than train robustness results imply: On test_dataset_final, rf_model AUC starts at 0.7530, increases slightly to 0.7590 at perturbation size 0.1, and then declines to 0.6883 at 0.5. Test-set observations pass through perturbation size 0.4, with only the 0.5 noise level marked as failed.

• Baseline generalization differs by model: At baseline, log_model_champion has nearly identical train and test AUC values (0.6855 vs. 0.6866), whereas rf_model shows a large train-test gap (1.0000 vs. 0.7530). This gap remains visible across perturbed runs, with training performance consistently above test performance for rf_model.

The robustness results show two distinct patterns across the evaluated models. log_model_champion maintains relatively stable AUC under all tested noise levels and records passing outcomes throughout, with only modest performance decay on both training and test data. rf_model exhibits materially larger sensitivity to perturbation, especially on the training dataset where failures begin at perturbation size 0.2 and performance decay reaches 0.2031 by 0.5; on the test dataset, degradation is more moderate, though performance falls below the pass condition at the highest perturbation level.

Tables

model	Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
log_model_champion	Baseline (0.0)	train_dataset_final	2585	0.6855	0.0000	True
log_model_champion	Baseline (0.0)	test_dataset_final	647	0.6866	0.0000	True
log_model_champion	0.1	train_dataset_final	2585	0.6848	0.0007	True
log_model_champion	0.1	test_dataset_final	647	0.6871	-0.0005	True
log_model_champion	0.2	train_dataset_final	2585	0.6823	0.0032	True
log_model_champion	0.2	test_dataset_final	647	0.6828	0.0038	True
log_model_champion	0.3	train_dataset_final	2585	0.6798	0.0057	True
log_model_champion	0.3	test_dataset_final	647	0.6765	0.0101	True
log_model_champion	0.4	train_dataset_final	2585	0.6779	0.0075	True
log_model_champion	0.4	test_dataset_final	647	0.6711	0.0154	True
log_model_champion	0.5	train_dataset_final	2585	0.6717	0.0138	True
log_model_champion	0.5	test_dataset_final	647	0.6629	0.0237	True
rf_model	Baseline (0.0)	train_dataset_final	2585	1.0000	0.0000	True
rf_model	Baseline (0.0)	test_dataset_final	647	0.7530	0.0000	True
rf_model	0.1	train_dataset_final	2585	0.9807	0.0192	True
rf_model	0.1	test_dataset_final	647	0.7590	-0.0060	True
rf_model	0.2	train_dataset_final	2585	0.9408	0.0592	False
rf_model	0.2	test_dataset_final	647	0.7388	0.0142	True
rf_model	0.3	train_dataset_final	2585	0.9020	0.0980	False
rf_model	0.3	test_dataset_final	647	0.7277	0.0253	True
rf_model	0.4	train_dataset_final	2585	0.8500	0.1500	False
rf_model	0.4	test_dataset_final	647	0.7176	0.0354	True
rf_model	0.5	train_dataset_final	2585	0.7969	0.2031	False
rf_model	0.5	test_dataset_final	647	0.6883	0.0648	False

Figures

2026-10-30 21:01:47,042 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.RobustnessDiagnosis:Champion_vs_LogRegression does not exist in model's document

Run feature importance tests

We also want to verify the relative influence of different input features on our models' predictions, as well as inspect the differences between our champion and challenger to see if a certain model offers more understandable or logical importance scores for features.

Use list_tests() to identify all the feature importance tests for classification:

# Store the feature importance tests
FI = vm.tests.list_tests(tags=["feature_importance"], task="classification",pretty=False)
FI

['validmind.model_validation.FeaturesAUC',
 'validmind.model_validation.sklearn.PermutationFeatureImportance',
 'validmind.model_validation.sklearn.SHAPGlobalImportance']

We'll only use our testing dataset (vm_test_ds) here, to provide a realistic, unseen sample that mimic future or production data, as the training dataset has already influenced our model during learning:

# Run and log our feature importance tests for both models for the testing dataset
for test in FI:
    vm.tests.run_test(
        "".join((test,':champion_vs_challenger')),
        input_grid={
            "dataset": [vm_test_ds], "model" : [vm_log_model,vm_rf_model]
        },
    ).log()

Features Champion Vs Challenger

The FeaturesAUC test evaluates the discriminatory power of each individual feature by calculating its standalone AUC against the binary target. The results are presented as feature-level AUC scores for the test_dataset_final dataset, allowing direct comparison of each variable’s univariate separation strength. The displayed AUC values span from approximately 0.39 to 0.59 across the listed features. Balance has the highest AUC, followed by Geography_Germany, while IsActiveMember has the lowest AUC among the features shown.

Key insights:

• Balance has the highest standalone AUC: Balance records the largest feature-level AUC at approximately 0.59, making it the strongest individual discriminator in the displayed set.

• Geography_Germany is the second strongest feature: Geography_Germany follows closely with an AUC of roughly 0.56, placing it above all other listed features except Balance.

• Most features cluster in the mid-0.4 range: HasCrCard, Geography_Spain, Tenure, EstimatedSalary, CreditScore, Gender_Male, and NumOfProducts appear between about 0.43 and 0.50, indicating a relatively compressed middle tier of univariate discriminatory performance.

• IsActiveMember is the weakest feature shown: IsActiveMember has the lowest AUC at approximately 0.39, below the rest of the displayed variables.

The results show a clear ranking of standalone feature discrimination on the test dataset, with Balance and Geography_Germany separating from the rest of the feature set. Most remaining variables are concentrated within a narrow mid-range band, indicating similar individual AUC performance. The overall spread across features is moderate, from about 0.39 to 0.59, with the weakest and strongest univariate predictors distinctly identifiable from the chart.

Figures

2026-10-30 21:01:59,630 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.FeaturesAUC:champion_vs_challenger does not exist in model's document

Permutation Feature Importance Champion Vs Challenger

The Permutation Feature Importance test evaluates feature significance by measuring the change in model performance after randomly permuting each input feature. The results are presented separately for the champion logistic model and the challenger random forest model, with larger positive permutation importance values indicating greater dependence on a given feature. In the champion model, the largest importances appear for IsActiveMember, Gender_Male, and Geography_Germany, followed by Balance. In the challenger model, NumOfProducts is the dominant feature, followed by Balance and Geography_Germany, while the remaining features have substantially smaller effects.

Key insights:

• Different primary drivers across models: The champion model shows its highest importance for IsActiveMember at approximately 0.05, while the challenger model is most sensitive to NumOfProducts at approximately 0.14. This indicates that the two models concentrate predictive importance on different inputs.

• Challenger importance is more concentrated: In the challenger model, NumOfProducts is materially larger than all other features, with the next highest values being Balance at roughly 0.055 and Geography_Germany at roughly 0.041. In contrast, the champion model distributes importance more across several variables, led by IsActiveMember, Gender_Male, and Geography_Germany.

• Geography_Germany is relevant in both models: Geography_Germany has a positive permutation importance in both models, at roughly 0.024 in the champion model and 0.041 in the challenger model. This places it among the more influential variables in both specifications.

• Several features have near-zero or negative importance: In the champion model, HasCrCard, CreditScore, Tenure, Geography_Spain, and EstimatedSalary are negative, and NumOfProducts is near zero. In the challenger model, EstimatedSalary and HasCrCard are negative, while Tenure is effectively zero and Gender_Male and CreditScore are small positive contributors.

The permutation importance results show that the champion and challenger models rely on different feature structures. The champion model places greatest weight on IsActiveMember and retains moderate reliance on Gender_Male, Geography_Germany, and Balance, whereas the challenger model is dominated by NumOfProducts with secondary dependence on Balance and Geography_Germany. Across both models, several variables contribute little or exhibit negative importance, indicating limited measured performance contribution under permutation.

Figures

2026-10-30 21:02:17,681 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.PermutationFeatureImportance:champion_vs_challenger does not exist in model's document

SHAP Global Importance Champion Vs Challenger

The SHAP Global Importance test evaluates global feature importance using absolute SHAP values and summary plots to show how individual features contribute to model output. The results are presented for both the champion logistic model (log_model_champion) and the challenger random forest model (rf_model). For each model, the normalized feature importance plot ranks variables by mean absolute SHAP value, and the summary plot shows the direction and dispersion of feature-level effects across observations. The reported outputs allow direct comparison of the relative concentration of importance and the distribution of SHAP impacts between the two models.

Key insights:

• Champion importance is highly concentrated: In log_model_champion, IsActiveMember is the most important feature at 100 normalized importance, followed by Geography_Germany at roughly 89 and Gender_Male at roughly 76. The next feature, Balance, drops to about 46, indicating a material concentration of explanatory weight in the top three variables.

• Challenger shifts dominance to NumOfProducts: In rf_model, NumOfProducts is the highest-ranked feature at 100 normalized importance, while IsActiveMember and Geography_Germany follow at approximately 46 and 41 respectively. This differs from the champion model, where NumOfProducts is a lower-ranked contributor.

• Top feature rankings differ across models: Both models assign relatively high importance to IsActiveMember, Geography_Germany, Balance, and Gender_Male, but their ordering changes materially. The champion ranks Gender_Male third and NumOfProducts seventh, whereas the challenger ranks NumOfProducts first and Gender_Male fifth.

• Directional effects are visible for several binary features: In both summary plots, binary variables such as IsActiveMember, Geography_Germany, and Gender_Male show separated SHAP clusters for low and high feature values, indicating distinct directional contributions associated with category membership. In the champion plot, these separations are especially pronounced for the highest-ranked variables.

• Challenger shows broader SHAP spread for NumOfProducts: The rf_model summary plot shows NumOfProducts spanning the widest SHAP range among challenger features, extending from negative values near -0.3 to positive values above 0.4. This spread is larger than the corresponding feature spread in the champion plot and aligns with its leading importance rank in the challenger model.

• Lower-ranked features remain comparatively limited: Geography_Spain and HasCrCard are among the least important features in both models, with normalized importances near the bottom of each ranking and SHAP values clustered closely around zero relative to the leading features.

Across both models, the SHAP results show that feature importance is concentrated in a limited set of predictors, with materially different importance structures between the champion and challenger specifications. The champion model is dominated by IsActiveMember, Geography_Germany, and Gender_Male, while the challenger places the greatest weight on NumOfProducts. Summary plots further show that the challenger exhibits a wider SHAP dispersion for its top feature, whereas lower-ranked variables in both models have comparatively small contributions centered near zero.

Figures

2026-10-30 21:02:32,678 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.SHAPGlobalImportance:champion_vs_challenger does not exist in model's document

In summary

In this third notebook, you learned how to:

• Initialize ValidMind model objects
• Assign predictions and probabilities to your ValidMind model objects
• Use tests from ValidMind to evaluate the potential of models, including comparative tests between champion and challengers
• Log an artifact in the ValidMind Platform

Next steps

Finalize validation and reporting

Now that you're familiar with the basics of using the ValidMind Library to run and log validation tests, let's learn how to implement some custom tests and wrap up our validation: 4 — Finalize validation and reporting

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial