ValidMind for validation 3 — Developing a potential challenger

Learn how to use ValidMind for your end-to-end validation process with our series of four introductory notebooks. In this third notebook, develop a potential challenger and then pass your challenger and its predictions to ValidMind.

A challenger is an alternate record (model) that attempts to outperform the champion, ensuring that the best performing fit-for-purpose record is always considered for deployment. Challengers also help avoid over-reliance on a single record, and allow testing of new features, algorithms, or data sources without disrupting the production lifecycle.

Learn by doing

Our course tailor-made for validators new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Validator Fundamentals

Prerequisites

In order to develop potential challengers with this notebook, you'll need to first have:

• Registered a model within the ValidMind Platform and granted yourself access to the model as a validator
• Installed the ValidMind Library in your local environment, allowing you to access all its features
• Learned how to import and initialize datasets for use with ValidMind
• Understood the basics of how to run and log tests with ValidMind
• Run data quality tests on the datasets used to train the champion, and logged the results of those tests to ValidMind
• Inserted your logged test results into your validation report

Need help with the above steps?

Refer to the first two notebooks in this series:

• 1 — Set up the ValidMind Library for validation
• 2 — Start the validation process

Setting up

This section should be quite familiar to you — as we performed the same actions in the previous notebook, 2 — Start the validation process.

Initialize the ValidMind Library

As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Validation from the Document drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="validation-report",
)

Note: you may need to restart the kernel to use updated packages.

2026-09-11 23:52:21,140 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model validation (ID: cmalguc9y02ok199q2db381ib)
📁 Document Type: validation_report

Import the sample dataset

Next, we'll load in the sample Bank Customer Churn Prediction dataset used to develop the champion that we will independently preprocess:

# Load the sample dataset
from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

Preprocess the dataset

We’ll apply a simple rebalancing technique to the dataset before continuing:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

Let’s also quickly remove highly correlated features from the dataset using the output from a ValidMind test.

As you know, before we can run tests you’ll need to initialize a ValidMind dataset object with the init_dataset function:

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:

# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among variables to identify potentially redundant or highly collinear feature pairs. The result table reports the top 10 pairwise Pearson correlation coefficients, along with Pass/Fail status based on the configured absolute-value threshold of 0.3. Observed coefficients in this output range from -0.1862 to 0.327. Only one pair, (Age, Exited), exceeds the threshold and is marked as Fail; all remaining reported pairs are marked as Pass.

Key insights:

• Single threshold breach observed: The pair (Age, Exited) has a Pearson correlation coefficient of 0.327, which exceeds the configured threshold of 0.3 and is the only reported Fail in the table.

• Remaining reported correlations are low: The other nine reported feature pairs have absolute correlation values between 0.0358 and 0.1862, all below the threshold and therefore classified as Pass.

• Largest negative correlation is modest: The most negative reported coefficient is -0.1862 for (IsActiveMember, Exited), remaining well below the threshold in absolute terms.

• Top reported relationships are concentrated near zero: Aside from (Age, Exited), the reported correlations are weak in magnitude, including (Balance, NumOfProducts) at -0.1632 and (Balance, Exited) at 0.1326, with the remaining listed pairs closer to zero.

The reported correlation structure shows one pair above the configured threshold and a broad set of remaining top correlations with low absolute magnitudes. Within the top 10 reported pairs, the only identified high-correlation relationship is between Age and Exited, while all other listed relationships remain below the test limit. Overall, the output indicates a largely low linear correlation pattern among the reported pairs, with one isolated threshold exceedance.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3270	Fail
(IsActiveMember, Exited)	-0.1862	Pass
(Balance, NumOfProducts)	-0.1632	Pass
(Balance, Exited)	0.1326	Pass
(HasCrCard, IsActiveMember)	-0.0591	Pass
(Tenure, IsActiveMember)	-0.0511	Pass
(NumOfProducts, Exited)	-0.0467	Pass
(Age, Balance)	0.0447	Pass
(NumOfProducts, IsActiveMember)	0.0398	Pass
(Age, HasCrCard)	-0.0358	Pass

# From result object, extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3270	Fail
1	(IsActiveMember, Exited)	-0.1862	Pass
2	(Balance, NumOfProducts)	-0.1632	Pass
3	(Balance, Exited)	0.1326	Pass
4	(HasCrCard, IsActiveMember)	-0.0591	Pass
5	(Tenure, IsActiveMember)	-0.0511	Pass
6	(NumOfProducts, Exited)	-0.0467	Pass
7	(Age, Balance)	0.0447	Pass
8	(NumOfProducts, IsActiveMember)	0.0398	Pass
9	(Age, HasCrCard)	-0.0358	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

# Extract feature names from the list of strings
high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

We can then re-initialize the dataset with a different input_id and the highly correlated features removed and re-run the test for confirmation:

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

# Re-run the test with the reduced feature set
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant or highly collinear variables. The result table reports the top 10 strongest feature-pair correlations after removing duplicate and self-correlations, along with each pair’s Pearson coefficient and Pass/Fail status relative to the configured threshold of 0.3. In this run, all reported correlations are below the threshold in absolute value and are therefore marked as Pass. The listed coefficients range from -0.1862 to 0.1326 across the reported feature pairs.

Key insights:

• No threshold breaches observed: All 10 reported feature pairs pass the test against the 0.3 maximum threshold. None of the observed absolute correlation coefficients exceed 0.1862.

• Strongest relationship remains modest: The largest absolute correlation in the reported results is between IsActiveMember and Exited at -0.1862. This is the most pronounced linear relationship in the table, but it remains below the configured threshold.

• Reported relationships are weak overall: The remaining coefficients are clustered close to zero, including -0.1632 for Balance and NumOfProducts and 0.1326 for Balance and Exited, with all other reported pairs between -0.0591 and 0.0398 or 0.0320 to -0.0251. This indicates limited linear dependence among the listed pairs.

• Both positive and negative associations appear: The table includes negative correlations such as Balance with NumOfProducts (-0.1632) and positive correlations such as Balance with Exited (0.1326). The observed directions vary by pair, with no single directional pattern dominating the reported results.

The reported correlation structure is characterized by uniformly low pairwise Pearson coefficients among the top-ranked feature pairs. No listed relationship exceeds the configured threshold, and the strongest observed association remains materially below it. Collectively, the results indicate that, within the reported top correlations, linear dependence is limited across these feature pairs.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1862	Pass
(Balance, NumOfProducts)	-0.1632	Pass
(Balance, Exited)	0.1326	Pass
(HasCrCard, IsActiveMember)	-0.0591	Pass
(Tenure, IsActiveMember)	-0.0511	Pass
(NumOfProducts, Exited)	-0.0467	Pass
(NumOfProducts, IsActiveMember)	0.0398	Pass
(Tenure, EstimatedSalary)	0.0320	Pass
(CreditScore, Exited)	-0.0306	Pass
(HasCrCard, Exited)	-0.0251	Pass

Split the preprocessed dataset

With our raw dataset rebalanced with highly correlated features removed, let's now spilt our dataset into train and test in preparation for model evaluation testing:

# Encode categorical features in the dataset
balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
1250	621	6	0.00	2	1	1	58883.91	0	False	False	True
3781	659	8	133436.52	1	1	0	56787.80	0	False	False	True
2559	524	1	0.00	2	1	0	126812.85	0	False	False	True
2455	734	3	55853.33	2	0	1	94811.85	1	False	True	False
4157	559	5	0.00	1	1	0	21006.10	1	False	False	False

from sklearn.model_selection import train_test_split

# Split the dataset into train and test
train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

# Initialize the split datasets
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

Import the champion model

With our raw dataset assessed and preprocessed, let's go ahead and import the champion submitted by the development team in the format of a .pkl file: lr_model_champion.pkl

# Import the champion model
import pickle as pkl

with open("lr_model_champion.pkl", "rb") as f:
    log_reg = pkl.load(f)

/opt/hostedtoolcache/Python/3.11.16/x64/lib/python3.11/site-packages/sklearn/base.py:442: InconsistentVersionWarning: Trying to unpickle estimator LogisticRegression from version 1.3.2 when using version 1.7.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(

Training a potential challenger model

We're curious how an alternate model compares to our champion, so let's train a challenger as a basis for our testing.

Our champion logistic regression model is a simpler, parametric model that assumes a linear relationship between the independent variables and the log-odds of the outcome. While logistic regression may not capture complex patterns as effectively, it offers a high degree of interpretability and is easier to explain to stakeholders. However, risk is not calculated in isolation from a single factor, but rather in consideration with trade-offs in predictive performance, ease of interpretability, and overall alignment with business objectives.

Random forest classification model

A random forest classification model is an ensemble machine learning algorithm that uses multiple decision trees to classify data. In ensemble learning, multiple models are combined to improve prediction accuracy and robustness.

Random forest classification models generally have higher accuracy because they capture complex, non-linear relationships, but as a result they lack transparency in their predictions.

# Import the Random Forest Classification model
from sklearn.ensemble import RandomForestClassifier

# Create the model instance with 50 decision trees
rf_model = RandomForestClassifier(
    n_estimators=50,
    random_state=42,
)

# Train the model
rf_model.fit(X_train, y_train)

RandomForestClassifier(n_estimators=50, random_state=42)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Initialize the ValidMind models

In addition to the initialized datasets, you'll also need to initialize a ValidMind model object (vm_model) that can be passed to other functions for analysis and tests on the data for each of our two models.

• Despite the naming convention, ValidMind model objects can be any type of record you want to test, document, validate, or monitor with the ValidMind Library.
• From classical statistical and machine learning models, to generative and agentic AI systems and more, the ValidMind model object provides a consistent wrapper around your record so it can be passed as a unified input to any ValidMind test or test suite, with results sent directly to the ValidMind Platform.

Initialize your model objects with vm.init_model():

# Initialize the champion logistic regression model
vm_log_model = vm.init_model(
    log_reg,
    input_id="log_model_champion",
)

# Initialize the challenger random forest classification model
vm_rf_model = vm.init_model(
    rf_model,
    input_id="rf_model",
)

Assign predictions

With our models registered, we'll move on to assigning both the predictive probabilities coming directly from each model's predictions, and the binary prediction after applying the cutoff threshold described in the Compute binary predictions step above.

• The assign_predictions() method from the Dataset object can link existing predictions to any number of models.
• This method links the model's class prediction values and probabilities to our vm_train_ds and vm_test_ds datasets.

If no prediction values are passed, the method will compute predictions automatically:

# Champion — Logistic regression model
vm_train_ds.assign_predictions(model=vm_log_model)
vm_test_ds.assign_predictions(model=vm_log_model)

# Challenger — Random forest classification model
vm_train_ds.assign_predictions(model=vm_rf_model)
vm_test_ds.assign_predictions(model=vm_rf_model)

2026-09-11 23:52:30,923 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-09-11 23:52:30,925 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-09-11 23:52:30,925 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-09-11 23:52:30,927 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-09-11 23:52:30,929 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-09-11 23:52:30,930 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-09-11 23:52:30,931 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-09-11 23:52:30,932 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-09-11 23:52:30,934 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-09-11 23:52:30,954 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-09-11 23:52:30,955 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-09-11 23:52:30,974 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-09-11 23:52:30,976 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-09-11 23:52:30,984 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-09-11 23:52:30,985 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-09-11 23:52:30,993 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Running model evaluation tests

With our setup complete, let's run the rest of our validation tests. Since we have already verified the data quality of the dataset used to train our champion, we will now focus on comprehensive performance evaluations of both the champion and challenger models.

Run model performance tests

Let's run some performance tests, beginning with independent testing of our champion logistic regression model, then moving on to our potential challenger model.

Use vm.tests.list_tests() to identify all the model performance tests for classification:

vm.tests.list_tests(tags=["model_performance"], task="classification")

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.model_validation.sklearn.CalibrationCurve	Calibration Curve	Evaluates the calibration of probability estimates by comparing predicted probabilities against observed...	True	False	['model', 'dataset']	{'n_bins': {'type': 'int', 'default': 10}}	['sklearn', 'model_performance', 'classification']	['classification']
validmind.model_validation.sklearn.ClassifierPerformance	Classifier Performance	Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy,...	False	True	['dataset', 'model']	{'average': {'type': 'str', 'default': 'macro'}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.ConfusionMatrix	Confusion Matrix	Evaluates and visually represents the classification ML model's predictive performance using a Confusion Matrix...	True	False	['dataset', 'model']	{'threshold': {'type': 'float', 'default': 0.5}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.sklearn.HyperParametersTuning	Hyper Parameters Tuning	Performs exhaustive grid search over specified parameter ranges to find optimal model configurations...	False	True	['model', 'dataset']	{'param_grid': {'type': 'dict', 'default': None}, 'scoring': {'type': 'Union', 'default': None}, 'thresholds': {'type': 'Union', 'default': None}, 'fit_params': {'type': 'dict', 'default': None}}	['sklearn', 'model_performance']	['clustering', 'classification']
validmind.model_validation.sklearn.MinimumAccuracy	Minimum Accuracy	Checks if the model's prediction accuracy meets or surpasses a specified threshold....	False	True	['dataset', 'model']	{'min_threshold': {'type': 'float', 'default': 0.7}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.MinimumF1Score	Minimum F1 Score	Assesses if the model's F1 score on the validation set meets a predefined minimum threshold, ensuring balanced...	False	True	['dataset', 'model']	{'min_threshold': {'type': 'float', 'default': 0.5}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.MinimumROCAUCScore	Minimum ROCAUC Score	Validates model by checking if the ROC AUC score meets or surpasses a specified threshold....	False	True	['dataset', 'model']	{'min_threshold': {'type': 'float', 'default': 0.5}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.ModelsPerformanceComparison	Models Performance Comparison	Evaluates and compares the performance of multiple Machine Learning models using various metrics like accuracy,...	False	True	['dataset', 'models']	{}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'model_comparison']	['classification', 'text_classification']
validmind.model_validation.sklearn.PopulationStabilityIndex	Population Stability Index	Assesses the Population Stability Index (PSI) to quantify the stability of an ML model's predictions across...	True	True	['datasets', 'model']	{'num_bins': {'type': 'int', 'default': 10}, 'mode': {'type': 'str', 'default': 'fixed'}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.PrecisionRecallCurve	Precision Recall Curve	Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve....	True	False	['model', 'dataset']	{}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.sklearn.ROCCurve	ROC Curve	Evaluates classification model performance by generating and plotting the Receiver Operating Characteristic...	True	False	['model', 'dataset']	{}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.sklearn.RegressionErrors	Regression Errors	Assesses the performance and error distribution of a regression model using various error metrics....	False	True	['model', 'dataset']	{}	['sklearn', 'model_performance']	['regression', 'classification']
validmind.model_validation.sklearn.TrainingTestDegradation	Training Test Degradation	Tests if model performance degradation between training and test datasets exceeds a predefined threshold....	False	True	['datasets', 'model']	{'max_threshold': {'type': 'float', 'default': 0.1}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.statsmodels.GINITable	GINI Table	Evaluates classification model performance using AUC, GINI, and KS metrics for training and test datasets....	False	True	['dataset', 'model']	{}	['model_performance']	['classification']
validmind.ongoing_monitoring.CalibrationCurveDrift	Calibration Curve Drift	Evaluates changes in probability calibration between reference and monitoring datasets....	True	True	['datasets', 'model']	{'n_bins': {'type': 'int', 'default': 10}, 'drift_pct_threshold': {'type': 'float', 'default': 20}}	['sklearn', 'binary_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.ongoing_monitoring.ClassDiscriminationDrift	Class Discrimination Drift	Compares classification discrimination metrics between reference and monitoring datasets....	False	True	['datasets', 'model']	{'drift_pct_threshold': {'type': '_empty', 'default': 20}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.ongoing_monitoring.ClassificationAccuracyDrift	Classification Accuracy Drift	Compares classification accuracy metrics between reference and monitoring datasets....	False	True	['datasets', 'model']	{'drift_pct_threshold': {'type': '_empty', 'default': 20}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.ongoing_monitoring.ConfusionMatrixDrift	Confusion Matrix Drift	Compares confusion matrix metrics between reference and monitoring datasets....	False	True	['datasets', 'model']	{'drift_pct_threshold': {'type': '_empty', 'default': 20}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.ongoing_monitoring.ROCCurveDrift	ROC Curve Drift	Compares ROC curves between reference and monitoring datasets....	True	False	['datasets', 'model']	{}	['sklearn', 'binary_classification', 'model_performance', 'visualization']	['classification', 'text_classification']

We'll isolate the specific tests we want to run in mpt:

• model_validation.sklearn.ClassifierPerformance
• model_validation.sklearn.ConfusionMatrix
• model_validation.sklearn.MinimumAccuracy
• model_validation.sklearn.MinimumF1Score
• model_validation.sklearn.ROCCurve

As we learned in the previous notebook 2 — Start the model validation process, you can use a custom result_id to tag the individual result with a unique identifier by appending this result_id to the test_id with a : separator. We'll append an identifier for our champion model here:

mpt = [
    "validmind.model_validation.sklearn.ClassifierPerformance:logreg_champion",
    "validmind.model_validation.sklearn.ConfusionMatrix:logreg_champion",
    "validmind.model_validation.sklearn.MinimumAccuracy:logreg_champion",
    "validmind.model_validation.sklearn.MinimumF1Score:logreg_champion",
    "validmind.model_validation.sklearn.ROCCurve:logreg_champion"
]

Evaluate performance of the champion model

Now, let's run and log our batch of model performance tests using our testing dataset (vm_test_ds) for our champion model:

• The test set serves as a proxy for real-world data, providing an unbiased estimate of model performance since it was not used during training or tuning.
• The test set also acts as protection against selection bias and model tweaking, giving a final, more unbiased checkpoint.

for test in mpt:
    vm.tests.run_test(
        test,
        inputs={
            "dataset": vm_test_ds, "model" : vm_log_model,
        },
    ).log()

Classifier Performance Logreg Champion

The Classifier Performance test evaluates classification effectiveness using precision, recall, F1-score, accuracy, and ROC AUC. The reported results present class-level precision, recall, and F1 for classes 0 and 1, together with weighted and macro averages. Separate summary metrics show overall accuracy of 0.643 and ROC AUC of 0.6797. The values allow comparison of performance across classes as well as assessment of aggregate classification and ranking performance.

Key insights:

• Class performance is closely aligned: Class 0 and class 1 show similar metric levels, with precision of 0.6558 vs. 0.6290, recall of 0.6577 vs. 0.6270, and F1 of 0.6568 vs. 0.6280. This indicates relatively even performance across the two classes, with class 0 performing modestly better on all three measures.

• Aggregate averages are nearly identical: Weighted average and macro average metrics are almost the same, with precision at 0.6429 vs. 0.6424, recall at 0.6430 vs. 0.6424, and F1 at 0.6429 vs. 0.6424. The close alignment indicates that aggregate performance summaries do not materially differ by averaging method in this result.

• Overall classification metrics are in the mid-0.64 range: Accuracy is 0.643, while weighted-average precision, recall, and F1 are each approximately 0.643. This shows consistency between overall accuracy and the averaged class-based performance metrics.

• ROC AUC exceeds accuracy and F1 levels: ROC AUC is reported at 0.6797, which is higher than accuracy and the reported average F1 scores. This indicates stronger ranking performance than the threshold-dependent classification metrics shown in the report.

The results show broadly consistent performance across both classes, with class 0 modestly outperforming class 1 on precision, recall, and F1. Aggregate metrics are highly stable across macro and weighted averaging, and overall accuracy aligns closely with the average class-level measures. ROC AUC is higher than the threshold-based classification metrics, indicating comparatively stronger separability than the final class assignment results alone.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6558	0.6577	0.6568
1	0.6290	0.6270	0.6280
Weighted Average	0.6429	0.6430	0.6429
Macro Average	0.6424	0.6424	0.6424

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6430
ROC AUC	0.6797

2026-09-11 23:52:39,549 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ClassifierPerformance:logreg_champion does not exist in model's document

Confusion Matrix Logreg Champion

The Confusion Matrix test evaluates the classification model’s predictive performance by comparing predicted class labels with observed class labels and displaying the resulting counts of true positives, true negatives, false positives, and false negatives. The heatmap shows four outcome cells for the binary classifier: 221 true negatives, 195 true positives, 115 false positives, and 116 false negatives. These values provide a direct view of how predictions are distributed across correct and incorrect classifications for classes 0 and 1.

Key insights:

• Correct classifications exceed errors: The model records 221 true negatives and 195 true positives, for a total of 416 correct classifications, compared with 231 total misclassifications from 115 false positives and 116 false negatives.
• Error types are nearly balanced: False positives and false negatives are almost identical in count, at 115 and 116 respectively, indicating that misclassification is distributed similarly across the two error types.
• Negative class slightly better identified: True negatives (221) exceed true positives (195), showing more correct classifications for class 0 than for class 1 in absolute terms.
• Observed class counts are similar: Based on the matrix totals, the true class 0 count is 336 (221 + 115) and the true class 1 count is 311 (116 + 195), indicating a relatively similar representation of both classes in the evaluated sample.

The confusion matrix indicates that the model produces more correct than incorrect classifications overall, with both true positive and true negative counts materially higher than the corresponding error counts. Misclassification is evenly split between false positives and false negatives, without a pronounced skew toward one error type. The evaluated sample also shows relatively similar observed counts for the two classes, supporting interpretation of both sides of the matrix on a comparable basis.

Figures

2026-09-11 23:52:47,666 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ConfusionMatrix:logreg_champion does not exist in model's document

❌ Minimum Accuracy Logreg Champion

The Minimum Accuracy test evaluates whether the model’s prediction accuracy meets or exceeds a specified minimum threshold. The result table reports the model’s accuracy score, the threshold applied in the test, and the resulting pass/fail outcome. In this run, the observed accuracy score is 0.643 against a threshold of 0.700, and the test result is recorded as Fail.

Key insights:

• Accuracy below threshold: The model achieved an accuracy score of 0.643, which is below the configured minimum threshold of 0.700 by 0.057.
• Test result is fail: The comparison between the observed score and the threshold produced a Fail outcome for this test.

The test result shows that the model’s observed classification accuracy did not meet the minimum level defined for this evaluation. The measured shortfall of 0.057 relative to the threshold is reflected directly in the failed test outcome.

Tables

Score	Threshold	Pass/Fail
0.643	0.7	Fail

2026-09-11 23:52:52,303 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumAccuracy:logreg_champion does not exist in model's document

✅ Minimum F1 Score Logreg Champion

The MinimumF1Score test evaluates whether the model’s F1 score on the validation dataset meets a predefined minimum threshold. The result table reports a validation F1 score of 0.628 alongside a minimum threshold of 0.5, with the outcome recorded as Pass. These values provide the basis for assessing whether the model’s observed balance between precision and recall satisfies the configured test criterion.

Key insights:

• F1 score exceeds threshold: The validation F1 score is 0.628 versus a minimum threshold of 0.5, placing the observed score 0.128 points above the required level.
• Test outcome is passing: The result is explicitly marked as Pass, indicating the configured minimum F1 score condition was met on the validation dataset.

The test result shows that the model satisfied the predefined minimum F1 score requirement on the validation set. The observed F1 score of 0.628 exceeded the threshold of 0.5, and the recorded outcome confirms a passing result for this performance check.

Tables

Score	Threshold	Pass/Fail
0.628	0.5	Pass

2026-09-11 23:52:54,642 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumF1Score:logreg_champion does not exist in model's document

ROC Curve Logreg Champion

The ROCCurve test evaluates classification performance by plotting the receiver operating characteristic curve and quantifying discrimination with the area under the curve (AUC). For logreg_champion on test_dataset_final, the result shows a single binary ROC curve against the random-classification reference line. The plotted curve remains above the diagonal baseline across most of the false positive rate range, and the reported AUC is 0.68.

Key insights:

• AUC is 0.68: The chart reports an area under the ROC curve of 0.68 for logreg_champion on the test dataset, summarizing the model’s overall ranking performance across classification thresholds.
• Curve stays above random baseline: The ROC curve lies above the 0.5 AUC reference line for most of the plot, indicating higher true positive rates than the random benchmark at comparable false positive rates.
• Discrimination varies across thresholds: The curve increases gradually rather than approaching the top-left corner early, showing that gains in true positive rate are achieved progressively as false positive rate rises.

The ROC result indicates that the model exhibits measurable discrimination on the test dataset, with performance above the random benchmark as reflected by an AUC of 0.68. The shape of the curve shows consistent but moderate separation between classes across thresholds rather than sharply concentrated performance in the low-false-positive region. Overall, the test documents positive classification signal with discrimination that is present but not strong.

Figures

2026-09-11 23:53:02,455 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ROCCurve:logreg_champion does not exist in model's document

Note the output returned indicating that a test-driven block doesn't currently exist in your documentation for some test IDs.

That's expected, as when we run validations tests the results logged need to be manually added to your report as part of your compliance assessment process within the ValidMind Platform.

Log an artifact

As we can observe from the output above, our champion doesn't pass the MinimumAccuracy based on the default thresholds of the out-of-the-box test, so let's log an artifact (finding) in the ValidMind Platform (Learn more: Add and manage artifacts):

1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

2. In the left sidebar that appears for your model, click Validation under Documents.

3. Click on 2.2.2. Model Performance to expand that section.

4. Under the Model Performance Metrics guideline, click to expand the Artifacts panel.

5. Click Link Artifact and select Validation Issue as the type of artifact.

6. Click + Add Validation Issue and enter in the details for your validation issue, for example:

   • Title — Champion Logistic Regression Model Fails Minimum Accuracy Threshold
   • Risk Area — Model Performance
   • Documentation Section — 3.2. Model Evaluation
   • Description — The logistic regression champion model was subjected to a Minimum Accuracy test to determine whether its predictive accuracy meets the predefined performance threshold of 0.7. The model achieved an accuracy score of 0.6136, which falls below the required minimum. As a result, the test produced a Fail outcome.

7. Click Add Validation Issue to submit the validation issue.

8. Select the validation issue you just added to link to your validation report.

9. Click Update Linked Artifacts to insert your validation issue.

10. Confirm that the validation issue you inserted has been correctly inserted into section 2.2.2. Model Performance of the report.

11. Click on the validation issue to expand the issue, where you can adjust details such as severity, owner, due date, status, etc. as well as include proposed remediation plans or supporting documentation as attachments.

Evaluate performance of challenger model

We've now conducted similar tests as the development team for our champion, with the aim of verifying their test results.

Next, let's see how our challengers compare. We'll use the same batch of tests here as we did in mpt, but append a different result_id to indicate that these results should be associated with our challenger:

mpt_chall = [
    "validmind.model_validation.sklearn.ClassifierPerformance:champion_vs_challenger",
    "validmind.model_validation.sklearn.ConfusionMatrix:champion_vs_challenger",
    "validmind.model_validation.sklearn.MinimumAccuracy:champion_vs_challenger",
    "validmind.model_validation.sklearn.MinimumF1Score:champion_vs_challenger",
    "validmind.model_validation.sklearn.ROCCurve:champion_vs_challenger"
]

We'll run each test once for each model with the same vm_test_ds dataset to compare them:

for test in mpt_chall:
    vm.tests.run_test(
        test,
        input_grid={
            "dataset": [vm_test_ds], "model" : [vm_log_model,vm_rf_model]
        }
    ).log()

Classifier Performance Champion Vs Challenger

The Classifier Performance test evaluates classification model performance using precision, recall, F1-score, accuracy, and ROC AUC. The results compare the champion model (log_model_champion) and the challenger model (rf_model) across class-level metrics for classes 0 and 1, along with macro average, weighted average, overall accuracy, and ROC AUC. The reported values show how each model performs at both the aggregate level and by class, allowing direct comparison of discrimination and balance across the two candidate models.

Key insights:

• Challenger outperforms champion overall: rf_model records higher aggregate performance across all reported summary metrics. Weighted-average precision, recall, and F1 are 0.6958, 0.6955, and 0.6946 versus 0.6429, 0.6430, and 0.6429 for log_model_champion, while accuracy is 0.6955 versus 0.6430 and ROC AUC is 0.7642 versus 0.6797.

• Class 0 performance is stronger in the challenger: For class 0, rf_model shows precision of 0.6925, recall of 0.7440, and F1 of 0.7174, compared with 0.6558, 0.6577, and 0.6568 for log_model_champion. The largest separation is in recall, where the challenger exceeds the champion by 0.0863.

• Class 1 precision improves with the challenger: For class 1, rf_model achieves precision of 0.6993 versus 0.6290 for log_model_champion, with F1 of 0.6700 versus 0.6280. Class 1 recall is also higher for rf_model at 0.6431 compared with 0.6270, though the gap is smaller than for precision.

• Metric balance is comparable within each model: log_model_champion shows closely aligned macro-average precision, recall, and F1 at 0.6424, 0.6424, and 0.6424, and rf_model shows similarly aligned macro-average values at 0.6959, 0.6936, and 0.6937. This indicates that, within each model, the reported aggregate precision and recall are broadly balanced.

The comparison shows a consistent performance advantage for rf_model over log_model_champion across class-specific and overall evaluation metrics. The challenger posts higher precision, recall, and F1 for both classes, along with higher accuracy and ROC AUC, indicating stronger classification performance in this test run. The aggregate averages for both models remain internally consistent, with the primary distinction being the uniformly higher metric levels observed for the challenger.

Tables

model	Class	Precision	Recall	F1
log_model_champion	0	0.6558	0.6577	0.6568
log_model_champion	1	0.6290	0.6270	0.6280
log_model_champion	Weighted Average	0.6429	0.6430	0.6429
log_model_champion	Macro Average	0.6424	0.6424	0.6424
rf_model	0	0.6925	0.7440	0.7174
rf_model	1	0.6993	0.6431	0.6700
rf_model	Weighted Average	0.6958	0.6955	0.6946
rf_model	Macro Average	0.6959	0.6936	0.6937

model	Metric	Value
log_model_champion	Accuracy	0.6430
log_model_champion	ROC AUC	0.6797
rf_model	Accuracy	0.6955
rf_model	ROC AUC	0.7642

2026-09-11 23:53:09,659 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ClassifierPerformance:champion_vs_challenger does not exist in model's document

Confusion Matrix Champion Vs Challenger

The Confusion Matrix test evaluates classification performance by comparing predicted labels with observed labels and displaying the counts of true positives, true negatives, false positives, and false negatives. The results show confusion matrices for two models: log_model_champion and rf_model. For log_model_champion, the matrix contains 195 true positives, 221 true negatives, 115 false positives, and 116 false negatives. For rf_model, the matrix contains 200 true positives, 250 true negatives, 86 false positives, and 111 false negatives.

Key insights:

• Lower error counts in rf_model: rf_model records fewer false positives and false negatives than log_model_champion, with false positives decreasing from 115 to 86 and false negatives decreasing from 116 to 111.

• Higher correct classification counts: rf_model shows higher counts in both correct prediction cells, with true positives increasing from 195 to 200 and true negatives increasing from 221 to 250 relative to log_model_champion.

• Largest difference in true negatives: The most pronounced cell-level change between the two matrices is in true negatives, which increase by 29 in rf_model compared with log_model_champion.

• Positive-class errors remain material in both models: Both models show substantial false negative counts relative to their true positive counts, with 116 false negatives versus 195 true positives for log_model_champion and 111 false negatives versus 200 true positives for rf_model.

Across the observed confusion matrix counts, rf_model demonstrates stronger classification results than log_model_champion by increasing both true positive and true negative classifications while reducing both false positive and false negative errors. The improvement is most visible in the negative class, where the true negative count is materially higher and the false positive count is lower. Both models continue to exhibit nontrivial misclassification in each error category, particularly in false negatives for the positive class.

Figures

2026-09-11 23:53:20,924 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ConfusionMatrix:champion_vs_challenger does not exist in model's document

❌ Minimum Accuracy Champion Vs Challenger

The Minimum Accuracy test evaluates whether each model’s prediction accuracy meets or surpasses the specified minimum threshold. The results table compares model-level accuracy scores against a threshold of 0.7 and records the corresponding pass/fail outcome for each model. Two models are included in the result set: log_model_champion with an accuracy score of 0.643 and rf_model with an accuracy score of 0.6955. Both scores are shown alongside the common threshold and associated test status.

Key insights:

• Both models failed the threshold: log_model_champion and rf_model are both marked as Fail, with neither accuracy score reaching the 0.7 minimum threshold.
• Random forest scored higher: rf_model achieved an accuracy of 0.6955 compared with 0.643 for log_model_champion, making it the stronger performer within this test result.
• Challenger was near the cutoff: rf_model falls short of the threshold by 0.0045, while log_model_champion is below the threshold by 0.057.

The test results show that neither evaluated model met the minimum accuracy criterion of 0.7 on the dataset used for this assessment. Among the two, rf_model produced the higher observed accuracy and was materially closer to the threshold than log_model_champion. The result indicates a failed minimum-accuracy outcome for both the champion and challenger models under the defined test condition.

Tables

model	Score	Threshold	Pass/Fail
log_model_champion	0.6430	0.7	Fail
rf_model	0.6955	0.7	Fail

2026-09-11 23:53:28,269 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumAccuracy:champion_vs_challenger does not exist in model's document

✅ Minimum F1 Score Champion Vs Challenger

The MinimumF1Score test evaluates whether each model’s validation-set F1 score meets the predefined minimum threshold for balanced precision and recall performance. The results table reports the F1 score, threshold, and pass/fail status for the compared models. Two models are listed: log_model_champion with an F1 score of 0.628 and rf_model with an F1 score of 0.67, and both are evaluated against the same threshold of 0.5.

Key insights:

• Both models passed the threshold: log_model_champion and rf_model both exceeded the minimum F1 score threshold of 0.5, and each received a pass result.
• Random forest scored higher: rf_model recorded the higher F1 score at 0.67 versus 0.628 for log_model_champion.
• Shared evaluation standard applied: Both models were assessed against an identical threshold of 0.5, allowing direct comparison of their reported F1 scores and pass/fail outcomes.

The test results show that both evaluated models met the minimum F1 score requirement on the validation set. Within this comparison, rf_model achieved the higher observed F1 score, while log_model_champion also remained above the threshold. Overall, the reported results indicate that both models satisfied the defined minimum criterion under a common evaluation standard.

Tables

model	Score	Threshold	Pass/Fail
log_model_champion	0.628	0.5	Pass
rf_model	0.670	0.5	Pass

2026-09-11 23:53:31,430 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumF1Score:champion_vs_challenger does not exist in model's document

ROC Curve Champion Vs Challenger

The ROCCurve:champion_vs_challenger test evaluates classification performance by plotting the ROC curve and calculating AUC for each model on the test dataset. The result includes ROC curves for log_model_champion and rf_model, each compared against the random-classification reference line. The plotted AUC values are 0.68 for log_model_champion and 0.76 for rf_model, with both curves remaining above the 0.5 baseline across most of the false positive rate range.

Key insights:

• Challenger shows higher AUC: rf_model records an AUC of 0.76 versus 0.68 for log_model_champion, indicating stronger rank-order discrimination on the evaluated test dataset.
• Both models exceed random baseline: Each ROC curve lies above the diagonal random reference line, and both AUC values are greater than 0.5.
• Separation is visible across thresholds: The rf_model ROC curve remains consistently above the log_model_champion curve visually, reflecting higher true positive rates at comparable false positive rate levels over much of the threshold range.

The ROC results show that both evaluated models demonstrate discriminatory ability on the test dataset, with rf_model exhibiting stronger overall separation than log_model_champion. The difference is reflected in both the higher AUC value and the higher ROC curve across most threshold regions. Collectively, the result indicates a measurable performance gap between the two models under this test.

Figures

2026-09-11 23:53:42,873 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ROCCurve:champion_vs_challenger does not exist in model's document

Based on the performance metrics, our challenger random forest classification model passes the MinimumAccuracy where our champion did not.

In your validation report, support your recommendation in your validation issue's Proposed Remediation Plan to investigate the usage of our challenger by inserting the performance tests we logged with this notebook into the appropriate section.

Run diagnostic tests

Next, we want to inspect the robustness and stability testing comparison between our champion and challenger.

Use list_tests() to list all available diagnosis tests applicable to classification tasks:

vm.tests.list_tests(tags=["model_diagnosis"], task="classification")

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.model_validation.sklearn.OverfitDiagnosis	Overfit Diagnosis	Assesses potential overfitting in a model's predictions, identifying regions where performance between training and...	True	True	['model', 'datasets']	{'metric': {'type': 'str', 'default': None}, 'cut_off_threshold': {'type': 'float', 'default': 0.04}}	['sklearn', 'binary_classification', 'multiclass_classification', 'linear_regression', 'model_diagnosis']	['classification', 'regression']
validmind.model_validation.sklearn.RobustnessDiagnosis	Robustness Diagnosis	Assesses the robustness of a machine learning model by evaluating performance decay under noisy conditions....	True	True	['datasets', 'model']	{'metric': {'type': 'str', 'default': None}, 'scaling_factor_std_dev_list': {'type': 'List', 'default': [0.1, 0.2, 0.3, 0.4, 0.5]}, 'performance_decay_threshold': {'type': 'float', 'default': 0.05}}	['sklearn', 'model_diagnosis', 'visualization']	['classification', 'regression']
validmind.model_validation.sklearn.WeakspotsDiagnosis	Weakspots Diagnosis	Identifies and visualizes weak spots in a machine learning model's performance across various sections of the...	True	True	['datasets', 'model']	{'features_columns': {'type': 'Optional', 'default': None}, 'metrics': {'type': 'Optional', 'default': None}, 'thresholds': {'type': 'Optional', 'default': None}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_diagnosis', 'visualization']	['classification', 'text_classification']

Let’s now assess the models for potential signs of overfitting and identify any sub-segments where performance may inconsistent with the model_validation.sklearn.OverfitDiagnosis test.

Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:

• Since the training dataset (vm_train_ds) was used to fit the model, we use this set to establish a baseline performance for how well the model performs on data it has already seen.
• The testing dataset (vm_test_ds) was never seen during training, and here simulates real-world generalization, or how well the model performs on new, unseen data.

vm.tests.run_test(
    test_id="validmind.model_validation.sklearn.OverfitDiagnosis:champion_vs_challenger",
    input_grid={
        "datasets": [[vm_train_ds,vm_test_ds]],
        "model" : [vm_log_model,vm_rf_model]
    }
).log()

Overfit Diagnosis Champion Vs Challenger

The Overfit Diagnosis test evaluates differences between training and test-set performance across feature-based slices to identify regions where the AUC gap exceeds the 0.04 threshold. Results are reported for both log_model_champion and rf_model, with slice-level training AUC, test AUC, record counts, and AUC gaps shown by feature. The output highlights the specific feature intervals where the observed gap crosses the threshold, allowing comparison of the concentration and magnitude of train-test divergence across the two models.

Key insights:

• Challenger shows broader overfit exposure: rf_model exceeds the 0.04 threshold across all reported feature groups in the table, covering CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, Geography_Germany, Geography_Spain, and Gender_Male.

• Challenger training AUC is uniformly 1.0: For every reported rf_model slice, training AUC equals 1.0, while test AUC ranges from 0.5090 to 0.9444 in the table, producing gaps from 0.0556 to 0.4910.

• Largest challenger gap occurs in Balance: The highest reported gap for rf_model is 0.4910 in Balance slice (150538.854, 175628.663], with training AUC 1.0 and test AUC 0.5090. Other large Balance gaps include 0.3560 for (75269.427, 100359.236] and 0.3333 for (25089.809, 50179.618].

• Challenger overfitting is persistent across Tenure and CreditScore: For rf_model, all listed Tenure slices exceed the threshold, with gaps from 0.1712 to 0.3905, and all listed CreditScore slices exceed the threshold, with gaps from 0.0981 to 0.2753.

• Champion overfit regions are fewer and more localized: log_model_champion has reported threshold exceedances in selected slices only: CreditScore, Tenure, Balance, NumOfProducts, EstimatedSalary, and Geography_Spain. No reported exceedances appear for HasCrCard, IsActiveMember, Geography_Germany, or Gender_Male.

• Champion’s largest gaps are concentrated in Balance: For log_model_champion, the largest reported gap is 0.3846 in Balance slice (200718.472, 225808.281], based on 15 training records and 4 test records, with training AUC 0.3846 and test AUC 0.0. Additional Balance gaps are 0.2243, 0.1998, and 0.1581 in adjacent higher-balance ranges.

• Champion non-Balance exceedances are moderate: Outside Balance, log_model_champion gaps remain materially smaller, including 0.0872 for Tenure (6.0, 7.0], 0.0831 for Geography_Spain (0.9, 1.0], 0.0588 for Tenure (-0.01, 1.0], and 0.0571 for NumOfProducts (0.997, 1.3].

• Some high-gap slices have small sample counts: Several reported slices with larger gaps are based on relatively few records, including log_model_champion Balance (200718.472, 225808.281] with 15 training and 4 test records, log_model_champion Balance (175628.663, 200718.472] with 46 training and 13 test records, and rf_model Balance (25089.809, 50179.618] with 20 training and 5 test records.

Overall, the test output shows a markedly different overfit profile between the two models. rf_model exhibits threshold exceedances across every reported feature family, with uniformly perfect training AUC and large train-test gaps throughout the feature space. log_model_champion also shows overfit regions, but these are fewer in number and are most pronounced in selected Balance intervals, with smaller additional gaps in limited CreditScore, Tenure, NumOfProducts, EstimatedSalary, and Geography_Spain slices.

Tables

model	Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
log_model_champion	CreditScore	(500.0, 550.0]	275	67	0.6929	0.6388	0.0540
log_model_champion	CreditScore	(700.0, 750.0]	394	96	0.6905	0.6461	0.0444
log_model_champion	Tenure	(-0.01, 1.0]	371	115	0.7267	0.6679	0.0588
log_model_champion	Tenure	(1.0, 2.0]	278	68	0.6578	0.6053	0.0525
log_model_champion	Tenure	(6.0, 7.0]	225	72	0.6825	0.5953	0.0872
log_model_champion	Balance	(50179.618, 75269.427]	94	19	0.6203	0.4205	0.1998
log_model_champion	Balance	(150538.854, 175628.663]	195	45	0.6563	0.4320	0.2243
log_model_champion	Balance	(175628.663, 200718.472]	46	13	0.5390	0.3810	0.1581
log_model_champion	Balance	(200718.472, 225808.281]	15	4	0.3846	0.0000	0.3846
log_model_champion	NumOfProducts	(0.997, 1.3]	1510	371	0.6785	0.6214	0.0571
log_model_champion	EstimatedSalary	(40007.76, 60005.85]	248	69	0.6775	0.6256	0.0519
log_model_champion	Geography_Spain	(0.9, 1.0]	577	152	0.6769	0.5938	0.0831
rf_model	CreditScore	(400.0, 450.0]	45	20	1.0000	0.7250	0.2750
rf_model	CreditScore	(450.0, 500.0]	119	37	1.0000	0.7588	0.2412
rf_model	CreditScore	(500.0, 550.0]	275	67	1.0000	0.7247	0.2753
rf_model	CreditScore	(550.0, 600.0]	361	100	1.0000	0.7292	0.2708
rf_model	CreditScore	(600.0, 650.0]	464	120	1.0000	0.7537	0.2463
rf_model	CreditScore	(650.0, 700.0]	496	103	1.0000	0.7960	0.2040
rf_model	CreditScore	(700.0, 750.0]	394	96	1.0000	0.7571	0.2429
rf_model	CreditScore	(750.0, 800.0]	243	66	1.0000	0.9019	0.0981
rf_model	CreditScore	(800.0, 850.0]	178	34	1.0000	0.7945	0.2055
rf_model	Tenure	(-0.01, 1.0]	371	115	1.0000	0.7285	0.2715
rf_model	Tenure	(1.0, 2.0]	278	68	1.0000	0.8184	0.1816
rf_model	Tenure	(2.0, 3.0]	280	63	1.0000	0.7453	0.2547
rf_model	Tenure	(3.0, 4.0]	264	59	1.0000	0.7759	0.2241
rf_model	Tenure	(4.0, 5.0]	255	56	1.0000	0.8288	0.1712
rf_model	Tenure	(5.0, 6.0]	265	52	1.0000	0.8162	0.1838
rf_model	Tenure	(6.0, 7.0]	225	72	1.0000	0.8016	0.1984
rf_model	Tenure	(7.0, 8.0]	256	62	1.0000	0.7142	0.2858
rf_model	Tenure	(8.0, 9.0]	254	71	1.0000	0.7825	0.2175
rf_model	Tenure	(9.0, 10.0]	137	29	1.0000	0.6095	0.3905
rf_model	Balance	(-250.898, 25089.809]	782	228	1.0000	0.8068	0.1932
rf_model	Balance	(25089.809, 50179.618]	20	5	1.0000	0.6667	0.3333
rf_model	Balance	(50179.618, 75269.427]	94	19	1.0000	0.7784	0.2216
rf_model	Balance	(75269.427, 100359.236]	303	63	1.0000	0.6440	0.3560
rf_model	Balance	(100359.236, 125449.045]	619	155	1.0000	0.7658	0.2342
rf_model	Balance	(125449.045, 150538.854]	509	115	1.0000	0.7505	0.2495
rf_model	Balance	(150538.854, 175628.663]	195	45	1.0000	0.5090	0.4910
rf_model	Balance	(175628.663, 200718.472]	46	13	1.0000	0.8214	0.1786
rf_model	NumOfProducts	(0.997, 1.3]	1510	371	1.0000	0.6471	0.3529
rf_model	NumOfProducts	(1.9, 2.2]	877	242	1.0000	0.6784	0.3216
rf_model	NumOfProducts	(2.8, 3.1]	160	28	1.0000	0.9444	0.0556
rf_model	HasCrCard	(-0.001, 0.1]	761	190	1.0000	0.7831	0.2169
rf_model	HasCrCard	(0.9, 1.0]	1824	457	1.0000	0.7564	0.2436
rf_model	IsActiveMember	(-0.001, 0.1]	1390	348	1.0000	0.7599	0.2401
rf_model	IsActiveMember	(0.9, 1.0]	1195	299	1.0000	0.7280	0.2720
rf_model	EstimatedSalary	(-188.401, 20009.67]	260	50	1.0000	0.6832	0.3168
rf_model	EstimatedSalary	(20009.67, 40007.76]	241	67	1.0000	0.8249	0.1751
rf_model	EstimatedSalary	(40007.76, 60005.85]	248	69	1.0000	0.6600	0.3400
rf_model	EstimatedSalary	(60005.85, 80003.94]	274	76	1.0000	0.8308	0.1692
rf_model	EstimatedSalary	(80003.94, 100002.03]	255	69	1.0000	0.6815	0.3185
rf_model	EstimatedSalary	(100002.03, 120000.12]	252	64	1.0000	0.7629	0.2371
rf_model	EstimatedSalary	(120000.12, 139998.21]	254	61	1.0000	0.7323	0.2677
rf_model	EstimatedSalary	(139998.21, 159996.3]	277	45	1.0000	0.8026	0.1974
rf_model	EstimatedSalary	(159996.3, 179994.39]	272	74	1.0000	0.8088	0.1912
rf_model	EstimatedSalary	(179994.39, 199992.48]	252	72	1.0000	0.7946	0.2054
rf_model	Geography_Germany	(-0.001, 0.1]	1773	440	1.0000	0.7356	0.2644
rf_model	Geography_Germany	(0.9, 1.0]	812	207	1.0000	0.7556	0.2444
rf_model	Geography_Spain	(-0.001, 0.1]	2008	495	1.0000	0.7633	0.2367
rf_model	Geography_Spain	(0.9, 1.0]	577	152	1.0000	0.7598	0.2402
rf_model	Gender_Male	(-0.001, 0.1]	1243	318	1.0000	0.7487	0.2513
rf_model	Gender_Male	(0.9, 1.0]	1342	329	1.0000	0.7685	0.2315

Figures

2026-09-11 23:54:00,893 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.OverfitDiagnosis:champion_vs_challenger does not exist in model's document

Let's also conduct robustness and stability testing of the two models with the model_validation.sklearn.RobustnessDiagnosis test.

Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets.

Again, we'll use both the training and testing datasets to establish baseline performance and to simulate real-world generalization:

vm.tests.run_test(
    test_id="validmind.model_validation.sklearn.RobustnessDiagnosis:Champion_vs_LogRegression",
    input_grid={
        "datasets": [[vm_train_ds,vm_test_ds]],
        "model" : [vm_log_model,vm_rf_model]
    },
).log()

❌ Robustness Diagnosis Champion Vs Log Regression

The Robustness Diagnosis test evaluates model resilience by measuring AUC changes after adding Gaussian noise to numeric input features across increasing perturbation sizes. Results are reported for log_model_champion and rf_model on both train_dataset_final and test_dataset_final, with baseline and perturbed AUC values, performance decay, and pass/fail outcomes shown for each noise level. The figures and table show how performance changes from perturbation sizes 0.1 through 0.5, allowing comparison of decay patterns between datasets and between the two models.

Key insights:

• Champion model shows limited AUC decay: log_model_champion declines from 0.6748 to 0.6615 on the training set and from 0.6797 to 0.6627 on the test set as perturbation size increases from baseline to 0.5. Corresponding performance decay remains between 0.0010 and 0.0187 across all perturbed runs.

• Champion model passes all robustness checks: Every log_model_champion run is marked Passed = true on both training and test datasets for all perturbation sizes from baseline through 0.5.

• Random forest has higher baseline AUC: rf_model starts at AUC 1.0000 on the training set and 0.7642 on the test set, exceeding the baseline AUC of log_model_champion on both datasets.

• Random forest training performance decays sharply: On train_dataset_final, rf_model AUC decreases from 1.0000 at baseline to 0.9827, 0.9443, 0.8908, 0.8379, and 0.7888 as perturbation size increases from 0.1 to 0.5. Performance decay rises accordingly from 0.0173 to 0.2112.

• Random forest training failures begin at 0.2 noise: For rf_model on the training set, the test passes at 0.1 but fails at perturbation sizes 0.2, 0.3, 0.4, and 0.5, where performance decay is 0.0557, 0.1092, 0.1621, and 0.2112 respectively.

• Random forest test performance is more stable than training: On test_dataset_final, rf_model AUC remains within 0.7117 to 0.7642 across all perturbation levels, with performance decay between 0.0030 and 0.0525. It passes through perturbation size 0.4 and fails only at 0.5.

• Test-set stability differs across models: At perturbation size 0.5, log_model_champion records test AUC 0.6627 with decay 0.0170 and passes, while rf_model records test AUC 0.7117 with decay 0.0525 and fails. This reflects a smaller observed relative degradation for the champion model under the highest tested noise level.

The robustness results show distinct decay profiles for the two models. log_model_champion exhibits gradual AUC reductions on both training and test datasets and remains within passing criteria at every perturbation level tested. rf_model begins with higher baseline AUC, but its training performance declines materially as noise increases, with failures starting at perturbation size 0.2, while its test performance remains comparatively steadier and does not fail until perturbation size 0.5.

Tables

model	Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
log_model_champion	Baseline (0.0)	train_dataset_final	2585	0.6748	0.0000	True
log_model_champion	Baseline (0.0)	test_dataset_final	647	0.6797	0.0000	True
log_model_champion	0.1	train_dataset_final	2585	0.6725	0.0023	True
log_model_champion	0.1	test_dataset_final	647	0.6787	0.0010	True
log_model_champion	0.2	train_dataset_final	2585	0.6712	0.0036	True
log_model_champion	0.2	test_dataset_final	647	0.6761	0.0036	True
log_model_champion	0.3	train_dataset_final	2585	0.6703	0.0045	True
log_model_champion	0.3	test_dataset_final	647	0.6762	0.0035	True
log_model_champion	0.4	train_dataset_final	2585	0.6613	0.0135	True
log_model_champion	0.4	test_dataset_final	647	0.6610	0.0187	True
log_model_champion	0.5	train_dataset_final	2585	0.6615	0.0133	True
log_model_champion	0.5	test_dataset_final	647	0.6627	0.0170	True
rf_model	Baseline (0.0)	train_dataset_final	2585	1.0000	0.0000	True
rf_model	Baseline (0.0)	test_dataset_final	647	0.7642	0.0000	True
rf_model	0.1	train_dataset_final	2585	0.9827	0.0173	True
rf_model	0.1	test_dataset_final	647	0.7568	0.0074	True
rf_model	0.2	train_dataset_final	2585	0.9443	0.0557	False
rf_model	0.2	test_dataset_final	647	0.7522	0.0120	True
rf_model	0.3	train_dataset_final	2585	0.8908	0.1092	False
rf_model	0.3	test_dataset_final	647	0.7612	0.0030	True
rf_model	0.4	train_dataset_final	2585	0.8379	0.1621	False
rf_model	0.4	test_dataset_final	647	0.7388	0.0254	True
rf_model	0.5	train_dataset_final	2585	0.7888	0.2112	False
rf_model	0.5	test_dataset_final	647	0.7117	0.0525	False

Figures

2026-09-11 23:54:19,426 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.RobustnessDiagnosis:Champion_vs_LogRegression does not exist in model's document

Run feature importance tests

We also want to verify the relative influence of different input features on our models' predictions, as well as inspect the differences between our champion and challenger to see if a certain model offers more understandable or logical importance scores for features.

Use list_tests() to identify all the feature importance tests for classification:

# Store the feature importance tests
FI = vm.tests.list_tests(tags=["feature_importance"], task="classification",pretty=False)
FI

['validmind.model_validation.FeaturesAUC',
 'validmind.model_validation.sklearn.PermutationFeatureImportance',
 'validmind.model_validation.sklearn.SHAPGlobalImportance']

We'll only use our testing dataset (vm_test_ds) here, to provide a realistic, unseen sample that mimic future or production data, as the training dataset has already influenced our model during learning:

# Run and log our feature importance tests for both models for the testing dataset
for test in FI:
    vm.tests.run_test(
        "".join((test,':champion_vs_challenger')),
        input_grid={
            "dataset": [vm_test_ds], "model" : [vm_log_model,vm_rf_model]
        },
    ).log()

Features Champion Vs Challenger

The FeaturesAUC test evaluates the discriminatory power of each individual feature by calculating a univariate AUC against the binary target. The result is presented as a ranked bar chart of feature-level AUC scores for test_dataset_final, with values spanning from just under 0.40 to slightly above 0.60. Geography_Germany appears as the highest-scoring feature, followed by Balance, EstimatedSalary, and Tenure, while NumOfProducts and IsActiveMember are the lowest-scoring features in the displayed set.

Key insights:

• Geography_Germany has the highest AUC: Geography_Germany records the strongest univariate discrimination at approximately 0.61, making it the top-ranked feature in the chart.

• Top features are moderately separated: Balance is shown at roughly 0.55, while EstimatedSalary and Tenure are both near 0.52, forming a small leading group below the highest-ranked feature.

• Most features cluster near 0.45–0.52: CreditScore, HasCrCard, Geography_Spain, and Gender_Male are concentrated in a relatively narrow band from about 0.43 to 0.48, indicating similar standalone discrimination across these variables.

• Lowest scores are below 0.42: IsActiveMember is near 0.41 and NumOfProducts is near 0.39, placing them at the bottom of the ranking shown in the figure.

The test result shows a clear ranking of univariate discriminatory strength across the displayed features, with Geography_Germany standing above the rest and a broad middle group concentrated around the mid-0.40s to low-0.50s. The spread across features is moderate rather than extreme, with only one feature slightly above 0.60 and the lowest feature just below 0.40. Overall, the chart indicates that standalone separation varies across predictors, with the strongest signal concentrated in a limited subset of features.

Figures

2026-09-11 23:54:35,866 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.FeaturesAUC:champion_vs_challenger does not exist in model's document

Permutation Feature Importance Champion Vs Challenger

The Permutation Feature Importance test evaluates the significance of each input feature by measuring the change in model performance after randomly permuting that feature. The results are presented separately for the champion model (log_model_champion) and the challenger model (rf_model) as ranked importance bars. In the champion model, the largest importances are associated with Geography_Germany, IsActiveMember, and Gender_Male, followed by Balance, while the remaining features have notably smaller values. In the challenger model, NumOfProducts is the dominant feature, with Geography_Germany and Balance as secondary contributors, and several features appear near zero or slightly negative.

Key insights:

• Champion importance is concentrated in three features: In log_model_champion, Geography_Germany has the highest importance at approximately 0.057, followed by IsActiveMember at about 0.052 and Gender_Male at about 0.039. These three features are separated from the rest of the feature set by a visible gap in importance.

• Challenger is dominated by NumOfProducts: In rf_model, NumOfProducts has the largest importance at approximately 0.138, materially exceeding the next two features, Geography_Germany at about 0.060 and Balance at about 0.056.

• Feature ranking differs materially across models: IsActiveMember and Gender_Male are among the most important features in the champion model, but both are near zero in the challenger model. Conversely, NumOfProducts has limited importance in the champion model and is the strongest driver in the challenger model.

• Several challenger features are non-contributory or negative: In rf_model, CreditScore, HasCrCard, and EstimatedSalary show slightly negative importance values, while IsActiveMember, Gender_Male, and Geography_Spain are effectively zero. This indicates little measured performance loss from permuting these features in that model.

The permutation importance results show that the champion and challenger models rely on different predictors to generate performance. The champion model distributes importance primarily across Geography_Germany, IsActiveMember, and Gender_Male, whereas the challenger model is driven predominantly by NumOfProducts with additional contribution from Geography_Germany and Balance. Several lower-ranked features contribute minimally in both models, and the challenger model includes multiple features with near-zero or slightly negative importance.

Figures

2026-09-11 23:54:56,460 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.PermutationFeatureImportance:champion_vs_challenger does not exist in model's document

SHAP Global Importance Champion Vs Challenger

The SHAP Global Importance test evaluates global feature importance using absolute SHAP values and summary plots to show how individual features contribute to model output. The results present both normalized SHAP importance rankings and SHAP summary distributions for the champion logistic model (log_model_champion) and the challenger random forest model (rf_model). In the champion model, IsActiveMember, Geography_Germany, and Gender_Male appear as the three most important features, followed by Balance, while the challenger model is led by NumOfProducts, followed by IsActiveMember, Geography_Germany, and Gender_Male. The summary plots additionally show the direction and spread of SHAP contributions for each feature across observations.

Key insights:

• Different leading drivers across models: The champion model assigns the highest normalized importance to IsActiveMember at 100, with Geography_Germany and Gender_Male next at roughly the low-80s and mid-70s. The challenger model instead ranks NumOfProducts first at 100, with IsActiveMember second at roughly the low-60s.

• Champion importance is more concentrated: In the champion model, the top four features (IsActiveMember, Geography_Germany, Gender_Male, and Balance) are materially separated from the remaining variables, with Balance near the midpoint of the scale and all other features below roughly 20. In the challenger model, importance declines more gradually after NumOfProducts, with several features (IsActiveMember, Geography_Germany, Gender_Male, and Balance) retaining moderate contribution levels.

• Binary features show directional separation: In both models, binary variables such as IsActiveMember, Geography_Germany, and Gender_Male display clear separation between low and high feature values on opposite sides of zero in the summary plots. For example, IsActiveMember shows low feature values associated with positive SHAP contributions and high feature values associated with negative contributions in both models.

• NumOfProducts is a distinguishing challenger driver: NumOfProducts is one of the least important features in the champion model but the dominant feature in the challenger model. Its summary plot for the challenger model shows both negative and strongly positive SHAP values, including a visible right tail extending above 0.4.

• Balance has positive right-skewed contribution: Balance is among the more important features in both models and shows a concentration of positive SHAP values, especially in the champion model where many observations cluster between roughly 0.05 and 0.30 with some larger positive values. The challenger model also shows positive Balance contributions, though with a tighter spread around zero.

• Lowest-ranked features are consistent: Geography_Spain and HasCrCard are among the least important variables in both models. Their SHAP distributions remain tightly concentrated near zero relative to higher-ranked features.

Overall, the SHAP results show that the champion and challenger models rely on overlapping feature sets but differ materially in how importance is allocated across them. The champion model places strongest emphasis on membership activity and selected demographic indicators, while the challenger model is more heavily driven by NumOfProducts and distributes importance more broadly across the top-ranked variables. Across both models, lower-ranked features contribute comparatively little, and several binary features exhibit clearly separated directional effects in the SHAP summary plots.

Figures

2026-09-11 23:55:11,124 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.SHAPGlobalImportance:champion_vs_challenger does not exist in model's document

In summary

In this third notebook, you learned how to:

• Initialize ValidMind model objects
• Assign predictions and probabilities to your ValidMind model objects
• Use tests from ValidMind to evaluate the potential of models, including comparative tests between champion and challengers
• Log an artifact in the ValidMind Platform

Next steps

Finalize validation and reporting

Now that you're familiar with the basics of using the ValidMind Library to run and log validation tests, let's learn how to implement some custom tests and wrap up our validation: 4 — Finalize validation and reporting

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial