ValidMind for development 3 — Integrate custom tests
Learn how to use ValidMind for your end-to-end documentation process with our series of four introductory notebooks. In this third notebook, supplement ValidMind tests with your own and include them as additional evidence in your documentation.
This notebook assumes that you already have a repository of custom made tests considered critical to include in your documentation. A custom test is any function that takes a set of inputs and parameters as arguments and returns one or more outputs:
The function can be as simple or as complex as you need it to be — it can use external libraries, make API calls, or do anything else that you can do in Python.
The only requirement is that the function signature and return values can be "understood" and handled by the ValidMind Library. As such, custom tests offer added flexibility by extending the default tests provided by ValidMind, enabling you to document any type of record (model) or use case.
For a more in-depth introduction to custom tests, refer to our Implement custom tests notebook.
Learn by doing
Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Developer Fundamentals
Prerequisites
In order to integrate custom tests with your documentation with this notebook, you'll need to first have:
# Make sure the ValidMind Library is installed%pip install -q validmind# Load your model identifier credentials from an `.env` file%load_ext dotenv%dotenv .env# Or replace with your code snippetimport validmind as vmvm.init(# api_host="...",# api_key="...",# api_secret="...",# model="...", document="documentation",)
Note: you may need to restart the kernel to use updated packages.
2026-11-20 23:02:19,050 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model development (ID: cmalgf3qi02ce199qm3rdkl46)
📁 Document Type: model_documentation
Import sample dataset
Next, we'll import the same public Bank Customer Churn Prediction dataset from Kaggle we used in the last notebook so that we have something to work with:
from validmind.datasets.classification import customer_churn as demo_datasetprint(f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}")raw_df = demo_dataset.load_data()
Loaded demo dataset with:
• Target column: 'Exited'
• Class labels: {'0': 'Did not exit', '1': 'Exited'}
We'll apply a simple rebalancing technique to the dataset before continuing:
import pandas as pdraw_copy_df = raw_df.sample(frac=1) # Create a copy of the raw dataset# Create a balanced dataset with the same number of exited and not exited customersexited_df = raw_copy_df.loc[raw_copy_df["Exited"] ==1]not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] ==0].sample(n=exited_df.shape[0])balanced_raw_df = pd.concat([exited_df, not_exited_df])balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)
Remove highly correlated features
Let's also quickly remove highly correlated features from the dataset using the output from a ValidMind test.
As you learned previously, before we can run tests you'll need to initialize a ValidMind dataset object:
# Register new data and now 'balanced_raw_dataset' is the new dataset object of interestvm_balanced_raw_dataset = vm.init_dataset( dataset=balanced_raw_df, input_id="balanced_raw_dataset", target_column="Exited",)
With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:
# Run HighPearsonCorrelation test with our balanced dataset as input and return a result objectcorr_result = vm.tests.run_test( test_id="validmind.data_validation.HighPearsonCorrelation", params={"max_threshold": 0.3}, inputs={"dataset": vm_balanced_raw_dataset},)
❌ High Pearson Correlation
The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant or highly collinear variable pairs. The results table lists the top reported feature pairs with their Pearson correlation coefficients and pass/fail status against the configured absolute correlation threshold of 0.3. Across the 10 reported pairs, coefficients range from -0.1837 to 0.3274, with one pair exceeding the threshold and the remaining pairs marked as passing.
Key insights:
One pair exceeds threshold: The pair (Age, Exited) records the highest reported correlation at 0.3274 and is the only result marked Fail against the 0.3 threshold.
All other reported pairs pass: The remaining nine reported feature pairs have absolute correlation values below 0.3, with pass-designated coefficients ranging from -0.1837 to 0.1528 and smaller magnitudes thereafter.
Correlations are mostly weak in magnitude: Aside from (Age, Exited), the largest absolute reported coefficient is 0.1837 for (IsActiveMember, Exited), indicating that the reported relationships outside the failed pair are comparatively limited in linear strength.
Reported relationships include both positive and negative directions: The table shows positive correlations such as (Balance, Exited) at 0.1528 and negative correlations such as (Balance, NumOfProducts) at -0.1759, reflecting mixed directional relationships across the reported pairs.
The reported correlation structure is limited in concentration at the configured threshold, with only one feature pair, (Age, Exited), exceeding 0.3. All other displayed pairwise relationships remain below the threshold and are materially smaller in magnitude. Overall, the result indicates that the top reported correlations are dominated by a single failing pair, while the remaining listed correlations are weak to modest in linear strength.
Parameters:
{
"max_threshold": 0.3
}
Tables
Columns
Coefficient
Pass/Fail
(Age, Exited)
0.3274
Fail
(IsActiveMember, Exited)
-0.1837
Pass
(Balance, NumOfProducts)
-0.1759
Pass
(Balance, Exited)
0.1528
Pass
(Tenure, IsActiveMember)
-0.0706
Pass
(NumOfProducts, Exited)
-0.0547
Pass
(HasCrCard, IsActiveMember)
-0.0547
Pass
(NumOfProducts, IsActiveMember)
0.0457
Pass
(Age, NumOfProducts)
-0.0436
Pass
(Tenure, EstimatedSalary)
0.0383
Pass
# From result object, extract table from `corr_result.tables`features_df = corr_result.tables[0].datafeatures_df
Columns
Coefficient
Pass/Fail
0
(Age, Exited)
0.3274
Fail
1
(IsActiveMember, Exited)
-0.1837
Pass
2
(Balance, NumOfProducts)
-0.1759
Pass
3
(Balance, Exited)
0.1528
Pass
4
(Tenure, IsActiveMember)
-0.0706
Pass
5
(NumOfProducts, Exited)
-0.0547
Pass
6
(HasCrCard, IsActiveMember)
-0.0547
Pass
7
(NumOfProducts, IsActiveMember)
0.0457
Pass
8
(Age, NumOfProducts)
-0.0436
Pass
9
(Tenure, EstimatedSalary)
0.0383
Pass
# Extract list of features that failed the testhigh_correlation_features = features_df[features_df["Pass/Fail"] =="Fail"]["Columns"].tolist()high_correlation_features
['(Age, Exited)']
# Extract feature names from the list of stringshigh_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]high_correlation_features
['Age']
We can then re-initialize the dataset with a different input_id and the highly correlated features removed and re-run the test for confirmation:
# Remove the highly correlated features from the datasetbalanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)# Re-initialize the dataset objectvm_raw_dataset_preprocessed = vm.init_dataset( dataset=balanced_raw_no_age_df, input_id="raw_dataset_preprocessed", target_column="Exited",)
# Re-run the test with the reduced feature setcorr_result = vm.tests.run_test( test_id="validmind.data_validation.HighPearsonCorrelation", params={"max_threshold": 0.3}, inputs={"dataset": vm_raw_dataset_preprocessed},)
✅ High Pearson Correlation
The High Pearson Correlation test evaluates pairwise linear relationships between features to identify potentially redundant variables or multicollinearity. The results table reports the top 10 feature pairs by absolute Pearson correlation coefficient using a threshold of 0.3 for pass/fail assessment. All reported coefficients fall below the threshold and are marked as Pass, with values ranging from -0.1837 to 0.1528. The listed relationships include both negative and positive associations, with the largest absolute correlations observed for (IsActiveMember, Exited), (Balance, NumOfProducts), and (Balance, Exited).
Key insights:
No pair exceeds threshold: All 10 reported feature pairs are classified as Pass under the 0.3 threshold, indicating that none of the strongest observed linear relationships crosses the test limit.
Largest absolute correlation is modest: The strongest relationship by absolute value is between IsActiveMember and Exited at -0.1837, which remains materially below the threshold.
Balance appears in multiple top pairs: Balance is present in two of the three largest reported correlations, with NumOfProducts at -0.1759 and Exited at 0.1528, making it the most recurrent variable among the highest-ranked pairs shown.
Observed relationships are generally weak: The reported coefficients are all close to zero in magnitude, and the lower-ranked pairs range from -0.0706 to 0.0251.
Overall, the reported correlation structure is limited in magnitude across the top-ranked feature pairs. The strongest observed associations remain below the configured threshold, and the table does not show evidence of high pairwise linear dependence among the listed variables. Within the displayed results, Balance is the only feature appearing repeatedly among the largest correlations, but those relationships also remain below the test limit.
Parameters:
{
"max_threshold": 0.3
}
Tables
Columns
Coefficient
Pass/Fail
(IsActiveMember, Exited)
-0.1837
Pass
(Balance, NumOfProducts)
-0.1759
Pass
(Balance, Exited)
0.1528
Pass
(Tenure, IsActiveMember)
-0.0706
Pass
(NumOfProducts, Exited)
-0.0547
Pass
(HasCrCard, IsActiveMember)
-0.0547
Pass
(NumOfProducts, IsActiveMember)
0.0457
Pass
(Tenure, EstimatedSalary)
0.0383
Pass
(CreditScore, Exited)
-0.0266
Pass
(Tenure, HasCrCard)
0.0251
Pass
Train the model
We'll then use ValidMind tests to train a simple logistic regression model on our prepared dataset:
# First encode the categorical features in our dataset with the highly correlated features removedbalanced_raw_no_age_df = pd.get_dummies( balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True)balanced_raw_no_age_df.head()
CreditScore
Tenure
Balance
NumOfProducts
HasCrCard
IsActiveMember
EstimatedSalary
Exited
Geography_Germany
Geography_Spain
Gender_Male
454
848
3
110929.96
1
1
1
30876.84
0
False
True
True
2286
657
5
119785.47
2
1
1
107534.32
0
False
True
True
915
626
1
0.00
2
1
0
66232.23
0
False
False
True
6186
777
4
137851.31
1
1
0
5008.23
1
False
False
False
4438
715
9
105489.31
1
0
0
143096.49
1
True
False
False
# Split the processed dataset into train and testfrom sklearn.model_selection import train_test_splittrain_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)X_train = train_df.drop("Exited", axis=1)y_train = train_df["Exited"]X_test = test_df.drop("Exited", axis=1)y_test = test_df["Exited"]
Let's initialize the ValidMind Dataset and Model objects in preparation for assigning predictions to each dataset:
# Initialize the datasets into their own ValidMind dataset objectsvm_train_ds = vm.init_dataset( input_id="train_dataset_final", dataset=train_df, target_column="Exited",)vm_test_ds = vm.init_dataset( input_id="test_dataset_final", dataset=test_df, target_column="Exited",)# Initialize the ValidMind model objectvm_model = vm.init_model(log_reg, input_id="log_reg_model_v1")
Assign predictions
Once the model is registered, we'll assign predictions to the training and test datasets:
2026-11-20 23:02:33,499 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-11-20 23:02:33,501 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-11-20 23:02:33,501 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-11-20 23:02:33,503 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-11-20 23:02:33,504 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-11-20 23:02:33,505 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-11-20 23:02:33,505 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-11-20 23:02:33,506 - INFO(validmind.vm_models.dataset.utils): Done running predict()
Implementing a custom inline test
With the set up out of the way, let's implement a custom inline test that calculates the confusion matrix for a binary classification model.
An inline test refers to a test written and executed within the same environment as the code being tested — in this case, right in this Jupyter Notebook — without requiring a separate test file or framework.
You'll note that the custom test function is just a regular Python function that can include and require any Python library as you see fit.
Create a confusion matrix plot
Let's first create a confusion matrix plot using the confusion_matrix function from the sklearn.metrics module:
import matplotlib.pyplot as pltfrom sklearn import metrics# Get the predicted classesy_pred = log_reg.predict(vm_test_ds.x)confusion_matrix = metrics.confusion_matrix(y_test, y_pred)cm_display = metrics.ConfusionMatrixDisplay( confusion_matrix=confusion_matrix, display_labels=[False, True])cm_display.plot()
Next, create a @vm.test wrapper that will allow you to create a reusable test. Note the following changes in the code below:
The function confusion_matrix takes two arguments dataset and model. This is a VMDataset and VMModel object respectively.
VMDataset objects allow you to access the dataset's true (target) values by accessing the .y attribute.
VMDataset objects allow you to access the predictions for a given record (model) by accessing the .y_pred() method.
The function docstring provides a description of what the test does. This will be displayed along with the result in this notebook as well as in the ValidMind Platform.
The function body calculates the confusion matrix using the sklearn.metrics.confusion_matrix function as we just did above.
The function then returns the ConfusionMatrixDisplay.figure_ object — this is important as the ValidMind Library expects the output of the custom test to be a plot or a table.
The @vm.test decorator is doing the work of creating a wrapper around the function that will allow it to be run by the ValidMind Library. It also registers the test so it can be found by the ID my_custom_tests.ConfusionMatrix.
@vm.test("my_custom_tests.ConfusionMatrix")def confusion_matrix(dataset, model):"""The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known. The confusion matrix is a 2x2 table that contains 4 values: - True Positive (TP): the number of correct positive predictions - True Negative (TN): the number of correct negative predictions - False Positive (FP): the number of incorrect positive predictions - False Negative (FN): the number of incorrect negative predictions The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure. """ y_true = dataset.y y_pred = dataset.y_pred(model=model) confusion_matrix = metrics.confusion_matrix(y_true, y_pred) cm_display = metrics.ConfusionMatrixDisplay( confusion_matrix=confusion_matrix, display_labels=[False, True] ) cm_display.plot() plt.close() # close the plot to avoid displaying itreturn cm_display.figure_ # return the figure object itself
You can now run the newly created custom test on both the training and test datasets using the run_test() function:
# Training datasetresult = vm.tests.run_test("my_custom_tests.ConfusionMatrix:training_dataset", inputs={"model": vm_model, "dataset": vm_train_ds},)
Confusion Matrix Training Dataset
The Confusion Matrix test evaluates classification outcomes by comparing predicted labels against true labels on the training dataset. The matrix shows 874 observations with true label False predicted as False, 433 observations with true label False predicted as True, 487 observations with true label True predicted as False, and 791 observations with true label True predicted as True. These four counts summarize the distribution of correct and incorrect classifications across both classes.
Key insights:
Correct classifications are the majority: The diagonal cells contain 874 true negatives and 791 true positives, both of which exceed the corresponding off-diagonal error counts of 433 false positives and 487 false negatives.
False negatives slightly exceed false positives: Misclassification counts are somewhat higher for true positives predicted as False (487) than for true negatives predicted as True (433), indicating slightly more missed positive cases than incorrect positive assignments.
Negative class has higher correct count: The number of correctly classified False cases (874) is higher than the number of correctly classified True cases (791), showing stronger absolute correct classification count for the negative class in this training result.
Errors are present in both classes: Both off-diagonal cells are substantial, with 433 false positives and 487 false negatives, indicating that misclassification is distributed across both class labels rather than concentrated in only one direction.
The training confusion matrix indicates that correct predictions outnumber incorrect predictions for both classes, with 1,665 observations on the diagonal compared with 920 off the diagonal. Misclassification appears relatively balanced across error types, although false negatives are modestly more frequent than false positives. The negative class records the largest single cell count through 874 true negatives, while the positive class also shows a substantial number of correct classifications through 791 true positives.
Figures
# Test datasetresult = vm.tests.run_test("my_custom_tests.ConfusionMatrix:test_dataset", inputs={"model": vm_model, "dataset": vm_test_ds},)
Confusion Matrix Test Dataset
The Confusion Matrix test evaluates classification performance by comparing predicted labels against observed labels on the test dataset. The matrix reports counts for each prediction outcome across the two classes: 194 observations with true label False were predicted as False, 115 observations with true label False were predicted as True, 131 observations with true label True were predicted as False, and 207 observations with true label True were predicted as True. These results present the distribution of correct and incorrect classifications across both negative and positive classes.
Key insights:
Correct predictions exceed errors: The diagonal cells total 401 correct classifications (194 true negatives and 207 true positives), compared with 246 off-diagonal misclassifications (115 false positives and 131 false negatives).
True positives are the largest cell: The highest single count in the matrix is 207 for observations with true label True predicted as True, exceeding the 194 true negatives.
False negatives slightly exceed false positives: Misclassification counts are close across error types, with 131 false negatives versus 115 false positives, indicating slightly more missed positive cases than incorrectly flagged negative cases.
Observed classes are relatively balanced: The test set contains 309 observations with true label False and 338 with true label True, showing similar representation of both classes in the evaluated sample.
The confusion matrix shows that correct classifications are more frequent than misclassifications, with both true positive and true negative counts materially contributing to overall performance. Errors are present in both directions and are relatively similar in magnitude, though false negatives are modestly higher than false positives. The evaluated sample also reflects a broadly balanced class distribution, which supports direct comparison of model outcomes across the two classes.
Figures
Add parameters to custom tests
Custom tests can take parameters just like any other function. To demonstrate, let's modify the confusion_matrix function to take an additional parameter normalize that will allow you to normalize the confusion matrix:
@vm.test("my_custom_tests.ConfusionMatrix")def confusion_matrix(dataset, model, normalize=False):"""The confusion matrix is a table that is often used to describe the performance of a classification model on a set of data for which the true values are known. The confusion matrix is a 2x2 table that contains 4 values: - True Positive (TP): the number of correct positive predictions - True Negative (TN): the number of correct negative predictions - False Positive (FP): the number of incorrect positive predictions - False Negative (FN): the number of incorrect negative predictions The confusion matrix can be used to assess the holistic performance of a classification model by showing the accuracy, precision, recall, and F1 score of the model on a single figure. """ y_true = dataset.y y_pred = dataset.y_pred(model=model)if normalize: confusion_matrix = metrics.confusion_matrix(y_true, y_pred, normalize="all")else: confusion_matrix = metrics.confusion_matrix(y_true, y_pred) cm_display = metrics.ConfusionMatrixDisplay( confusion_matrix=confusion_matrix, display_labels=[False, True] ) cm_display.plot() plt.close() # close the plot to avoid displaying itreturn cm_display.figure_ # return the figure object itself
Pass parameters to custom tests
You can pass parameters to custom tests by providing a dictionary of parameters to the run_test() function.
The parameters will override any default parameters set in the custom test definition. Note that dataset and model are still passed as inputs.
Since these are VMDataset or VMModel inputs, they have a special meaning.
When declaring a dataset, model, datasets or models argument in a custom test function, the ValidMind Library will expect these get passed as inputs to run_test() or run_documentation_tests().
Re-running the confusion matrix with normalize=True and our testing dataset looks like this:
# Test dataset with normalize=Trueresult = vm.tests.run_test("my_custom_tests.ConfusionMatrix:test_dataset_normalized", inputs={"model": vm_model, "dataset": vm_test_ds}, params={"normalize": True})
Confusion Matrix Test Dataset Normalized
The ConfusionMatrix:test_dataset_normalized test evaluates classification outcomes by comparing predicted labels with true labels in a normalized confusion matrix. The result is presented as a 2×2 heatmap with rows corresponding to true labels and columns corresponding to predicted labels. The normalized cell values are 0.30 for true negatives, 0.18 for false positives, 0.20 for false negatives, and 0.32 for true positives.
Key insights:
Correct classifications exceed errors: The diagonal cells sum to 0.62, comprising 0.30 true negatives and 0.32 true positives, while the off-diagonal error cells sum to 0.38.
True positives are the largest cell: The highest normalized value is 0.32 in the true positive cell, indicating that correct positive classifications represent the single largest share of outcomes in the matrix.
False negatives exceed false positives: The false negative cell is 0.20 versus 0.18 for false positives, showing slightly more missed positive cases than incorrect positive assignments.
Prediction outcomes are fairly balanced: Predicted negatives account for 0.50 of observations (0.30 + 0.20), and predicted positives also account for 0.50 (0.18 + 0.32).
The normalized confusion matrix shows that correct classifications make up a larger share of outcomes than misclassifications, with both true positive and true negative results contributing materially to overall performance. The largest individual component is the true positive rate at 0.32, and the two error types are close in magnitude, with false negatives slightly higher than false positives. Predicted positive and predicted negative outcomes are evenly split at 0.50 each.
Parameters:
{
"normalize": true
}
Figures
Log the confusion matrix results
As we learned in 2 — Start the model development process under Documenting results > Run and log an individual tests, you can log any result to the ValidMind Platform with the .log() method of the result object, allowing you to then add the result to the documentation.
You can now do the same for the confusion matrix results:
result.log()
2026-11-20 23:02:52,523 - INFO(validmind.vm_models.result.result): Test driven block with result_id my_custom_tests.ConfusionMatrix:test_dataset_normalized does not exist in model's document
Note the output returned indicating that a test-driven block doesn't currently exist in your documentation for this particular test ID.
That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.
Using external test providers
Creating inline custom tests with a function is a great way to customize your documentation. However, sometimes you may want to reuse the same set of tests across multiple records (models) and share them with others in your organization. In this case, you can create an external custom test provider that will allow you to load custom tests from a local folder or a Git repository.
In this section you will learn how to declare a local filesystem test provider that allows loading tests from a local folder following these high level steps:
Create a folder of custom tests from existing inline tests (tests that exist in your active Jupyter Notebook)
Let's start by creating a new folder that will contain reusable custom tests from your existing inline tests.
The following code snippet will create a new my_tests directory in the current working directory if it doesn't exist:
tests_folder ="my_tests"import os# create tests folderos.makedirs(tests_folder, exist_ok=True)# remove existing testsfor f in os.listdir(tests_folder):# remove files and pycacheif f.endswith(".py") or f =="__pycache__": os.system(f"rm -rf {tests_folder}/{f}")
After running the command above, confirm that a new my_tests directory was created successfully. For example:
~/notebooks/tutorials/development/my_tests/
Save an inline test
The @vm.test decorator we used in Implementing a custom inline test above to register one-off custom tests also includes a convenience method on the function object that allows you to simply call <func_name>.save() to save the test to a Python file at a specified path.
While save() will get you started by creating the file and saving the function code with the correct name, it won't automatically include any imports, or other functions or variables, outside of the functions that are needed for the test to run. To solve this, pass in an optional imports argument ensuring necessary imports are added to the file.
The confusion_matrix test requires the following additional imports:
import matplotlib.pyplot as pltfrom sklearn import metrics
Let's pass these imports to the save() method to ensure they are included in the file with the following command:
confusion_matrix.save(# Save it to the custom tests folder we created tests_folder, imports=["import matplotlib.pyplot as plt", "from sklearn import metrics"],)
2026-11-20 23:02:53,370 - INFO(validmind.tests.decorator): Saved to /home/runner/work/documentation/documentation/site/notebooks/EXECUTED/development/my_tests/ConfusionMatrix.py!Be sure to add any necessary imports to the top of the file.
2026-11-20 23:02:53,370 - INFO(validmind.tests.decorator): This metric can be run with the ID: <test_provider_namespace>.ConfusionMatrix
# Saved from __main__.confusion_matrix
# Original Test ID: my_custom_tests.ConfusionMatrix
# New Test ID: <test_provider_namespace>.ConfusionMatrix
Now that your my_tests folder has a sample custom test, let's initialize a test provider that will tell the ValidMind Library where to find your custom tests:
ValidMind offers out-of-the-box test providers for local tests (tests in a folder) or a Github provider for tests in a Github repository.
You can also create your own test provider by creating a class that has a load_test method that takes a test ID and returns the test function matching that ID.
For most use cases, using a LocalTestProvider that allows you to load custom tests from a designated directory should be sufficient.
The most important attribute for a test provider is its namespace. This is a string that will be used to prefix test IDs in model documentation. This allows you to have multiple test providers with tests that can even share the same ID, but are distinguished by their namespace.
Let's go ahead and load the custom tests from our my_tests directory:
from validmind.tests import LocalTestProvider# initialize the test provider with the tests folder we created earliermy_test_provider = LocalTestProvider(tests_folder)vm.tests.register_test_provider( namespace="my_test_provider", test_provider=my_test_provider,)# `my_test_provider.load_test()` will be called for any test ID that starts with `my_test_provider`# e.g. `my_test_provider.ConfusionMatrix` will look for a function named `ConfusionMatrix` in `my_tests/ConfusionMatrix.py` file
Run test provider tests
Now that we've set up the test provider, we can run any test that's located in the tests folder by using the run_test() method as with any other test:
For tests that reside in a test provider directory, the test ID will be the namespace specified when registering the provider, followed by the path to the test file relative to the tests folder.
For example, the Confusion Matrix test we created earlier will have the test ID my_test_provider.ConfusionMatrix. You could organize the tests in subfolders, say classification and regression, and the test ID for the Confusion Matrix test would then be my_test_provider.classification.ConfusionMatrix.
Let's go ahead and re-run the confusion matrix test with our testing dataset by using the test ID my_test_provider.ConfusionMatrix. This should load the test from the test provider and run it as before.
result = vm.tests.run_test("my_test_provider.ConfusionMatrix", inputs={"model": vm_model, "dataset": vm_test_ds}, params={"normalize": True},)result.log()
Confusion Matrix
The ConfusionMatrix test evaluates classification performance by comparing predicted labels against true labels in a 2x2 matrix. In this result, the matrix is shown with normalization enabled, so each cell is presented as a proportion rather than a raw count. The four displayed values are 0.30 for true negatives, 0.18 for false positives, 0.20 for false negatives, and 0.32 for true positives. The diagonal cells represent correct classifications, while the off-diagonal cells represent misclassifications across the two classes.
Key insights:
Correct classifications total 0.62: The diagonal entries sum to 0.62, consisting of 0.30 true negatives and 0.32 true positives.
Misclassifications total 0.38: The off-diagonal entries sum to 0.38, with 0.18 false positives and 0.20 false negatives.
True positive cell is largest: The largest single proportion in the matrix is the true positive cell at 0.32, followed closely by the true negative cell at 0.30.
Error types are relatively balanced: False positives and false negatives are close in magnitude, at 0.18 and 0.20 respectively, indicating similar contribution from both error types.
The normalized confusion matrix shows that correct classifications account for a larger share of outcomes than misclassifications, with 0.62 on the diagonal versus 0.38 off-diagonal. Correct predictions are split relatively evenly between the negative and positive classes, and the two error types are also similar in size. Overall, the result reflects a moderately balanced classification pattern across both classes and both misclassification directions.
Parameters:
{
"normalize": true
}
Figures
2026-11-20 23:02:58,174 - INFO(validmind.vm_models.result.result): Test driven block with result_id my_test_provider.ConfusionMatrix does not exist in model's document
Again, note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for this particular test ID.
That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.
Add test results to documentation
With our custom tests run and results logged to the ValidMind Platform, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation (Learn more:Work with test results):
From the Inventory in the ValidMind Platform, go to the model you connected to earlier.
In the left sidebar that appears for your model, click Development under Documents.
Locate the Data Preparation section and click on 3.2. Model Evaluation to expand that section.
Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a + button appears, indicating that you can insert a new block.
Click + and then select Test-Driven Block under FROM LIBRARY:
Click on Custom under TEST-DRIVEN in the left sidebar.
Select the two custom ConfusionMatrix tests you logged above:
Finally, click Insert 2 Test Results to Document to add the test results to the documentation.
Confirm that the two individual results for the confusion matrix tests have been correctly inserted into section 3.2. Model Evaluation of the documentation.
In summary
In this third notebook, you learned how to:
Next steps
Finalize testing and documentation
Now that you're proficient at using the ValidMind Library to run and log tests, let's put the last pieces in place to prepare our fully documented sample model for review: 4 — Finalize testing and documentation