ValidMind Library

Published

May 26, 2026

The ValidMind Library streamlines development and validation by automating testing. Run tests, log those test results to the ValidMind Platform, and have fully supported drafts of documentation or reporting ready for you to fine-tune.

What is the ValidMind Library?

The ValidMind Library provides a rich collection of documentation tools and test suites, from documenting descriptions of your datasets to validation testing your records (such as models) for weak spots and overfit areas.

ValidMind offers two primary methods for automating documentation:

Generate documentation¹ — Through automation, the library extracts metadata from associated datasets and records for you and generates documentation based on a template. You can also add more documentation and tests manually using the documentation editing capabilities in the ValidMind Platform.
Run validation tests² — The library provides a suite of validation tests for common financial services use cases. For cases where these tests do not cover everything you need, you can also extend existing test suites with your own proprietary tests or testing providers.

¹ ValidMind for development

² ValidMind for validation

The ValidMind Library is designed to be agnostic. For example, if you have a model built with Python, the library provides all the standard functionality you may need without requiring you to rewrite any functions.

Key ValidMind concepts

record: A tool tracked in the ValidMind Platform inventory, such as a model. Records include traditional statistical models, legacy systems, artificial intelligence/machine learning models, large language models (LLMs), agentic AI systems, and other documentable items that benefit from oversight, testing, and lifecycle management.

model: SR 26-2 (which supersedes SR 11-7) defines a model as a "complex quantitative method, system, or approach that applies statistical, economic, or financial theories to process input data into quantitative estimates." Simple arithmetic, deterministic rule-based processes, or software without statistical, economic, or financial theories underpinning their design or use are generally outside SR 26-2’s definition of a model.

Within ValidMind, a model is a type of record tracked in the inventory.

documentation, model documentation: A structured and detailed record pertaining to a record (such as a model), encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses.

validation report: A formal document produced after a model validation process, outlining the artifacts, assessments, and recommendations related to a specific model's performance, appropriateness, and limitations. Provides a comprehensive review of the model's conceptual framework, data sources and integrity, calibration methods, and performance outcomes.

ongoing monitoring, ongoing monitoring report, ongoing monitoring plan, monitoring plan: A comprehensive and structured periodic report assessing the record's performance and compliance over time, ensuring it remains valid under changing conditions. Monitoring includes key elements such as data sources, inputs, performance metrics, and periodic evaluations, ensuring transparency and visibility of the record's performance in the production environment.

document template: Lays out the structure of documents, segmented into various sections and sub-sections, and functions as a test suite to help automate your development, validation, monitoring, and other risk management processes. Document templates are available for default ValidMind document types as well as custom document types.
documentation template: A default ValidMind document type that serves as a standardized framework for developing and documenting records (such as models), including sections designated for record details, data descriptions, test results, and performance metrics. By outlining required documentation and recommended analyses, documentation templates ensure consistency and completeness across documentation and help guide developers through a systematic development process while promoting comparability and traceability of development outcomes.
validation report template: A default ValidMind document type that serves as a standardized framework for conducting and documenting validation, including sections designated for attaching test results, evidence, or artifacts (findings). By outlining required documentation, recommended analyses, and expected validation tests, validation report templates ensure consistency and completeness across validation reports and help guide validators through a systematic review process while promoting comparability and traceability of validation outcomes.
monitoring template, monitoring report template: A default ValidMind document type that serves as a standardized framework for ongoing monitoring, including sections designated for test results, performance metrics, and drift analyses. By outlining required monitoring checks and expected routine tests, monitoring templates ensure consistency and completeness across monitoring reports and help guide owners through a systematic monitoring process while promoting early detection of performance degradation.

test: A function contained in the library, designed to run a specific quantitative test on the dataset or record (such as a model). Test results are logged to the ValidMind Platform, where they are attached to documents.

Tests are the building blocks of ValidMind, used to evaluate and document records and datasets, and can be run individually or as part of a suite defined by your templates.

test suite: A collection of tests designed to run together to automate and generate documentation end-to-end for specific use cases. (Learn more: test_suites)

metrics, custom metrics: Metrics are a subset of tests that do not have thresholds. Custom metrics are functions that you define to evaluate your record (such as a model) or dataset. These functions can be registered via the ValidMind Library to be used with the ValidMind Platform.

In the context of ValidMind's Jupyter Notebooks, metrics and tests can be thought of as interchangeable concepts.

inputs: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:

model: A single record (such as a model) that has been initialized in ValidMind. Despite the naming convention, model objects can be any type of record you want to test, document, validate, or monitor with ValidMind. Refer to the vm.init_model() function for more information.
dataset: A single dataset that has been initialized in ValidMind. Refer to the vm.init_dataset() function for more information.
models: A list of ValidMind records — usually this is used when you want to compare multiple records in your custom tests.
datasets: A list of ValidMind datasets — usually this is used when you want to compare multiple datasets in your custom tests. (Learn more: Run tests with multiple datasets)

parameters: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.

outputs: Custom tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.

Quickstart

After you sign up for ValidMind to get access, try our Jupyter Notebook quickstarts for documentation or validation:

Quickstart for documentation

Learn the basics of using ValidMind to document records as part of a development workflow. Set up the ValidMind Library in your environment, and generate a draft of documentation using ValidMind tests for a binary classification model.

Quickstart for validation

Learn the basics of using ValidMind to validate records as part of a validation workflow. Set up the ValidMind Library in your environment, and generate a draft of a validation report using ValidMind tests for a binary classification model.

End-to-end tutorials

Learn how to use the ValidMind Library with our high-level Jupyter Notebook courses covering usage of ValidMind for specific roles or concepts.

ValidMind for development

Learn how to use ValidMind for your end-to-end documentation process based on common development scenarios with our ValidMind for development series of four introductory notebooks:

1 — Set up the ValidMind Library

Get to know ValidMind by setting up the ValidMind Library in your own environment, and registering a sample binary classification model in the ValidMind Platform for use with this series of notebooks.

2 — Start the development process

Learn to run and log tests with a variety of methods and in different situations with the ValidMind Library, then add the results or evidence to your documentation for the sample model you registered.

3 — Integrate custom tests

After you become familiar with the basics of the ValidMind Library, learn how to supplement ValidMind tests with your own and include them as additional evidence in your documentation.

4 — Finalize testing and documentation

Wrap up by learning how to ensure that custom tests are included in your documentation template. By the end of this series, you will have a fully documented sample model ready for review.

ValidMind for validation

Learn how to use ValidMind for your end-to-end validation process based on common scenarios with our ValidMind for validation series of four introductory notebooks:

1 — Set up the ValidMind Library for validation

Get to know ValidMind by setting up the ValidMind Library in your own environment, and gaining access as a validator to a sample model in the ValidMind Platform for use with this series of notebooks.

2 — Start the validation process

Independently verify the data quality tests performed on datasets used to train the dummy champion using tests from the ValidMind Library, then add the results or evidence to your validation report.

3 — Developing a potential challenger

After you become familiar with the basics of the ValidMind Library, use it to develop a potential challenger and run thorough comparison tests, such as performance, diagnostic, and feature importance tests.

4 — Finalize validation and reporting

Wrap up by learning how to include custom tests and verifying that all tests conducted during development were run and reported accurately. By the end of this series, you will have a validation report complete with artifacts ready for review.

Learn how to use the ValidMind Library

Learn how to use the comprehensive out-of-the-box tests and test suites, and other features in the ValidMind Library that make it easy for you to automate building, documenting, validating your records and more.

How to run tests and test suites

ValidMind provides many built-in tests and test suites, which help you produce documentation during stages of your risk management lifecycle, where you need to ensure that your work satisfies regulatory and risk management requirements.

How to use ValidMind Library features

Browse our range of Jupyter Notebooks demonstrating how to use the core features of the ValidMind Library. Use these how-to notebooks to get familiar with the library's capabilities and apply them to your own use cases.

Try code samples by use case

Try our Jupyter Notebook code samples that showcase the capabilities of the ValidMind Library and cover a variety of sample use cases.

Code samples by use case

Examples that you can build on and adapt for your own usage include:

Work with documentation

After you have tried out the ValidMind Library, continue working with your documentation in the ValidMind Platform:³

³ Working with documentation

Work with test results

Once generated via the ValidMind Library, view and add the test results to your documentation in the ValidMind Platform.

Work with content blocks

Make edits to your documents by adding or removing content blocks directly in the online editor.