Run dataset based tests
run_test
function to run built-in or custom tests that take any dataset or model as input. These tests generate outputs in the form of text, tables, and images that get populated in model documentation.
EU AI Act Compliance — Read our original regulation brief on how the EU AI Act aims to balance innovation with safety and accountability, setting standards for responsible AI use
December 5, 2024
ValidMind provides many built-in tests and test suites, which help you produce documentation during stages of the model development lifecycle where you need to validate that your work satisfies MRM (model risk management) requirements.
Within the realm of model risk management, this documentation serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.
Within model risk management, the validation report is crucial for ensuring transparency, demonstrating regulatory compliance, and offering actionable insights for model refinement or adjustments.
ValidMind templates come with pre-defined sections, similar to test placeholders, including boilerplates and spaces designated for documentation and test results. When rendered, produces a document that model developers can use for model validation.
Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.
In the context of ValidMind’s Jupyter Notebooks, metrics and tests can be thought of as interchangeable concepts.
vm.init_model()
. See the Model Documentation or the for more information.vm.init_dataset()
. See the Dataset Documentation for more information.For example, the classifier_full_suite
test suite runs tests from the tabular_dataset
and classifier
test suites to fully document the data and model sections for binary classification model use cases.
Start by running a pre-made test, then modify it, and finally create your own test:
Next, find available tests and test suites using the library or the interactive test sandbox:
Building on previous sections, add your own test provider, set up datasets, run tests on individual sections in your model documentation, and more:
Need more? Try some of the advanced features provided by the library:
While you have the flexibility to decide when to use which ValidMind tests, we have identified a few typical scenarios with their own characteristics and needs:
To document and validate your dataset:
tabular_dataset
test suite.time_series_dataset
test suite.To document and validate your model:
classifier
test suite.timeseries
test suite.To document a binary classification model and the relevant dataset end-to-end:
Use the classifier_full_suite
test suite.
Absolutely! ValidMind supports custom tests that you develop yourself or that are provided by third-party test libraries, also referred to as test providers. We provide instructions with code examples that you can adapt: