- model documentation
-
A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses.
Within the realm of model risk management, this documentation serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.
- validation report
-
A formal document produced after a model validation process, outlining the artifacts, assessments, and recommendations related to a specific model’s performance, appropriateness, and limitations. Provides a comprehensive review of the model’s conceptual framework, data sources and integrity, calibration methods, and performance outcomes.
Within model risk management, the validation report is crucial for ensuring transparency, demonstrating regulatory compliance, and offering actionable insights for model refinement or adjustments.
- document template
-
Lays out the structure of model documents, segmented into various sections and sub-sections, and function as test suites to help automate your development, validation, monitoring, and other risk management processes. Document templates are available for default ValidMind document types as well as custom document types.
- documentation template
-
A default ValidMind document type that serves as a standardized framework for developing and documenting models, including sections designated for model details, data descriptions, test results, and performance metrics. By outlining required documentation and recommended analyses, document templates ensure consistency and completeness across model documentation and help guide developers through a systematic development process while promoting comparability and traceability of development outcomes.
- validation report template
-
A default ValidMind document type that serves as a standardized framework for conducting and documenting model validation, including sections designated for attaching test results, evidence, or artifacts (findings). By outlining required documentation, recommended analyses, and expected validation tests, validation report templates ensure consistency and completeness across validation reports and help guide validators through a systematic review process while promoting comparability and traceability of validation outcomes.
- monitoring template, monitoring report template
-
A default ValidMind document type that serves as a standardized framework for ongoing model monitoring, including sections designated for test results, performance metrics, and drift analyses. By outlining required monitoring checks and expected routine tests, monitoring templates ensure consistency and completeness across monitoring reports and help guide model owners through a systematic monitoring process while promoting early detection of model performance degradation.
- test
-
A function contained in the library, designed to run a specific quantitative test on the dataset or model. Test results are sent to the ValidMind Platform to generate the model documentation according to the template that is associated with the documentation.
Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.
- metrics, custom metrics
-
Metrics are a subset of tests that do not have thresholds. Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered via the ValidMind Library to be used with the ValidMind Platform.
In the context of ValidMind’s Jupyter Notebooks, metrics and tests can be thought of as interchangeable concepts.
- inputs
-
Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:
- model: A single model that has been initialized in ValidMind. Refer to the
vm.init_model() function for more information.
- dataset: Single dataset that has been initialized in ValidMind. Refer to the
vm.init_dataset() function for more information.
- models: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom tests.
- datasets: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom tests. (Learn more: Run tests with multiple datasets)
- parameters
-
Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.
- outputs
-
Custom tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.
- test suite
-
A collection of tests which are run together to generate model documentation end-to-end for specific use cases.
For example, the classifier_full_suite test suite runs tests from the tabular_dataset and classifier test suites to fully document the data and model sections for binary classification model use cases.