How do the out-of-the-box tests developed by ValidMind work?
All the default tests are developed using open-source Python and R libraries.
The ValidMind Library1 test interface is a light wrapper that defines utility functions to agnostically interact with different dataset and record (model) backends, and contains functions to collect and post results to the ValidMind Platform2 using a generic results schema.
Dataset testing — To document and validate your dataset.
Model testing — To document and validate your record, such as a model.
End-to-end testing — To document a binary classification model and the relevant dataset end-to-end.
Can we configure, customize, or add our own tests?
Yes, ValidMind allows tests to be manipulated at several levels:
You can configure which tests are required to run programmatically depending on the record’s use case.4
You can change the thresholds and parameters for default tests already available in the library — for instance, changing the threshold parameter for the class imbalance flag.5
You can also connect your own custom tests with the ValidMind Library. These custom tests are configurable and are able to run programmatically, just like the rest of the library.6
Personalize tests further for your use case by using ValidMind’s RawData feature7 to customize the output of tests.
In addition to custom tests, you can also add use case and test-specific context for any test to enhance the LLM-generated test descriptions using the ValidMind Library.8
You use the ValidMind Library to run and log tests during development, the results of which are then inserted into your documentation within the ValidMind Platform.9 The library also automatically generates draft test descriptions for your test results — generations that can be modified for your custom use cases.10
You use the ValidMind Library to run and log tests during validation, the results of which are then inserted into your validation report within the ValidMind Platform.13 The library also automatically generates draft test descriptions for your test results — generations that can be modified for your custom use cases.14
To log tests as a validator with the ValidMind Library:
You must have the Validator role15 or another role with sufficient permissions to access records (models) for validation, to review documentation, and to work with validation reports and artifacts.
You must be the record validator, but not the record owner or record developer,16 for the record you want to log tests and update documentation for.
Do you include explainability-related testing and documentation?
Yes, ValidMind includes explainability-related testing and documentation as part of our offerings. Our approach incorporates a comprehensive suite of tests designed to evaluate interpretability and identify potential risks, ensuring transparency and reliability in outcomes.
Below is an overview of our key explainability-related tests17 with models as an example:
Features AUC — Assesses the discriminatory power of individual features in binary classification models, providing insights into how well each feature differentiates between classes. This test supports explainability by isolating the contribution of each feature to the classification task.
Feature Importance — Generates feature importance scores to identify and compare impactful features across different models and datasets. By highlighting the relative significance of features, this test clarifies how inputs influence model predictions.
Overfit Diagnosis — Detects potential overfitting by comparing performance between training and testing sets for specific feature segments, highlighting areas of significant deviation. This test aids explainability by revealing where model behavior is inconsistent, offering insights into its generalization capability.
Permutation Feature Importance — Measures feature significance by analyzing the impact of randomly rearranging feature values on model performance. This test quantifies the dependency of model performance on each feature, making it clear which inputs drive the predictions.
SHAP Global Importance — Uses SHAP (SHapley Additive exPlanations) values to assign global importance to features, offering a clear explanation of model outcomes and supporting risk identification. SHAP values provide a mathematically sound attribution of model predictions to specific features, enhancing interpretability.
Weakspots Diagnosis — Identifies and visualizes regions of suboptimal model performance across the feature space, highlighting areas that may require further attention. This test explains where and why the model struggles by connecting poor performance to specific feature regions.
NoneWhen logged for documentation, each test automatically generates a comprehensive report as soon as it is executed.
ValidMind leverages generative AI to produce tailored, detailed summaries that include the test description, key insights, and a concise summary of results.
This automated documentation ensures that every test outcome is transparently recorded, clearly communicated, and immediately actionable.
Does ValidMind support using synthetic datasets?
The ValidMind Library supports you bringing your own datasets, including synthetic datasets, for testing and benchmarking purposes such as for fair lending and bias testing.18
If you are unable to share your real-world data with us, ValidMind is happy to work with you to generate custom synthetic datasets based on characteristics of your data, or provide scripts to assist with synthetic dataset generation if details cannot be shared.
Does ValidMind support monitoring records after deployment?
Yes, ValidMind offers ongoing monitoring support to help you regularly assess a record’s accuracy, stability, and robustness to ensure it remains reliable after deployment:
You can enable monitoring for both new and existing records.19
You use the ValidMind Library to automatically populate the monitoring template for your record with data, providing a comprehensive view of your record’s performance over time.
You then access and examine these results within the ValidMind Platform, allowing you to identify any deviations from expected performance and take corrective actions as needed.20
Once generated via the ValidMind Library, view and add metrics over time to your ongoing monitoring reports in the ValidMind Platform.21