Testing

How did ValidMind develop the tests that are currently in the library?

All the existing tests were developed using open-source Python and R libraries.

The developer framework test interface is a light wrapper that defines some utility functions to interact with different dataset and model backends in an agnostic way, and other functions to collect and post results to the ValidMind backend using a generic results schema.

Can tests be configured or customized, and can we add our own tests?

ValidMind allows tests to be configured at several levels:

• Administrators can configure which tests are required to run programmatically depending on the model use case
• You can change the thresholds and parameters for tests already available in the developer framework (for instance, changing the threshold parameter for class imbalance flag).
• In addition, ValidMind is implementing a feature that allows you to add your own tests to the developer framework. You will also be able to connect your own custom tests with the developer framework. These custom tests will be configurable and able to run programmatically, just like the rest of the developer framework libraries (roadmap item – Q3’2023).

Do you include explainability-related testing and documentation?

Our developer framework currently includes test kits to test and document global explainability features of the model, specifically, permutation feature importance and Shapley values.

In addition, ValidMind is implementing standard documentation via the developer framework for the following items and modeling techniques:

• Conceptual soundness
  • Model use case description (Q2’2023)
  • Model selection rationale (Q2’2023)
• Data evaluation
  • Data quality metrics
  • Sampling method validation
  • Population distribution (PSI)
  • Correlations & interactions
  • Data lineage (Q3’2023)
  • Feature engineering (Q3’2023)
• Model Evaluation
  • Performance & accuracy evaluation
  • Goodness of fit (Q2’2023)
  • Stability & sensitivity to perturbations (Q3’2023)
  • Model robustness & weak regions (Q3’2023)
  • Global explainability - permutation feature importance, SHAP
  • Local explainability- LIME (Q3’2023)
  • Model testing at implementation / post-production (2024)
• Model techniques
  • Time series (ARIMA, Error correction)
  • Regression (OLS, Logistic, GLM, XGBoost)
  • Decision trees (tree-based ML models)
  • Random forests
  • K-means clustering (Q2 2023)
  • NLP (2024)
  • Deep learning (2024)
  • Computer vision (2024)

Is there a use case for synthetic data on the platform?

ValidMind’s developer framework supports you bringing your own datasets, including synthetic datasets, for testing and benchmarking purposes, such as for fair lending and bias testing.

We are happy to discuss exploring specific use cases for synthetic data generation with you further.