Integrations and support

Published

January 6, 2025

Which languages, libraries, and environments do you support?

  • The ValidMind Library1 is designed to be platform-agnostic and compatible with most popular open-source programming languages and model development environments in Python and R,2 from XGBoost to more sophisticated libraries such as Pytorch and TensorFlow — and many more.
  • We directly support Matplotlib3 and Plotly4 plotting libraries for visual representations, and you’re able to return images from other libraries as bytes-like objects.5
Currently, we support Python ≧3.8 and <3.11 and the most popular AI/ML and data science libraries.

Support for commercial and closed-source programming languages such as SAS and Matlab depends on specific deployment details and commercial agreements with customers.

What test ingestion or modeling techniques are supported?

  • ValidMind supports ingesting test results from your training and evaluation pipeline, such as using batch prediction or online prediction mechanisms.6
  • We are also offer standard documentation via the library for additional modeling techniques.7

Do you support including images in model documentation?

Yes, as long as you can produce the image with Python or open the image from a file, you can include it in your documentation with ValidMind:8

  • If you want to log an image as a test result, you can do so by passing the path to the image as a parameter to the custom test and then opening the file in the test function.
  • If you are using a plotting library that isn’t directly supported by ValidMind, you can still return the image directly as a bytes-like object.

What large language model (LLM) features are offered?

ValidMind offers several specialized features that use large language models (LLMs) to streamline model risk management and ensure regulatory compliance:

  • Test interpretation — Interprets results from tests run within ValidMind.
  • Qualitative checks — Leverages metadata from the model inventory, test outcomes, and additional data provided to create qualitative sections within model documentation.
  • Risk assessment — Using data from test results, generates a tailored risk assessment for each section of model documentation.
  • Document checker — Reviews model development documentation to ensure it aligns with relevant regulatory requirements.

Do you include explainability-related testing and documentation?

Yes, ValidMind includes explainability-related testing and documentation as part of our offerings. Our approach incorporates a comprehensive suite of tests designed to evaluate model interpretability and identify potential risks, ensuring transparency and reliability in model outcomes.

Below is an overview of our key explainability-related tests:

  • Features AUC9 — Assesses the discriminatory power of individual features in binary classification models, providing insights into how well each feature differentiates between classes. This test supports explainability by isolating the contribution of each feature to the classification task.
  • Feature Importance10 — Generates feature importance scores to identify and compare impactful features across different models and datasets. By highlighting the relative significance of features, this test clarifies how inputs influence model predictions.
  • Overfit Diagnosis11 — Detects potential overfitting by comparing performance between training and testing sets for specific feature segments, highlighting areas of significant deviation. This test aids explainability by revealing where model behavior is inconsistent, offering insights into its generalization capability.
  • Permutation Feature Importance12 — Measures feature significance by analyzing the impact of randomly rearranging feature values on model performance. This test quantifies the dependency of model performance on each feature, making it clear which inputs drive the predictions.
  • SHAP Global Importance13 — Uses SHAP (SHapley Additive exPlanations) values to assign global importance to features, offering a clear explanation of model outcomes and supporting risk identification. SHAP values provide a mathematically sound attribution of model predictions to specific features, enhancing interpretability.
  • Weakspots Diagnosis14 — Identifies and visualizes regions of suboptimal model performance across the feature space, highlighting areas that may require further attention. This test explains where and why the model struggles by connecting poor performance to specific feature regions.
When logged for documentation, each test automatically generates a comprehensive report as soon as it is executed.
  • ValidMind leverages generative AI to produce tailored, detailed summaries that include the test description, key insights, and a concise summary of results.
  • This automated documentation ensures that every test outcome is transparently recorded, clearly communicated, and immediately actionable.

What deployment options are supported by ValidMind?

Our deployment options provide a balance of control and cost-efficiency while integrating seamlessly with your infrastructure. For added flexibility, you can deploy on Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure.

We offer two deployment models:

  • Multi-tenant cloud — Multiple organizations (tenants) share infrastructure while keeping data isolated, providing cost-efficiency and scalability. For secure connectivity, a private link can be established to ensure traffic stays within your network, avoiding the public internet.

  • Virtual Private ValidMind (VPV) — A single-tenant architecture where one organization uses dedicated infrastructure, offering greater control, customization, and enhanced security. This option is ideal for high-compliance needs, and secure connectivity can also be established via a private link.

Learn more