October 22, 2024

Release highlights

We’ve added more flexible features to enable better model risk management, including support for ongoing monitoring plans, the ability to archive and delete models, the ability to insert metric over time blocks into your documentation, and more.

Support for ongoing monitoring

Monitoring is a critical component of model risk management, as emphasized in regulations such as SR 11-7, SS1/23, and E-24. With this release of ValidMind, we officially support ongoing monitoring. You can enable this feature for both existing and new models.

Ongoing monitoring of a model

Scenarios where ongoing monitoring is warranted:

• Pre-approval monitoring of new models
• Monitoring during significant updates
• Post-production monitoring

The monitoring template for your model automatically populates with data as your code runs, providing a comprehensive view of performance over time. You can access these results in the ValidMind Platform UI, identify deviations, and take corrective actions as needed.

To start uploading ongoing monitoring results for a model to ValidMind, you enable monitoring in your code and then select a monitoring template. Our user guide can walk you through the process of setting up ongoing monitoring step-by-step.

Ongoing monitoring

We’ve also added a new notebook that demonstrates how to log ongoing monitoring results for a model.

This notebook provides a hands-on example of setting up and conducting monitoring tests, helping you ensure your models are performing consistently over time.

Try notebook on JupyterHub

Archive and delete models

When you need to decommission models that you no longer need, you can now archive and then delete them. This feature helps keep your model inventory accurate and up to date with your organization’s current resources.

You now have new stages for inventory models, including ACTIVE, ARCHIVED, and DELETED, which are shown as a new column in the model inventory and as field in the model overview.

Archive and delete models

• Archived models are read-only and do not contribute to statistics or reports.
• Models must be inactive and archived before they can be deleted.
• Model archiving and deletion are recorded in the activity feed.
• An administrator can restore archived models if needed.

Support for bias and fairness tests

The latest update to the ValidMind Developer Framework introduces new features and improvements that enhance bias and fairness testing.

To let you evaluate model bias and fairness effectively across protected classes, four new metrics are available:

• ProtectedClassesCombination
• ProtectedClassesDescription
• ProtectedClassesDisparity
• ProtectedClassesThresholdOptimizer

There is also an additional parameter class_of_interest in the SHAPGlobalImportance tool for better SHAP value selection.

A new Jupyter Notebook guides you through building a credit risk model with integrated bias and fairness analysis.

Try notebook on JupyterHub

Accompanying this, a dataset called lending_club_biased.csv.gz and its processing script have been added for testing purposes, alongside an update to the dataset module initializer.

New Metric Over Time content block

You now can add Metrics Over Time blocks to model documentation or as part of your ongoing monitoring of a model, in addition to the previously available Text and Test-Driven blocks.

Work with metrics over time

• This new block type which retrieves values for a particular unit metric key over a specified time range.
• A line graph and a table of underlying values are shown as part of your model documentation.

Adding a Metric Over Time block to model documentation

Selecting one of the available blocks

Metric over time graph in model documentation

Filtering columns when running tests

When running a test, you now have the option to filter specific columns in your dataset.

• This feature helps simplify and analyze your results more efficiently.
• For example, you can use this enhancement to focus on relevant data.

Tests can run on a subset of columns in a dataset without creating a new dataset with columns removed.

The following example shows how to run the DatasetDescription test with only a single column from a dataset:

from validmind.tests import run_test

result = run_test(
    test_id="validmind.data_validation.DatasetDescription",
    inputs={
        "dataset": {
            "input_id": "my_dataset",
            "columns": ["col1"]
        }
    },
)

Notice how the dataset input is set to a dictionary whose key input_id maps to a particular input dataset. And whose columns key sets the subset of columns from the original dataset to pass to the DatasetDescription test.

A new Jupyter Notebook demonstrates how to selectively include columns in your analysis based on your specific criteria.

Try notebook on JupyterHub

Enhancements

Email verification improvements

This update enhances the user experience by introducing a new email verification error page.

Register with ValidMind

If you need to verify your email address but have been unable to do so, this page allows you to resend the verification email and navigate to the dashboard to access ValidMind once verification is complete.

Enhanced custom field management with Python support

This update enhances the custom field functionality, making it possible to define and test custom Python code to calculate an output.

The results of your code execution are displayed clearly, and you can test your custom code using a new testing feature directly within the interface.

Manage model inventory fields

• The custom field editor has also been upgraded, now including a code editor and testing area, along with a drag-and-drop interface for selecting fields.
• The overall user experience has been improved with better error handling and feedback.
• Additionally, managing text fields is now more intuitive with expandable text displays, making it simpler to navigate and modify custom fields.

Watch the demo:

Define attachment fields for your model inventory

You can now define attachments as model inventory fields for supporting documentation.

Inventory field types

• Each attachment field can be unique.
• This enhancement allows you to attach different sets of files for a model.

The inventory model page displays attachments when the attachment field is defined:

Sample attachments

Manage attachments in findings

You can now manage attachments within findings.

Manage supporting documentation

• This feature allows you to upload, delete, and view attachments directly in the findings section.
• Permissions have been enhanced, with a new Manage Attachment permission for roles, ensuring that only authorized users can handle attachment-related tasks.
• This update makes it easier to organize and manage any files associated with your findings.

New currency number field option

We added a new Currency type for Number fields which provides formatting for currencies and large number abbreviations. This field allows you to record pricing data within a model, for example.

You can configure the currency type as well as the precision (decimals) when creating the field:

Adding the currency number type

Currency type in model documentation

Add or edit inventory fields

Delete findings

You can now delete findings as a validator.

Delete model findings

• When deleting a finding, a confirmation dialog appears to ensure that you want to proceed.
• If any errors occur during the process or if the finding is not found, you will be redirected to the home page with a detailed message.

Rename organization roles

As an organization admin, you can now rename any system role, except for the Customer Admin role, and edit the role description.

Rename existing roles

• This update makes it easier to manage roles within your organization by allowing you to edit the name and description of roles directly on the settings page.
• Custom fields for role names and descriptions can now be modified seamlessly, improving the overall experience of managing organizational roles.

Better filtering for model activity

You can now filter model activity by type, allowing you to easily view specific actions.

The recent activity section on the model details page also provides a clearer overview of updates. This enhancement simplifies navigating through activity logs, making it easier to find relevant information based on the selected filters.

The new activity type filters are available on:

• The Model Activity page
• The Recent Activity section on model details pages

Filter activity

Rich text editor for validation guidelines

You can now use rich text editing in the validation guideline description dialog, available under Settings > Risk Areas & Validation Guidelines.

• This enhancement allows for better formatting and presentation of your guidelines, improving how content is managed and displayed.
• The rich text editor provides a more flexible way to create and edit guideline descriptions, making the process smoother and more user-friendly.

Edit guidelines

Rich text guidelines editor

Bug fixes

Scrollbar display on Windows

Fixed an issue where scrollbars were unnecessarily displayed in the layout sidebar on browsers running on Windows.

• This update improves the user interface by ensuring scrollbars are hidden where they are not needed, resulting in a cleaner and more consistent browsing experience across different components and pages.
• Scrolling behavior has been optimized to make the layout more visually appealing and user-friendly.

Documentation

Updated training content

Our training modules now feature ValidMind’s new color scheme, making them more visually appealing and easier to use.

• Sections are linked directly from the training overview for quicker navigation.

• These changes aim to improve the overall look and usability of our training materials.

Additionally, we’ve introduced a new, easy-to-remember URL for accessing our training content.

docs.validmind.ai/training

Validating models 101 videos

We created a series of short videos to help you better understand the model validation process. These videos introduce you to the steps you need to follow when validating models on our platform.

Adding tests video

We added a short FAQ-style video to show you how to find the tests ValidMind provides and add them to your own model documentation.

New deployment options overview

We expanded our documentation on deployment options for ValidMind.

Deployment options

You can now find detailed information on both multi-tenant cloud and single-tenant options, giving you more clarity on how to deploy ValidMind based on your needs.

Find our open-source on GitHub

To make it easier to find our open-source software on GitHub, we added a link to the code samples page.

Code samples

How to upgrade

ValidMind Platform UI

To access the latest version of the ValidMind Platform UI,¹ hard refresh your browser tab:

¹ Log in to ValidMind

• Windows: Ctrl + Shift + R OR Ctrl + F5
• MacOS: ⌘ Cmd + Shift + R OR hold down ⌘ Cmd and click the Reload button

ValidMind Developer Framework

To upgrade to the latest version of the ValidMind Developer Framework:²

² Get started with the ValidMind Developer Framework

1. In your Jupyter Notebook:

   • Using JupyterHub:³ Hard refresh your browser tab.
   • In your own developer environment:⁴ Restart your notebook.

2. Locate the %pip install -q validmind cell and replace the contents with:

   %pip install --upgrade validmind

3. Run the edited code cell.

³ Try it with JupyterHub

⁴ Try it in your own developer environment

You may need to restart your kernel after running the upgrade package for changes to be applied.