Synchronize with Databricks

Link ​ValidMind inventory records to models, datasets, and agents in Databricks Unity Catalog for governance across your ML ecosystem and bidirectional metadata synchronization.

You can also run validation notebooks directly against Databricks-hosted data.

How does the integration with Databricks work?

Databricks Unity Catalog provides a unified governance solution for all data and AI assets across your Databricks workspaces. The integration exposes three Unity Catalog resource types:

• Models — Registered ML models in Unity Catalog’s model registry.
• Datasets — Tables and datasets managed by Unity Catalog.
• Agents — AI agents and applications built on Databricks.

After linking your inventory record to Databricks, you can run validation tests directly against data hosted in Unity Catalog.

flowchart LR
    subgraph databricks [Databricks]
        UC[Unity Catalog]
        Data[Tables/Datasets]
    end
    
    subgraph notebook [Validation Notebook]
        SDK[ValidMind SDK]
    end
    
    subgraph validmind [ValidMind Platform]
        Report[Validation Report]
        Inventory[Inventory Record]
    end
    
    UC --> Data
    Data --> SDK
    SDK --> Report
    Report --> Inventory
    Inventory -.->|linked| UC

Data flow between Databricks and ValidMind

Prerequisites

• You are logged in to ValidMind.
• You are a Customer Admin or assigned another role with sufficient permissions to configure connections.¹
• You have access to your Databricks workspace and can authorize the connection (OAuth) or create a personal access token (PAT).
• For Personal Access Token authentication only: a secret is configured for your Databricks PAT.²

¹ Manage permissions

² Manage secrets

Sync ​ValidMind records with Databricks

1. Configure Databricks connection

1. In the left sidebar, click Settings.

2. Under Integrations, select Connections.

3. Click Add Connection.

4. In the modal that opens, select Databricks.³

5. Complete the shared fields:

   • Integration Name — How other admins can identify the connection.
   • Description (optional) — The intended usage or additional details.
   • Authentication Method — Choose OAuth (user login) (recommended) or Personal Access Token.
   • Workspace URL — Your Databricks workspace URL, found in the browser address bar, such as https://adb-1234567890.azuredatabricks.net.
   • SQL Warehouse ID (optional) — The ID of your SQL Warehouse, found in SQL Warehouses settings. Required only when querying table row data via SQL.
   • Job ID (optional) — The ID of a Databricks Job that already runs your validation notebook. Required only if you want to run tests in Databricks from ​ValidMind.

6. Complete the fields for your authentication method:

   OAuth (user login)

   • OAuth Client ID (optional) — Leave blank to use the default ValidMind partner OAuth app, or enter a client ID from your own Databricks OAuth application.
   • OAuth Client Secret (optional) — Select a secret only for confidential OAuth apps. Leave blank for the ValidMind partner app or public PKCE apps.

   Click Connect with Databricks. You are redirected to Databricks to sign in and authorize access. After you authorize, you return to Connections with the new connection created. Access and refresh tokens are stored securely and are never shown in the form.

   Personal Access Token

   • Personal Access Token — Select a secret containing your Databricks Personal Access Token.⁴
   • Initial Status — Set to Operational to enable immediately or Disabled if you plan to finish setup later.

   Click Save Integration.

7. Test the connection:

   1. Hover over the Databricks connection you just created.
   2. When the menu appears, click on it and select Test Connection.

   If the test is successful, the message Connection successful is displayed.

³ 

Configure the Databricks connection

⁴ Manage secrets

NoteReconnect OAuth connections

To refresh OAuth tokens later, edit the Databricks connection and click Reconnect with Databricks. Complete the Databricks sign-in flow again. Switching an existing connection from Personal Access Token to OAuth also requires reconnecting.

2. Link records to Databricks

Once the connection is configured, you can link ​ValidMind inventory records to your Databricks Unity Catalog resources:

1. In the left sidebar, click Inventory.

2. Select a record by clicking on it or find your record by applying a filter or searching for it.⁵

3. Scroll down until you locate the Databricks connection box in the right sidebar.

4. Hover over the Databricks box.

5. When the menu appears, click on it and select Link Model.

6. In the modal that opens:

   • Select Model — Choose the Databricks model to link from the dropdown.
   • Sync Frequency — Set how often ValidMind automatically syncs data from the external system.

7. Click Link Model to complete the link.

⁵ Working with the inventory

After linking, metadata from the Unity Catalog resource syncs to ​ValidMind. You can use linked fields in custom calculated fields to surface Databricks metadata directly in your inventory views.

Run tests in Databricks

You can start a Databricks Job that runs your validation notebook from a record in ​ValidMind. ​ValidMind starts the Job through the Databricks Jobs API. Compute, secrets, and concurrency settings stay in Databricks.

When the Job starts, ​ValidMind passes two notebook parameters: model_cuid, which holds the identifier of the record you started the run from, and triggered_by, set to validmind. Your notebook can read model_cuid as a widget to log results to the right record.

Prerequisites

• A Databricks Job already runs a notebook that uses the ValidMind Library to log test results to ​ValidMind.
• The Databricks identity used by the connection has permission to run that Job.
• The record is linked to Databricks.⁶
• You are assigned to the record, or your role has permission to update records.⁷

⁶ Link records to Databricks

⁷ Manage permissions

Add the Job ID to the connection

Requires a Customer Admin or another role with sufficient permissions to configure connections.

1. In the left sidebar, click Settings.

2. Under Integrations, select Connections.

3. Click your Databricks connection.

4. In the modal that appears, click Edit Connection.

5. Under Job ID, enter the numeric ID of your Databricks Job.

6. Click Update Integration to save your changes.

Start a test run

1. In the left sidebar, click Inventory.

2. Select a record by clicking on it or find your record by applying a filter or searching for it.⁸

3. In the left sidebar that appears for your record, click Data.⁹

4. Select the Test Results tab.

5. If the record is linked to more than one Databricks resource, select the link to run under Databricks link.

6. Click Run tests now.

⁸ Working with the inventory

⁹ View record data

A confirmation that the run started in Databricks appears, with a View in Databricks link that opens the run in your Databricks workspace. While the Job is queued or running, the button shows its progress and you cannot start another run.

When the Job finishes, the date and result of the last run display below the button. If the run failed, the tab also shows the truncated notebook output. Open the run in Databricks to see the full log.

A Job that completes in Databricks does not mean that test results have been logged. Test results appear on the Test Results tab once your notebook logs them to ​ValidMind.

NoneCan’t start a run?

• If Run tests now is unavailable, the Databricks connection does not have a Job ID yet. Ask an administrator to add the Job ID to the connection.
• If the button does not display at all, check that the record is linked to Databricks and that you can update the record.
• If you are asked to reconnect Databricks, an administrator needs to edit the connection and reconnect it.
• If the Databricks identity cannot run Jobs, check the Job’s permissions in Databricks.

What’s next