setup

Databricks

Purpose

Syncs one or more Databricks workspaces into Subimage, including workspace identity, compute, workflows, SQL assets, networking, apps, model serving, and the Unity Catalog data plane. Workspaces on AWS, Azure, and GCP are all supported, and one module can sync workspaces from any mix of them.

tip

Secret fields below accept either an AWS Secrets Manager ARN or a value pasted directly into Subimage's managed vault. See Secrets for details.

Workspaces

Add an entry under workspaces for each workspace you want Subimage to sync. Paste the workspace URL from your browser's address bar, for example https://adb-1234567890123456.7.azuredatabricks.net or https://dbc-xxxx.cloud.databricks.com. Subimage keeps only the host, so a URL that still carries a path or a ?o= workspace selector works too.

Each workspace signs in with one of:

  • Default: the default credentials described below. Use this when one identity has access to several workspaces, for example a service principal that is assigned to all of them.
  • Access token: a personal access token created in that workspace.
  • Service principal: an OAuth M2M client ID and client secret for that workspace.

A workspace that fails to sync does not stop the others. The sync is marked degraded and its logs name the workspace that failed.

Default credentials

The default credentials are used by every workspace set to Default. They are optional when every workspace has its own. Choose one method:

Field Secret? Description
databricks_token Yes Workspace-admin personal access token with all APIs access
databricks_client_id No Workspace-admin service principal application ID
databricks_client_secret Yes OAuth client secret for that service principal

Set either databricks_token, or both databricks_client_id and databricks_client_secret. OAuth M2M is recommended for persistent deployments because it uses short-lived access tokens instead of a long-lived personal credential.

Steps

  1. In each Databricks workspace, create the credential:
    • PAT: go to Settings, Developer, Access tokens, and Generate new token. Select Other APIs and all APIs (not recommended). Use a short lifetime and revoke the token after testing.
    • OAuth M2M: create a service principal, then generate an OAuth secret for it (Settings, Identity and access, Service principals). To reuse one service principal across workspaces, create it in the account console and assign it to each workspace.
  2. Grant the user or service principal the workspace admin role in every workspace it will sync. Full ingestion requires workspace admin access to enumerate SCIM identities and the token management API.
  3. In Subimage, open the Databricks module and add each workspace under workspaces. Leave a workspace on Default to use the default credentials, or give it its own access token or service principal.
  4. If any workspace uses Default, fill in the default credentials with a token, or a client ID and client secret.
  5. For each secret, choose Secret to paste the credential into Subimage's managed vault, or choose AWS ARN to reference a plaintext secret in your AWS Secrets Manager.

Subimage's inventory requests use read-only GET operations. The credential itself is not read-only: Databricks API scopes authorize families of operations that can include both reads and mutations. Cartography requests the OAuth all-apis scope automatically, and full PAT-based ingestion requires the equivalent all APIs selection. See the Databricks API scope reference.

Removing a workspace from the list stops syncing it. The data it already synced stays in the graph.

Subimage currently configures workspace-level ingestion only. Cartography's separate Databricks account-level configuration is not exposed through the Subimage module yet.