Connecting a Databricks account

Overview

To onboard your Databricks environment to DigiUsher, create a service principal and grant the relevant permissions as described below, connecting to a SQL warehouse of your choice. This document describes exactly what permissions are requested, why each is needed, and what credentials to provide.

Summary of Access Required

DigiUsher cannot create, modify, or delete any of your Databricks resources.

Prerequisites

Information to Gather

Roles Required by the Person Performing Setup

Option A: Terraform (Recommended)

The DigiUsher Terraform configuration automates the entire setup — it creates the service principal, generates an OAuth secret, grants Can use on the SQL warehouse, and applies all the Unity Catalog grants.

Prerequisites

• Terraform — version 1.3 or higher
• Python 3 with databricks-sql-connector, databricks-sdk, pandas, pyarrow (only needed to run the verification script)
• A Databricks personal access token with All APIs scope, used once to bootstrap the setup (not stored after terraform apply completes)
• Account Admin and Workspace Admin rights

Create a Personal Access Token

1. Go to your workspace → top-right user icon → Settings → Developer → Access tokens.
2. Click Generate new token.
3. Set a name (e.g. terraform-setup-digiusher) and a lifetime.
4. Under Scope, select All APIs.
5. Copy the token immediately — it won't be shown again.

Find Your Configuration Values

You need three values before running Terraform:

Deployment

git clone https://github.com/digiusher/digiusher-iac.git
cd digiusher-iac/databricks

Create terraform.tfvars with your values:

databricks_account_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
workspace_host        = "your-workspace.cloud.databricks.com"
warehouse_id          = "abc123def456"

Export your personal access token, then apply:

export TF_VAR_databricks_token="XXXXXXXXXXXXXXXXXXXX"

terraform init
terraform plan
terraform apply

Retrieve the Credentials

After apply completes, the four values DigiUsher needs are available as outputs:

echo "DATABRICKS_HOST=$(terraform output -raw workspace_hostname)"
echo "DATABRICKS_HTTP_PATH=$(terraform output -raw http_path)"
echo "DATABRICKS_CLIENT_ID=$(terraform output -raw client_id)"
echo "DATABRICKS_CLIENT_SECRET=$(terraform output -raw client_secret)"

These map to the Server Hostname, HTTP Path, Client ID, and Client Secret fields in DigiUsher (see Connect in DigiUsher below).

Verify (Optional)

The repository includes a verification script that confirms authentication, system catalog access, and billing data availability:

export DATABRICKS_HOST="$(terraform output -raw workspace_hostname)"
export DATABRICKS_HTTP_PATH="$(terraform output -raw http_path)"
export DATABRICKS_CLIENT_ID="$(terraform output -raw client_id)"
export DATABRICKS_CLIENT_SECRET="$(terraform output -raw client_secret)"

python3 verify_exports.py

See the digiusher-iac README for full Terraform documentation and troubleshooting.

Option B: Manual Setup

GRANT USE CATALOG ON CATALOG system TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.billing TO `<client-id>`;
GRANT SELECT ON TABLE system.billing.usage TO `<client-id>`;
GRANT SELECT ON TABLE system.billing.list_prices TO `<client-id>`;
GRANT SELECT ON TABLE system.billing.account_prices TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.access TO `<client-id>`;
GRANT SELECT ON TABLE system.access.workspaces_latest TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.compute TO `<client-id>`;
GRANT SELECT ON TABLE system.compute.clusters TO `<client-id>`;
GRANT SELECT ON TABLE system.compute.warehouses TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.lakeflow TO `<client-id>`;
GRANT SELECT ON TABLE system.lakeflow.pipelines TO `<client-id>`;

Connect in DigiUsher

After completing either the Terraform or Manual setup, enter the following into DigiUsher to complete the connection:

Verification Checklist

• Service principal created in the Databricks account console
• OAuth client secret generated and both Client ID and Secret saved securely
• Service principal assigned to the target workspace
• Service principal granted Can use on the SQL warehouse
• All GRANT statements executed successfully in the SQL Editor — system.billing, system.access, system.compute, system.lakeflow
• Workspace hostname, HTTP path, Client ID, and Client Secret entered in DigiUsher
• Verification probe passed — DigiUsher confirms access to system.billing.usage and system.billing.list_prices and returns an account_id
• *.digiusher.com allowlisted in your network/firewall (if applicable)
• digiusher.com allowlisted as an approved sender domain in your email security gateway (if applicable)

Security

What DigiUsher CAN Access (Read-Only)

• system.billing.usage — DBU and credit consumption records for all workspaces in the account
• system.billing.list_prices — Databricks public list prices for SKU cost calculation
• system.billing.account_prices — contracted rates, if your account is enrolled in the Account Prices Preview (AWS/GCP only)
• system.access.workspaces_latest — workspace name metadata used to enrich billing records
• system.compute.clusters and system.compute.warehouses — resource names used to enrich billing records
• system.lakeflow.pipelines — pipeline names used to enrich billing records

What DigiUsher CANNOT Do

• Access any table outside the schemas explicitly granted above
• Read notebooks, job definitions, query results, or any business data
• Access data in any workspace catalog (e.g. hive_metastore, user-created catalogs)
• Create, modify, or delete any object in Databricks
• View or modify billing settings or account configuration
• Make purchases or changes to your account
• Access any underlying cloud infrastructure (S3, ADLS, GCS) directly

Monitoring

Monitor service principal activity in the Databricks workspace under Settings → Identity and access → Service Principals → your service principal. For a full audit trail, query system.access.audit if your workspace has that schema enabled — filter on user_identity.email = '<client-id>' to see all actions taken by the DigiUsher service principal.

Credential Rotation

Terraform: Run terraform apply -replace="databricks_service_principal_secret.digiusher_focus_reader" to generate a new secret, then update the Client Secret in DigiUsher with the new terraform output -raw client_secret.

Manual:

1. In the account console, go to Service Principals → select the DigiUsher service principal → Secrets tab.
2. Click Generate secret to create a new client secret.
3. Update the Client Secret in DigiUsher immediately.
4. Delete the old secret from the Secrets tab.

Revocation

Terraform: Run terraform destroy — this removes the service principal, its secret, the warehouse permission, and all Unity Catalog grants, instantly invalidating access.

Manual: Delete the service principal from the account console under Service Principals. This immediately invalidates the OAuth secret and removes all associated permissions from the workspace. Alternatively, delete only the secret under the SP's Secrets tab to pause access without removing the principal — useful if you intend to re-connect later.

Troubleshooting

Permission denied on system.billing.usage or other system tables

• Confirm all GRANT statements completed without errors in the SQL Editor. Re-run the full grant block — grants are idempotent and safe to repeat.
• Verify the user who ran the grants was both an Account Admin and a Metastore Admin at the time. Grants run by a workspace-only admin silently fail or error.
• Confirm the service principal's Client ID (Application ID UUID) was used as the grantee, not its display name.

"User is not an account admin" during setup

• Go to accounts.cloud.databricks.com → User Management → Users → click the setup user's email → toggle Account admin on.
• This role is separate from workspace admin group membership — being in the workspace admins group is not sufficient.

Verification probe passes but BilledCost is zero

• BilledCost is computed as usage_quantity × account_unit_price. If list_prices has no rows matching your account's region and SKUs yet, the price join returns NULL and cost coalesces to 0.
• Confirm system.billing.list_prices is populated: run SELECT COUNT(*) FROM system.billing.list_prices in the SQL Editor. Newly provisioned trial accounts may see an empty or sparse price table until Databricks backfills it for your region — this resolves within 24–48 hours.

No data in system.billing.usage

• The table only contains records from the date the account was provisioned. A brand-new account will have one or two rows at most (warehouse startup storage). This is expected and not an error — data accumulates as the account is used.
• Confirm the date range: SELECT MIN(usage_date), MAX(usage_date) FROM system.billing.usage.

Connection hangs with no output

• Verify DATABRICKS_SERVER_HOSTNAME does not include https://. The SDK adds the scheme automatically — a double https://https:// prefix causes a silent 5-minute timeout.
• Confirm the SQL warehouse is in Running or Idle (auto-startable) state. A warehouse in Deleted or Stopped (manually) state will not start on connection.

Warehouse access denied

• Confirm the service principal has Can use permission on the warehouse: workspace → SQL Warehouses → your warehouse → Permissions.
• If the warehouse was recreated after the initial setup, the permission must be re-granted — it does not follow the SP.

OAuth authentication failure

• Confirm the Client Secret entered in DigiUsher matches the secret generated for this specific service principal, not a PAT or another SP's secret.
• OAuth secrets do not expire by default, but if the secret was manually deleted and regenerated in the Databricks console the old value is permanently invalid — generate a new one and update DigiUsher.

Need Help?

If you encounter any issues not covered above, contact DigiUsher support at support@digiusher.com and the team will help you get set up.