Connecting a Databricks account

Overview

To connect your Databricks environment to DigiUsher, create a service principal. Then give it the permissions on this page, and connect it to one of your SQL warehouses. This page gives the permissions, the reason for each permission, and the credentials that you enter in DigiUsher.

Summary of Access Required

DigiUsher cannot create, change, or delete a Databricks resource.

Prerequisites

Information to Gather

Roles Required by the Person Performing Setup

Option A: Terraform (Recommended)

The DigiUsher Terraform configuration does most of the setup. It creates the service principal, it creates an OAuth secret, and it gives the permission Can use on the SQL warehouse. It also applies the Unity Catalog grants for the tables that the cost query of DigiUsher reads, in system.billing, system.access, system.compute, and system.lakeflow. You run the other grants yourself after the apply.

Prerequisites

• Terraform, version 1.3 or higher.
• Python 3 with databricks-sql-connector, databricks-sdk, pandas, and pyarrow. You need Python only for the verification script.
• A Databricks personal access token with the scope All APIs. The setup uses it one time, and it does not store the token after terraform apply completes.
• The rights of an Account Admin and of a Workspace Admin.

Create a Personal Access Token

1. Go to your workspace → top-right user icon → Settings → Developer → Access tokens.
2. Click Generate new token.
3. Enter a name, for example terraform-setup-digiusher, and a lifetime.
4. Under Scope, select All APIs.
5. Copy the token immediately. Databricks does not show it again.

Find Your Configuration Values

You need three values before you run Terraform:

Deployment

git clone https://github.com/digiusher/digiusher-iac.git
cd digiusher-iac/databricks

Create the file terraform.tfvars with your own values:

databricks_account_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
workspace_host        = "your-workspace.cloud.databricks.com"
warehouse_id          = "abc123def456"

Export your personal access token. Then run the apply:

export TF_VAR_databricks_token="XXXXXXXXXXXXXXXXXXXX"

terraform init
terraform plan
terraform apply

Complete the Remaining Grants

GRANT SELECT ON TABLE system.lakeflow.jobs TO `<client-id>`;
GRANT SELECT ON TABLE system.compute.instance_pools TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.serving TO `<client-id>`;
GRANT SELECT ON TABLE system.serving.served_entities TO `<client-id>`;

GRANT SELECT ON TABLE system.compute.node_timeline TO `<client-id>`;
GRANT SELECT ON TABLE system.compute.warehouse_events TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.query TO `<client-id>`;
GRANT SELECT ON TABLE system.query.history TO `<client-id>`;

GRANT SELECT ON TABLE system.serving.endpoint_usage TO `<client-id>`;

You can also grant SELECT ON TABLE system.billing.account_prices. This grant is optional. When the table exists, DigiUsher reads your contracted rates from it. When the table does not exist, DigiUsher reads system.billing.list_prices. An absent grant is therefore not an error.

Retrieve the Credentials

After the apply completes, four outputs give the values that DigiUsher needs:

echo "DATABRICKS_HOST=$(terraform output -raw workspace_hostname)"
echo "DATABRICKS_HTTP_PATH=$(terraform output -raw http_path)"
echo "DATABRICKS_CLIENT_ID=$(terraform output -raw client_id)"
echo "DATABRICKS_CLIENT_SECRET=$(terraform output -raw client_secret)"

These four values go into the fields Server Hostname, HTTP Path, Client ID, and Client Secret in DigiUsher. The section Connect in DigiUsher gives the fields.

Make Sure the Setup Works (Optional)

The repository has a script that makes sure that the authentication works, that the system catalog is readable, and that the billing data exists:

export DATABRICKS_HOST="$(terraform output -raw workspace_hostname)"
export DATABRICKS_HTTP_PATH="$(terraform output -raw http_path)"
export DATABRICKS_CLIENT_ID="$(terraform output -raw client_id)"
export DATABRICKS_CLIENT_SECRET="$(terraform output -raw client_secret)"

python3 verify_exports.py

The digiusher-iac README is the full Terraform documentation. It also gives the troubleshooting steps.

Option B: Manual Setup

GRANT USE CATALOG ON CATALOG system TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.billing TO `<client-id>`;
GRANT SELECT ON TABLE system.billing.usage TO `<client-id>`;
GRANT SELECT ON TABLE system.billing.list_prices TO `<client-id>`;
GRANT SELECT ON TABLE system.billing.account_prices TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.access TO `<client-id>`;
GRANT SELECT ON TABLE system.access.workspaces_latest TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.compute TO `<client-id>`;
GRANT SELECT ON TABLE system.compute.clusters TO `<client-id>`;
GRANT SELECT ON TABLE system.compute.warehouses TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.lakeflow TO `<client-id>`;
GRANT SELECT ON TABLE system.lakeflow.pipelines TO `<client-id>`;
GRANT SELECT ON TABLE system.lakeflow.jobs TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.query TO `<client-id>`;
GRANT SELECT ON TABLE system.query.history TO `<client-id>`;

GRANT SELECT ON TABLE system.compute.node_timeline TO `<client-id>`;
GRANT SELECT ON TABLE system.compute.instance_pools TO `<client-id>`;
GRANT SELECT ON TABLE system.compute.warehouse_events TO `<client-id>`;

GRANT USE SCHEMA ON SCHEMA system.serving TO `<client-id>`;
GRANT SELECT ON TABLE system.serving.served_entities TO `<client-id>`;
GRANT SELECT ON TABLE system.serving.endpoint_usage TO `<client-id>`;

Connect in DigiUsher

After you complete the Terraform setup or the manual setup, go to Connectors > Add Source in DigiUsher and select Databricks. On the Configure step, enter the values in the table. Then go to Connect and click Connect source:

At the connection, DigiUsher makes sure that the service principal can read every necessary system table. Then it starts to read the data. The first sync reads 12 calendar months, which is the retention window of system.billing.usage. Every sync after the first one reads the current month and the month before it again. The values of those two months therefore change when Databricks finalizes them.

Setup Checklist

• Service principal created in the Databricks account console
• OAuth client secret created, and the Client ID and the Secret saved in a safe place
• Service principal added to the workspace
• Service principal has the permission Can use on the SQL warehouse
• All GRANT statements ran without an error in the SQL Editor, for system.billing, system.access, system.compute, system.lakeflow, system.query, and system.serving
• Workspace hostname, HTTP path, Client ID, and Client Secret entered in DigiUsher
• DigiUsher reads system.billing.usage, system.billing.list_prices, system.access.workspaces_latest, system.lakeflow.pipelines, system.lakeflow.jobs, system.compute.clusters, system.compute.warehouses, system.compute.instance_pools, and system.serving.served_entities, and it returns an account_id
• *.digiusher.com in the allowlist of your network and firewall (if your organization restricts this)
• digiusher.com in the allowlist of permitted sender domains in your email security gateway (if your organization restricts this)

Security

What DigiUsher CAN Access (Read-Only)

• system.billing.usage, which holds the DBU records and the credit consumption of all workspaces of the account
• system.billing.list_prices, which holds the public list prices of Databricks for the cost of each SKU
• system.billing.account_prices, which holds your contracted rates. This table exists only with the Account Prices Preview, on AWS and GCP
• system.access.workspaces_latest, which holds the workspace names for the billing records
• system.compute.clusters and system.compute.warehouses, which hold the resource names for the billing records
• system.lakeflow.pipelines, which holds the pipeline names for the billing records
• system.lakeflow.jobs, which holds the job definitions for the billing records
• system.compute.instance_pools and system.serving.served_entities, which hold the resource inventory
• system.compute.node_timeline, system.compute.warehouse_events, system.query.history, and system.serving.endpoint_usage, which hold the utilization metrics for the rightsizing recommendations and the idle recommendations

What DigiUsher CANNOT Do

• Read a table outside the schemas of the grants on this page
• Read notebooks, job definitions, query results, or your business data
• Read the data of a workspace catalog, such as hive_metastore or a catalog of your users
• Create, change, or delete an object in Databricks
• Read or change the billing configuration or the account configuration
• Buy a product or change your account
• Read the cloud infrastructure under Databricks, such as S3, ADLS, or GCS

Monitoring

To monitor the activity of the service principal, open the Databricks workspace under Settings → Identity and access → Service Principals → your service principal. For the full audit trail, query system.access.audit when your workspace has that schema. Filter on user_identity.email = '<client-id>' to see every action of the DigiUsher service principal.

Credential Rotation

With Terraform, run terraform apply -replace="databricks_service_principal_secret.digiusher_focus_reader" to create a new secret. Then read the new secret with terraform output -raw client_secret, and enter it in the field Client Secret in DigiUsher.

By hand, do these four steps:

1. In the account console, go to Service Principals, select the DigiUsher service principal, and open the Secrets tab.
2. Click Generate secret to create a new client secret.
3. Enter the new secret in the field Client Secret in DigiUsher immediately.
4. Delete the old secret on the Secrets tab.

Revocation

With Terraform, run terraform destroy. It removes the service principal, its secret, the permission on the warehouse, and all Unity Catalog grants. The access ends immediately.

By hand, delete the service principal in the account console under Service Principals. Databricks then immediately invalidates the OAuth secret and removes all permissions of the principal from the workspace. To stop the access for a short time, delete only the secret on the Secrets tab of the service principal. Use this method when you want to connect again later.

Troubleshooting

Permission denied on system.billing.usage or other system tables

• Make sure that all GRANT statements ran without an error in the SQL Editor. Run the full block again. The grants are idempotent, so you can repeat them.
• Make sure that the user of the grants had the role Account Admin and the role Metastore Admin at that time. For an admin of the workspace only, the grants fail or have no effect.
• Make sure that the grantee is the Client ID of the service principal, which is the Application ID UUID, and not its display name.

"User is not an account admin" during setup

• Go to accounts.cloud.databricks.com → User Management → Users. Click the email address of the setup user, and turn Account admin on.
• This role is not the membership of the workspace admin group. The group admins of the workspace is not sufficient.

Verification probe passes but BilledCost is zero

• DigiUsher computes BilledCost as usage_quantity × account_unit_price. If list_prices has no row for the region and the SKUs of your account, the join of the price returns NULL, and the cost becomes 0.
• Make sure that system.billing.list_prices has rows. Run SELECT COUNT(*) FROM system.billing.list_prices in the SQL Editor. A new trial account can have an empty or small price table until Databricks fills it for your region. This takes 24 to 48 hours.

No data in system.billing.usage

• The table contains records from the creation date of the account only. A new account has one or two rows at most, for the storage of the warehouse start. This is correct, and it is not an error. The data grows with the use of the account.
• Read the date range with SELECT MIN(usage_date), MAX(usage_date) FROM system.billing.usage.

Connection hangs with no output

• Make sure that the state of the SQL warehouse is Running, or Idle with an automatic start. A warehouse in the state Deleted or Stopped (manually) does not start at a connection.

Warehouse access denied

• Make sure that the service principal has the permission Can use on the warehouse. Open the workspace → SQL Warehouses → your warehouse → Permissions.
• If somebody created the warehouse again after the setup, give the permission again. The permission does not move with the service principal.

OAuth authentication failure

• Make sure that the Client Secret in DigiUsher is the secret of this service principal. A personal access token does not work, and the secret of another service principal does not work.
• An OAuth secret does not expire by default. But if somebody deleted the secret and created a new one in the Databricks console, the old value is invalid forever. Create a new secret and enter it in DigiUsher.

Need Help?

If this page does not answer your question, write to DigiUsher support at support@digiusher.com. The team will help you.