> For the complete documentation index, see [llms.txt](https://docs.onelens.cloud/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.onelens.cloud/integrations/cloud-and-cost-sources/connecting-to-databricks.md). # Connecting to Databricks Connect your Databricks account to OneLens for cost visibility, analysis, and optimization — using a read-only service principal and Unity Catalog system tables. ## TL;DR **What this does:** Connects OneLens to your Databricks account via Unity Catalog **system tables** to pull DBU consumption, SKU pricing, compute configuration, and job/pipeline metadata — giving you unified cost visibility, right-sizing recommendations, idle-compute detection, tag-based allocation, and anomaly detection across every workspace in your Databricks account. **Time required:** \~20 minutes **Who you need:** A Databricks user who is **both an account admin and a metastore admin** (required to grant access to system schemas), plus a **workspace admin** to grant SQL warehouse access. **What OneLens reads:** Read-only `SELECT` on a specific list of `system` catalog schemas — `system.billing`, `system.compute`, `system.lakeflow`, and `system.access.workspaces_latest`. OneLens does **not** read any of your data catalogs, tables, notebooks, or audit logs. {% hint style="warning" %} **Least privilege matters.** OneLens needs `SELECT` on a named subset of `system` schemas — not on your data catalogs, and not on `system.access.audit`. Grant the service principal exactly the schemas listed in [What OneLens Will Access](#what-onelens-will-access). Do **not** make the service principal an account admin or metastore admin. {% endhint %} ## What You'll Get Once Connected | Capability | What it does for you | | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | **Unified Cost Explorer** | See Databricks DBU spend alongside your AWS, Azure, GCP, OCI, and AI provider costs in one dashboard. | | **Cost by Workspace, SKU & Product** | Break spend down by workspace, SKU, and billing origin product (Jobs, All-Purpose, SQL, DLT, Model Serving). | | **List-Price Cost Attribution** | Joins `system.billing.usage` to `system.billing.list_prices` to convert raw DBU quantities into dollar cost per record. | | **Job & Pipeline Cost Attribution** | Attribute DBU spend down to individual jobs, tasks, runs, and Lakeflow pipelines — not just to a workspace. | | **Tag-Based Allocation & Chargeback** | Use custom tags on clusters, warehouses, and jobs to allocate cost to teams, cost centers, or products. | | **Idle & Oversized Compute Detection** | Flags clusters with long auto-termination windows, warehouses with generous auto-stop, and consistently under-utilized nodes. | | **All-Purpose vs. Jobs Compute Waste** | Identifies scheduled workloads running on All-Purpose compute, where the same work on Jobs compute costs materially less. | | **Warehouse Right-Sizing** | Correlates warehouse size and uptime against actual query load to recommend a smaller size or tighter auto-stop. | | **Cost Anomaly Detection** | Alerts when Databricks spend deviates from historical baselines — catch a runaway cluster the day it starts. | | **Budgets & Forecasting** | Set budgets at the account, workspace, or tag level and track run-rate against them with OneLens alerts. | ## Security at a Glance | Question | Answer | | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Does OneLens read our data — tables, notebooks, or files?** | No. OneLens is granted `SELECT` only on named `system` catalog schemas. It has no privileges on any user catalog, schema, table, volume, or model, and cannot browse or query them. | | **Does OneLens read our audit logs?** | No. `system.access.audit` is explicitly **not** granted. The only table OneLens reads from the `access` schema is `system.access.workspaces_latest`, which provides workspace IDs and names for labelling. | | **Does OneLens see user identities or object names?** | **Yes — this is unavoidable and worth understanding.** `system.billing.usage` includes an `identity_metadata` struct (`run_as`, `owned_by`, `created_by`) containing user emails, and a `usage_metadata` struct that can include `job_name`, `notebook_path`, `app_name`, and — for materialized views — `uc_table_catalog`, `uc_table_schema`, and `uc_table_name`. These are **names and paths, never contents**. They are what makes per-owner and per-job chargeback possible. See the FAQ for what this does and does not reveal. | | **Does OneLens see SQL query text?** | **Only if you opt in.** `system.query.history` contains a `statement_text` column. It is **not** part of the default grant. Enable it only if you want query-level cost attribution — see [Optional: Query-Level Attribution](#optional-query-level-attribution). | | **Is access read-only?** | Yes. System tables are read-only by design and cannot be modified by anyone. The service principal is granted `USE CATALOG`, `USE SCHEMA`, and `SELECT` — no `MODIFY`, `CREATE`, `MANAGE`, or ownership. | | **What authentication is used?** | A Databricks-managed **service principal** using **OAuth M2M** (client credentials). Access tokens are short-lived — one hour — and minted per request. No long-lived personal access token is used. | | **Can the service principal run expensive compute?** | It is granted `CAN USE` on one SQL warehouse only. It cannot create clusters, warehouses, jobs, or pipelines, and cannot attach to or modify any other compute. | | **How is data transmitted and stored?** | All calls use HTTPS/TLS 1.2+ to your workspace URL. The client secret is entered directly in the OneLens integration form and encrypted at rest using GCP KMS. | | **Can I restrict by IP?** | Yes. Add OneLens's static egress IP ranges to your Databricks workspace **IP access list**. Contact for the current ranges. | | **What about Databricks' guidance on exporting system table data?** | Databricks advises care when moving system table data off-platform, because system tables can expose sensitive operational detail. This is why OneLens's default grant excludes audit logs, lineage, and query text — the ingested set is billing, pricing, compute configuration, and job metadata only. | ## Cost of the Integration OneLens creates **no new infrastructure** in your Databricks account. System tables themselves are free — Databricks charges only for the compute used to query them. The only cost is a small amount of SQL warehouse uptime once per day. The figures below assume a dedicated **2X-Small serverless SQL warehouse** at the AWS US list rate of **4 DBU/hour × $0.70/DBU = $2.80/hour ($0.047/minute)**, with `auto_stop_mins` set to 5 — the lowest value the UI allows for serverless. | Item | What it is | Typical cost | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | **Serverless SQL warehouse compute** | \~5–10 min of 2X-Small uptime per day: 3–5 min of query execution plus the auto-stop idle tail. 5 min/day = $0.23/day; 10 min/day = $0.47/day | **$7–$14/month** | | **System table storage & retention** | Hosted by Databricks, free for the standard retention period | $0 | | **Data egress** | Result sets leave Databricks-hosted serverless compute, not your VPC | $0 | | **Storage in your account** | None — OneLens writes nothing back | $0 | | **New resources created** | None | $0 | | **Estimated total** | Sum of above | **$7–$14/month** | **If you reuse an existing warehouse** that is already running during your business day, the marginal cost is effectively **$0** — OneLens's queries execute inside uptime you are already paying for. This is the cheapest option, and OneLens's queries are small enough not to affect concurrency. **If you use a classic or pro warehouse** instead of serverless, add the underlying cloud VM cost (a 2X-Small classic warehouse runs one `i3.2xlarge` driver plus one worker) for the duration of the poll. Serverless is recommended precisely because it avoids this. {% hint style="info" %} **Rates are list price and region-dependent.** Serverless SQL is \~$0.70/DBU in AWS US regions and higher elsewhere. If you have a Databricks commit (DBCU) or negotiated discount, your effective cost will be lower than the figures above. {% endhint %} ## How It Works Databricks system tables are a Databricks-hosted analytical store of your account's operational data, exposed as read-only Delta tables in the `system` catalog of every Unity Catalog metastore. They are governed by Unity Catalog, and no principal has access to them by default — access must be granted explicitly by a user who is both an account admin and a metastore admin. Critically, system tables are **account-wide, not workspace-wide**. `system.billing.usage` contains billable usage for every workspace in your account, and is replicated globally. `system.compute` and `system.lakeflow` contain regional data covering every workspace in that region — including workspaces that are not themselves Unity Catalog-enabled. This means a single OneLens connection, made through one UC-enabled workspace, covers your whole account in that region. OneLens authenticates as a Databricks service principal using OAuth M2M, exchanges the client ID and secret for a one-hour access token at your workspace's OIDC token endpoint, and runs parameterised `SELECT` statements through the **SQL Statement Execution API** (`/api/2.0/sql/statements`) against a SQL warehouse you nominate. Queries are date-bounded and column-selective. Costs are computed by joining usage records to the SKU price effective at the time of usage: ```sql SELECT u.workspace_id, u.sku_name, u.billing_origin_product, u.usage_date, u.custom_tags, SUM(u.usage_quantity) AS dbus, SUM(u.usage_quantity * COALESCE( p.pricing.effective_list.default, -- resolves list + promotional p.pricing.default -- fallback )) AS cost_usd FROM system.billing.usage u JOIN system.billing.list_prices p ON u.cloud = p.cloud AND u.sku_name = p.sku_name AND u.usage_start_time >= p.price_start_time AND (u.usage_end_time <= p.price_end_time OR p.price_end_time IS NULL) WHERE u.usage_date BETWEEN :start_date AND :end_date GROUP BY ALL ``` Two details in that query matter for accuracy. First, OneLens reads `pricing.effective_list.default` rather than `pricing.default`, because `effective_list` is the field Databricks resolves list and promotional pricing into — using `pricing.default` alone overstates cost during a promotional period. Second, the query aggregates with `SUM` rather than reading individual rows. `system.billing.usage` handles corrections by appending a `RETRACTION` record with a negated `usage_quantity` followed by a `RESTATEMENT` record with the corrected values. Summing lets the retraction cancel the original automatically; reading rows individually would double-count. ## Prerequisites * A Databricks account on a **Premium plan or above** — Unity Catalog and system tables are not available on Standard or Trial workspaces. * **At least one Unity Catalog-enabled workspace.** System tables can only be queried from a UC-enabled workspace, though they contain data from all workspaces. * The metastore must be on **Unity Catalog Privilege Model Version 1.0**. Older metastores need to be upgraded to privilege inheritance first. * A user with **both the account admin and metastore admin roles** — required to grant `SELECT` on system schemas. Neither role alone is sufficient. * A **SQL warehouse** in the UC-enabled workspace. A dedicated 2X-Small serverless warehouse is recommended. * Your **workspace URL** (for example `https://dbc-a1b2c3d4-e5f6.cloud.databricks.com`) and the target **SQL warehouse ID**. ## What OneLens Will Access The service principal is granted `USE CATALOG` on `system`, then `USE SCHEMA` + `SELECT` on exactly these schemas: | Schema | Key tables | Why OneLens needs it | | ----------------- | --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `system.billing` | `usage`, `list_prices` | The core cost dataset: DBU quantity per record, SKU, workspace, custom tags, and the historical price log used to convert DBUs to dollars. | | `system.compute` | `clusters`, `warehouses`, `warehouse_events`, `node_timeline`, `node_types`, `instance_pools` | Compute configuration and utilization — powers right-sizing, idle detection, and auto-stop/auto-termination recommendations. | | `system.lakeflow` | `jobs`, `job_tasks`, `job_run_timeline`, `job_task_run_timeline`, `pipelines`, `pipeline_update_timeline` | Job and pipeline metadata, so DBU spend can be attributed to a named job or pipeline rather than an anonymous cluster ID. | | `system.access` | `workspaces_latest` **only** | Maps workspace IDs to human-readable workspace names. Granted at table level — **not** at schema level, so `system.access.audit` and the lineage tables stay out of scope. | {% hint style="info" %} **Preview status.** `system.billing.usage`, `system.billing.list_prices`, and the core `system.compute` and `system.lakeflow` tables are generally available. Several tables OneLens reads are still Public Preview — including `system.access.workspaces_latest`, `system.compute.instance_events`, `system.compute.instance_pools`, `system.lakeflow.pipelines`, and `system.query.history` — and the AI tables below are Beta. Preview tables can change, so a small number of secondary insights may lag behind Databricks changes. Core cost reporting does not depend on any preview table. {% endhint %} ### Optional: AI & Model Serving Costs If you run Mosaic AI Model Serving or AI Gateway and want that spend broken out in OneLens alongside your OpenAI, Anthropic, and Bedrock costs, additionally grant: | Schema | Tables | Adds | | ------------------- | ----------------------------------- | --------------------------------------------------------------------------- | | `system.serving` | `served_entities`, `endpoint_usage` | Per-endpoint token counts and served model metadata. | | `system.ai_gateway` | `usage`, `external_model_spend` | AI Gateway request metrics and estimated spend on externally routed models. | ### Optional: Query-Level Attribution `system.query.history` enables per-query cost attribution — showing which queries drove warehouse uptime and which users are responsible for spend. {% hint style="warning" %} **This table contains query text.** `system.query.history` includes a `statement_text` column holding the literal SQL your users ran, which may embed table names, filter values, or other business-sensitive detail. It is **excluded from the default grant** for exactly this reason. Grant it only after your security team has reviewed the trade-off. OneLens can also be configured to ingest this table with `statement_text` dropped at read time — request redacted mode from . If your workspace uses **customer-managed keys for managed services**, `statement_text` and `error_message` are encrypted in the system table by default and return empty unless an account admin adds a key configuration to the `system` catalog. In that setup you get query-level attribution with no text exposure at all. {% endhint %} ### What OneLens Will NOT Access * **Any user catalog, schema, table, view, volume, function, or model** — the service principal holds no privileges outside the `system` catalog. * **`system.access.audit`** — audit logs are never granted. * **`system.access.table_lineage` and `system.access.column_lineage`** — lineage data is out of scope. * **`system.query.history`** — excluded unless you explicitly opt in. * **Notebooks, repos, workspace files, and secrets** — no workspace object permissions are granted. * **Model invocation** — the service principal cannot call model serving endpoints or run any inference. * **Cluster, warehouse, job, or pipeline creation or modification** — it holds `CAN USE` on a single warehouse and nothing more. * **Account or workspace administration** — the service principal is not an account admin, metastore admin, or workspace admin. * **Billing and payment method details** — OneLens reads usage and list prices, not your invoice or payment instrument. {% stepper %} {% step %} ## Confirm System Schemas Are Enabled Some system schemas are enabled by default and others must be turned on. Check which are active, and enable any that are missing. Run this from a machine with the Databricks CLI configured against your account. Find your metastore ID in **Catalog Explorer → your metastore → Details**. ```bash # List the system schemas and their current state databricks system-schemas list # Enable any that show as unavailable/disabled databricks system-schemas enable compute databricks system-schemas enable lakeflow databricks system-schemas enable access ``` `billing` is enabled by default on all accounts. The caller must be an account admin or metastore admin. {% hint style="info" %} Enabling a system schema is free — Databricks does not charge for system tables. Newly enabled schemas begin collecting data from the point of enablement, so historical backfill for those tables starts from that date. {% endhint %} {% endstep %} {% step %} ## Create a Service Principal and OAuth Secret Create a dedicated Databricks-managed service principal so its access can be audited, rotated, and revoked independently of any human user. 1. In the **account console**, go to **User management → Service principals → Add service principal**. 2. Name it **OneLens Integration**. Do **not** grant it account admin. 3. Open the service principal, go to the **Permissions** tab, and assign it to the Unity Catalog-enabled workspace you will connect through. Assign the **User** entitlement only — not workspace admin. 4. Generate the OAuth secret. In the account console this is the service principal's **Credentials & secrets** tab; from inside a workspace it is **Settings → Identity and access → Service principals → Manage →** select the principal **→ Secrets**. Click **Generate secret** and set a lifetime (maximum 730 days). 5. Copy the **Client ID** (the same value as the Application ID) and the **Secret**. Both account admins and workspace admins can create an OAuth secret. {% hint style="warning" %} **The secret is displayed once.** Copy it before closing the dialog. A service principal can hold up to five OAuth secrets, which makes zero-downtime rotation straightforward — see the FAQ. {% endhint %} {% endstep %} {% step %} ## Grant Read Access on the System Schemas Run the following in a SQL editor or notebook in your UC-enabled workspace. **The user running this must hold both the account admin and metastore admin roles** — an account admin alone will get `PERMISSION_DENIED: User is not an owner of Schema 'system.compute'`. Replace `` with the service principal's application ID (the UUID, not the display name). ```sql -- Catalog-level access to the system catalog GRANT USE CATALOG ON CATALOG system TO ``; -- Billing: DBU usage records and the historical SKU price log GRANT USE SCHEMA ON SCHEMA system.billing TO ``; GRANT SELECT ON SCHEMA system.billing TO ``; -- Compute: cluster/warehouse configuration and utilization GRANT USE SCHEMA ON SCHEMA system.compute TO ``; GRANT SELECT ON SCHEMA system.compute TO ``; -- Lakeflow: job and pipeline metadata for cost attribution GRANT USE SCHEMA ON SCHEMA system.lakeflow TO ``; GRANT SELECT ON SCHEMA system.lakeflow TO ``; -- Access: workspace names ONLY - granted at table level, not schema level, -- so audit logs and lineage remain inaccessible GRANT USE SCHEMA ON SCHEMA system.access TO ``; GRANT SELECT ON TABLE system.access.workspaces_latest TO ``; ``` **Optional — AI and model serving spend:** ```sql GRANT USE SCHEMA ON SCHEMA system.serving TO ``; GRANT SELECT ON SCHEMA system.serving TO ``; GRANT USE SCHEMA ON SCHEMA system.ai_gateway TO ``; GRANT SELECT ON SCHEMA system.ai_gateway TO ``; ``` **Optional — query-level attribution (contains query text, see the warning above):** ```sql GRANT USE SCHEMA ON SCHEMA system.query TO ``; GRANT SELECT ON TABLE system.query.history TO ``; ``` Verify the grants landed: ```sql SHOW GRANTS `` ON SCHEMA system.billing; SHOW GRANTS `` ON TABLE system.access.workspaces_latest; ``` {% hint style="info" %} **On table-level grants.** Databricks' own documentation describes granting `SELECT` at the *schema* level on system schemas. Unity Catalog also supports table-level `SELECT`, which is what the `system.access` grant above relies on to keep audit logs out of scope. If the table-level grant is rejected in your metastore, contact — OneLens can run without `workspaces_latest`, in which case workspaces appear by ID rather than by name. Do **not** fall back to granting `SELECT` on all of `system.access`. {% endhint %} {% endstep %} {% step %} ## Grant Warehouse Access The service principal needs a warehouse to execute its queries on. A dedicated 2X-Small serverless warehouse keeps the cost predictable and isolates OneLens from your users' workloads. 1. Go to **SQL → SQL Warehouses**. 2. Either create a new warehouse (**Serverless**, size **2X-Small**, max clusters **1**) or pick an existing one. On **Auto stop**: serverless warehouses default to 10 minutes and the UI will not let you go below 5. If you want to trim the idle tail further, create or edit the warehouse through the SQL Warehouses API, where `auto_stop_mins` can be set as low as **1**. That takes the integration's monthly cost from roughly $7–14 down to $4–6. 3. Open the warehouse → **Permissions** → add the **OneLens Integration** service principal with **CAN USE**. 4. Copy the **warehouse ID** from the warehouse's **Connection details** tab, or from the end of the HTTP path (`/sql/1.0/warehouses/`). `CAN USE` permits running queries on the warehouse. It does not permit editing, resizing, or deleting it. {% endstep %} {% step %} ## (Optional) Restrict Access by IP If your workspace uses IP access lists, add OneLens's static egress ranges so the integration isn't blocked. 1. Request the current OneLens egress IP ranges from . 2. In the workspace, go to **Settings → Security → IP access list**. 3. Add the ranges to your allow list. If you do not use IP access lists, skip this step — nothing further is required. {% endstep %} {% step %} ## Connect to OneLens In OneLens, open the **Integrations** page, select the **Databricks** integration, and enter the following. The client secret is encrypted at rest — there is no need to share it over email or chat. | Field | Value | Example | | ---------------- | ------------------------------------------------ | ------------------------------------------------ | | Integration Name | A display name for this connection | `Acme Corp Databricks` | | Workspace URL | Your UC-enabled workspace host, no trailing path | `https://dbc-a1b2c3d4-e5f6.cloud.databricks.com` | | Client ID | The service principal's application ID | `a1b2c3d4-e5f6-7890-abcd-ef1234567890` | | Client Secret | The OAuth secret generated in Step 2 | `dose...` | | SQL Warehouse ID | The warehouse the service principal can use | `abc123def4567890` | | Cloud | Which cloud your Databricks account runs on | `AWS` / `Azure` / `GCP` | | Backfill Window | How far back to load history (max 12 months) | `12 months` | OneLens will begin ingesting on the next scheduled poll. {% endstep %} {% step %} ## Verify the Connection After connecting, check the OneLens dashboard for: * Databricks spend appearing in the cost explorer * Workspace, SKU, and billing-origin-product breakdowns populating * Job and pipeline level attribution appearing under cost allocation * Historical usage backfilling across the requested window If nothing appears within 24 hours, confirm the credentials and grants work independently. Mint a token and run a probe query: ```bash # 1. Exchange client credentials for a one-hour access token export DATABRICKS_HOST="https://dbc-a1b2c3d4-e5f6.cloud.databricks.com" TOKEN=$(curl -s -X POST "$DATABRICKS_HOST/oidc/v1/token" \ -u ":" \ -d "grant_type=client_credentials" \ -d "scope=all-apis" | jq -r .access_token) # 2. Run a probe query through the SQL Statement Execution API curl -s -X POST "$DATABRICKS_HOST/api/2.0/sql/statements" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "warehouse_id": "", "wait_timeout": "30s", "statement": "SELECT usage_date, sku_name, SUM(usage_quantity) AS dbus FROM system.billing.usage WHERE usage_date >= CURRENT_DATE() - INTERVAL 3 DAYS GROUP BY 1,2 ORDER BY 1 DESC LIMIT 10" }' ``` A `SUCCEEDED` status with rows confirms authentication, warehouse permission, and Unity Catalog grants are all correct. {% endstep %} {% endstepper %} ## Data Refresh Schedule * **Billing and pricing** (`system.billing.usage`, `system.billing.list_prices`): polled **once per 24 hours**. Databricks explicitly does not support real-time monitoring of system tables — data lands throughout the day with inherent latency, so polling more often surfaces no fresher data while adding warehouse uptime cost. * **Compute and job metadata** (`system.compute`, `system.lakeflow`): polled in the same daily window, since configuration changes slowly. * **Backfill on connection**: OneLens loads up to 12 months of history on first connect, bounded by Databricks' free retention period for `system.billing.usage` (365 days). `node_timeline` retains 90 days; `list_prices` is retained indefinitely. * **Current-period figures may shift**, for two reasons. Usage records arrive with a lag, so the current month's total moves as records land. Separately, Databricks issues corrections by appending `RETRACTION` and `RESTATEMENT` records, which can revise a figure for a date that has already closed. OneLens re-reads a trailing window on each poll to pick these up. ## Data Privacy & Security * **Least-privilege scope** — `SELECT` on four named system schemas, with `system.access` granted at table level so audit logs and lineage stay out of reach. * **No data-plane access** — zero privileges on any user catalog, schema, table, volume, or model. OneLens sees object *names* and owner identities from the billing metadata, never contents. * **Read-only by construction** — system tables are read-only for every principal; no `MODIFY`, `CREATE`, `MANAGE`, or ownership is granted. * **Short-lived credentials** — OAuth M2M access tokens expire after one hour and are minted per request. No long-lived personal access token. * **Query text excluded by default** — `system.query.history` is opt-in, with a redacted mode available. * **TLS in transit** — all calls to your workspace use HTTPS/TLS 1.2+. * **Encrypted at rest** — the client secret is entered directly in the OneLens form and encrypted using GCP KMS; no external key-sharing tool is involved. * **Network restriction supported** — OneLens egress IPs can be pinned in your workspace IP access list. * **Data retention** — 12-month default, configurable. Deletion within 30 days on request with confirmation. ## Troubleshooting | Symptom | Cause | Fix | | ------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `PERMISSION_DENIED: User is not an owner of Schema 'system.compute'` | The person running the `GRANT` statements holds only the account admin role, not metastore admin. | In the account console, open **Catalog → your metastore → Edit** and assign yourself as metastore admin. Re-run the grants. You can unassign yourself afterwards. | | `Schema 'system.compute' does not exist` or the schema is empty | The system schema has not been enabled on the metastore. | Run `databricks system-schemas enable compute`, then repeat for `lakeflow` and `access`. | | `401 Unauthorized` from the token endpoint | Wrong token issuer, or the secret has expired. M2M tokens must come from the Databricks OIDC endpoint, not your IdP or cloud provider. | Confirm you are posting to `/oidc/v1/token` with `grant_type=client_credentials&scope=all-apis`. Generate a fresh secret if the old one has lapsed. | | `Workspace not found` / `host not found` | The service principal has not been assigned to the workspace, or the workspace URL includes a trailing `/api` or path segment. | Assign the service principal to the workspace in the account console. Use the bare workspace URL with no suffix. | | `Warehouse ... does not exist or you do not have permission` | `CAN USE` was never granted on the warehouse, or the warehouse ID belongs to a different workspace. | Add the service principal with **CAN USE** on the warehouse, and confirm the ID matches the workspace in the connection. | | `System Table query returned too much data. Please repeat query with more selective predicates.` | A query was not sufficiently selective — Databricks rejects unbounded system table scans to protect performance. | OneLens date-bounds every query. If this appears during backfill, shorten the backfill window and let it load incrementally. | | Connected, but no data after 24–48 hours | Either the first poll has not run, or the warehouse is failing to start. | Run the verification query from Step 7. Check **SQL Warehouses → Monitoring** for start failures, and confirm the warehouse is not stopped by a policy. | | Cost figures look lower than the Databricks invoice | System tables report **list price**. They do not apply DBCU commit discounts, private offers, or negotiated rates, and exclude the underlying cloud VM cost for classic compute. | Expected behaviour. Configure your commit discount in OneLens, and connect your AWS/Azure/GCP account so the infrastructure side of the bill is captured too. | | New columns cause ingestion warnings | Databricks may add columns to system tables or fields to struct columns at any time. | No action needed — OneLens selects columns explicitly and tolerates additive schema changes. Report persistent warnings to support. | | Some workspaces are missing from the data | System tables are regional for `compute` and `lakeflow`. Workspaces in another cloud region belong to a different metastore. | Add a second OneLens Databricks integration through a UC-enabled workspace in that region. Note that `system.billing.usage` is global and already covers everything. | ## Appendix: Personal Access Token Authentication **Not recommended.** OneLens also supports authenticating as a service principal using a personal access token (PAT) instead of OAuth M2M. This exists only for environments where OAuth is blocked by policy. PATs are long-lived bearer credentials. If one leaks, it stays valid until someone notices and revokes it, whereas an OAuth M2M access token expires within the hour. Databricks is steadily moving customers toward OAuth across its drivers and SDKs, so PAT-based setups should be treated as a migration liability rather than a steady state. If you must use a PAT: generate it on behalf of the service principal (never a human user), scope the same Unity Catalog grants and warehouse permission described above, set the shortest workable expiry, and put a calendar reminder on rotation. Supply the workspace URL, the token, and the warehouse ID in the OneLens integration form. Plan to move to OAuth M2M when policy allows. ## Frequently Asked Questions
Can OneLens see our actual data — tables, notebooks, or files? No. The service principal is granted privileges only inside the `system` catalog. It holds no `USE CATALOG` on any of your data catalogs, so it cannot list, browse, or query a single business table. Unity Catalog enforces this — it is not a matter of OneLens choosing not to look.
Can OneLens modify our Databricks environment or run up our bill? No. System tables are read-only for every principal, and the service principal holds no `CREATE`, `MODIFY`, or `MANAGE` privileges anywhere. It has `CAN USE` on exactly one SQL warehouse, which permits running queries and nothing else — it cannot resize the warehouse, create clusters, or trigger jobs. Its own footprint is roughly 5–10 minutes of 2X-Small warehouse uptime per day.
Will OneLens see our SQL query text? Not by default. `system.query.history` contains a `statement_text` column with literal SQL, and it is deliberately excluded from the standard grant. You can opt in for query-level cost attribution, and OneLens can ingest the table with `statement_text` dropped at read time if you want the attribution without the text. Request redacted mode from .
What user identities and object names does OneLens actually ingest? `system.billing.usage` carries two metadata structs that OneLens reads, and it is worth being precise about them. `identity_metadata` contains user emails: `run_as` (who ran the workload), `owned_by` (the SQL warehouse owner), and `created_by` (for Databricks Apps and agents). This is what enables per-owner chargeback — attributing a runaway cluster to a person rather than a UUID. `usage_metadata` contains object names and IDs: `job_name`, `notebook_path`, `app_name`, `run_name`, `cluster_id`, `warehouse_id`, and — for materialized views specifically — `uc_table_catalog`, `uc_table_schema`, and `uc_table_name`. So OneLens can see that a job called `nightly_pii_reconciliation` ran a notebook at `/Repos/finance/etl/customer_merge`, who ran it, and what it cost. It cannot see a single row, column value, or line of code inside any of it. If your naming conventions themselves encode sensitive information, that is the one thing to weigh here. If your workspace is enabled for the FedRAMP compliance standard, Databricks replaces all non-null `identity_metadata` values with `__REDACTED__` before the data reaches the table, so per-user attribution is unavailable in those environments.
Does OneLens read our audit logs? No. `system.access.audit` is never granted. The `system.access` grant is scoped to a single table — `workspaces_latest` — which contains workspace IDs and display names used to label costs. Because the grant is issued at table level rather than schema level, audit logs and the lineage tables remain inaccessible even as new tables are added to that schema.
Do we need a separate integration for every workspace? No. System tables are account-scoped. `system.billing.usage` is global and covers every workspace in the account. `system.compute` and `system.lakeflow` are regional and cover every workspace in that region — including workspaces that are not themselves Unity Catalog-enabled. You need one integration per cloud region where you have a Unity Catalog metastore.
Why does the service principal need a SQL warehouse at all? System tables are Delta tables governed by Unity Catalog, not an API endpoint. Querying them requires SQL compute. The warehouse is the execution engine; OneLens stores nothing there and writes nothing back.
Will this affect our developers or our warehouse concurrency? No, if you use a dedicated 2X-Small warehouse — it is fully isolated from your users' compute. If you point OneLens at a shared warehouse, its queries are small, date-bounded, and run once daily, so the concurrency impact is negligible. A dedicated warehouse is still the cleaner choice because it makes the integration's own cost visible as a separate line.
Why don't OneLens figures match our Databricks invoice exactly? System tables report **list price** via `system.billing.list_prices`. They do not reflect DBCU commit discounts, private offers, or negotiated enterprise rates, and they cover only the Databricks platform charge — not the underlying cloud VM, disk, and networking cost for classic compute, which lands on your AWS, Azure, or GCP bill. Configure your commit discount in OneLens and connect your cloud provider account to close both gaps.
How much history will we get on first connection? Up to 12 months, bounded by Databricks' free retention period of 365 days for `system.billing.usage`. Some tables retain less — `node_timeline` keeps 90 days — while `list_prices` is retained indefinitely. If a system schema was only recently enabled, its history begins at the enablement date.
How do we rotate the client secret? A service principal can hold up to five OAuth secrets simultaneously, which makes rotation zero-downtime. Generate a new secret in the account console, update it in the OneLens integration form, confirm the next poll succeeds, then delete the old secret.
How do we disconnect OneLens? Remove the integration in OneLens, then revoke access in Databricks: ```sql REVOKE SELECT ON SCHEMA system.billing FROM ``; REVOKE USE SCHEMA ON SCHEMA system.billing FROM ``; REVOKE SELECT ON SCHEMA system.compute FROM ``; REVOKE USE SCHEMA ON SCHEMA system.compute FROM ``; REVOKE SELECT ON SCHEMA system.lakeflow FROM ``; REVOKE USE SCHEMA ON SCHEMA system.lakeflow FROM ``; REVOKE SELECT ON TABLE system.access.workspaces_latest FROM ``; REVOKE USE SCHEMA ON SCHEMA system.access FROM ``; REVOKE USE CATALOG ON CATALOG system FROM ``; ``` Then remove `CAN USE` on the warehouse and delete the service principal in the account console.
Can we review exactly what OneLens queried? Yes. Every statement the service principal runs appears in your own `system.query.history` and in your audit logs, attributed to the service principal's identity. You can audit the integration's activity without OneLens having any access to those records itself.
## Need Help? **Databricks Documentation:** * [System tables reference](https://docs.databricks.com/aws/en/admin/system-tables/) * [Billable usage system table reference](https://docs.databricks.com/aws/en/admin/system-tables/billing) * [Pricing system table reference](https://docs.databricks.com/aws/en/admin/system-tables/pricing) * [Authorize service principal access to Databricks with OAuth](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m) * [Manage privileges in Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/) * [Monitor costs using system tables](https://docs.databricks.com/aws/en/admin/usage/system-tables) **OneLens Support:** --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.onelens.cloud/integrations/cloud-and-cost-sources/connecting-to-databricks.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.