Connecting to Databricks

Connect to Databricks

Connect your Databricks account to OneLens to import usage and billing data for cost analysis, anomaly detection, and optimization insights.

Before you begin

Ensure the following:

• You have Account Admin access in Databricks

• Your workspace is Unity Catalog-enabled

• You can create Service Principals and SQL Warehouses

Required Permissions

The Service Principal used for this integration must have the following permissions:

SQL Warehouse

• CAN USE on the SQL Warehouse

Unity Catalog (System Tables)

Grant Data Reader access on:

• system.billing

• system.compute

• system.access

---

Configuring Databricks

Complete the following steps in your Databricks account before integrating with OneLens

1

Collect account and workspace details

• Log in to the Databricks console

• Copy your Databricks Account ID

• Navigate to Workspaces and select your workspace

• Copy the Workspace URL

• Open the workspace

2

Create a Service Principal

• Go to: Settings → Identity and access → Service principals

• Click Add service principal

• Enter a name (for example, onelens-integration-sp) & click on add

• Open the Service Principal and go to the Secrets tab

• Click Generate secret. When the secret expires, you’ll need to create a new one and reconfigure integration with OneLens.

• Copy the following and save to use in OneLens :

  • Client ID

  • Client Secret

3

Create a SQL Warehouse

• Navigate to: SQL → SQL Warehouses

• Click Create warehouse

• Configure:

  • Type: Serverless

  • Size: 2X-Small

• Create the warehouse

• Copy the SQL Warehouse ID

4

Grant warehouse access

• Open the SQL Warehouse

• Go to Permissions

• Add the Service Principal

• Grant CAN USE permission

5

Grant system table access

• Navigate to Catalog → system

• Grant Data Reader access to the Service Principal on:

  • system.billing

  • system.compute

  • system.access

---

Connecting Databricks in OneLens

Complete the following steps in OneLens.

1

Open the integration flow

1. Navigate to OneLens → Integrations → Databricks

2. Click Connect Databricks

2

Enter account details

1. Integration Name

2. Databricks Account ID

3. Workspace URL

4. SQL Warehouse ID

3

Enter authentication details

Provide Service Principal credentials :

• Client ID

• Client Secret

Your client secret is encrypted at rest and never exposed to end users. OneLens only accesses billing and metadata tables. We do not access query results or customer datasets.

4

Validate Connection

OneLens will automatically verify:

• Authentication

• SQL Warehouse access

• System table permissions

• Ability to query usage data

Once all checks pass, click Connect.

---

Data handling and security

• OneLens reads only billing and usage metadata from Databricks system tables

• No access to query results or user data

• Credentials are encrypted at rest

---

Data availability

• Data ingestion begins immediately after connection

• Historical data is imported based on availability in system tables

• Data is refreshed continuously

---

Troubleshooting

Authentication failed

Verify the Client ID and Client Secret

Warehouse access denied

Ensure the Service Principal has CAN USE permission

Missing system table permissions

Ensure Data Reader access is granted on all required schemas

No data available

Confirm Unity Catalog is enabled and system tables contain data