Integrating with Databricks

Prerequisites

Databricks Guide Create Personal Access Token Create and Connect a new SQL Warehouse Connect an existing SQL warehouse or all-purpose compute

Prerequisites

To connect Databricks to Drivetrain, you need:

1. An active Databricks admin account

2. At least one SQL warehouse or a compute cluster to sync data from your catalog.

Databricks Guide

Step 1: Create Personal Access Token

1. Drivetrain uses a secure token to connect to Databricks. You can read more about the Databricks token management guide here

2. In your Databricks workspace, click your Databricks username in the top bar, and then select Settings from the drop down.

3. Click Developer.

4. Next to Access tokens, click Manage.

5. Click Generate new token.

6. (Optional) Enter a comment that helps you to identify this token in the future, and change the token’s default lifetime of 90 days. To create a token with no lifetime (not recommended), leave the Lifetime (days) box empty (blank).

7. Click Generate.

8. Copy the displayed token to a secure location, and then click Done.

Please note during the data sync, if we find a table which doesn’t have ChangeDataFeed enabled, we try to activate it. Make sure the personal access token has MODIFY permissions on the table. The command to enable ChangeDataFeed for a table is

ALTER TABLE catalog_name.schema_name.table_name SET TBLPROPERTIES (delta.enableChangeDataFeed=true)

Step 2: Create and Connect a new SQL Warehouse

If there is already an unity catalog enabled SQL warehouse or an all-purpose compute present in your Databricks setup, please jump to step 3

1. In the Databricks console, go to SQL > SQL warehouses > Create SQL warehouse

1. In the New SQL warehouse window, enter a Name for your warehouse

2. Choose your Cluster Size and configure the other warehouse options

3. Click Create

1. After creating the warehouse, Go to the Connection details tab

1. Make a note of the following values and keep them in the secured location along with personal access token.

   1. Server Hostname

   2. Port

   3. HTTP Path

Step 3: Connect an existing SQL warehouse or all-purpose compute

1. Go to your compute cluster or warehouse.

2. Click Configuration > Advanced Options > JDBC/ODBC

1. Make a note of the following values and keep them in the secured location along with personal access token.

   1. Server Hostname

   2. Port

   3. HTTP Path

That’s it, share server hostname, port, HTTP path & personal access token to your customer success manager.