# Integrating with Databricks

[Prerequisites](#prerequisites)

[Databricks Guide](#databricks-guide)\
   [Create Personal Access Token](#step-1-create-personal-access-token)\
   [Create and Connect a new SQL Warehouse](#step-2-create-and-connect-a-new-sql-warehouse)\
   [Connect an existing SQL warehouse or all-purpose compute](#step-3-connect-an-existing-sql-warehouse-or-all-purpose-compute)

## Prerequisites

To connect Databricks to Drivetrain, you need:

1. An active Databricks admin account
2. At least one [SQL warehouse](https://docs.databricks.com/en/compute/sql-warehouse/index.html) or a compute cluster to sync data from your catalog.

## Databricks Guide <a href="#databricks-guide" id="databricks-guide"></a>

### Step 1: Create Personal Access Token <a href="#step-1-create-personal-access-token" id="step-1-create-personal-access-token"></a>

1. Drivetrain uses a secure token to connect to Databricks. You can read more about the Databricks token management guide [here](https://docs.databricks.com/en/dev-tools/auth/pat.html)
2. In your Databricks workspace, click your Databricks username in the top bar, and then select **Settings** from the drop down.
3. Click **Developer**.
4. Next to **Access tokens**, click **Manage**.
5. Click **Generate new token**.
6. (Optional) Enter a comment that helps you to identify this token in the future, and change the token’s default lifetime of 90 days. To create a token with no lifetime (not recommended), leave the **Lifetime (days)** box empty (blank).
7. Click **Generate**.
8. Copy the displayed token to a secure location, and then click **Done**.

{% hint style="info" %}
Please note during the data sync, if we find a table which doesn’t have ChangeDataFeed enabled, we try to activate it. Make sure the personal access token has MODIFY permissions on the table. The command to enable ChangeDataFeed for a table is

&#x20;`ALTER TABLE catalog_name.schema_name.table_name SET TBLPROPERTIES (delta.enableChangeDataFeed=true)`
{% endhint %}

### Step 2: Create and Connect a new SQL Warehouse <a href="#step-2-create-and-connect-a-new-sql-warehouse" id="step-2-create-and-connect-a-new-sql-warehouse"></a>

{% hint style="warning" %}
If there is already an unity catalog enabled SQL warehouse or an all-purpose compute present in your Databricks setup, please jump to step 3
{% endhint %}

1. In the Databricks console, go to **SQL** > **SQL warehouses** > **Create SQL warehouse**

<figure><img src="https://4220329029-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FmBZR0D9lq6jrI895OuOU%2Fuploads%2FiPDErzV3f7TliFxizviO%2FDB-1.avif?alt=media&#x26;token=e6d38bb5-f169-4043-b4ae-d3fcf9796405" alt=""><figcaption></figcaption></figure>

2. In the New SQL warehouse window, enter a **Name** for your warehouse
3. Choose your **Cluster Size** and configure the other warehouse options
4. Click **Create**

<figure><img src="https://4220329029-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FmBZR0D9lq6jrI895OuOU%2Fuploads%2F5hgj3mAT9G3PorPzIFLg%2FDB-2.avif?alt=media&#x26;token=3f50a2a6-0ffd-476d-952f-41503c16532d" alt=""><figcaption></figcaption></figure>

5. After creating the warehouse, Go to the **Connection details** tab

<figure><img src="https://4220329029-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FmBZR0D9lq6jrI895OuOU%2Fuploads%2FQ8rcnCdYZWgBTFR95q1w%2FDB-3.avif?alt=media&#x26;token=d3990c31-50d9-43e6-bd93-12ba0588a0b0" alt=""><figcaption></figcaption></figure>

6. Make a note of the following values and keep them in the secured location along with personal access token.
   1. Server Hostname
   2. Port
   3. HTTP Path

### Step 3: Connect an existing SQL warehouse or all-purpose compute <a href="#step-3-connect-an-existing-sql-warehouse-or-all-purpose-compute" id="step-3-connect-an-existing-sql-warehouse-or-all-purpose-compute"></a>

1. Go to your compute cluster or warehouse.
2. Click **Configuration** > **Advanced Options** > **JDBC/ODBC**

<figure><img src="https://4220329029-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FmBZR0D9lq6jrI895OuOU%2Fuploads%2FHNKNotC8a7pjHbCLtws8%2FDB-4.avif?alt=media&#x26;token=683a931b-cf80-4d8e-a7b4-7f9459a16031" alt=""><figcaption></figcaption></figure>

3. Make a note of the following values and keep them in the secured location along with personal access token.
   1. Server Hostname
   2. Port
   3. HTTP Path

{% hint style="info" %}
That’s it, share server hostname, port, HTTP path & personal access token to your customer success manager.
{% endhint %}