Set up Pendo Data Sync to Databricks

Use this article to connect Pendo Data Sync to a Databricks SQL warehouse. Pendo creates the target schema you specify and syncs Pendo tables directly to it.

Pendo copies the data into your Databricks workspace. Your organization owns and manages the copied data and is responsible for the associated Databricks storage and compute costs.

Note: Data Sync to Databricks is currently available for Pendo subscriptions in the US and EU regions only.

Prerequisites

Before you begin, make sure that you:

• Are a Pendo subscription admin or have the Configure Data Sync permission.
• Can create or use a Databricks SQL warehouse and connection identity.
• Can grant the connection identity access to the target warehouse, catalog, and schema.
• Know whether your workspace uses Unity Catalog or Hive Metastore. Unity Catalog is the default. Hive Metastore is also supported and requires staging-bucket credentials.

We recommend OAuth client credentials with a Databricks service principal for production connections. Personal access token (PAT) authentication is also supported.

Step 1. Create or select a SQL warehouse

1. In Databricks, go to SQL Warehouses.
2. Create a SQL warehouse for Pendo Data Sync, or select an existing warehouse that the connection identity can use.
3. If you use Unity Catalog, confirm that the warehouse is configured to use Unity Catalog.
4. Keep the warehouse available while you configure access. You collect its connection details later in this article.

Step 2. Configure OAuth client credentials

1. In Databricks, go to Settings > Identity and access > Manage service principals.
2. Add a service principal dedicated to Pendo Data Sync.
3. Generate an OAuth secret for the service principal. Save the client ID and secret securely. The secret isn't displayed again.
4. Grant the service principal Can use permission on the SQL warehouse.
5. Grant USE CATALOG on the target catalog.
6. Grant the service principal access at a scope that covers the target schema Pendo creates. Grant either ALL PRIVILEGES or all of the following privileges:
   • USE SCHEMA
   • SELECT
   • APPLY TAG
   • WRITE VOLUME
   • MODIFY
   • CREATE MATERIALIZED VIEW
   • READ VOLUME
   • CREATE TABLE
   • CREATE VOLUME

Keep the client ID and secret available for the Pendo destination form.

Note: Missing USE CATALOG is the most frequent cause of test-connection failures.

Step 3. Configure a personal access token

If you use PAT authentication instead of OAuth, create a token for the Databricks identity used by Data Sync. A dedicated service-principal token is preferable to an individual employee's token because its lifecycle isn't tied to one employee.

1. In your Databricks workspace, go to Settings > Developer > Access tokens.
2. Create a token with an appropriate name and lifetime. If your workspace uses scoped tokens, include the sql scope required for SQL warehouse access.
3. Copy and store the token securely. It isn't displayed again.
4. Confirm that the token's identity has Can use on the SQL warehouse, USE CATALOG on the target catalog, and the required schema-level privileges.

For additional instructions, including creating a PAT for a service principal, see Authenticate with Databricks personal access tokens.

You enter the token in Pendo as the Personal access token.

Step 4. Collect the Databricks connection details

In the Databricks SQL warehouse connection details, collect:

• Server hostname
• Port, typically 443
• Catalog name
• HTTP path

Also choose the name of the schema that Pendo should create.

Step 5. Add the Databricks destination in Pendo

1. Open the Data Sync setup flow in Pendo and select Databricks as the destination.
2. Enter the destination details:
   • Server hostname
   • Port
   • Schema
   • Catalog
   • HTTP path
3. For Auth method, choose one of the following:
   • Client Credentials (recommended). Enter the OAuth client ID and client secret for the Databricks service principal.
   • PAT. Enter the personal access token.
4. For Metastore, select Unity Catalog or Hive Metastore.
5. For Use SSH tunnel, select the setting required by your organization's network configuration.
6. If you selected Hive Metastore, enter credentials for a staging bucket that the connection identity can write to and delete from. Pendo removes staged transfer files after they are loaded. You can also configure a two-day lifecycle rule as a safeguard.
7. Validate the connection, then continue to Select data.

Step 6. Select data to sync

Choose the Pendo applications to sync. You can also choose account and visitor metadata. Data selection is optional during initial setup and can be completed later.

The first recurring sync begins overnight after the destination is configured and data is selected. Subsequent syncs run approximately every 24 hours.

Each recurring sync incrementally loads new and updated rows. Full reloads aren't required.

For the available tables and fields, see Data Sync schema definitions and the Databricks ERD.

Troubleshooting