Pendo Data Sync delivers Pendo product-usage data and account and visitor metadata to a data destination that your organization controls. Use it to combine Pendo data with your other business data for reporting, customer-health analysis, churn modeling, and other business intelligence and data-science use cases.
Data Sync is a scheduled, one-way export from Pendo. It doesn't import data into Pendo and isn't a real-time stream.
Prerequisites
You must be a Pendo subscription admin or have the Configure Data Sync permission to set up Data Sync. Contact your Pendo representative for access.
You can also evaluate Data Sync with a data estimate or a test export before setting up recurring exports.
Choose how to receive Pendo data
Data Sync supports two delivery methods. Both include the same underlying Pendo dataset, but differ in where the data lands and the pipeline work your organization owns.
| Delivery method | Destinations | What Pendo delivers | What your organization manages |
|---|---|---|---|
| Warehouse | Snowflake, Databricks, or BigQuery | Tables created and synced directly in your warehouse | Destination access, warehouse configuration, storage, and compute |
| Cloud storage | S3, Google Cloud Storage (GCS), Azure Storage, or Pendo-hosted GCS | Avro files and an export manifest JSON file | Storage configuration and, for customer-owned storage, the ETL or ELT pipeline that loads the files into your data platform |
Choose a warehouse destination when you want Pendo to create and maintain queryable tables in a supported warehouse. Choose cloud storage when your organization wants to control how the exported files are transformed and loaded.
Choose the data to sync
After you configure a destination, choose the Pendo data to deliver:
- Event data is configured at the application level. You choose which applications to include.
- Account and visitor metadata is optional and configured at the subscription level.
For the available tables and fields, see Data Sync schema definitions.
Warehouse destinations
Data Sync can create and sync Pendo tables directly in Snowflake, Databricks, or BigQuery. After you configure the destination, choose the applications and optional account and visitor metadata to sync.
Note: Data Sync to Databricks and BigQuery is currently available for Pendo subscriptions in the US and EU regions only.
Use the setup article for your destination:
- Set up Pendo Data Sync to Snowflake
- Set up Pendo Data Sync to Databricks
- Set up Pendo Data Sync to BigQuery
For Snowflake data-flow and operational details, see Data Sync to Snowflake architecture.
Cloud storage destinations
Data Sync can deliver Avro files to Amazon S3, Google Cloud Storage, or Microsoft Azure Storage. Each export also includes an export manifest JSON file that your organization uses to process the exported files.
For setup instructions, see:
- Set up Data Sync with Google Cloud
- Set up Data Sync with Amazon S3 using user access keys
- Set up Data Sync with Amazon S3 using IAM roles
- Set up Data Sync with Microsoft Azure Storage
- Set up Data Sync with a Pendo-hosted Google Cloud Storage destination
Cloud storage destinations support:
- Recurring daily exports.
- Backfills of up to three calendar years of historical data.
- Automatic updates when Page or Feature rules are added or updated in Pendo.
Build a pipeline from cloud storage to your data warehouse
Cloud storage destinations deliver files rather than queryable warehouse tables. Assign a data engineer to build and maintain an ETL or ELT pipeline that reads the exports from cloud storage and loads them into your data warehouse.
After you configure the destination, create the exports that the pipeline consumes. Event exports are configured at the application level and can run once or recur daily. Account and visitor metadata exports are configured at the subscription level and can run once or continue with daily updates.
At a high level, the pipeline must:
- Monitor each export manifest and process only completed exports that haven't already been loaded.
- Use the bill of materials to locate the Avro files and their schemas, then create or update the corresponding warehouse tables.
- Apply replacement exports correctly. Replace event data for the affected UTC period, and replace account or visitor records with matching IDs.
- Process retroactive event exports after Page or Feature rules change.
- Monitor failed loads and retry them without creating duplicate data.
For implementation details, see:
For performance, we recommend checking for new exports hourly, loading files in parallel, and partitioning event data by UTC day.
Estimate Data Sync volume
If you want to understand how much data can be synced before setting up Data Sync, run a data estimate for accounts, visitors, and application-level event data.
- In Pendo, go to Settings > Data Sync.
- Select Run data estimate.
- Select the sources to estimate.
- Select Run estimate.
The estimate runs in the background. Pendo emails you when it is ready.
For each selected application, the estimate includes one day of event data in cloud storage and a 12-month historical backfill. For accounts and visitors, it includes a full sync of account and visitor metadata. Snowflake estimates also include estimated Snowflake storage based on an approximate 50% compression rate. Actual compression can vary.
Create a test export
If you aren't yet a Data Sync customer, you can create a single test export containing one day of event data. Test exports help your data-engineering team inspect the Avro files and plan a cloud storage ETL pipeline. They aren't available for account or visitor metadata and aren't available for warehouse destinations at this time.
To create one, go to Settings > Data Sync, set up a supported cloud storage destination, create an export, and select Test export.
Understand when syncs run
Data Sync exports event data in whole UTC calendar days. Every event record belongs to a UTC day from 00:00 through 23:59 UTC, regardless of your subscription's configured time zone. This keeps historical partitions stable across exports.
Recurring event exports run nightly after midnight in your subscription's local time zone. Account and visitor metadata syncs run on a separate 24-hour cycle. For new Databricks and BigQuery destinations, the first recurring sync begins overnight after the destination is configured and data is selected. Subsequent syncs run approximately every 24 hours.
Subscriptions in time zones ahead of UTC can see their latest exported day land about two calendar days before the sync date. The scheduled sync can begin after local midnight before the current UTC day has ended.
Subscriptions at or behind UTC run late enough in the UTC day that the prior UTC day is usually complete. Their most recent exported day is approximately one day before the sync date.
Some time zones move between UTC and UTC+1 for daylight saving time. The most recent exported day can change with the seasonal time change.
Note: We don't recommend changing your subscription time zone to change Data Sync day labeling. A time-zone change affects day boundaries throughout Pendo and begins a full reprocessing of the subscription's data.