Set up Data Sync with Google Cloud

Pendo Data Sync provides the ability to export Pendo data (excluding Pendo Feedback data) to your Google Cloud Storage (GCS) account. This article describes how to set up Data Sync with GCS.

Prerequisites

• Data Sync is a paid feature. Contact your Pendo representative for access.
• You must be a Pendo subscription admin or have the Configure Data Sync permission to set up a cloud storage destination as part of this process.
• The user must have storage.objects.create, storage.objects.delete, and storage.objects.update permissions in GCS.

Setup overview

Data Sync GCS is set up in Pendo and configured in your Google Cloud environment, and involves the following steps.

1. Create a Google Cloud Platform (GCP) service account.
2. Generate a JSON key file for your service account.
3. Create a GCS bucket.
4. Give the GCS bucket access to your GCP service account.
5. Choose a provider in Pendo.
6. Enter your service account key and bucket URL in Pendo.

Step 1. Create a GCP service account

If you don’t have one already, create a Google Cloud Platform (GCP) service account in your Google Cloud environment.

1. In the Google Cloud console, go to IAM & Admin > Service Accounts and select Create service account.
2. Add your service account details, including service account name and description.
3. Optionally, choose an access level for your service account.
4. Select Done. This creates a service account and role on the IAM Service Accounts page.
5. Copy the service account name, which is in email format. Save this for use in Step 4 of this article.

For more information and instructions, see Google’s Create and manage service account keys.

Step 2. Generate a JSON key file for your service account

Regardless of whether you’re creating a new service account for Data Sync, or using an existing one, you must generate a JSON key file to add to your Data Sync destination in Pendo. To generate this key:

1. In GCP, go to the IAM Service Accounts page and find the service account you want to use for Data Sync.
2. Under the Actions column, select the vertical dots next to your service account and select Manage keys.
3. Select Create new key from the ADD KEY dropdown menu.
4. Select the JSON key type and then CREATE.
5. This generates a message that tells you that your key has been saved to your computer, which you can then close.

Your key is now on your computer. Ensure you save your key for use in Step 6 of this article.

Step 3. Create a GCS bucket

Pendo-exported data is copied to a GCS bucket. To create a GCS bucket:

1. In Cloud Storage, go to Buckets and select + CREATE at the top of the page.
2. Create a meaningful name for your bucket.
3. Follow the Google workflow to choose a location and storage class for your data, how to control access to objects, and how to protect object data.
4. Select CREATE.
5. This generates a message that tells you that public access is prevented. Select CONFIRM.

You’re then taken to your empty bucket in Google Cloud.

Step 4. Give your GCS bucket access to your service account

Grant Storage Admin access to the service account by making your GCP service account a member of the GCS bucket.

1. In Cloud Storage, find your Data Sync bucket in Buckets, and either:
   • Open your Data Sync bucket, select the PERMISSIONS tab at the top and then GRANT ACCESS.
   • Select the vertical dots next to your Data Sync bucket in the Buckets page and choose Edit access.
2. Grant access to your bucket by selecting Add principal and pasting your service account name, which you copied as part of Step 1 in this article.
3. Under Select a role, select Cloud Storage > Storage Admin. This grants full control of GCS resources. Alternatively, create your own role, with specific requirements (storage.objects.create, storage.objects.delete, and storage.objects.update).
4. Select SAVE and verify that your service account has Storage Admin access to your Data Sync bucket.

Step 5. Choose a provider in Pendo

Before you can send data to your storage bucket, you must set a target destination in Pendo.

1. In Pendo, go to Settings > Data Sync.
2. Select Connect to destination.
3. Hover over the Google Cloud Storage tile in the panel that slides out on the right, and then select Choose destination. The panel then summarizes the next steps for you.

Step 6. Configure your GCS account destination in Pendo

The final step is to upload the service account key and provide the bucket URL to Pendo.

1. Enter your bucket URL. You can find this in the CONFIGURATION tab of your Data Sync bucket created in Google Cloud Storage > Buckets. Copy the URL in the gsutil URI row of the Overview table.
2. Select Upload JSON file and upload the JSON key file that was created and downloaded in Step 2 of this article.
3. If you want to manually enter the service account and secret key:
   • Enter the service account email name that corresponds to the JSON key file.
   • Enter the private_key value from the service account JSON key.
     
     
4. Select Validate destination to test the credentials you entered. The validation result appears below the button. If validation is unsuccessful, review the displayed message or select View error when available, correct the destination information, and try again.
5. When successfully validated, select Create destination.
   
   

Rotate your credentials

There are two ways to update your credentials after you've configured a destination in Pendo:

• Manually rotate credentials through the Pendo UI.
• Automatically rotate your credentials through the public API.

To use the public API endpoint for rotating Data Sync credentials, you must have an integration key with write access.

Through the Pendo UI

1. In Pendo, go to Settings > Data Sync.
2. If you have more than one destination, select this GCS destination at the top of the page.
3. Select Manage destination in the top-right corner of the page.
4. Provide new credentials.

Through the public API

1. Generate a new Pendo integration key using the instructions outlined in the Pendo integration key article.
2. In our public API documentation, go to Data Sync.
3. Choose the PUT endpoint that corresponds to your cloud storage destination.
4. Follow the instructions on how to use the public API endpoint to rotate your cloud storage credentials.

Update your cloud storage destination

You have the option to move your cloud storage destination from one GCS bucket to another, or to change your cloud storage service to another provider, such as Microsoft Azure or Amazon S3.

Update your GCS bucket

If you want to change from one GCS bucket to another existing GCS bucket, update the bucket URL in Pendo. A subscription can have only one customer-owned GCS destination, so update the existing destination rather than adding a second one.

1. In Pendo, go to Settings > Data Sync.
2. If you have more than one destination, select this GCS destination at the top of the page.
3. Select Manage destination in the top-right corner of the page.
4. Replace the entry in the Bucket URL field with the name of the new GCS bucket that you'd like to use for Data Sync.
5. Select Update destination.

Change your cloud storage provider

To move from GCS to a different destination type, such as Amazon S3, Microsoft Azure Storage, or a warehouse destination, add the new destination while this GCS destination is still operational. Don't delete the GCS destination first.

1. Set up the new destination by following the setup article for that destination type:
   • Set up Data Sync with Amazon S3 using user access keys
   • Set up Data Sync with Amazon S3 using IAM roles
   • Set up Data Sync with Microsoft Azure Storage
   • Set up Data Sync with a Pendo-hosted GCS destination
2. Create the exports you need on the new destination. Settings and recurring exports aren't copied from the GCS destination.
3. Confirm that exports are arriving in the new destination.
4. Delete the GCS destination:
   1. In Pendo, go to Settings > Data Sync.
   2. Select this GCS destination at the top of the page.
   3. Select Manage destination in the top-right corner of the page.
   4. Select Delete destination.
   5. Confirm that you want to delete the destination.

Deleting a destination automatically turns off the recurring exports and ongoing metadata updates for that destination and cancels any of its in-progress exports. Other configured destinations continue to sync.

A subscription can have only one destination of each type. Pendo-hosted and customer-owned GCS count as different types. For more information about Pendo-hosted GCS, contact your Pendo account team.

Next steps

After the destination is created, create exports and build the pipeline that loads them:

• To create and load event exports, see Data Sync event export handling.
• To create and load account and visitor exports, see Data Sync account and visitor export handling.
• For the fields in each exported file, see Data Sync schema definitions.