Set up Data Sync with Amazon S3 using user access keys

Pendo Data Sync provides the ability to export Pendo data (excluding Pendo Feedback data) to your Amazon S3 account. This article describes how to set up Data Sync with Amazon S3 using IAM user access keys. Alternatively you can set up Data Sync with Amazon S3 using IAM roles.

Prerequisites

• Data Sync is a paid feature. Contact your Pendo representative for access.
• You must be a Pendo subscription admin or have the Configure Data Sync permission to set up a cloud storage destination as part of this process.
• The user must have putObject permissions in S3.

Setup overview

Data Sync with Amazon S3 is set up in Pendo and configured in your S3 account. Setting up Data Sync with Amazon S3 using IAM user access keys involves the following steps:

1. Create an S3 bucket.
2. Create a folder in your S3 bucket.
3. Generate an access key for your service account.
4. Choose a provider in Pendo.
5. Enter your service account access key and bucket URL in Pendo.

For a production setting, you should create a service account that isn't tied to an individual person. For internal development or testing, you can use your AWS personal account so long as it's tied to the Pendo organization.

Step 1. Create an S3 bucket

Pendo data is copied to an S3 bucket. To create an S3 bucket:

1. Go to the AWS Management Console.
2. Open the Amazon S3 Console and choose Buckets.
3. Choose Create bucket at the top right of the page.
4. Create a meaningful name for your S3 bucket.
5. Follow the workflow to set the appropriate settings for your S3 bucket.
6. Select Create bucket at the bottom of the form.

Step 2. Create a folder in your S3 bucket

While in the bucket you just created for Step 1:

1. Select Create folder.
2. Create a meaningful name for your folder.
3. Follow the workflow to set the appropriate settings for your S3 folder.
4. Select Create folder at the bottom of the form.

Step 3. Generate an access key

If you don't already have an access key associated with your account, you must generate one to add to your Data Sync destination in Pendo. To generate this key:

1. Go to the AWS Management Console.
2. Select your account from the top right and select Security Credentials.
3. Under Access Keys, select Create access key.
4. Enter a description tag value and then select Create access key.
5. Save the secret access key provided or download the .csv file with the credentials. You won't see this again. You need this for Step 5.
6. Select Done.

Step 4. Choose a provider in Pendo

Before you can send data to your storage bucket, you must set a target destination in Pendo.

1. In Pendo, go to Settings > Data Sync.
2. Select Connect to destination.
3. Hover over the Amazon Simple Storage Service (Amazon S3) tile in the panel that slides out on the right.
4. Select Choose destination. The panel then summarizes the next steps for you.

Step 5. Configure your S3 account destination in Pendo

The final step is to upload the service account key and provide the bucket name to Pendo.

1. Enter your bucket URL. You can find this in the Properties tab of your Data Sync bucket created in S3 > Buckets > [Bucket Name] > [Folder Name]. Copy the URL in the S3 URI column of the Folder overview table.
2. Enter your secret access key. You should have this saved from Step 3, either as a CSV file or manually copied.
   • If you downloaded the CSV file with your credentials, select Upload CSV file to upload it here.
   • If you want to manually enter the service account and secret key, first enter your Access Key, which is found in IAM > Security Credentials > Access Keys. Then, enter the Secret Access Key you saved in Step 3.
3. Select Validate destination to test the credentials you entered. The validation result appears below the button. If validation is unsuccessful, review the displayed message or select View error when available, correct the destination information, and try again.
4. When successfully validated, select Create destination.

Rotate your credentials

There are two ways to update your credentials after you've configured a destination in Pendo:

• Manually rotate credentials through the Pendo UI.
• Automatically rotate your credentials through the public API.

To use the public API endpoint for rotating Data Sync credentials, you must have an integration key with write access.

Through the Pendo UI

1. In Pendo, go to Settings > Data Sync.
2. If you have more than one destination, select this Amazon S3 destination at the top of the page.
3. Select Manage destination in the top-right corner of the page.
4. Provide new credentials.

Through the public API

1. Generate a new Pendo integration key using the instructions outlined in the Pendo integration key article.
2. In our public API documentation, go to Data Sync.
3. Choose the PUT endpoint that corresponds to your cloud storage destination.
4. Follow the instructions on how to use the public API endpoint to rotate your cloud storage credentials.

Update your cloud storage destination

You have the option to move your cloud storage destination from one S3 bucket to another, or to change your cloud storage service to another provider, such as Google Cloud Storage (GCS) or Microsoft Azure.

Update your S3 bucket

If you want to change from one S3 bucket to another existing S3 bucket, update the S3 URI field in Pendo. A subscription can have only one Amazon S3 destination, so update the existing destination rather than adding a second one.

1. In Pendo, go to Settings > Data Sync.
2. If you have more than one destination, select this Amazon S3 destination at the top of the page.
3. Select Manage destination in the top-right corner of the page.
4. Replace the entry in the S3 URI field with the name of the new S3 bucket that you'd like to use for Data Sync.
5. Select Update destination.

Change your cloud storage provider

To move from Amazon S3 to a different destination type, such as Google Cloud Storage, Microsoft Azure Storage, or a warehouse destination, add the new destination while this Amazon S3 destination is still operational. Don't delete the Amazon S3 destination first.

1. Set up the new destination by following the setup article for that destination type:
   • Set up Data Sync with Google Cloud
   • Set up Data Sync with Microsoft Azure Storage
   • Set up Data Sync with a Pendo-hosted GCS destination. For more information, contact your Pendo account team.
2. Create the exports you need on the new destination. Settings and recurring exports aren't copied from the Amazon S3 destination.
3. Confirm that exports are arriving in the new destination.
4. Delete the Amazon S3 destination:
   1. In Pendo, go to Settings > Data Sync.
   2. Select this Amazon S3 destination at the top of the page.
   3. Select Manage destination in the top-right corner of the page.
   4. Select Delete destination.
   5. Confirm that you want to delete the destination.

Deleting a destination automatically turns off the recurring exports and ongoing metadata updates for that destination and cancels any of its in-progress exports. Other configured destinations continue to sync.

To use IAM roles instead of user access keys, see Set up Data Sync with Amazon S3 using IAM roles.

Next steps

After the destination is created, create exports and build the pipeline that loads them:

• To create and load event exports, see Data Sync event export handling.
• To create and load account and visitor exports, see Data Sync account and visitor export handling.
• For the fields in each exported file, see Data Sync schema definitions.