Recipe: Set up an A/B Test Using Pendo – Pendo Help Center

Overview

An A/B test is an experimentation tool used when you want to measure an effect of a certain change or intervention. For example, you may want to see whether a Guide that introduces a new feature will lead to a higher percent of visitors clicking the feature after launch. This recipe is intended to walk through the considerations and steps of creating an A/B test using Pendo sample groups.

Note: You can repurpose this recipe’s Sample Groups to do an A/B/C test (3 Groups), but be mindful of the size of the groups. By creating your segments on the Visitor’s page, you can easily see the number of users in your segments to try to keep your groups as equal as possible.

Ingredients (What Do I Need?)

Pendo Admin Permissions or the following:
- User permission
- User permission to create Guides
- User permission to publish Guides

How to Make It

In this Recipe, you will create 2 custom segments (which are your 2 test groups) using Pendo’s sample groups feature. To create your segments:

Step 1. Navigate to the “Visitors” section in Pendo.

Step 2. Create two segments to target the right test groups by using “AND” rules with Sample Groups. (Learn how “Sample Groups" work) To create a 50/50 test, use the following Sample Group rules:

Segment 1: Sample Group is less than or equal to 49 (This includes groups 0-49)
Segment 2: Sample Group is greater than or equal to 50 (This includes groups 50-99)

Tip: Name one segment (Control) and one (Test). The control group will not receive a Guide introducing a certain feature, while the Test group will receive a Guide.

Step 3. Create a Pendo Guide and set the segment to the Test group you just created.

Note: When you launch the guide, guide should be automatically activated next time the user is in the app (If you’re launching an automatic Guide, make sure to update Guide throttling settings appropriately.

Best practices for Setting Up Sample Groups

How do you know if your sample size is large enough?

A sample that is too small will be prone to outliers and the data that results from it may become skewed. The larger the group, the better. But, particular numbers will depend on the volume of visitors, their frequency, and time spent on site.

How do you know if your groups are equal?

It’s highly recommended to analyze the different types of visitor attributes and understand the averages of login frequency, event frequency, time since first login, and other standard actions to know that the groups are similar in scope. If you find any significant outliers that could skew the data (internal accounts, QA testing bots), please utilize the Exclude List feature or add additional segment conditions to exclude them.

How do I analyze the data afterward?

Pendo provides a variety of different ways to help you compare actions, trends, interactions with features and guides by utilizing the segment dropdown.

Learn more on all analytics options