An A/B test is an experimentation tool you use when you want to measure an effect of a certain change or intervention. For example, you might want to see whether a guide that introduces a new Feature leads to a higher percent of visitors using the Feature after launch.
This article details the considerations and steps of creating an A/B test using Pendo sample groups.
Note: You can repurpose the sample groups created here to do an A/B/C test (three groups), but be mindful of the size of each group. By creating your segments on the Segments Page, you can easily see the number of users in your segments to try and keep your groups as equal as possible.
Understand segment groups
Sample groups in Pendo allow you to perform A/B tests by targeting random subsets of visitors. You can use them to test different guide designs, rotate random groups for user research, or limit the number of visitors eligible for an experiment.
The Sample Group segment rule assigns a static value (0 to 99) to each visitor when they're created. These values are evenly distributed across all visitors over time, ensuring fair randomization. For example, to target 25% of your user base, create two rules: Sample Group greater or equal to 50, AND Sample Group less or equal to 74.
You can view a visitor's assigned sample group value on their details page. To find it, go to People > Visitors, select their Visitor ID from the Visitors table, and open the Details section.
Create your segments and guide
- Navigate to People > Segments in Pendo.
- Create two segments to target the correct test groups by using “AND” rules with sample groups. To create a 50/50 test, use the following sample group rules, labeling one segment as Control and one as Test in the segment names.
-
Segment 1. Sample group is less than or equal to 49. This includes groups 0 to 49.
-
Segment 2. Sample group is greater than or equal to 50. This includes groups 50 to 99.
-
Segment 1. Sample group is less than or equal to 49. This includes groups 0 to 49.
- Create a guide and set the segment to the test segment you created in step 2. Ensure the guide is automatically activated for the next time your users are in your app. (If you're launching an automatic guide, ensure that your guide throttling settings are appropriately set.)
Best practices for setting up sample groups
When setting up these sample groups for your A/B test, consider the following recommendations:
- Ensure that your sample size is large enough. A sample that's too small is prone to outliers, and the data that results from it might become skewed. The larger the group, the better. However, particular numbers depend on the volume of visitors, their frequency, and their time spent on site.
- Ensure that your sample groups are equal. It’s highly recommended to analyze the different types of visitor attributes and understand the averages of login frequency, event frequency, time since first login, and other standard actions to know that the groups are similar in scope. If you find any significant outliers that could skew the data (such as internal accounts or QA testing bots), use the exclude list or add additional segment conditions to exclude these visitors or accounts.
- Analyze and monitor the data. Use Pendo analytics and filter by your sample group segments to help you compare actions, trends, and interactions with Features and guides.