Set up an A/B test using Pendo

Last updated:

An A/B test is an experimentation tool you use when you want to measure an effect of a certain change or intervention. For example, you might want to see whether a guide that introduces a new feature will lead to a higher percent of visitors using the feature after launch.

This article details the considerations and steps of creating an A/B test using Pendo sample groups. For more information about sample groups, see Segment sample groups.

Note: You can repurpose the sample groups created here to do an A/B/C test (three groups), but be mindful of the size of each group. By creating your segments on the Segments page, you can easily see the number of users in your segments to try and keep your groups as equal as possible. 

Create your segments and guide

  1. Navigate to People > Segments in Pendo.
  2. Create two segments to target the correct test groups by using “AND” rules with sample groups. To create a 50/50 test, use the following sample group rules, labeling one segment as Control and one as Test in the segment names.
    • Segment 1. Sample group is less than or equal to 49. This includes groups 0 to 49.

      SegmentBuilder_ControlSampleGroup.png

    • Segment 2. Sample group is greater than or equal to 50. This includes groups 50 to 99.

      SegmentBuilder_TestSampleGroup.png

  3. Create a guide, and set the segment to the test segment you created in step 2. Ensure the guide is automatically activated for the next time your users are in your app. (If you're launching an automatic guide, ensure that your guide throttling settings are appropriately set.)

Best practices for setting up sample groups

When setting up these sample groups for your A/B test, consider the following recommendations:

  • Ensure that your sample size is large enough. A sample that's too small is prone to outliers, and the data that results from it might become skewed. The larger the group, the better. However, particular numbers depend on the volume of visitors, their frequency, and their time spent on site.
  • Ensure that your sample groups are equal. It’s highly recommended to analyze the different types of visitor attributes and understand the averages of login frequency, event frequency, time since first login, and other standard actions to know that the groups are similar in scope. If you find any significant outliers that could skew the data (such as internal accounts or QA testing bots), use the Exclude List or add additional segment conditions to exclude these visitors or accounts.
  • Analyze and monitor the data. Use Pendo analytics and filter by your sample group segments to help you compare actions, trends, and interactions with Features and guides. Learn more about our analytics.
Was this article helpful?
6 out of 9 found this helpful