A/B tests are a type of experiment that allows you to test two different guide experiences in your app to see which one performs better against a specific goal. You can test variations in activation and display method across the same experiment.
Use cases
Each experiment compares two variants (guide versus no guide, or guide versus guide) against a shared conversion goal.
Use cases include:
- Testing different headlines or layouts to improve engagement.
- Comparing embedded versus overlay delivery.
- Using different activation styles for the same message.
Once active, the experiment controls each guide's status, schedule, and segment. The guide with the higher conversion rate is considered the winner, but you can manually choose which guide to promote.
Prerequisites
Before you can use guide experiments, you must have:
- Guides Pro included in your Pendo subscription.
- The Guide Creator or Content Editor permission assigned to your user role.
Step 1. Set up your experiment
You can create a new experiment from the Experiments page or from the details page of a guide you want to test. Each experiment includes two variants, and each guide inherits the settings you define.
Note: You can only include guides in one experiment at a time, and both guides must belong to the same application.
While the experiment is in Draft status, you can edit the content, activation, and localization of the selected guides. When you activate the experiment, both guides become read-only and inherit the experiment settings:
- Status changes to Public.
- Segment is defined by the experiment.
- Schedule is set by the experiment duration.
Create an experiment from the Experiments page
- Go to Guides > Experiments, then select Create experiment.
- Under A, select + Add, then choose one of the following options:
- Select an existing guide.
- Clone from existing guide, then choose one to duplicate and modify.
- Create a new guide. After building your guide, return to the experiment by selecting Go to experiment from the guide's details page.
- Add a "No guide" group to test only one guide.
- Repeat the same steps to select the second guide under B.
- Continue to the next section to configure your experiment settings.
Create an experiment from a guide's details page
- Open the details page for the guide you want to test.
- Select Add to experiment. This guide becomes Variant A.
- Under B, select + Add, then choose one of the following:
- Add an existing guide, then select it from the list.
- Clone from existing guide, then choose one to duplicate and modify.
- Create a new guide, then choose overlay or embedded. After building your guide, return to the experiment by selecting Go to experiment from the guide's details page.
- Add a "No guide" group to test only one guide.
- Continue to the next section to configure your experiment settings.
Step 2. Configure experiment settings
Use the A/B Test Settings panel to define how your experiment works and what outcome you want to measure.
At the top of the A/B Test section, select Edit. You can also select + Add conversion metric to open the same menu.
Choose how to measure success by selecting a conversion metric from the first dropdown. This defines the type of event you're tracking, such as a Page view, Feature click, or Track Event. For example, if you’re testing abandoned cart messages, the metric could be a click on a purchase button.
Choose an event to track from the second dropdown. The options shown depend on the type of event selected above.
Define your Attribution window. This instructs Pendo on the number of days to track conversions after the first guide view. The maximum is 14 days.
Select your Audience by choosing a segment of visitors to include. For help creating one, see Segments.
Adjust the Distribution to control what percentage of visitors see each guide. The default is a 50/50 split.
Use Scheduling to set the duration of the experiment. This determines how long the experiment is active before it ends automatically. The maximum is three months.
-
Choose what happens to the guides when the experiment ends.
- Keep guides public: The experiment stops collecting data. Both guides remain live in their segments and will be moved to Needs Review. They stay active until you either promote a winner or manually mark the experiment as Completed, which turns off both guides.
- Turn off guides: The experiment stops collecting data and automatically moves to Completed, turning off both guides immediately.
Select Save to confirm and apply your settings.
Use a controlled rollout (optional)
You can use a rollout to gradually increase the number of visitors who see the experiment.
- In the A/B Test section, select Add rollout.
- Use the slider or text field to set your rollout percentage. For example, you might start at 10% and increase over time. You can adjust your rollout settings while the experiment is in draft or active status.
Step 3. Conduct the experiment
After you've configured your experiment, you're ready to start it.
Activate the experiment
To begin collecting data, change the experiment status to Active. This action sets both guides to Public and publishes them to the experiment segment.
Note: You can't edit guides while the experiment is active. To make changes, switch the experiment status to Completed.
Complete the experiment
An experiment ends when the duration expires or you manually set the status to Completed.
When the experiment is complete:
- Both guides are automatically set to Inactive.
- Data may continue to be processed for visitors who viewed a guide near the end of the experiment, but are still within the attribution window.
- The guides are released from the experiment and become editable again. You can add them to other experiments.
Step 4. Determine the winning guide
After the experiment becomes active, Pendo starts collecting conversion data. We recommend waiting until the experiment completes before deciding which guide performed best.
Conversion rate is the percentage of visitors who completed the target event after viewing the guide. It’s calculated as:
\[ Conversion Rate = 100 \times \frac{Visitors Who Completed Event}{Visitors Who Viewed Guide} \]
If needed, export detailed results as a CSV by using the ellipsis (...) in the top-right corner of the page, then choosing Download results (.csv).
You can promote (publish) the guide with the higher conversion rate or choose a different guide manually. To publish the guide you want to continue showing to visitors:
- Select Promote next to the guide.
- Choose whether to use the original segment or define a new one.
- Select Promote to publish the selected guide and unpublish the other.
Create a segment from the experiment
You can create segments based on visitors who saw a specific variant in an experiment. This lets you compare behaviors and outcomes with other metrics.
Note: Segments based on experiment participation can’t be used for guide targeting.
To create a segment for variant A or B:
- In the left-side menu, go to People > Segments, then select Create segment.
- Enter a name for the segment.
- In the rule builder, choose Guide Interactions > Experiment as the filter.
- Select the relevant app if prompted (for example: All Apps).
- Choose the experiment from the list. Only active or completed experiments will appear.
- Select either Variant A was seen or Variant B was seen.
- Add any additional rules if needed.
- Select Create segment.
You can now apply this segment to filters and reports, like Data Explorer, to compare variant performance.
Manage your experiment
You can view, update, or remove experiments as needed.
View experiments
To see all existing experiments, go to Guides > Experiments.
Edit guides in an experiment
You can edit or replace guides while the experiment is in Draft status.
- To edit a guide, hover over it under A or B, then select Edit Guide. This opens the guide's details page in a new browser tab.
- To replace a guide, hover over it under A or B, then select Replace and choose a method of replacement.
- To remove both guides, use the ellipsis (...) in the top-right corner of the page, then choose Clear Variants.
Delete an experiment
To permanently remove an experiment:
- Go to Guides > Experiments.
- Find the experiment you want to delete.
- Select the ellipsis (...) in the top-right corner of the page, then select Delete experiment.
Understand confidence scores
Confidence calculates the statistical probability that an experiment's outcome is a direct result of the guides rather than random chance. The score is based on comparing conversion rates, variance, and sample size across visitor groups.
When an experiment reaches a confidence score of 95% or greater, it's flagged as Significant, indicating a statistically meaningful result.
To view the confidence score, hover over the results in the experiment summary.
What a high confidence score means
A confidence score of 95% or higher suggests that the difference in outcomes is unlikely to be random. However, it doesn't indicate the magnitude of the outcome, only that the observed difference is statistically valid. This gives you statistical support for deciding whether to continue or stop showing a guide. It's not a forecast of future conversion rates.
Statistical explanation
In a split test, the goal is to determine whether the outcome of one variant is more probable than what we'd expect by chance, based on the null hypothesis.
Pendo calculates:
- The conversion rates and variance for each variant.
- The resulting p-value, which measures the probability that the observed difference is due to chance.
The confidence score is then derived from the p-value. It shows how likely it is that the guide made a difference, rather than the results happening by chance.
Formula reference
The confidence score is based on the cumulative distribution function (CDF) of the standard normal distribution:
\[ Confidence = 0.5 \times [1 + erf(\frac{z}{2})] \]
In this formula,
- z is the z-score, which represents how extreme the observed difference is compared to what we'd expect by chance.
- erf is the error function, used to calculate the area under the normal curve.
This formula transforms the z-score into a probability between 0 and 1, showing the likelihood that the result is statistically significant.
If the confidence score is 0.95 or higher, it means there’s a 95% or greater chance that the guide had an actual effect and wasn't a random fluctuation.