Guide Control Group Experiments Overview

Overview

Guide experiments determine if behavior is the result of a guide intervention or other confounding criteria. A lot of time, energy, and hard work is put into creating quality guides that improve user experience. In any guide program, it's important to prioritize the guides that influence user behavior and fix or discontinue the guides that don't produce results. Guide experiments help you say with certainty that guide content produces real behavior change.

Guide Control Group Experiments enable you to measure and validate the impact that a guide has on product outcomes. In a controlled experiment, the goal is to measure the impact a single independent variable has on a predicted outcome across a population, while controlling as many other variables as possible. To measure the change produced by the intervention, we need to compare behavior between a group that has seen the invention, the experimental group, and a different group with no outside influence, the control group.

In a Guide Control Group Experiment, the guide is the independent variable influencing behavior change. The predicted outcome is a feature, page, or track event that you expect visitors to use after seeing a guide, identified as the conversion target. The guide's target segment is the population for the experiment. In an active experiment, when an eligible visitor triggers the guide, they're randomly assigned to the experimental group and see the guide or the control group and do not see the guide. Pendo automatically tracks which group visitors are in and monitors their behavior after entering the experiment. The experiment tracks the conversion rates for the experimental and control group. An effective guide will show a higher conversion rate in the experimental group compared to the control group.

 

Splash.png

 

Requirements

  • Pendo Guide Creator or Content Editor user roles
  • Web Agent 2.87.1 or higher
  • Mobile SDK 2.8.0 or higher
  • Pro or Enterprise-tier Pendo subscription

How to run an experiment

All good experiments start with a hypothesis. In many cases, you're publishing a guide because you want a user to do something. A new feature announcement is pushing users to try the feature. A marketing guide is trying to increase new subscriptions. A self-help guide is pointing users in the right direction to continue a workflow.

The basic hypothesis is "If I publish a guide telling users about [my product] then users are more likely to engage with [my product] and improve outcomes related to low or no usage." Selecting the target of the experiment, predicting the outcomes resulting from increased usage, and designing the guide content is up to you. Use Pendo to run a control group experiment and measure whether or not your guide changed user behavior.

A control group experiment has three primary steps:

  1. Set the conversion target
  2. Set up the control group and notification time
  3. Review the results and take action.

Setup Guide Conversion

Conversion tracks the number of users who engaged with the conversion target after seeing the guide. The conversion target is the feature, page, or track event you expect users to engage with after viewing the guide. This is the success criteria for the guide. An effective guide will result in more users converting versus no guide at all. Select the conversion target in the Conversion tile on the Guide Details page. Conversion can be set up independent of an experiment and is useful for guide analytics without an experiment.

 

1. On the Guide Details page, find the Conversion tile and click Edit.

conversion_blank.png

 

2. Define Conversion type and select the target. 

ConversionTarget.png

Conversion can be a tagged feature, page, or track event.

ConversionDropdown.png

The target dropdown menu will load a list of all tagged features, pages, or track events for the current app, corresponding with the conversion selection.

SelectTarget.png

 

3. Click Save to set the conversion target.

Save.png

SavedConversion.png

 

Setup Experiment

The Experiment tile is used to setup a control group experiment that measures how effectively your guide influences user behavior in the experimental group, who see the guide, versus the control group, who do not see the guide. The experiment should be configured after other guide details have been setup. Conversion is required prior to experiment setup but segment is also crucial for understanding the size of the overall experiment population. The Experiment tile is on the Guide Details page.

 

1. On the Guide Details page, find the Experiment tile and click + Create Experiment.

CreateExperiment.png

 

2. Set the experiment follow-up notification time and experiment group size.

ExperimentSetup.png

The follow-up notification will display an alert in Pendo when it has expired reminding you to check the results of your experiment. Experiments must be stopped manually. You may decide to end an experiment early if the results are clear or allow the experiment to continue if you need more data. This is difficult to guess before you see the data.

Experiment Group Size sets the percentage of the target segment that will see the guide and conversely the size of the control group that won't see the guide. Don't worry, the control group can see the guide when the experiment ends.

  • Try to get more than 1,000 visitors total, with at least 500 visitors in each group
  • A larger sample size, generated by the eligible visitors in the target segment for the guide, will generally produce more data and better results
  • Increasing the size of the experiment group with a small overall sample size increases the number of opportunities for users to convert after seeing the guide
  • Having a statistically significant control group size is important to say with confidence that your guide is influencing behavior and your conversion rate isn't the result of other factors

Tip: Use caution with extremely large sample sizes in large populations, the sample should be large enough that the results aren't swayed by random chance but not so large that the sheer volume of data suggests a meaningful result

  • After thousands of visitors have participated in the experiment, the experimental group may have several hundred more conversions than the control group
  • This appears to be statistically significant but the difference between the experimental and control group conversion rates may be so small that it isn't practically significant, meaning the guide won't have a tangible affect on your users or any expected business outcomes

3. Click Save to save the experiment.

Save.png

ExperimentReady.png

 

4. Set the guide to Public to start the experiment.

SetGuideExperimentPublic.png

 

Review results

Your experiment will start running automatically when the guide is published. You can monitor the results from the Experiment Results tab. On this tab, you will see a summary of the experiment results, a chart displaying conversions over time for the experimental and control groups, and downloadable experiment data table.

You should let your experiment run until you are satisfied with the results. The confidence score is included with the conversion data to help you weigh the significance of the results and decide when to end the experiment. A Confidence percentage greater than 95% will be marked as "Significant". You will receive a follow-up notification when the reminder you set expires.

Experiment_Results_cropped.png

 

Confidence Score

Confidence calculates the statistical probability that the outcome was based on the guide and not random chance. The score is determined by comparing the conversion rates, variance, and number of results, and determining the probability that the results of the experiment are statistically significant. When an experiment reaches a confidence score of 95% or greater, it will be flagged with (Significant) next to the confidence percentage.

This does not mean that the magnitude of the outcome is significant, only that the results are probably not due to random chance. For example, an experiment could show that visitors are more likely to convert, had no change in the rate of conversion, or are less likely to convert and the results will still be significant with a confidence score greater than 95%. This gives you the confidence to make the decision to keep or discontinue your guide based on the results of the experiment. This is not an indication that the magnitude of the outcome of the experiment is the expected conversation rate over time. Determining the average outcome over time requires running multiple experiments and comparing the results or monitoring the conversion rate over time to track the sustained impact your guide is having on your users.

For statisticians, we want to find the probability that in our split test the results of the experimental group are more probable than the null hypothesis represented by the control group. The conversion rates and variance for the experimental and control groups are found. These values are used in the final calculation that determines the p value, and then the final confidence score showing the probability the null hypothesis is false.

confidencecalc.png

p is the distribution of the difference between the experimental and control groups distributions

sigma is the variance of the experimental and control groups

 

End the experiment

Experiments must be stopped manually. You may decide to end an experiment early if the results are clear or allow the experiment to continue if you need more data. This is difficult to guess before you see the data. If possible, you should wait until you have at least 500 visitors in each group.

You can end an experiment from the Experiment tile on the Summary tab or the Results tile on the Experiment Results tab in Guide Details or Experiments tab in the Guides List.

Experiment Tile in Summary Tab                            Results Tile in Experiments Tab

EndExperimentTile.png     EndExperiment.png

 

1. Click End Experiment

EndExperimentModal.png

 

2. Decide to End & Keep Public or End & Disable

  • End & Keep Public shows the guide to users who were in the Control Group and did not receive the guide during the experiment and any users who have not seen it yet. We recommend you select this option if your experiment ended with a positive outcome.
  • End & Disable ends the experiment and disables the guide. Users in the targeted segment will no longer receive the guide, but you will be able to access the experiment metrics from the Experiment Results tab or the Experiments tab on the Guide List.

Share results

Experiments Tab

All Experiments can be accessed by all Pendo users from the Experiments tab in the Guide List. The Experiments tab shows results summaries for any guides that have experiments configured even if they haven't started yet. Each summary has links to the conversion target and the guide. You can filter the summaries on this page by experiment status and newest to oldest. Deleting a guide removes the experiment summary from Experiment tab.

Experiments_Tab_cropped.png

 

Experiment Results Dashboard Widget

Experiment summaries can be added as a dashboard widget and shared when you share a dashboard with other Pendo users in your subscription.

Add experiment results to the dashboard from the experiment results summary in the Experiments tab of the Guide List or the Experiment Results tab in Guide Details.

 

1. Click Add to Dashboard in the Options menu on the Experiment Results summary.

AddtoDashboarddropdown.png

 

2. Select the Dashboard from a dropdown menu of your current dashboards.

AddtoDashboardModal.png

 

3. Click Add to Dashboard to add the widget to the selected dashboard.

AddtoDashboard.png