An A/B test is an experimentation tool used when you want to measure an effect of a certain change or intervention. For example, you may want to see whether a Guide that introduces a new feature will lead to a higher percent of visitors clicking the feature after launch. This recipe is intended to walk through the considerations and steps of creating an A/B test using Pendo sample groups.
Ingredients (What Do I Need?)
- Pendo Admin Permissions or the following:
- User permission
- User permission to create Guides
- User permission to publish Guides
How to Make It
In this Recipe, you will create 2 custom segments (which are your 2 test groups) using Pendo’s sample groups feature. To create your segments:
Step 1. Navigate to the “Visitors” section in Pendo.
Step 2. Create two segments to target the right test groups by using “AND” rules with Sample Groups. (Learn how “Sample Groups" work) To create a 50/50 test, use the following Sample Group rules:
- Segment 1: Sample Group is less than or equal to 49 (This includes groups 0-49)
- Segment 2: Sample Group is greater than or equal to 50 (This includes groups 50-99)
Step 3. Create a Pendo Guide and set the segment to the Test group you just created.
Best practices for Setting Up Sample Groups
How do you know if your sample size is large enough?
A sample that is too small will be prone to outliers and the data that results from it may become skewed. The larger the group, the better. But, particular numbers will depend on the volume of visitors, their frequency, and time spent on site.
How do you know if your groups are equal?
It’s highly recommended to analyze the different types of visitor attributes and understand the averages of login frequency, event frequency, time since first login, and other standard actions to know that the groups are similar in scope. If you find any significant outliers that could skew the data (internal accounts, QA testing bots), please utilize the blacklist feature or add additional segment conditions to exclude them.
How do I analyze the data afterward?
Pendo provides a variety of different ways to help you compare actions, trends, interactions with features and guides by utilizing the segment dropdown.