A/B Testing with Multivariate Flags
We are currently working on support for A/B testing with multivariate flags. For now, if you would like to do A/B testing, you must use a boolean feature flag.
Setting up LaunchDarkly to run A/B tests (aka experiments) involves a few additional setup steps beyond what's needed to control feature flags. We'll walk through the steps one by one. We'll assume that you've already gone through the basic Setup steps and can successfully toggle features on and off.
Goals are the metrics used to measure the effectiveness of a feature. To run an A/B test, you need to define the goals you care about. LaunchDarkly supports three kinds of goals:
- Click goals - track whether a user clicks on a specific page element.
- Page view goals - track whether a user lands on a specific page (for example, a confirmation page).
- Custom goals - track other user interactions that don't correspond to page views or clicks.
You can create goals directly in LaunchDarkly, or import them from Optimizely.
Once you've created your goals, you'll need to decide which goals to track for each feature.
Goals are defined per project in LaunchDarkly and can be re-used for multiple feature flags. In order to indicate which goals are relevant to each flag, you need to associate them with the feature flag. You can do this on the feature flag's Experiments tab by clicking "Manage Goals".
Associating a goal with a feature flag
Once your goals have been set up (and you've verified that you're receiving events for your goals on the dev console, you'll begin to see results in the Experiments tab.
A/B testing results on the Experiments tab
We'll only show results once each variation of the feature flag has received at least one impression.
Event processing time
We process events on a five minute delay. If you're not seeing the numbers you expect on the analytics tab, ensure that you've waited at least five minutes for your events to be processed.
Once we've determined a winner, we'll show a green checkbox next to the winning variation.
Determining a winner
We'll select a winner when each variation has at least 1000 distinct users (impressions), and the confidence interval is above 95%.