• Home
  • Integrations
  • SDKs
  • Guides
  • API docs
No results for ""
EXPAND ALL

EDIT ON GITHUB

Allocating experiment audiences

Read time: 4 minutes
Last edited: Jun 27, 2022

Overview

This topic explains how to include specific groups of users in an experiment audience using audience allocation.

Understanding experiment audiences

You have the option to only include a subset of users in your experiments, which gives you accurate experiment results more quickly. This subset of users is called your "experiment audience."

For example, imagine you plan to test alternate copy for your checkout button. You target all Canadian and US users with the true variation for the button, which shows the new, alternate copy, but you only want to run an experiment measuring click conversions for users in the United States. To accomplish this, you select the user targeting rule that affects US-based users and de-select the rule that targets users in Canada. This limits the users who evaluate the flag to only those who are based in the United States.

You may want to refine your experiment audience for any of the following reasons:

  • To run targeted experiments for a subset of your flag-targeted users.
  • To exclude user groups whose events you do not need to measure. For example, users affected by 'Default' rules.
  • To reduce the volume of users in an experiment.

You can create a custom experiment audience by selecting specific flag targeting rules to include in your active experiments. When increasing or decreasing the number of users in your variations, LaunchDarkly properly randomizes users and provides the configuration and guidance to prevent carryover bias. This method of managing your experiment audience is called "audience allocation." To learn more about carryover bias, read Understanding variation reassignment.

Audience allocation gives you flexibility when selecting your experiment audience and ensures accurate experiment results. Only users that you elect to be part of the experiment will be analyzed for those results.

Verify that your SDK versions support Experimentation

Before you begin, compare your SDK versions to our list of required SDK versions to confirm that you're using an SDK that supports our most recent Experimentation offering. Most versions of our SDKs support Experimentation, but if you use older versions of SDKs or you used an older version of Experimentation, you may need to update your SDKs.

To find the supported SDK versions, read Prerequisites.

Creating experiment audiences

You determine the initial experiment audience when you create a new experiment. You must include at least two variations in the experiment for the experiment to be valid. To learn more, read Building experiments.

Here is an image of a flag's targeting rule with 20% of users included in the experiment:

The user targeting section of a flag's "Targeting" tab with an active experiment.
The user targeting section of a flag's "Targeting" tab with an active experiment.

If you decide to start a new iteration of your experiment and increase or decrease the number of users in your experiment audience, LaunchDarkly will automatically add or remove users to or from the variations using variation reassignment. To learn how to change the audience for a running experiment, read Managing experiment audiences.

Understanding variation reassignment

Experiments are subject to day-of-week or hour-of-day effects. For example, a user’s behavior on a website is often different depending on if they are visiting on a Saturday or a Monday. This can cause problems in properly computing experiment results if you use traffic from two different time frames when you increased the percent of traffic going to various users.

For example, say you allocated 6% of your traffic to your experiment and it included traffic from weekdays only, and later when you increase your allocated traffic to 30%, it also included traffic from weekends. Because you had more traffic on weekends, the data would overrepresent that traffic and you would get a biased result. You can avoid this by allowing "variation reassignment."

Here is an example of audience allocation using variation reassignment:

First, you add an experiment to a flag with three variations: A, B, and C. You roll out the three variations to 6% of the users, while the remainder receives variation A.

An example starting traffic allocation appears below:

Audience allocation at 6%
Audience allocation at 6%

The traffic on the right is receiving variation A, but is not part of the experiment nor its analysis.

Next, you ramp up your experiment to 30% of traffic. This creates a new iteration of the experiment.

Here is an example of the modified allocation:

Audience allocation at 30%
Audience allocation at 30%

In the new iteration, the traffic that was receiving variations A, B, and C, is now reverted back to A, and new traffic is used for the 30% allocated to the experiment.

Finally, you roll out variations A and B to 50% of traffic each. This creates another iteration of the experiment, and the entire audience is reallocated randomly. For example, 0%-1% partition will not have the exact same set of users as the previous 0%-1% partition.

Here is an example image:

Audience allocation at 50%
Audience allocation at 50%

This new random assignment prevents carryover bias by ensuring that users from previous variations are evenly distributed between variations A and B.

Using variation reassignment

It may seem counterintuitive to allow the experiment to reassign users to different variations, especially if you are used to the way rollouts work in LaunchDarkly outside of experimentation. However, allowing reassignment has advantages, provided the change you are experimenting on is not disruptive to users. In practice, this is the case for many experiments. A common example is "user-day experiments", which run for only one day at a time.

For experiments where users should not leave the variation they're initially allocated to, such as significant navigation menu changes or large user interface (UI) changes, you can disable variation reassignment. However, keeping users in the same variations incurs additional risk.

Using fresh traffic and freeing the previous ramp’s traffic prevents carry-over effects and ensures you don't run out of traffic for your experiment.

Disabling variation reassignment

We do not recommend this option for the majority of use cases

Allowing variation reassignment should be the default for your experiments. If you are unsure if you need to allow or disallow variation reassignment, we suggest you allow variation reassignment.

You should only rarely prevent users from switching variations during an experiment by disabling variation reassignment. You may want to disable variation reassignment for substantial changes, such as major navigation menu or UI changes. You can disable variation reassignment by clicking Advanced, then checking the Keep users assigned to initial variation checkbox.

Here is an example of running an experiment with variation reassignment disabled:

You add an experiment to a flag with three variations: A, B, and C. You roll out the three variations to 6% of the users, while the remainder receives variation A.

Here is an example starting traffic allocation:

Audience allocation at 6%
Audience allocation at 6%

The traffic on the right is receiving variation A, but is not part of the experiment nor its analysis.

Next, you increase the experiment to 30% of traffic.

Here is an example experiment with increased traffic:

Audience allocation at 30%
Audience allocation at 30%

We suggest you reset your experiment results when you increase traffic to avoid time-varying effects.