Allocating experiment audiences

Read time: 7 minutes

Last edited: Mar 17, 2023

Overview

This topic explains how to include specific groups of contexts in an experiment audience using audience allocation.

Understanding experiment audiences

You have the option to only include a subset of contexts in your experiments, which gives you accurate experiment results more quickly. This subset of contexts is called your "experiment audience."

For example, imagine you plan to test alternate copy for your checkout button. You target all Canadian and US contexts with the true variation for the button, which shows the new, alternate copy, but you only want to run an experiment measuring click conversions for end users in the United States.

To accomplish this, you would select the targeting rule on the flag's Targeting tab that affects US-based contexts and de-select the rule that targets contexts in Canada. This limits the end users who evaluate the flag to only those who are based in the United States.

You may want to refine your experiment audience for any of the following reasons:

To run targeted experiments for a subset of your flag-targeted contexts.
To exclude groups whose events you do not need to measure. For example, those affected by 'Default' rules.
To reduce the volume of contexts in an experiment.

Randomization units

An experiment's randomization unit is the context kind the experiment uses to assign traffic to each of its variations.

For example, if you choose the context kind user as an experiment's randomization unit, then the experiment will divide contexts into the experiment's variations by individual users. User Anna might be assigned to variation A, and user Jesse assigned to variation B. However, if you choose organization as a randomization unit, and Anna and Jesse are part of multi-contexts with the same organization key, then they will both be assigned to the same variation.

Here is what Anna and Jesse's multi-contexts might look like:

Two multi-contexts with different users but the same organization.

There are six industry-standard randomization units that you can use in Experimentation:

user
user-time
guest
guest-time
organization
request

Attempting to use other randomization units would result in invalid experiment results.

The built-in user context kind is automatically mapped to the user standard randomization unit. You will need to map any additional context kinds that you want to use in Experimentation to one of the standard randomization units. Multiple context kinds can be mapped to the same standard randomization unit.

Here is an example of mapping a new context kind of "customer" to the randomization unit "user":

The "New context kind" dialog with the randomization unit mapping field called out.

To learn more, read Creating context kinds.

Here is a table of the industry-standard randomization units and example context kinds that might be associated with them:

Standard randomization unit	Example context kind mappings	Description
User	`user`, `member`, `customer`	For each person that encounters your feature, randomly choose a variation for that particular person
User-Time	`user`+hour-of-day, `user`+day-of-week Takes the form of a composite key of user-key and time-at-some-granularity	For each person that encounters your feature at a particular time of day or on a particular day of the week, randomly choose a variation for that person/time combination
Guest	`device`, `cookie`, `session`, `logged-out user`, `guest`, `non-authorized user`	For each non-logged-in person that encounters your feature, randomly choose a variation for that particular guest
Guest-time	`guest`+hour-of-day, `guest`+day-of-week Takes the form of a composite key of guest-key and time-at-some-granularity	For each non-logged-in person that encounters your feature at a particular time of day or on a particular day of the week, randomly choose a variation for that guest/time combination
Organization	`organization`, `company`, `business`	For each organization that encounters your feature, randomly choose a variation for that organization
Request	`HTTP request`, `operation`, `transaction`, `interaction`	For each particular action that someone performs on your platform, randomly choose a variation for that particular request, operation, or action

Experiments must use valid randomization units

If you're unsure of which randomization unit to map to your context kind, contact Support for help.

When you create a metric, you will also need to map the new metric to a context kind you have marked as available for experiments:

The "Metric information" panel with the randomization unit mapping field called out.

To learn more, read Creating metrics.

Finally, when you create an experiment, you will choose a context kind to serve as a randomization unit that determines both:

which context kind the experiment will allocate traffic to different variations by, and
which metrics you can use in the experiment.

In this example, the experiment has a randomization unit of customer. It is compatible with the chosen metric, because the metric can measure events from customer context kinds.

The experiment creation screen with randomization units for the experiment and the metric called out.

Creating experiment audiences

You determine the initial experiment audience when you create a new experiment. You must include at least two variations in the experiment for the experiment to be valid. To learn more, read Building experiments.

Using targeting rules

You can run an experiment on a flag's default rule, or you can create a custom experiment audience by selecting a specific flag targeting rule to include in your experiment. You can target by any context attribute you collect. To learn how, read Targeting rules.

Allocating audiences

When you build your experiment, you can allocate all or a percentage of the traffic that encounters a flag in an experiment. Audience allocation gives you flexibility when selecting your experiment audience and ensures accurate experiment results. LaunchDarkly analyzes only contexts that you choose to be part of the experiment.

If you decide to increase or decrease the number of contexts in an experiment, LaunchDarkly will create a new iteration of your experiment. To learn more, read Starting experiment iterations.

LaunchDarkly automatically performs checks on the allocation, to make sure that actual traffic matches the allocation you set. To learn more, read Understanding sample ratios.

Changing traffic allocation

When the number of contexts in your variations increases or decreases, LaunchDarkly properly reassigns contexts to different variations to prevent carryover bias. This method of managing your experiment audience is called "audience allocation." To learn more about carryover bias, read Understanding variation reassignment.

Here is an image of a flag's targeting rule with 20% of contexts included in the experiment:

The targeting section of a flag's "Targeting" tab with an active experiment.

If you decide to start a new iteration of your experiment and increase or decrease the amount of traffic in your experiment audience, LaunchDarkly will automatically add or remove contexts to or from the variations using variation reassignment. To learn how to change the audience for a running experiment, read Changing an experiment's audience.

If you start a new iteration of your experiment but don't change the amount of traffic in your experiment audience, LaunchDarkly will not reassign contexts to different variations.

Understanding variation reassignment

Experiments are subject to day-of-week or hour-of-day effects. For example, an end user’s behavior on a website is often different depending on if they are visiting on a Saturday or a Monday. This can cause problems in properly computing experiment results if you use traffic from two different time frames when you increased the percent of traffic going to various contexts.

For example, say you allocated 6% of your traffic to your experiment and it included traffic from weekdays only, and later when you increase your allocated traffic to 30%, it also included traffic from weekends. Because you had more traffic on weekends, the data would overrepresent that traffic and you would get a biased result. You can avoid this by allowing "variation reassignment."

Here is an example of audience allocation using variation reassignment: first, you add an experiment to a flag with three variations: A, B, and C. You roll out the three variations to 6% of contexts, while the remaining 94% receives the control, variation A. The control traffic is not part of the experiment nor its analysis.

Here is a visualization of the starting traffic allocation, with the control group on the right:

An experiment audience with 6% in the experiment and 94% in the control group.

Next, you ramp up your experiment to 30% of traffic. This creates a new iteration of the experiment. In the new iteration, the 6% that were receiving variations A, B, and C have reverted to the control, variation A. The 30% allocated to the experiment is from new traffic.

Here is an example of the modified allocation, with the control group on both the left and right:

An experiment audience with 30% in the experiment and 70% in the control group.

Finally, you roll out variations A and B to 50% of traffic each. This creates another iteration of the experiment, and the entire audience is reallocated randomly. For example, 0%-1% partition will not have the exact same set of contexts as the previous 0%-1% partition.

Here is an example of the allocation with no control group:

An experiment audience with 100% in the experiment.

This new random assignment prevents carryover bias by ensuring that contexts from previous variations are evenly distributed between variations A and B.

Using variation reassignment

It may seem counterintuitive to allow the experiment to reassign contexts to different variations, especially if you are used to the way rollouts work in LaunchDarkly outside of experimentation. However, allowing reassignment has advantages, provided the change you are experimenting on is not disruptive to end users. In practice, this is the case for many experiments. A common example is "user-day experiments", which run for only one day at a time.

For experiments where contexts should not leave the variation they're initially allocated to, such as significant navigation menu changes or large user interface (UI) changes, you can disable variation reassignment. However, keeping contexts in the same variations incurs additional risk.

Using fresh traffic and freeing the previous ramp’s traffic prevents carry-over effects and ensures you don't run out of traffic for your experiment.

Disabling variation reassignment

We do not recommend this option for the majority of use cases

Allowing variation reassignment should be the default for your experiments. If you are unsure if you need to allow or disallow variation reassignment, we suggest you allow variation reassignment.

You should only rarely prevent contexts from switching variations during an experiment by disabling variation reassignment. You may want to disable variation reassignment for substantial changes, such as major navigation menu or UI changes. You can disable variation reassignment by clicking Advanced, then checking the Prevent variation reassignment when increasing traffic checkbox.

Traffic will be assigned to new variations if you stop an experiment

Checking the Prevent variation reassignment when increasing traffic checkbox prevents variation reassignment only if you change the amount of traffic in your experiment audience by editing a running experiment using the experiment's Edit button. To learn more, read Editing experiments.

If you instead use the Stop button to stop an experiment iteration, change the amount of traffic in your experiment audience, then start a new iteration, LaunchDarkly will still reshuffle traffic into new variations.

Here is an example of running an experiment with variation reassignment disabled: you add an experiment to a flag with three variations: A, B, and C. You roll out the three variations to 6% of contexts, while the remaining 94% receives the control, variation A. The control traffic is not part of the experiment nor its analysis.