No results for ""
EXPAND ALL
  • Home
  • API docs

GIVE DOCS FEEDBACK

Allocating experiment audiences

Read time: 12 minutes
Last edited: Jun 14, 2024

Overview

This topic explains how to include specific groups of contexts in an experiment audience using audience allocation.

Experiment audiences

You have the option to only include a subset of contexts in your experiments, which gives you accurate experiment results more quickly. This subset of contexts is called your "experiment audience."

For example, imagine you plan to test alternate copy for your checkout button. You target all Canadian and US contexts with the true variation for the button, which shows the new, alternate copy, but you only want to run an experiment measuring click conversions for end users in the United States.

To accomplish this, you would select the targeting rule on the flag's Targeting tab that affects US-based contexts and de-select the rule that targets contexts in Canada. This limits the end users who evaluate the flag to only those who are based in the United States.

You may want to refine your experiment audience for any of the following reasons:

  • To run targeted experiments for a subset of your flag-targeted contexts.
  • To exclude groups whose events you do not need to measure. For example, those affected by 'Default' rules.
  • To reduce the volume of contexts in an experiment.

Randomization units

An experiment's randomization unit is the context kind the experiment uses to assign traffic to each of its variations. If you choose the user context kind as an experiment's randomization unit, then the experiment will divide contexts into the experiment's variations by individual users.

For example, imagine users Anne and Jesse both work for the same organization, and are part of an experiment comparing two variations. Both Anne and Jesse are each part of a different multi-context, with different user keys but the same organization key.

Here is what their multi-contexts would look like:

"context": {
"kind": "multi",
"user": {
"key": "user-key-anna", // Anna has a unique user key
"name": "Anna",
},
"organization": {
"key": "org-key-global-health", // The org key is the same in both multi-contexts
"name": "Global Health Services",
}
}
"context": {
"kind": "multi",
"user": {
"key": "user-key-jesse", // Jesse has a unique user key
"name": "Jesse",
},
"organization": {
"key": "org-key-global-health", // The org key is the same in both multi-contexts
"name": "Global Health Services",
}
}

There are two ways you could randomize the multi-contexts in an experiment:

  • if you randomize by user, Anna could be assigned to one variation, and Jesse could be assigned to the other variation because they have different user keys
  • if you randomize by organization, Anna and Jesse will both always be assigned to the same variation because they share the same organization key

Before you begin an experiment, you must:

  • choose an industry-standard randomization unit for the experiment,
  • map the randomization unit to an appropriate context kind,
  • map the context kind to the appropriate metrics, then
  • add the randomization unit to the experiment.

These steps are explained below.

Industry standard randomization units

There are six industry-standard randomization units that you can use in Experimentation:

  • user
  • user-time
  • guest
  • guest-time
  • organization
  • request

LaunchDarkly limits experiments to randomizing by these six industry-standard units because other units might result in invalid results. If you have a context kind that doesn't logically map to one of these industry-standard units, it may not be appropriate to use as a randomization unit. For examples of common context kind mapping, read the table in Map randomization units to context kinds.

For example, if you tried to use "country" as a randomization unit, your experiment would serve the same variation to everyone in a particular country. If your experiment had end users in the United States, Mexico, and Canada, the experiment might serve variation A to all end users in the United States and Canada, and variation B to all end users in Mexico. This could result in significantly uneven numbers of end users in each variation, and each variation would not include a random sampling of end users. At the end of the experiment, there would be no way to tell if variation A or B performed better in any given country.

Instead, if you wanted to view results by country, you could slice your results by context attribute. To learn more, read Slicing experiment results.

Map randomization units to context kinds

The built-in user context kind is automatically mapped to the user standard randomization unit. If you have created additional context kinds you want to use in experiments, then you must first map them to one of the standard randomization units. Multiple context kinds can be mapped to the same standard randomization unit. To learn how, read Create context kinds.

Here is an example of mapping a new context kind of "customer" to the randomization unit "user":

The "New context kind" dialog with the randomization unit mapping field called out.
The "New context kind" dialog with the randomization unit mapping field called out.


This table includes the industry-standard randomization units and example context kinds that might be associated with them:

Standard randomization unitExample context kind mappingsDescription
Useruser, member, customerFor each person that encounters your feature, randomly choose a variation for that particular person
User-Timeuser+hour-of-day, user+day-of-week

Takes the form of a composite key of user-key and time-at-some-granularity
For each person that encounters your feature at a particular time of day or on a particular day of the week, randomly choose a variation for that person/time combination
Guestdevice, cookie, session, logged-out user, guest, non-authorized userFor each non-logged-in person that encounters your feature, randomly choose a variation for that particular guest
Guest-timeguest+hour-of-day, guest+day-of-week

Takes the form of a composite key of guest-key and time-at-some-granularity
For each non-logged-in person that encounters your feature at a particular time of day or on a particular day of the week, randomly choose a variation for that guest/time combination
Organizationorganization, company, businessFor each organization that encounters your feature, randomly choose a variation for that organization
RequestHTTP request, operation, transaction, interactionFor each particular action that someone performs on your platform, randomly choose a variation for that particular request, operation, or action
Experiments must use valid randomization units

If you're unsure of which randomization unit to map to your context kind, contact Support for help.

Map randomization units to metrics

When you create a metric, you must map the new metric to a context kind you have marked as available for experiments:

The "Metric information" panel with the randomization unit mapping field called out.
The "Metric information" panel with the randomization unit mapping field called out.

This context kind determines which randomization units the metric is compatible with. To learn more, read Metrics.

Choose randomization units for experiments

Finally, when you create an experiment, you will choose a randomization unit that determines both:

  • which context kind the experiment will allocate traffic to different variations by, and
  • which metrics you can use in the experiment.

In this example, the experiment has a randomization unit of user. It is compatible with the chosen metric, because the metric can measure events from user context kinds.

The experiment creation screen with the randomization unit for the experiment called out.
The experiment creation screen with the randomization unit for the experiment called out.

Create experiment audiences

You determine the initial experiment audience when you create a new experiment. You must include at least two variations in the experiment for the experiment to be valid. To learn more, read Creating experiments.

Targeting rules

You can run an experiment on a flag's default rule, or you can create a custom experiment audience by selecting a specific flag targeting rule to include in your experiment. You can target by any context attribute you collect. To learn how, read Target with flags.

Allocate audiences

When you build your experiment, you can allocate all or a percentage of the traffic that encounters a flag in an experiment. Audience allocation gives you flexibility when selecting your experiment audience and ensures accurate experiment results. LaunchDarkly analyzes only contexts that you choose to be part of the experiment.

If you decide to increase or decrease the number of contexts in an experiment, LaunchDarkly will create a new iteration of your experiment. To learn more, read Start experiment iterations.

LaunchDarkly automatically performs checks on the allocation, to make sure that actual traffic matches the allocation you set. To learn more, read Understanding sample ratios.

Change traffic allocation

When the number of contexts in your variations increases or decreases, LaunchDarkly properly reassigns contexts to different variations to prevent carryover bias. This method of managing your experiment audience is called "audience allocation." To learn more, read About carryover bias.

If you decide to start a new iteration of your experiment and increase or decrease the amount of traffic in your experiment audience, LaunchDarkly will automatically add or remove contexts to or from the variations using variation reassignment. To learn how to change the audience for a running experiment, read Change experiment audiences.

If you start a new iteration of your experiment but don't change the amount of traffic in your experiment audience, LaunchDarkly will not reassign contexts to different variations.

About carryover bias

Experiments are subject to day-of-week or hour-of-day effects. For example, an end user’s behavior on a website is often different depending on if they are visiting on a Saturday or a Monday. This can cause problems in properly computing experiment results if you use traffic from two different time frames when you increase the percent of traffic going to various contexts. This is called carryover bias.

For example, say you allocated 6% of your traffic to your experiment and it included traffic from weekdays only, and later when you increase your allocated traffic to 30%, it also included traffic from weekends. Because you had more traffic on weekends, the data would over represent that traffic and you would get a biased result. You can avoid this by allowing "variation reassignment."

Here is an example of audience allocation using variation reassignment: first, you add an experiment to a flag with three variations: A, B, and C. You roll out the three variations to 6% of contexts, while the remaining 94% receives the control, variation A. The control traffic is not part of the experiment nor its analysis.

Here is a visualization of the starting traffic allocation, with the control group on the right:

An experiment audience with 6% in the experiment and 94% in the control group.
An experiment audience with 6% in the experiment and 94% in the control group.

Next, you ramp up your experiment to 30% of traffic. This creates a new iteration of the experiment. In the new iteration, the 6% that were receiving variations A, B, and C have reverted to the control, variation A. The 30% allocated to the experiment is from new traffic.

Here is an example of the modified allocation, with the control group on both the left and right:

An experiment audience with 30% in the experiment and 70% in the control group.
An experiment audience with 30% in the experiment and 70% in the control group.

Finally, you roll out variations A and B to 50% of traffic each. This creates another iteration of the experiment, and the entire audience is reallocated randomly. For example, 0%-1% partition will not have the exact same set of contexts as the previous 0%-1% partition.

Here is an example of the allocation with no control group:

An experiment audience with 100% in the experiment.
An experiment audience with 100% in the experiment.

This new random assignment prevents carryover bias by ensuring that contexts from previous variations are evenly distributed between variations A and B.

Variation reassignment

It may seem counterintuitive to allow the experiment to reassign contexts to different variations, especially if you are used to the way rollouts work in LaunchDarkly outside of experimentation. However, allowing reassignment has advantages, provided the change you are experimenting on is not disruptive to end users. In practice, this is the case for many experiments. A common example is "user-day experiments," which run for only one day at a time.

For experiments where contexts should not leave the variation they're initially allocated to, such as significant navigation menu changes or large user interface (UI) changes, you can disable variation reassignment. However, keeping contexts in the same variations incurs additional risk.

Using fresh traffic and freeing the previous ramp’s traffic prevents carry-over effects and ensures you don't run out of traffic for your experiment.

Disable variation reassignment

We do not recommend this option for the majority of use cases

Allowing variation reassignment should be the default for your experiments. If you are unsure if you need to allow or disallow variation reassignment, we suggest you allow variation reassignment.

Expand Disabling variation assignment

You should only rarely prevent contexts from switching variations during an experiment by disabling variation reassignment. You may want to disable variation reassignment for substantial changes, such as major navigation menu or UI changes. You can disable variation reassignment by clicking Advanced, then checking the Prevent variation reassignment when increasing traffic checkbox.

Traffic will be assigned to new variations if you stop an experiment

Checking the Prevent variation reassignment when increasing traffic checkbox prevents variation reassignment only if you change the amount of traffic in your experiment audience by editing a running experiment using the experiment's Edit button. To learn more, read Edit experiments.

If you instead use the Stop button to stop an experiment iteration, change the amount of traffic in your experiment audience, then start a new iteration, LaunchDarkly will still reshuffle traffic into new variations.

Here is an example of running an experiment with variation reassignment disabled: you add an experiment to a flag with three variations: A, B, and C. You roll out the three variations to 6% of contexts, while the remaining 94% receives the control, variation A. The control traffic is not part of the experiment nor its analysis.

Here is a visualization of the starting traffic allocation, with the control group on the right:

An experiment audience with 6% in the experiment and 94% in the control group.
An experiment audience with 6% in the experiment and 94% in the control group.

Next, you ramp up your experiment to 30% of traffic. This creates a new iteration of the experiment. In the new iteration, the 6% that were receiving variations A, B, and C, continue to receive those variations, but are no longer included in the experiment nor its analysis. New traffic is used for the 30% allocated to the experiment.

Here is an example of the modified allocation, with the control group on the right and the original experiment audience on the left:

An experiment audience with 6% no longer in the experiment, 30% in the experiment, and 64% in the control group.
An experiment audience with 6% no longer in the experiment, 30% in the experiment, and 64% in the control group.