Allocating experiment audiences
Read time: 12 minutes
Last edited: May 01, 2024
Overview
This topic explains how to include specific groups of contexts in an experiment audience using audience allocation.
Experiment audiences
You have the option to only include a subset of contexts in your experiments, which gives you accurate experiment results more quickly. This subset of contexts is called your "experiment audience."
For example, imagine you plan to test alternate copy for your checkout button. You target all Canadian and US contexts with the true
variation for the button, which shows the new, alternate copy, but you only want to run an experiment measuring click conversions for end users in the United States.
To accomplish this, you would select the targeting rule on the flag's Targeting tab that affects US-based contexts and de-select the rule that targets contexts in Canada. This limits the end users who evaluate the flag to only those who are based in the United States.
You may want to refine your experiment audience for any of the following reasons:
- To run targeted experiments for a subset of your flag-targeted contexts.
- To exclude groups whose events you do not need to measure. For example, those affected by 'Default' rules.
- To reduce the volume of contexts in an experiment.
Randomization units
An experiment's randomization unit is the context kind the experiment uses to assign traffic to each of its variations. If you choose the user
context kind as an experiment's randomization unit, then the experiment will divide contexts into the experiment's variations by individual users.
For example, imagine users Anne and Jesse both work for the same organization, and are part of an experiment comparing two variations. Both Anne and Jesse are part of two different multi-contexts, with different user keys but the same organization key.
There are two ways you could randomize the multi-contexts in an experiment:
- if you randomize by
user
, Anna could be assigned to one variation, and Jesse could be assigned to the other variation because they have different user keys - if you randomize by
organization
, Anna and Jesse will both always be assigned to the same variation because they share the same organization key
Here is what their multi-contexts would look like:
"context": {"kind": "multi","user": {"key": "user-key-anna", // Anna has a unique user key"name": "Anna",},"organization": {"key": "org-key-global-health", // The org key is the same in both multi-contexts"name": "Global Health Services",}}"context": {"kind": "multi","user": {"key": "user-key-jesse", // Jesse has a unique user key"name": "Jesse",},"organization": {"key": "org-key-global-health", // The org key is the same in both multi-contexts"name": "Global Health Services",}}
Before you begin an experiment, you must:
- choose an industry-standard randomization unit for the experiment,
- map the randomization unit to an appropriate context kind,
- map the context kind to the appropriate metrics, then
- add the randomization unit to the experiment.
These steps are explained below.
Industry standard randomization units
There are six industry-standard randomization units that you can use in Experimentation:
- user
- user-time
- guest
- guest-time
- organization
- request
LaunchDarkly limits experiments to randomizing by these six industry-standard units because other units might result in invalid results. If you have a context kind that doesn't logically map to one of these industry-standard units, it may not be appropriate to use as a randomization unit. For examples of common context kind mapping, read the table in Map randomization units to context kinds.
For example, if you tried to use "country" as a randomization unit, your experiment would serve the same variation to everyone in a particular country. If your experiment had end users in the United States, Mexico, and Canada, the experiment might serve variation A to all end users in the United States and Canada, and variation B to all end users in Mexico. This could result in significantly uneven numbers of end users in each variation, and each variation would not include a random sampling of end users. At the end of the experiment, there would be no way to tell if variation A or B performed better in any given country.
Instead, if you wanted to view results by country, you could slice your results by context attribute. To learn more, read Slicing experiment results.
Map randomization units to context kinds
The built-in user
context kind is automatically mapped to the user
standard randomization unit. If you have created additional context kinds you want to use in experiments, then you must first map them to one of the standard randomization units. Multiple context kinds can be mapped to the same standard randomization unit. To learn how, read Create context kinds.
Here is an example of mapping a new context kind of "customer" to the randomization unit "user":
This table includes the industry-standard randomization units and example context kinds that might be associated with them:
Standard randomization unit | Example context kind mappings | Description |
---|---|---|
User | user , member , customer | For each person that encounters your feature, randomly choose a variation for that particular person |
User-Time | user +hour-of-day, user +day-of-weekTakes the form of a composite key of user-key and time-at-some-granularity | For each person that encounters your feature at a particular time of day or on a particular day of the week, randomly choose a variation for that person/time combination |
Guest | device , cookie , session , logged-out user , guest , non-authorized user | For each non-logged-in person that encounters your feature, randomly choose a variation for that particular guest |
Guest-time | guest +hour-of-day, guest +day-of-weekTakes the form of a composite key of guest-key and time-at-some-granularity | For each non-logged-in person that encounters your feature at a particular time of day or on a particular day of the week, randomly choose a variation for that guest/time combination |
Organization | organization , company , business | For each organization that encounters your feature, randomly choose a variation for that organization |
Request | HTTP request , operation , transaction , interaction | For each particular action that someone performs on your platform, randomly choose a variation for that particular request, operation, or action |
If you're unsure of which randomization unit to map to your context kind, contact Support for help.
Map randomization units to metrics
When you create a metric, you must map the new metric to a context kind you have marked as available for experiments:
This context kind determines which randomization units the metric is compatible with. To learn more, read Metrics.
Choose randomization units for experiments
Finally, when you create an experiment, you will choose a randomization unit that determines both:
- which context kind the experiment will allocate traffic to different variations by, and
- which metrics you can use in the experiment.
In this example, the experiment has a randomization unit of user
. It is compatible with the chosen metric, because the metric can measure events from user
context kinds.
Create experiment audiences
You determine the initial experiment audience when you create a new experiment. You must include at least two variations in the experiment for the experiment to be valid. To learn more, read Creating experiments.
Targeting rules
You can run an experiment on a flag's default rule, or you can create a custom experiment audience by selecting a specific flag targeting rule to include in your experiment. You can target by any context attribute you collect. To learn how, read Targeting rules.
Allocate audiences
When you build your experiment, you can allocate all or a percentage of the traffic that encounters a flag in an experiment. Audience allocation gives you flexibility when selecting your experiment audience and ensures accurate experiment results. LaunchDarkly analyzes only contexts that you choose to be part of the experiment.
If you decide to increase or decrease the number of contexts in an experiment, LaunchDarkly will create a new iteration of your experiment. To learn more, read Start experiment iterations.
LaunchDarkly automatically performs checks on the allocation, to make sure that actual traffic matches the allocation you set. To learn more, read Understanding sample ratios.
Change traffic allocation
When the number of contexts in your variations increases or decreases, LaunchDarkly properly reassigns contexts to different variations to prevent carryover bias. This method of managing your experiment audience is called "audience allocation." To learn more, read About carryover bias.
If you decide to start a new iteration of your experiment and increase or decrease the amount of traffic in your experiment audience, LaunchDarkly will automatically add or remove contexts to or from the variations using variation reassignment. To learn how to change the audience for a running experiment, read Change experiment audiences.
If you start a new iteration of your experiment but don't change the amount of traffic in your experiment audience, LaunchDarkly will not reassign contexts to different variations.
About carryover bias
Experiments are subject to day-of-week or hour-of-day effects. For example, an end user’s behavior on a website is often different depending on if they are visiting on a Saturday or a Monday. This can cause problems in properly computing experiment results if you use traffic from two different time frames when you increase the percent of traffic going to various contexts. This is called carryover bias.
For example, say you allocated 6% of your traffic to your experiment and it included traffic from weekdays only, and later when you increase your allocated traffic to 30%, it also included traffic from weekends. Because you had more traffic on weekends, the data would over represent that traffic and you would get a biased result. You can avoid this by allowing "variation reassignment."
Here is an example of audience allocation using variation reassignment: first, you add an experiment to a flag with three variations: A, B, and C. You roll out the three variations to 6% of contexts, while the remaining 94% receives the control, variation A. The control traffic is not part of the experiment nor its analysis.
Here is a visualization of the starting traffic allocation, with the control group on the right:
Next, you ramp up your experiment to 30% of traffic. This creates a new iteration of the experiment. In the new iteration, the 6% that were receiving variations A, B, and C have reverted to the control, variation A. The 30% allocated to the experiment is from new traffic.
Here is an example of the modified allocation, with the control group on both the left and right:
Finally, you roll out variations A and B to 50% of traffic each. This creates another iteration of the experiment, and the entire audience is reallocated randomly. For example, 0%-1% partition will not have the exact same set of contexts as the previous 0%-1% partition.
Here is an example of the allocation with no control group:
This new random assignment prevents carryover bias by ensuring that contexts from previous variations are evenly distributed between variations A and B.
Variation reassignment
It may seem counterintuitive to allow the experiment to reassign contexts to different variations, especially if you are used to the way rollouts work in LaunchDarkly outside of experimentation. However, allowing reassignment has advantages, provided the change you are experimenting on is not disruptive to end users. In practice, this is the case for many experiments. A common example is "user-day experiments," which run for only one day at a time.
For experiments where contexts should not leave the variation they're initially allocated to, such as significant navigation menu changes or large user interface (UI) changes, you can disable variation reassignment. However, keeping contexts in the same variations incurs additional risk.
Using fresh traffic and freeing the previous ramp’s traffic prevents carry-over effects and ensures you don't run out of traffic for your experiment.
Disable variation reassignment
Allowing variation reassignment should be the default for your experiments. If you are unsure if you need to allow or disallow variation reassignment, we suggest you allow variation reassignment.
Expand Disabling variation assignment
You should only rarely prevent contexts from switching variations during an experiment by disabling variation reassignment. You may want to disable variation reassignment for substantial changes, such as major navigation menu or UI changes. You can disable variation reassignment by clicking Advanced, then checking the Prevent variation reassignment when increasing traffic checkbox.
Checking the Prevent variation reassignment when increasing traffic checkbox prevents variation reassignment only if you change the amount of traffic in your experiment audience by editing a running experiment using the experiment's Edit button. To learn more, read Edit experiments.
If you instead use the Stop button to stop an experiment iteration, change the amount of traffic in your experiment audience, then start a new iteration, LaunchDarkly will still reshuffle traffic into new variations.
Here is an example of running an experiment with variation reassignment disabled: you add an experiment to a flag with three variations: A, B, and C. You roll out the three variations to 6% of contexts, while the remaining 94% receives the control, variation A. The control traffic is not part of the experiment nor its analysis.
Here is a visualization of the starting traffic allocation, with the control group on the right:
Next, you ramp up your experiment to 30% of traffic. This creates a new iteration of the experiment. In the new iteration, the 6% that were receiving variations A, B, and C, continue to receive those variations, but are no longer included in the experiment nor its analysis. New traffic is used for the 30% allocated to the experiment.
Here is an example of the modified allocation, with the control group on the right and the original experiment audience on the left: