Creating mutually exclusive experiments

Overview

This guide explains how to create mutually exclusive experiments. Mutually exclusive experiments are two or more experiments configured to prevent contexts from being included in more than one at a time.

Generally, it is not a problem to include a context in multiple experiments. Our Experimentation offering minimizes interaction effects using audience allocation, and you can be confident in the winning variation even when running multiple concurrent experiments with overlapping audiences.

However, there are some cases in which you may not want to include each context in more than one experiment at a time. For example, you may be concerned about collisions between experiments that are testing similar parts of your app, like two different changes to the same section of your app’s user interface (UI), or experiments running on both the back end and front end of the same functionality.

Ideally, you would run experiments with potential collisions sequentially or include all of the related changes in the same experiment. However, if you are unable to do that due to time or other constraints, you can create mutually exclusive experiments using prerequisite flags.

In this tutorial you will:

• Create a prerequisite flag
• Create two dependent flags
• Include each dependent flag in its own mutually exclusive experiment

Controlling experiment audiences with flag variations

You can control experiment audiences with flag variations. First, you can create a prerequisite flag with variations that correspond to each of the mutually exclusive experiments you want to run. Then, you can use a percentage rollout to randomly assign each context to only one of those exclusive experiments.

If you want LaunchDarkly to include all of your contexts in one of the experiments, add the percentage rollout to the default rule in the prerequisite flag. If you want LaunchDarkly to include only some of your contexts in the experiments, add the percentage rollout to a targeting rule in the prerequisite flag.

In this example, the prerequisite flag uses a targeting rule to limit contexts in the experiments to only those on an iOS device:

Prerequisites

To complete this tutorial, you must have the following prerequisites:

• An active LaunchDarkly account with Experimentation enabled, and with permissions to create flags and edit experiments.
• Familiarity with the Experimentation feature. To learn more, read Experimentation.

Concepts

To complete this guide, you should understand the following concepts:

Prerequisite flags

Prerequisites allow you to control feature dependencies in LaunchDarkly. You can configure feature flags to depend on other flags being enabled to take effect, making those other flags prerequisites to enable the feature. To learn more, read Flag prerequisites.

Creating the prerequisite flag

In this example, you will create a string prerequisite flag. You will assign 50% of contexts to the Experiment 1 variation, and 50% of contexts to the Experiment 2 variation. Later, when you create your two mutually exclusive experiments, the flags those experiments include will be dependents of this flag.

To create the prerequisite flag:

1. Navigate to the flags list.
2. Click Create flag.
3. Enter "Mutually exclusive experiment prereq" in the Name field.
4. Enter a description such as "The prerequisite flag for two mutually exclusive experiments" in the Description field.
5. (Optional) Update the Maintainer for the flag.
6. Select the String flag variation type.
7. Enter "Experiment 1" in the first Name field.
8. Enter "1" in the first Value field.
9. Enter "Experiment 2" in the second Name field.
10. Enter "2" in the second Value field.

11. Click Create flag. The flag’s Targeting tab appears.
12. In the "Default rule" serve menu, select "a percentage rollout."

• (Optional) If you want to include only some of your contexts in the experiments, targeted by attribute, add the percentage rollout to a targeting rule instead of the default rule.

13. Update the rollout so 50% of contexts are served Experiment 1, and 50% of contexts are served Experiment 2.

14. Click Review and save.

Next, you will create your dependent flags.

Creating dependent flags

Next, create the two dependent flags you will include in your two experiments.

To create the first dependent flag:

1. Navigate to the flags list.
2. Click Create flag.
3. Enter "Mutually exclusive flag 1" in the Name field.
4. Enter a description such as "The dependent flag in mutually exclusive experiment #1" in the Description field.
5. (Optional) Update the Maintainer for the flag.
6. (Optional) Check the Include flag in this project's release pipeline box.
7. Select the Experiment flag template.
8. Select the flag type and variations as needed for your experiment.
9. Click Create flag. The flag’s Targeting tab appears.
10. Click + Add rule and select "Set prerequisites."
11. Choose "Mutually exclusive experiment prereq" from the Flag menu.
12. Leave the variation set to "Experiment 1."

13. Click Review and save.

To create the second dependent flag:

1. Navigate to the flags list.
2. Click Create flag.
3. Enter "Mutually exclusive flag 2" in the Name field.
4. Enter a description such as "The dependent flag in mutually exclusive experiment #2" in the Description field.
5. (Optional) Update the Maintainer for the flag.
6. (Optional) Check the Include flag in this project's release pipeline box.
7. Select the Experiment flag template.
8. Select the flag type and variations as needed for your experiment.
9. Click Create flag. The flag’s Targeting tab appears.
10. Click + Add rule and select "Set prerequisites."
11. Choose "Mutually exclusive experiment prereq" from the Flag menu.
12. Change the variation to "Experiment 2."
13. Click Review and save.

You now have three related flags in your flags list. Next, you will include the two dependent flags in two separate experiments.

Creating experiments

Creating an experiment requires four steps:

1. Creating a metric
2. Building the experiment
3. Turning on the feature flag
4. Starting an iteration

Creating your metrics

You can use the same metric or two different metrics for each of your experiments. This tutorial demonstrates using two different custom conversion/binary metrics for each experiment. To learn more, read Custom conversion/binary metrics.

To create the first custom metric:

1. Navigate to the Metrics list.
2. Click Create metric. The "Create metric" panel appears.
3. Enter "Experiment 1 custom metric" in the Name field.
4. Choose Custom in the "Event information" section.
5. Select Conversion/binary:

6. Enter the existing event key from your codebase in the Event key field. In this example, the event key is "Experiment 1 metric."

7. Select "Average individual unit values" for the Unit aggregation method.
8. Choose the user Randomization unit for the metric to measure events from.
9. Click Create metric.

To create the second custom metric, repeat steps 1-8, except name the metric "Experiment 2 custom metric" and use the appropriate event key from your codebase.

Building your experiments

Now it’s time to build your two mutually exclusive experiments.

To build your first experiment:

1. Navigate to the Experiments list.
2. Click Create experiment.
3. Enter "Mutually exclusive experiment 1" in the Experiment name field.
4. Enter a Hypothesis for your experiment.
5. Choose the user context kind as the randomization unit.
6. Click Next. The "Select metrics" step opens.
7. Choose "Experiment 1 custom metric" from the Primary metric menu.

8. Click Next. The "Define variations" step opens.
9. Choose the "Mutually exclusive flag 1" flag from the Select flag menu.
10. Click Next. The "Set audience" step opens.
11. Enter the percentage of traffic for each variation you want to include in the experiment.
12. Select which variation you want LaunchDarkly to serve to the remaining population.

13. Click Finish. You are returned to the experiment's Design tab.

To create your second experiment, repeat steps 1-13, except choose "Experiment 2 custom metric" and "Mutually exclusive flag 2."

Toggling on your flags

When you are ready to run your experiments, toggle your prerequisite flag and both dependent flags to On.

Starting experiment iterations

The last step is to start an iteration for each of your mutually exclusive experiments.

To start your experiment iterations:

1. Navigate to the Experiments list.
2. Click on "Mutually exclusive experiment 1."
3. Click Start.
4. Repeat steps 1-3 for "Mutually exclusive experiment 2."

You are now running two mutually exclusive experiments, with no overlap between the contexts included in each.

Conclusion

In this guide, you learned how to create a prerequisite flag and two dependent flags to prevent contexts from being included in two concurrent experiments.

Mutually exclusive experiments are not limited to two at a time. When it's time to create your own, you can create as many mutually exclusive experiments as you need to prevent collisions between experiments running on similar parts of your app or infrastructure.