No results for ""
EXPAND ALL
  • Home
  • API docs

GIVE DOCS FEEDBACK

Creating mutually exclusive experiments

Read time: 9 minutes
Last edited: May 01, 2024
Experimentation is available for Pro and Enterprise plans

Experimentation is available as an add-on to customers on a Pro or Enterprise plan. To learn more, read about our pricing. To add Experimentation to your plan, contact Sales.

Overview

This guide explains how to create mutually exclusive experiments. Mutually exclusive experiments are two or more experiments configured to prevent contexts from being included in more than one at a time.

Generally, it is not a problem to include a context in multiple experiments. Our Experimentation offering minimizes interaction effects using audience allocation, and you can be confident in the winning variation even when running multiple concurrent experiments with overlapping audiences.

However, there are some cases in which you may not want to include each context in more than one experiment at a time. For example, you may be concerned about collisions between experiments that are testing similar parts of your app, like two different changes to the same section of your app’s user interface (UI), or experiments running on both the back end and front end of the same functionality.

Ideally, you would run experiments with potential collisions sequentially or include all of the related changes in the same experiment. However, if you are unable to do that due to time or other constraints, you can create mutually exclusive experiments using prerequisite flags.

In this tutorial you will:

  • Create a prerequisite flag
  • Create two dependent flags
  • Include each dependent flag in its own mutually exclusive experiment

Control experiment audiences with flag variations

You can control experiment audiences with flag variations. First, you can create a prerequisite flag with variations that correspond to each of the mutually exclusive experiments you want to run. Then, you can use a percentage rollout to randomly assign each context to only one of those exclusive experiments.

If you want LaunchDarkly to include all of your contexts in one of the experiments, add the percentage rollout to the default rule in the prerequisite flag. If you want LaunchDarkly to include only some of your contexts in the experiments, add the percentage rollout to a targeting rule in the prerequisite flag.

In this example, the prerequisite flag uses a targeting rule to limit contexts in the experiments to only those on an iOS device:

A prerequisite flag with a targeting rule that targets contexts identified as iOS devices.
A prerequisite flag with a targeting rule that targets contexts identified as iOS devices.

Prerequisites

To complete this tutorial, you must have the following prerequisites:

  • An active LaunchDarkly account with Experimentation enabled, and with permissions to create flags and edit experiments.
  • Familiarity with the Experimentation feature. To learn more, read Experimentation.

Concepts

To complete this guide, you should understand the following concepts:

Prerequisite flags

Prerequisites allow you to control feature dependencies in LaunchDarkly. You can configure feature flags to depend on other flags being enabled to take effect, making those other flags prerequisites to enable the feature. To learn more, read Flag prerequisites.

Create the prerequisite flag

In this example, you will create a string prerequisite flag. You will assign 50% of contexts to the Experiment 1 variation, and 50% of contexts to the Experiment 2 variation. Later, when you create your two mutually exclusive experiments, the flags those experiments include will be dependents of this flag.

To create the prerequisite flag:

  1. Navigate to the flags list.
  2. Click Create flag.
  3. Enter "Mutually exclusive experiment prereq" for the Name.
  4. Enter a Description such as "The prerequisite flag for two mutually exclusive experiments."
  5. Select the String flag variation type.
  6. Enter "Experiment 1" in the first Name field.
  7. Enter "1" in the first Value field.
  8. Enter "Experiment 2" in the second Name field.
  9. Enter "2" in the second Value field.
The "Flag variations" section of a flag's creation panel.
The "Flag variations" section of a flag's creation panel.
  1. Click Create flag. The flag’s targeting page appears.
  2. In the "Default rule" serve menu, select "a percentage rollout."
  • (Optional) If you want to include only some of your contexts in the experiments, targeted by attribute, add the percentage rollout to a targeting rule instead of the default rule.
  1. Update the rollout so 50% of contexts are served Experiment 1, and 50% of contexts are served Experiment 2.
A default rule on a prerequisite flag serving a rollout including 50% of contexts in Experiment 1, and 50% of contexts in Experiment 2.
A default rule on a prerequisite flag serving a rollout including 50% of contexts in Experiment 1, and 50% of contexts in Experiment 2.
  1. Click Review and save.
Create an additional variation for each experiment

If you want to create three or more mutually exclusive experiments, add an additional variation for each experiment you want to include, and divide traffic between each variation.

Next, you will create your dependent flags.

Create dependent flags

Next, create the two dependent flags you will include in your two experiments.

To create the first dependent flag:

  1. Navigate to the flags list.
  2. Click Create flag.
  3. Enter "Mutually exclusive flag 1" for the Name.
  4. Enter a Description such as "The dependent flag in mutually exclusive experiment #1."
  5. Select the Experiment template.
  6. Add variations as needed for your experiment.
  7. Click Create flag. The flag’s targeting page appears.
  8. Click +Add rule and select Set prerequisites.
  9. Choose "Mutually exclusive experiment prereq" from the Flag menu.
  10. Leave the variation set to "Experiment 1."
The "Prerequisites" section of the dependent flag with a prerequisite flag added.
The "Prerequisites" section of the dependent flag with a prerequisite flag added.
  1. Click Review and save.

To create the second dependent flag:

  1. Navigate to the flags list.
  2. Click Create flag.
  3. Enter "Mutually exclusive flag 2" for the Name.
  4. Enter a Description such as "The dependent flag in mutually exclusive experiment #2."
  5. Select the Experiment template.
  6. Add variations as needed for your experiment.
  7. Click Create flag. The flag’s targeting page appears.
  8. Click +Add rule and select Set prerequisites.
  9. Choose "Mutually exclusive experiment prereq" from the Flag menu.
  10. Change the variation to "Experiment 2."
  11. Click Review and save.

You now have three related flags in your flags list. Next, you will include the two dependent flags in two separate experiments.

Create experiments

Creating an experiment requires four steps:

  1. Creating a metric
  2. Building the experiment
  3. Turning on the feature flag
  4. Starting an iteration

Create metrics

You can use the same metric or two different metrics for each of your experiments. This tutorial demonstrates using two different custom conversion/binary metrics for each experiment. To learn more, read Custom conversion/binary metrics.

To create the first custom metric:

  1. Navigate to the Metrics list.
  2. Click Create metric. The "Create metric" dialog appears.
  3. Enter "Experiment 1 custom metric" in the Name field.
  4. Choose Custom in the "Event information" section.
  5. Select Conversion/binary:
The "Event information" section of the "Create a new metric" panel with a conversion metric configured.
The "Event information" section of the "Create a new metric" panel with a conversion metric configured.
  1. Enter the existing event key from your codebase in the Event key field. In this example, the event key is "Experiment 1 metric."
Metric keys and event keys are different

LaunchDarkly automatically generates a metric key when you create a metric. You can use the metric key to identify the metric in API calls. To learn more, read Creating metrics.

Custom conversion/binary and custom numeric metrics also require an event key. You can set the event key to anything you want. Adding this event key to your codebase lets your SDK track actions customers take in your app as events. To learn more, read Sending custom events.

  1. Select Average individual unit values for the Unit aggregation method.
  2. Choose the user Randomization unit for the metric to measure events from.
  3. Click Create metric.

To create the second custom metric, repeat steps 1-8, except name the metric "Experiment 2 custom metric" and use the appropriate event key from your codebase.

Build the experiments

Now it’s time to build your two mutually exclusive experiments.

To build your first experiment:

  1. Navigate to the Experiments list.
  2. Click Create experiment.
  3. Enter "Mutually exclusive experiment 1" in the Experiment name field.
  4. Enter a Hypothesis for your experiment.
  5. Choose the user context kind as the randomization unit.
  6. Click Next. The "Select metrics" step opens.
  7. Choose "Experiment 1 custom metric" from the Primary metric menu.
The "Select metrics" section of a new experiment.
The "Select metrics" section of a new experiment.
  1. Click Next. The "Define variations" step opens.
  2. Choose the "Mutually exclusive flag 1" flag from the Select flag menu.
  3. Click Next. The "Set audience" step opens.
  4. Enter the percentage of traffic for each variation you want to include in the experiment.
  5. Select which variation you want LaunchDarkly to serve to the remaining population.
The "Set audience" section of a new experiment.
The "Set audience" section of a new experiment.
  1. Click Finish. You are returned to the experiment's Design tab.

To create your second experiment, repeat steps 1-13, except choose "Experiment 2 custom metric" and "Mutually exclusive flag 2."

Toggle on flags

When you are ready to run your experiments, toggle your prerequisite flag and both dependent flags to On.

Start experiment iterations

The last step is to start an iteration for each of your mutually exclusive experiments.

To start your experiment iterations:

  1. Navigate to the Experiments list.
  2. Click on "Mutually exclusive experiment 1."
  3. Click Start.
  4. Repeat steps 1-3 for "Mutually exclusive experiment 2."

You are now running two mutually exclusive experiments, with no overlap between the contexts included in each.

Conclusion

In this guide, you learned how to create a prerequisite flag and two dependent flags to prevent contexts from being included in two concurrent experiments.

Mutually exclusive experiments are not limited to two at a time. When it's time to create your own, you can create as many mutually exclusive experiments as you need to prevent collisions between experiments running on similar parts of your app or infrastructure.

Want to know more? Start a trial.

Your 14-day trial begins as soon as you sign up. Get started in minutes using the in-app Quickstart. You'll discover how easy it is to release, monitor, and optimize your software.

Want to try it out? Start a trial.