No results for ""
  • Home
  • API docs

Creating feature change experiments

Read time: 6 minutes
Last edited: Jul 10, 2024


This topic explains how to set up and configure a feature change experiment in LaunchDarkly.

Configuring a feature change experiment requires four steps:

  1. Creating a metric,
  2. Building an experiment,
  3. Turning on the feature flag, and
  4. Starting an iteration.

These steps are explained in detail below.


Before you build an experiment, you should read about and understand the following concepts:

Create metrics

Metrics measure audience behaviors affected by the flags in your experiments. You can use metrics to track all kinds of things, from how often end users access a URL to how long that URL takes to load a page. You can reuse metrics in multiple experiments, or create new ones for your feature change experiment. To learn how to create a new metric, read Metrics.

If you want to learn which flag variation performs better, it must be possible for that metric to measure something in all of the flag variations within the experiment. To learn more, read Metrics and flag variations.

You can use one or more metrics or standard metric groups in feature change experiments. However, you cannot use funnel metric groups in feature change experiments. To learn more, read Metric groups.

Build experiments

You can view all of the experiments in your environment on the Experiments list.

To build an experiment:

  1. Navigate to the Experiments list.
  2. Click Create experiment.
  3. The "Experiment details" step opens. Enter a Name.
  4. Enter a Hypothesis.
  5. Select the Feature change experiment type.
  6. (Optional) If you want to include the experiment in a holdout, click Advanced, then select a Holdout name.
Experiments cannot be in a holdout and in a layer at the same time

Experiments can either be in a holdout, or in a layer, but not both. If an experiment is part of a holdout, you will not see the option to add it to a layer when you create it. To learn more, read Mutually exclusive experiments.

  1. Click Next. The "Choose randomization unit and attributes" step opens.
  2. Choose a context kind from the Randomization unit menu. If you included the experiment in a holdout, the experiment's randomization unit is automatically set to the holdout's randomization unit.
  3. (Optional) Choose up to five context attributes to slice results by.
  4. Click Next. The "Select metrics" step opens.
  5. Add a Metric or Metric group, or click Create to create and use a new metric or new standard metric group. The metrics you select must use the same randomization unit as the experiment.
    • A list of environments displays. It shows which environments have received events for these metrics. If no environments are receiving events, check that your SDKs are configured correctly.
  6. (Optional) If you have added multiple metrics, click on the crown icon to change the primary metric:
A metric group with the primary metric denoted by a crown icon.
A metric group with the primary metric denoted by a crown icon.
  1. Click Next. The "Choose flag variations" step opens.
  2. Choose a Flag to use in the experiment, or click Create flag to create and use a new flag.
  3. Click Next. The "Set audience" step opens.
  4. (Optional) Check the Add experiment to layer checkbox if you want to exclude contexts in this experiment from other experiments.
Expand layer options

A layer is a set of experiments that cannot share traffic with each other. All of the experiments within a layer are mutually exclusive, which means that if a context is included in one experiment, LaunchDarkly will exclude it from any other experiments in the same layer. To learn more, read Mutually exclusive experiments.

To add the experiment to a layer:

  • Click Select layer to add the experiment to an existing layer.
    • Select the layer you want to add the experiment to.
  • If you do not have a layer configured or need to create a new layer, click Create layer.
    • Add a Name and Description.
    • Click Next.
  • Add a Reservation amount. This is the percentage of the contexts within this layer you want LaunchDarkly to include in this experiment.
  1. In the "Define audience" section, choose which targeting rule to run the experiment on.
  • If you want to restrict your experiment audience to only contexts with certain attributes, create a targeting rule on the flag you include in the experiment and run the experiment on that rule.
  • If you don't want to restrict the audience for your experiment, run the experiment on the flag's default rule. If the flag doesn't have any targeting rules, the flag's default rule will be the only option.
The "Define audience" section with the default rule chosen.
The "Define audience" section with the default rule chosen.
  1. Enter the percentage of traffic you want to include in the experiment.
  2. Select which control variation you want LaunchDarkly to serve to the remaining population.
  • (Optional) Advanced: We strongly recommend leaving the Advanced options on their default settings. To learn more, read About carryover bias.

    The audience section of a new experiment.
    The audience section of a new experiment.
  1. Choose how you want to split the experiment audience:
  • Equally splits the experiment traffic equally between all variations in the experiment.
  • Percentage allows you to select specific percentages for each variation.
  1. Click Finish. You are returned to the experiment's Design tab.

After you have created your experiment, the next steps are to toggle on the flag and start an iteration.

You can also use the REST API: Create experiment

Turn on feature flags

For an experiment to begin recording data, the flag used in the experiment must be on. To learn how, read Turning flags on and off.

You can build multiple experiments on the same flag, but you can run only one of those experiments on the flag at a time.

Start experiment iterations

After you create an experiment and toggle on the flag, you can start an experiment iteration in one or more environments.

To start an experiment iteration:

  1. Navigate to the Experiments list.
  2. Click on the environment section containing the experiment you want to start.
  • If the environment you need isn't visible, click the + next to the list of environment sections. Search for the environment you want, and select it from the list.
The environment selection menu.
The environment selection menu.
  1. Click on the name of the experiment you want to start an iteration for. The Design tab appears.
  2. Click Start.
  3. Repeat steps 1-4 for each environment you want to start an iteration in.
An experiment with the "Start" button called out.
An experiment with the "Start" button called out.

Experiment iterations allow you to record experiments in individual blocks of time. To ensure accurate experiment results, when you make changes that impact an experiment, LaunchDarkly starts a new iteration of the experiment.

To learn more about starting and stopping experiment iterations, read Managing experiments.

You can also use the REST API: Create iteration