Creating experiments

Overview

This topic explains how to set up and configure an experiment in LaunchDarkly. It introduces the concepts of metrics, explains different metric types, and explains how metrics interact with feature flags to create experiments.

In LaunchDarkly, the combination of metrics and flags is an experiment. Experiments let you measure the effect of flags on end users by mapping them to the metrics your team cares about. To learn more, read Designing experiments.

Configuring experiments

When an end user performs a metric-tracked action in your application after they encounter a feature flag, the experiment logs an event.

For example, an experiment might show that end users are more likely to convert by clicking "Checkout" when the checkout button is a certain color. The metric you would track is the number of times end users click on the checkout button. You would connect the metric to a flag serving four variations, each of which is a different color for the button.

Configuring an experiment requires four steps:

1. Creating a metric,
2. Building an experiment,
3. Turning on the feature flag, and
4. Starting an iteration.

These steps are explained in detail below.

Prerequisites

To use Experimentation, you must have the following prerequisites:

You must use a supported SDK version

Most versions of LaunchDarkly SDKs support Experimentation, but if you use older versions of our SDKs, or you used an older version of Experimentation, compare your SDK versions to our list of supported SDKs to confirm that you're using a version that supports our most recent Experimentation offering.

To find the supported SDK versions, read Prerequisites in Experimentation.

Your SDKs must be configured to send events

All of your SDKs must send events. If you have disabled sending events for testing purposes, you must re-enable it. To learn more about the events SDKs send to LaunchDarkly, read Analytics events.

The all flags method sends events for some SDKs, but not others. For SDKs that do not send events with the all flags method, you must call the variation method instead. If you call the variation method, you must use the right variation type.

Your network must be allowed to send events

Your network must be allowed to send events to LaunchDarkly. Ensure that event streaming endpoints, mobile.launchdarkly.com, and events.launchdarkly.com are on your allow list.

You must be using unique context keys

To use Experimentation, you must configure your SDK to assign anonymous contexts their own unique context keys.

You must configure the Relay Proxy to send events

You do not have to use the Relay Proxy to use Experimentation. However, if you do use the Relay Proxy, you must configure it to send events. To learn more, read Configuring an SDK to use the Relay Proxy.

Creating metrics

Metrics measure audience behaviors affected by the flags in your experiments. You can use metrics to track all kinds of things, from how often end users access a URL to how long that URL takes to load a page. If end users load a URL, click an element, or otherwise participate in the behavior the metric tracks, LaunchDarkly sends an event to your experiment.

You don't need to create a new metric for each new experiment. You can reuse existing metrics in multiple experiments, which allows you to compare how the metric performs with different flags. Similarly, a single experiment can use primary and secondary metrics, which allow you to observe how the variations perform against various measurements.

Using primary and secondary metrics

You can designate only one metric as the primary metric in an experiment, but you can attach secondary metrics to your experiments if you want to track the performance of additional measurements. We recommend using no more than ten metrics per experiment.

The primary metric is sometimes called the "overall evaluation criterion." When you are making decisions about the winning variation in an experiment, you should base your decision making only on your primary metric, because decision making becomes much more complicated when you include multiple metrics in a decision.

If you are using just one metric, there are two possible outcomes: up or down. If you are using two metrics, there are four possible outcome combinations: up/down, down/up, up/up, or down/down.

For each metric you add to an experiment, the possible outcomes increase quickly. If you are using three metrics, there are eight different possible outcome combinations. If you are using ten metrics, there are 1,024 possible outcome combinations. For this reason we recommend basing your decision-making on only your primary metric.

To learn how to create a new metric, read Metrics.

Building experiments

You can view all of the experiments in your environment on the Experiments list.

Before you build an experiment for the first time, you should read about and understand randomization units, primary and secondary metrics, and attribute filters.

To build an experiment:

1. Navigate to the Experiments list.
2. Click Create experiment.
3. The "Experiment details" step opens. Enter a Name.
4. Enter a Hypothesis.
5. Click Next. The "Choose randomization unit and attributes" step opens.
6. Choose a context kind from the Randomization unit menu.
7. (Optional) Choose up to five context attributes to slice results by.
8. Click Next. The "Select metrics" step opens.
9. Choose a metric from the Primary metric menu, or click Create a new metric.
10. (Optional) Choose additional metrics from the Secondary metrics menu.

11. Click Next. The "Choose flag variations" step opens.
12. Choose a flag to use in the experiment from the Flag menu.
13. Click Next. The "Set audience" step opens.
14. Choose which targeting rule to run the experiment on.

• If you want to restrict your experiment audience to only contexts with certain attributes, create a targeting rule on the flag you include in the experiment and run the experiment on that rule.
• If you don't want to restrict the audience for your experiment, run the experiment on the flag's default rule. If the flag doesn't have any targeting rules, the flag's default rule will be the only option.

15. Enter the percentage of traffic for each variation you want to include in the experiment. You must include at least two variations for the experiment to be valid.
16. Select which variation you want LaunchDarkly to serve to the remaining population.

• (Optional) Advanced: We strongly recommend leaving the Advanced options on their default settings. To learn more, read Understanding variation reassignment.

17. Click Finish. You are returned to the experiment's Design tab.

After you have created your experiment, the next steps are to toggle on the flag and start an iteration.

Turning on feature flags

For an experiment to begin recording data, the flag used in the experiment must be on. To learn how, read Turning flags on and off.

You can build multiple experiments on the same flag, but you can run only one of those experiments on the flag at a time.

Starting experiment iterations

After you create an experiment and toggle on the flag, you can start an experiment iteration in one or more environments.

To start an experiment iteration:

1. Navigate to the Experiments list in the environment you want to start an iteration in.
2. Click on the name of the experiment you want to start an iteration for. The Results tab appears.
3. Click Start.
4. Repeat steps 1-3 for each environment you want to start an iteration in.

Experiment iterations allow you to record experiments in individual blocks of time. To ensure accurate experiment results, when you make changes that impact an experiment, LaunchDarkly starts a new iteration of the experiment.

To learn more about starting and stopping experiment iterations, read Managing experiments.

Archiving experiments

You can archive experiments that have concluded, as well as the flags and metrics attached to them, but you cannot permanently delete any of them. Archiving experiments preserves the results so you can refer to them in the future.

When you archive an experiment, you cannot start new iterations for it. LaunchDarkly hides archived experiments from the Experiments list by default.

To archive an experiment:

1. Navigate to the Experiments list in the environment you want to archive an experiment in.
2. Click on the name of the experiment you want to archive. The experiment detail page appears.
3. Click Archive experiment.

To view archived experiments on the Experiments list, click on the overflow menu and select "View archived experiments." To switch back to viewing active experiments, click on the overflow menu and select "View active experiments."

To restore an experiment, click on an experiment's Overflow menu from the Experiments list and select "Restore experiment. Or, click Restore experiment from the experiment details page.