• Home
  • Integrations
  • SDKs
  • Guides
  • API docs
No results for ""
EXPAND ALL

EDIT ON GITHUB

Creating experiments

Read time: 6 minutes
Last edited: May 20, 2022

Overview

This topic explains how to set up and configure an experiment in LaunchDarkly. It introduces the concepts of metrics, explains different metric types, and explains how metrics interact with feature flags to create experiments.

In LaunchDarkly, the combination of metrics and flags is an experiment. Experiments let you measure the effect of flags on users by mapping them to the metrics your team cares about.

Understanding Experimentation

When a user performs a metric-tracked action in your application after they encounter a feature flag, the experiment logs an event.

For example, an experiment might show that users are more likely to convert by clicking "Checkout" when the checkout button is a certain color. The metric you would track is the number of times users click on the checkout button. You would connect the metric to a flag serving four variations, each of which is a different color for the button.

Configuring an experiment requires five steps:

  1. Creating a metric,
  2. Defining the experiment population,
  3. Building an experiment,
  4. Starting an experiment iteration, and
  5. Turning on the feature flag

These steps are explained in detail below.

Prerequisites

To use Experimentation, you must have the following prerequisites:

Your SDKs must be configured to send events

All of your SDKs must send events. If you have disabled sending events for testing purposes, you must re-enable it. To learn more about the events SDKs send to LaunchDarkly, read Analytics events.

The all flags method sends events for some SDKs, but not others. For SDKs that do not send events with the all flags method, you must call the variation method instead. If you call the variation method, you must use the right variation type. To learn more, read Evaluating flags.

Your network must be allowed to send events

Your network must be allowed to send events to LaunchDarkly. Ensure that event streaming endpoints, mobile.launchdarkly.com, and events.launchdarkly.com are on your allow list.

You must be using unique user keys

Your anonymous users must have their own unique user keys to use Experimentation. To learn more, read Designating anonymous users.

You must configure the Relay Proxy to send events

You do not have to use the Relay Proxy to use Experimentation. However, if you do use the Relay Proxy, you must configure it to send events. To learn more, read Configuring an SDK to use the Relay Proxy.

Creating metrics

Metrics are the measure against which you evaluate flag events. You can use metrics to track all kinds of things, from how often users access a URL to how long that URL takes to load a page.

After you create a metric, you can connect it to one or more feature flags. If users load a URL, click an element, or otherwise participate in the behavior the metric tracks, LaunchDarkly sends an event to your experiment.

You don't need to create a new metric for each new experiment. You can reuse existing metrics by connecting them to multiple flags, which allows you to compare how this metric fares across numerous flags. Similarly, a single flag can have more than one metric connected to it, which allows you to observe how this flag performs against various measurements.

To learn how to create a new metric, read Creating metrics.

Defining experiment populations

You have the option to only include a subset of users in your experiments, which gives you accurate experiment results more quickly. This subset of users is called your "experiment population."

You can create a custom experiment population by selecting specific flag targeting rules to include in your active experiments.

For example, imagine you are testing alternate copy for your checkout button. You're currently targeting all your Canadian and US users with the true variation for the alternate button, which shows your new copy, but you only want to run an experiment measuring click conversions for users in the United States. To accomplish this, you select the user targeting rule that affects US-based users and de-select the rule that targets users in Canada.

You may want to refine your experiment population for any of the following reasons:

  • To run targeted experiments for a subset of your flag-targeted users.
  • To exclude user groups whose events you do not need to measure. For example, users affected by 'Default' rules.
  • To reduce the volume of events an experiment generates in order to manage billing costs.

When making changes to ramp up or ramp down users in your variations, LaunchDarkly properly randomizes users and provides the configuration and guidance to prevent carryover bias. This method of managing your experiment population is called "audience allocation." To learn more, read Allocating experiment populations.

Building experiments

You can view all the experiments in your environment by clicking Experiments in the sidebar menu.

Here is an image of the Experiments dashboard:

The "Experiments" dashboard.
The "Experiments" dashboard.
Creating an experiment is different for Early Access Program customers

The process of building an experiment is different for Early Access Program customers. If you are an Early Access Program customer, read Building experiments using the experiment builder.

If you are not an Early Access Program customer, read Building experiments from the dashboard.

Building experiments from the dashboard

There are two ways to build experiments:

  • From the Experiments dashboard, and
  • From a flag's Experimentation tab.

To build an experiment from the Experiments dashboard:

  1. Navigate to the Experiments dashboard.
  2. Click Create experiment:
The "Experiments" dashboard with the "Create experiment" button called out.
The "Experiments" dashboard with the "Create experiment" button called out.

The "Create experiment" panel appears.

  1. Choose a flag from the Select flag menu.
  2. Choose a metric from the Connect metric menu.
  3. Choose a baseline variation from the Select baseline menu.
  4. Click Create experiment. Your new experiment appears on the dashboard.
  5. Navigate to the Targeting tab of the feature flag connected to your experiment.
  6. Click the graph icon next to the rule you want to run an experiment on, and select “Run an experiment on this rule.”
The "graph" icon with the "Run an experiment on this rule" option called out.
The "graph" icon with the "Run an experiment on this rule" option called out.
  1. Enter the percentage of traffic for each variation you want to include in the experiment.
  2. Select which variation you want LaunchDarkly to serve to the remaining population.
  3. (Optional) To disable traffic variation reassignment, click Advanced, then check the Keep users assigned to initial variation checkbox. We strongly recommend leaving this box unchecked. To learn more, read Understanding variation reassignment.

You can also use the REST API: Create experiment

After you create an experiment, you must start a new iteration in every environment where you want it to run. To learn more, read Starting experiment iterations.

Click to expand details on building experiments from a flag's Experimentation tab

To build an experiment from a flag's Experimentation tab:

  1. Navigate to the flag's Experimentation tab.
  2. Click Create an Experiment or, if you have already configured an experiment using this flag, click Manage metrics:
A flag's "Experimentation" tab with the "Manage metrics" button called out.
A flag's "Experimentation" tab with the "Manage metrics" button called out.

The "Experiment Setup" panel appears.

  1. Select a metric from the Add new metric menu. The metric you choose appears in the table.
  2. Select a baseline from the Baseline menu.
  3. Click Save experiment settings. The new experiment appears in the flag's Experimentation tab.
The "Experiment Setup" panel, configured to connect a metric to a flag.
The "Experiment Setup" panel, configured to connect a metric to a flag.
  1. Navigate to the flag's Targeting tab.
  2. Click the graph icon next to the rule you want to run an experiment on, and select “Run an experiment on this rule.”
The "graph" icon with the "Run an experiment" option called out.
The "graph" icon with the "Run an experiment" option called out.
  1. Enter the percentage of traffic for each variation you want to include in the experiment.
  2. Select which variation you want LaunchDarkly to serve to the remaining population.
  3. (Optional) To disable traffic variation reassignment, click Advanced, then check the Keep users assigned to initial variation checkbox. We strongly recommend leaving this box unchecked. To learn more, read Understanding variation reassignment.

After you create an experiment, you must start a new iteration in every environment where you want it to run. To learn how, read Starting experiment iterations.

To learn how to change the experiment population to something other than the flag's current targeting rules, read Managing experiment populations.

Building experiments using the experiment builder

The experiment builder is for Early Access Program customers only

The experiment builder is limited to a subset of customers in LaunchDarkly's Early Access Program (EAP). If you want access to this feature, join the EAP.

Click to expand details on building experiments using the experiment builder

To build an experiment using the experiment builder:

  1. Navigate to the Experiments dashboard or the Experimentation tab of a flag.
  2. Click Create experiment:
The "Experiments" dashboard with the "Create experiment" button called out.
The "Experiments" dashboard with the "Create experiment" button called out.
  1. Select a metric from the Metric menu.
  2. Click Next.
  3. Select a flag from the Select flag menu. You cannot select any flags that are already part of an experiment.
  4. Click Next.
  5. Enter the percentage of traffic for each variation you want to include in the experiment:
The "Set audience allocation" dialog.
The "Set audience allocation" dialog.
  1. Click Finish. You are returned to the Experimentation tab of the feature flag you included in the experiment.

After you create an experiment, you must start a new iteration in every environment where you want it to run. To learn how, read Starting experiment iterations.

To learn how to update the audience allocation after you create an experiment, read Managing experiment populations.

Starting experiment iterations

After you create an experiment, you can start an experiment iteration in one or more environments. From a flag's Experimentation tab, click the Start recording button in every environment where you want to record data.

Experiment iterations allow you to record experiments in discrete blocks of time. To ensure accurate experiment results, when you make changes that impact an experiment’s audience or variations it is best practice to start a new iteration of the experiment.

To learn more about starting and stopping experiment iterations, read The experiment lifecycle.

Experiment settings are environment-specific

When you create an experiment, it appears in every environment in your project. However, the Start recording button only impacts the experiment in one environment at a time. If you want to start a new iteration of the experiment in multiple environments, you must click Start recording for each environment individually. For example, you might run an experiment in Staging to gather data internally before turning it on in Production to gather customer data.

You can also use the REST API: Create iteration

Turning on feature flags

For an experiment to begin recording data, the flags connected to its metrics must be toggled on. To learn how, read Turning flags on and off.