Experiment types

Overview

This topic explains the different kinds of experiments you can run in LaunchDarkly, including feature change experiments and funnel optimization experiments.

In LaunchDarkly, the combination of metrics and flags is an experiment. Experiments let you measure the effect of flags on end users by mapping them to the metrics your team cares about. To learn more, read Designing experiments.

Feature change experiments

Feature change experiments let you measure the effect different flag variations have on a metric. You can use feature change experiments to test a wide variety of feature changes. For examples of feature change experiments, read Example experiments.

Here is an example Results tab for a feature change experiment:

Metrics

Metrics measure audience behaviors affected by the flag variations in your experiments. You can use metrics to track all kinds of things, from how often end users access a URL to how long that URL takes to load a page. If end users load a URL, click an element, or otherwise participate in the behavior the metric tracks, LaunchDarkly sends an event to your experiment.

You don't need to create a new metric for each new experiment. You can reuse existing metrics in multiple experiments, which allows you to compare how the metric performs with different flags. Similarly, a single experiment can use primary and secondary metrics, which allow you to observe how the variations perform against various measurements.

Using primary and secondary metrics

You can designate only one metric as the primary metric in a feature change experiment, but you can attach secondary metrics to your experiments if you want to track the performance of additional measurements. We recommend using no more than ten metrics per experiment.

The primary metric is sometimes called the "overall evaluation criterion." When you are making decisions about the winning variation in an experiment, you should base your decision-making only on your primary metric, because decision-making becomes much more complicated when you include multiple metrics in a decision.

If you are using just one metric, there are two possible outcomes: up or down. If you are using two metrics, there are four possible outcome combinations: up/down, down/up, up/up, or down/down.

For each metric you add to an experiment, the possible outcomes increase quickly. If you are using three metrics, there are eight different possible outcome combinations. If you are using ten metrics, there are 1,024 possible outcome combinations. For this reason we recommend basing your decision-making on only your primary metric.

To learn how to create a new metric, read Metrics.

Funnel optimization experiments

A "funnel" is a marketing model that describes a customer's journey through your purchasing or conversion cycle, typically from the awareness stage to the purchasing stage. LaunchDarkly's funnel optimization experiments use multiple metrics to track the performance of each of the steps in your funnel over time.

Funnel optimization experiments let your product and marketing teams safely and rapidly increase the performance of your funnels by:

• measuring the performance of your main acquisition and engagement channels
• identifying strong points and weak points in your funnel flow
• testing changes to ensure your funnel's effectiveness improves over time

Before you build a funnel optimization experiment, you must first create a funnel metric group that includes metrics that measure all of the steps in your funnel.

Here is an example Results tab for a funnel optimization experiment:

Metrics

Custom conversion/binary metrics and click metrics are most often used with funnel optimization experiments. To learn more, read Choosing a metric type.

Funnel experiments can only use metrics that use the "mean" analysis method. To learn more, read Analysis method.

Metric groups

A metric group is a reusable, ordered list of metrics you can use to standardize metrics across multiple experiments. You can use funnel metric groups with funnel optimization experiments. To learn more, read Metric groups.