Experiment types

Overview

This topic explains the different kinds of experiments you can run in LaunchDarkly, including feature change experiments, funnel optimization experiments, and Data Export experiments. Experiments let you measure the effect of flags on end users by mapping flags to the metrics your team cares about. To learn more, read Designing experiments.

Feature change experiments

Feature change experiments let you measure the effect different flag variations have on a metric. You can use feature change experiments to test a wide variety of feature changes. For examples of feature change experiments, read Example experiments.

Here is an example Results tab for a feature change experiment:

Metrics

Metrics measure audience behaviors affected by the flag variations in your experiments. You can use metrics to track all kinds of things, from how often end users access a URL to how long that URL takes to load a page. If end users load a URL, click an element, or otherwise participate in the behavior the metric tracks, LaunchDarkly sends an event to your experiment.

You don't need to create a new metric for each new experiment. You can reuse existing metrics in multiple experiments, which allows you to compare how the metric performs with different flags. Similarly, a single experiment can use primary and secondary metrics, which allow you to observe how the variations perform against various measurements.

Primary and secondary metrics

You can designate only one metric as the primary metric in a feature change experiment, but you can attach secondary metrics to your experiments if you want to track the performance of additional measurements. We recommend using no more than ten metrics per experiment.

The primary metric is sometimes called the "overall evaluation criterion." When you are making decisions about the winning variation in an experiment, you should base your decision-making only on your primary metric, because decision-making becomes much more complicated when you include multiple metrics in a decision.

When you create a feature change experiment, the primary metric is designated with a crown icon. After you add two or more metrics to an experiment, you can change the primary metric by clicking the crown icon of the new metric you want to designate as primary. To learn more, read Creating feature change experiments.

If you are using just one metric, there are two possible outcomes: better or worse. If you are using two metrics, there are four possible outcome combinations: better/worse, worse/better, better/better, or worse/worse.

For each metric you add to an experiment, the possible outcomes increase quickly. If you are using three metrics, there are eight different possible outcome combinations. If you are using ten metrics, there are 1,024 possible outcome combinations. For this reason we recommend basing your decision-making on only your primary metric.

To learn how to create a new metric, read Metrics.

Funnel optimization experiments

A "funnel" is a marketing model that describes a customer's journey through your purchasing or conversion cycle, typically from the awareness stage to the purchasing stage. LaunchDarkly's funnel optimization experiments use multiple metrics to track the performance of each of the steps in your funnel over time.

Funnel optimization experiments let your product and marketing teams safely and rapidly increase the performance of your funnels by:

• measuring the performance of your main acquisition and engagement channels
• identifying strong points and weak points in your funnel flow
• testing changes to ensure your funnel's effectiveness improves over time

Before you build a funnel optimization experiment, you must first create a funnel metric group that includes metrics that measure all of the steps in your funnel.

Metrics and metric groups

A metric group is a reusable, ordered list of metrics you can use to standardize metrics across multiple experiments. You can use funnel metric groups with funnel optimization experiments. To learn more, read Metric groups.

Custom conversion binary metrics and clicked or tapped metrics are most often used with funnel optimization experiments. To learn more, read Choose a metric type.

Funnel experiments can only use metrics that use the "Average" analysis method. To learn more, read Analysis method.

Data Export experiments

Data Export experiments let you export experiment data to a third-party tool for analysis using Data Export. Data Export experiments do not require a LaunchDarkly metric, and do not generate analysis results in the LaunchDarkly UI. This experiment type is useful when you want to use LaunchDarkly flags to manage your experiment, but have a preferred third-party analysis tool that allows you to import experiment data from other systems. To learn more, read Creating Data Export experiments.