No results for ""
EXPAND ALL
  • Home
  • API docs

Holdouts

Read time: 4 minutes
Last edited: Jul 03, 2024
You must be using the multi-environment view to use holdouts

To use holdouts, you must be using the multi-environment view in the LaunchDarkly user interface. To learn more, read the latest Release notes.

Overview

This topic explains how to use holdouts to measure the effectiveness of your Experimentation program over time.

About holdouts

Holdouts let you exclude a percentage of your audience from your Experimentation program. This enables you to see the overall effect of your experiments on your customer base, and helps determine how effective the experiments you're running are.

The holdouts list.
The holdouts list.

Prerequisites

To use holdouts, you should understand the following concepts:

  • Experiment audiences
  • Randomization units
  • Analyzing experiment results by attribute

Create holdouts

Before you create a holdout, you must decide the following:

  • How long to run the holdout for: we recommend anywhere from 1-3 months.
  • What percentage of your customer base to include in the holdout: we recommend 1-5%.
  • Whether to include all of your experiments in the holdout, or if the holdout will be specific to a certain product area.

To create a holdout:

  1. Navigate to the Holdouts list from the left sidenav.
  2. Click Create holdout. The "Holdout details" section appears.
  3. Enter a Name.
  4. Enter a Description that includes information about what kinds of experiments should be included in this holdout.
  5. Enter a Holdout amount. We recommend holding out between 1-5% of your audience.
  6. Click Next. The "Choose randomization unit and attributes" section appears.
  7. Select a Randomization unit.
  8. Add any Attributes to analyze your results by.
  9. Click Next. The "Select metrics" section appears.
  10. Select one or more metric or metric groups.
  11. Click Finish.

You can also use the REST API: Create holdout

Add experiments to holdouts

Experiments cannot be in a holdout and in a layer at the same time

Experiments can either be in a holdout, or in a layer, but not both. If an experiment is part of a holdout, you will not see the option to add it to a layer when you create it. To learn more, read Mutually exclusive experiments.

When you create a new experiment, you can decide whether to include it in an existing holdout. To learn how, read Creating feature change experiments and Creating funnel experiments.

When you add an experiment to a holdout, the holdout appears as a prerequisite to the flag used in the experiment:

An experiment's flag's details page with a holdout prerequisite.
An experiment's flag's details page with a holdout prerequisite.

To include an experiment in a holdout, the experiment flag cannot have any other prerequisite flags.

You can also use the REST API: Create experiment

Manage holdouts

You can view all of a project's holdouts on the holdouts list:

The holdouts list.
The holdouts list.

Holdouts can have four possible states:

  • Created: you have not yet added any experiments to the holdout.
  • Ready: you have added at least one experiment added to the holdout, but none of the experiments have started.
  • Analyzing: at least one experiment is in the holdout, and it is currently running or it has one or more completed iterations.
  • Ended: the holdout is finished analyzing the holdout audience against the experiments' audiences, and everyone is now receiving the same experience.

Click on the name of a holdout to view its details page.

The details page lists the holdout's name, description, holdout amount, randomization unit, and any experiments included in the holdout:

The details page for a holdout with no experiments added yet.
The details page for a holdout with no experiments added yet.

You can also use the REST API: Get all holdouts

Read results

On the holdout details page, the Probability report tab displays the results of the metric for two variations:

  • "Not in holdout" includes any contexts included in an experiment within the holdout.
  • "In holdout" includes the contexts that were excluded from experiments within the holdout.

You can see which variation is performing better if least one of the experiments in the holdout is running. However, we don't recommend making any decisions about your holdout experiment until all of the experiments within the holdout are finished and you have shipped winning variations for all of them. To learn how, read Winning variations.

A holdout including experiments that are still running.
A holdout including experiments that are still running.

When you have recorded enough data, you can stop the holdout and analyze its results. To do this, navigate to the holdout details page and click End. The contexts that were in the holdout will no longer be excluded from future experiments.

You can then make a decision about the results of your holdout based on which variation is performing better:

  • If the "Not in holdout" variation is performing better, then this means your experiments are overall having a positive impact on the metric.
  • If the "In holdout" variation is performing better, then this means your experiments are overall having a negative impact on the metric, and you may want to examine the experiments you're running and the variations you're testing to figure out how you can build better or more effective experiments.

You can also use the REST API: Get holdout