Creating holdouts

Read time: 2 minutes

Last edited: Dec 06, 2024

Overview

This topic explains how to create and add experiments to holdouts.

Create holdouts

Before you create a holdout, you must decide the following:

How long to run the holdout for: we recommend anywhere from 1-3 months.
What percentage of your customer base to include in the holdout: we recommend 1-5%.
Whether to include all of your experiments in the holdout, or if the holdout will be specific to a certain product area.

To create a holdout:

Navigate to the Holdouts list from the left sidenav.
Click Create holdout. The "Holdout details" section appears.
Enter a Name.
Enter a Description that includes information about what kinds of experiments should be included in this holdout.
Enter a Holdout amount. We recommend holding out between 1-5% of your audience.
Click Next. The "Choose randomization unit and attributes" section appears.
Select a Randomization unit.
Add any Attributes to analyze your results by.
Click Next. The "Select metrics" section appears.
Select one or more metric or metric groups.
Click Finish.

You can also use the REST API: Create holdout

Add experiments to holdouts

Experiments cannot be in a holdout and in a layer at the same time

Experiments can either be in a holdout, or in a layer, but not both. If an experiment is part of a holdout, you will not see the option to add it to a layer when you create it. To learn more, read Mutually exclusive experiments.

When you create a new experiment, you can decide whether to include it in an existing holdout. To learn how, read Creating feature change experiments, Creating funnel experiments, and Creating Data Export experiments.

When you add an experiment to a holdout, the holdout appears as a prerequisite to the flag used in the experiment:

An experiment's flag's details page with a holdout prerequisite.

To include an experiment in a holdout, the experiment flag cannot have any other prerequisite flags.