Creating holdouts
Read time: 2 minutes
Last edited: Sep 11, 2024
Holdout experiments are only available with server-side SDKs. You cannot use holdout experiments with client-side SDKs.
Overview
This topic explains how to create and add experiments to holdouts.
Create holdouts
Before you create a holdout, you must decide the following:
- How long to run the holdout for: we recommend anywhere from 1-3 months.
- What percentage of your customer base to include in the holdout: we recommend 1-5%.
- Whether to include all of your experiments in the holdout, or if the holdout will be specific to a certain product area.
To create a holdout:
- Navigate to the Holdouts list from the left sidenav.
- Click Create holdout. The "Holdout details" section appears.
- Enter a Name.
- Enter a Description that includes information about what kinds of experiments should be included in this holdout.
- Enter a Holdout amount. We recommend holding out between 1-5% of your audience.
- Click Next. The "Choose randomization unit and attributes" section appears.
- Select a Randomization unit.
- Add any Attributes to analyze your results by.
- Click Next. The "Select metrics" section appears.
- Select one or more metric or metric groups.
- Click Finish.
You can also use the REST API: Create holdout
Add experiments to holdouts
Experiments can either be in a holdout, or in a layer, but not both. If an experiment is part of a holdout, you will not see the option to add it to a layer when you create it. To learn more, read Mutually exclusive experiments.
When you create a new experiment, you can decide whether to include it in an existing holdout. To learn how, read Creating feature change experiments and Creating funnel experiments.
When you add an experiment to a holdout, the holdout appears as a prerequisite to the flag used in the experiment:
To include an experiment in a holdout, the experiment flag cannot have any other prerequisite flags.
You can also use the REST API: Create experiment