Holdouts
Read time: 2 minutes
Last edited: Oct 18, 2024
Holdout experiments are only available with server-side SDKs. You cannot use holdout experiments with client-side SDKs.
Overview
This topic introduces holdouts, which you can use to measure the effectiveness of your Experimentation program over time.
About holdouts
Holdouts let you exclude a percentage of your audience from all experiments over a set amount of time. This can help you understand how much of an impact your experiments have on certain metrics.
For example, you can hold back 5% of your user base from all of the experiments you run in a certain quarter. At the end of the quarter, you can compare the 95% that were included in experiments to the 5% that were excluded to find out which group spent more money overall, or which group signed up for services at a higher rate. If there are no measurable differences between the two groups you may want to reconsider the number, scope, and design of the experiments you're running.
Holdout experiments and mutually exclusive experiments are similar, but are used for different reasons. Holdout experiments withhold a set of end users from all experiments. Mutually exclusive experiments use layers to ensure end users are not included in two or more specific experiments at the same time. To learn more, read Mutually exclusive experiments.
Prerequisites
To use holdouts, you should understand the following concepts:
To learn how to create and manage holdouts, read Creating holdouts and Managing holdouts.
To learn how to build and run an example holdout, read Measuring Experimentation impact with holdouts.