Avoiding carryover bias

Overview

This topic explains what carryover bias is and how to avoid it in your experiments.

About carryover bias

Experiments are subject to day-of-week or hour-of-day effects. For example, an end user’s behavior on a website is often different depending on if they are visiting on a Saturday or a Monday. This can cause problems in properly computing experiment results if you use traffic from two different time frames when you increase the percent of traffic going to various contexts. This is called carryover bias.

To prevent carryover bias, LaunchDarkly uses variation reassignment when you increase traffic in an experiment.

Variation reassignment

It may seem counterintuitive to allow the experiment to reassign contexts to different variations, especially if you are used to the way rollouts work in LaunchDarkly outside of experimentation. However, allowing reassignment has advantages, provided the change you are experimenting on is not disruptive to end users. In practice, this is the case for many experiments. A common example is "user-day experiments," which run for only one day at a time.

For experiments where contexts should not leave the variation they're initially allocated to, such as significant navigation menu changes or large user interface (UI) changes, you can disable variation reassignment. However, keeping contexts in the same variations incurs additional risk.

Disable variation reassignment

Expand Disabling variation assignment

You should only rarely prevent contexts from switching variations during an experiment by disabling variation reassignment. You may want to disable variation reassignment for substantial changes, such as major navigation menu or UI changes. You can disable variation reassignment by clicking Advanced, then checking the Prevent variation reassignment when increasing traffic checkbox.

Traffic will be assigned to new variations if you stop an experiment

Checking the Prevent variation reassignment when increasing traffic checkbox prevents variation reassignment only if you change the amount of traffic in your experiment audience by editing a running experiment using the experiment's Edit button. To learn more, read Edit experiments.

If you instead use the Stop button to stop an experiment iteration, change the amount of traffic in your experiment audience, then start a new iteration, LaunchDarkly will still reshuffle traffic into new variations.

Here is an example of running an experiment with variation reassignment disabled: you add an experiment to a flag with three variations: A, B, and C. You roll out the three variations to 6% of contexts, while the remaining 94% receives the control, variation A. The control traffic is not part of the experiment nor its analysis.

Here is a visualization of the starting traffic allocation, with the control group on the right:

Next, you ramp up your experiment to 30% of traffic. This creates a new iteration of the experiment. In the new iteration, the 6% that were receiving variations A, B, and C, continue to receive those variations, but are no longer included in the experiment nor its analysis. New traffic is used for the 30% allocated to the experiment.

Here is an example of the modified allocation, with the control group on the right and the original experiment audience on the left: