Regression thresholds for guarded rollouts

Overview

This topic explains how to determine a custom regression threshold for a guarded rollout.

For each metric you use in a guarded rollout, you can use the LaunchDarkly default regression threshold, or a custom threshold:

• The Default option uses LaunchDarkly's standard regression threshold. This option is appropriate for most guarded rollouts.
• The Set custom thresholds option lets you specify a custom threshold. This option requires familiarity with statistical analysis concepts, which are explained below.

Regression thresholds

When you create a guarded rollout, the regression threshold represents the level of underperformance you are willing to tolerate in the new variation you're rolling out, called the "treatment group," as compared to the current variation, called the "control group." The threshold value ranges from 0% to 100%.

Here is what the values mean:

• Thresholds closer to 0%: This indicates a more conservative approach where you have little tolerance for risk that the treatment group performs worse than the control group.
• Thresholds closer to 100%: This indicates a higher tolerance for risk, meaning you are more willing to accept the possibility of the treatment group performing worse than the control group.

If you prefer to minimize the risk of regression and prioritize sensitivity in detecting underperformance, set the threshold to 0%. However, if you are comfortable with some risk and want to reduce sensitivity, you can increase the threshold to a value greater than 0%.

Custom threshold calculation methods

The custom threshold calculation methods for different types of metrics include the following:

• For metrics that use the "Average" analysis method: you can set the threshold to a reference value that LaunchDarkly compares to the relative difference from the control to detect regressions.
• For metrics that use a percentile analysis method:
  • with a success criterion of "higher is better": you can set the threshold to a reference value against which LaunchDarkly compares a specific ratio to detect regression. This ratio is calculated as the amount of difference between the treatment's upper bound and the control's lower bound, relative to the control's percentile estimate.
  • with a success criterion of "lower is better": you can set the threshold to a reference value against which LaunchDarkly compares a specific ratio to detect regression. This ratio is calculated as the amount of difference between the treatment's lower bound and the control's upper bound, relative to the control's percentile estimate.

Example: A metric using the "Average" analysis method

Imagine you are using a metric with the "Average" analysis method in a guarded rollout. The true conversion rate of the control group is 2%, and you set the regression threshold to 10%.

In this example, LaunchDarkly detects a regression when .

In other words, LaunchDarkly detects a regression when there is a 95% probability that the true, unknown conversion rate of the treatment group would be smaller than 1.8%, given the evidence provided by the observed data. The threshold is with respect to the relative difference of the treatment’s mean and control’s mean.

Example: A metric using the "Percentile" analysis method

Imagine you are using a metric with the "Percentile" analysis method in a guarded rollout. The percentile of the control group is 1,000, and you set the regression threshold to 2%.

In this example, the regression detection behavior is as follows:

• If the metric's success criteria is "lower is better," LaunchDarkly detects a regression when .
• If the metric's success criteria is "higher is better," LaunchDarkly detects a regression when .