Release guardian

Overview

This topic explains how to monitor metrics on flag releases and configure LaunchDarkly to take action on the results. You can configure LaunchDarkly to notify you or automatically roll back a release when it detects problems.

Monitoring metrics

Metrics measure end-user and system behaviors affected by flag variations. You can use metrics to track a variety of system health indicators and end-user behaviors, from engineering metrics like errors and latencies, to product metrics like clicks and conversions.

You can connect metrics to LaunchDarkly using a LaunchDarkly SDK, the metric import API, the Segment integration, or the Sentry integration. To learn more, read Metrics.

Regressions

When you toggle on a flag, metric monitoring lets you know whether the variation change is having any negative impact on your app or audience. This negative effect is called a "regression." You can configure LaunchDarkly to either notify you of the regression, or notify you of the regression and automatically roll back the release.

For metrics where a lower value is better, LaunchDarkly considers it a regression when the lower bound of the treatment’s confidence interval exceeds the upper bound of the control’s confidence interval.

For metrics where a higher value is better, LaunchDarkly considers it a regression when the upper bound is less than zero.

This graphic displays the confidence interval for an example numeric metric for which lower values are better, and what a regression looks like:

You can monitor metrics on any flag rule except those serving a percentage rollout.

Add metrics to a flag

Before you can add metrics to release, you must create the metrics you want to monitor. To learn how, read Creating metrics.

To add metrics to an existing flag:

1. From the flags list, click on the flag you want to add a metric to.
2. Click Add metrics.
3. Select one or more Metrics to monitor.
4. Click Save. Or, If the flag is already toggled on, click Monitor and save and follow the instructions under Begin monitoring.

Begin monitoring

LaunchDarkly will prompt you to begin monitoring metrics when:

• you toggle the flag on, or
• if the flag is already toggled on, when you make a change to a flag variation.

If any of the metrics you attached to the flag have more than one randomization unit, you must decide on a single randomization unit when you begin monitoring. The randomization unit is the context kind that LaunchDarkly uses to assign traffic to each of a flag's variations. To learn more, read Randomization units.

To begin monitoring metrics on a flag:

1. After you have added metrics to a flag, toggle flag targeting On.

2. Click Monitor and save. The "Create a release strategy" dialog appears.

3. If you have more than one rule on the flag, choose a Rule to monitor. If you do not have any targeting rules on the flag, or only have targeting rules that serve a percentage rollout, the default rule is automatically selected.

4. If your metrics have more than one randomization unit, choose a randomization unit to allocate your traffic by.

5. Choose a monitoring Duration of 2 days, 24 hours, or one hour.

   • (Optional) If you want a different monitoring duration, click Customize traffic allocation and duration. You can then specify the rule's Traffic allocation and choose a custom Monitoring duration.

6. (Optional) Check the Automatically roll back rule if a regression is detected option if you want LaunchDarkly to automatically roll back your change when it detects a regression, in addition to an email and in-app notification.

   

7. Click Next. The "Save changes" dialog appears.

8. Complete the "Save changes" fields as needed and click Save changes.

The flag now displays a "Monitoring in progress" message. If LaunchDarkly detects a regression, LaunchDarkly sends you an email and an in-app notification.

You can view the attached metrics' performance on the targeting rule on the flag's Targeting tab:

To view more information about metric performance, click View insights. To learn more, read Flag observability charts.

Edit metrics attached to a flag

To edit the metrics attached to a flag, click the Metrics button:

Click Edit metrics to add or remove metrics from the flag.

Remove metrics from a flag

To remove all metrics from a flag, click the Metrics button and select Remove all metrics.

Stop monitoring early

If you want to stop monitoring before the monitoring window is over:

1. From the flag's Targeting tab, click Stop monitoring. A "Stop rollout early" dialog appears.
2. Choose which Variation to serve to all contexts after you stop monitoring. The field defaults to the control variation.
3. Click Stop.

Flag observability charts

If you are monitoring metrics on the flag, a message displays at the top of the Targeting and Insights tabs:

• Monitoring in progress on [rule]: LaunchDarkly is actively monitoring metrics on the flag rule.
• No regressions found, [rule] enabled fully: the monitoring window is over and LaunchDarkly found no regressions.
• Measured rollout suspended early: the release was rolled back before the end of the monitoring window, either manually or by LaunchDarkly.
• Regression mitigated: LaunchDarkly found a regression and automatically rolled back the release. This message may appear during or after the monitoring window has ended.
• Regression detected on the metric: LaunchDarkly found a regression, but you did not enable automatic rollback so you must manually roll back the release. This message may appear during or after the monitoring window has ended.

Roll back releases

To roll back a release after LaunchDarkly has detected a regression:

1. From the flag's Targeting tab, find the rule with the detected regression.

2. Click Roll back rule. The "Stop rollout early" dialog appears.

   

3. Choose which Variation to serve to all contexts after you stop monitoring. The field defaults to the control variation.

4. Click Stop.

The Insights tab

After you add a metric to a flag, the "Flag measurements" section on the flag's Insights tab displays the following information:

• The name of the monitored rule, and if automatic rollback is enabled
• The percentage of a rule's traffic you assigned to be monitored
• The length of the monitoring window
• The number of contexts monitored during the monitoring window

Probability charts

Each metric you add to the flag displays a probability chart with the lower and upper bounds of the metric results for each variation.

Hover over the percentage in the top right corner of each chart to view the confidence interval for the metric. The confidence interval is the range of values you can expect 95% of the contexts that encounter the metric to fall within.

Each variation for which there was a regression is highlighted in red: