Metric analysis

Overview

This topic explains the different ways you can analyze a metric, and how to select the analysis options appropriate for your needs. When you create a metric, you must decide how you want to handle its metric and unit analysis. Units are the context instances that encounter a metric.

Randomization unit

When you create a metric, you must select a randomization unit. The randomization unit is the context kind, such as user, device, or request, that the metric can measure events from.

Some examples of common metrics and their randomization units include "clicks by user" and "latency by request":

To learn more, read Randomization units.

Unit aggregation method

When you create a metric, you must select whether to aggregate event values per unit by average or by sum, depending on your use case.

For example, if your metric is tracking visitor purchases, you can use average and sum to understand the following information as part of an experiment:

• average: the proportion of customers who make at least one purchase
• sum: the number of purchases totaled to find the average number of purchases per customer

Imagine you have three visitors in your online store who make the following number of purchases:

• Visitor A: 3 purchases
• Visitor B: 0 purchases
• Visitor C: 2 purchases

In the example above, three customers make a total of five purchases. Because you have configured the number of purchases as your metric, this number is what LaunchDarkly uses to calculate average and sum.

This table shows an example of how LaunchDarkly would calculate the average versus the sum of your metric's results:

Analysis method

You can analyze a metric by the following methods:

• mean: the mean is the default analysis method
• p50: the 50th percentile, which is also the median
• p75: the 75th percentile
• p90: the 90th percentile
• p95: the 95th percentile
• p99: the 99th percentile

Funnel experiments can only use metrics that use the mean analysis method.

The closer the percentile is to 50, the closer it is to the median. Percentiles closer to the median, such as p75, are useful for analyzing general trends. For example, you might want to use the p50 or p75 methods for analyzing things like API latency.

Higher percentiles are better for detecting outliers. For example, if you have an endpoint that generally works well, but handles a single customer with large amounts of data poorly, you might want to use the p95 or p99 methods for the related metric.

Units without events

Sometimes a context instance will encounter a metric without completing the action the metric is measuring. When this happens the SDK does not send a metric event to LaunchDarkly. For example, if a conversion metric is measuring completed sign-ups through an email submission form, the SDK will not send an event if the user context opens the form but never clicks the "submit" button. LaunchDarkly automatically sets the value for these units without events to 0 for custom conversion/binary, click, and page view metrics. However, for custom numeric metrics, you can configure LaunchDarkly to exclude these units from the analysis entirely.

For custom numeric metrics that use a percentile analysis method:

• LaunchDarkly automatically excludes units without events from the analysis

For custom numeric metrics that use the mean analysis method:

• Setting the value for units without events to zero is best for metrics where an incomplete process can be treated the same as 0, such as tracking cart totals for an online store. In this example, customers who put items in their cart but never completed the checkout process are treated as if they purchased $0.
• Excluding units without events from the analysis is best for latency metrics. If LaunchDarkly never receives an event for a context instance, you do not want to default to 0 because LaunchDarkly would interpret this as an extremely fast latency time, which would skew or invalidate the results.