Example metrics

Overview

This guide provides examples of metrics you can use with LaunchDarkly features such as Experimentation and release guardian.

Metrics measure audience behaviors in your app or product and how those behaviors are affected by different flag variations. You can use metrics to track all kinds of things, from how often end users access a URL to how long that URL takes to load a page. How you set up a metric varies depending on what you want to measure. The examples included in this guide will help you understand how to configure metrics to meet your business needs.

Metrics and metric events

An "event" happens when someone takes an action in your app, such as clicking on a button, or when a system takes an action, such as loading a page. Your SDKs send these metric events to LaunchDarkly, where it can aggregate and analyze them using metrics. LaunchDarkly can then quantify the overall performance and health of your product and provide suggestions on how to respond.

For example, you can set up a purchase event in your app that sends the following information to LaunchDarkly:

• who made the purchase
• the amount of money spent
• the number of items purchased

Then you can configure a metric to calculate the average purchase total per user. With this metric, you can use Experimentation to compare two different versions of your shopping cart against each other to see which results in higher purchase totals, or use release guardian to track purchase totals after you toggle on a site redesign to see if totals go up or down.

To learn more about events, read about click events, page view events, and custom events. To learn about how SDKs send events to LaunchDarkly, read Sending custom events.

Creating metrics

There are four different types of metrics in LaunchDarkly:

• Custom conversion/binary: a conversion metric that tracks any event you set up in your codebase
• Custom numeric: a metric that tracks the amount of something, such as dollars spent or latency time
• Click: a conversion metric that tracks clicks on a link or button
• Page view: a conversion metric that tracks page views

You do not need to understand each of these options in-depth to use this guide, because each of the provided examples includes the suggested metric types. However, if you want a full explanation of each of these options, read Choosing a metric type.

Metric analysis options

"Units" are the context instances that encounter a metric. When you create a metric, you must decide how you want to handle its unit analysis. The unit analysis options include:

• Unit aggregation method: aggregating by average or by sum
• Analysis method: analyzing by mean or by percentile
• Units without events: setting the value for units without events to 0, or excluding them

You do not need to understand each of these options in-depth to use this guide, because each of the provided examples includes the suggested unit analysis options. However, if you want a full explanation of each of these options, read Metric analysis.

Examples: average purchase price per user

This section includes the configuration options for common custom numeric metrics related to average purchase price per user. For all of the examples in this section, the randomization unit is "user."

Average purchase price per user: only users who made a purchase

Expand Average purchase price per user: only users who made a purchase

You can use this metric to learn the average amount of money spent by users who made a purchase.

Event information:

• Metric type: Custom numeric
• Unit of measure: USD
• Success criteria: Higher than baseline

Metric analysis options:

• Unit aggregation method: Average individual unit values
• Analysis method: Mean
• Units without events: Do not include units with no events in analysis

Here is what the metric setup looks like:

Here is an example of how the metric calculates its results:

Step 1	Step 2	Step 3
Users mde the following purchases: User A: $5, $15 User B: $10 User C: $7 User D: $0 (no purchases)	The metric calculates the average for each user, excluding those who didn't purchase anything: User A: $10 User B: $10 User C: $7	The metric then calculates the average purchase price per user: Result: $9

This means that buyers spend an average of $9.

Average purchase price per user: all users

Expand Average purchase price per user: all users

You can use this metric to learn the average amount of money spent by all users, whether or not they made a purchase.

Event information:

• Metric type: Custom numeric
• Unit of measure: USD
• Success criteria: Higher than baseline

Metric analysis options:

• Unit aggregation method: Average individual unit values
• Analysis method: Mean
• Units without events: Set the value for units with no events to zero

Here is what the metric setup looks like:

Here is an example of how the metric calculates its results:

Step 1	Step 2	Step 3
Users mde the following purchases: User A: $5, $15 User B: $10 User C: $7 User D: $0 (no purchases)	The metric calculates the average for each user, including those who didn't purchase anything: User A: $10 User B: $10 User C: $7 User D: $0	The metric then calculates the average purchase price per user: Result: $6.75

This means that all users, including buyers and non-buyers, spend an average of $6.75.

Example: percentile purchase price per user

This section includes the configuration options for common custom numeric metrics related to percentile purchase price per user. For the example in this section, the randomization unit is "user."

Percentile purchase price per user

Expand Percentile purchase price per user

In this example, you are only interested in learning about your average spenders, and want to exclude the few customers who spend hundreds more than most and skew your data upward. You can use this metric to learn the average amount spent per purchase, excluding the top 10% of users.

This metric only includes users who made a purchase. When using a percentile analysis method, LaunchDarkly automatically excludes units without events.

Event information:

• Metric type: Custom numeric
• Unit of measure: USD
• Success criteria: Higher than baseline

Metric analysis options:

• Unit aggregation method: Average individual unit values
• Analysis method: p90

Here is what the metric setup looks like:

Here is an example of how the metric calculates its results:

Step 1	Step 2	Step 3
Users mde the following purchases: User A: $100, $300 User B: $10 User C: $7 User D: $0 (no purchases)	The metric calculates the average for each user: User A: $200 User B: $10 User C: $7	The metric then calculates the average for the 90th percentile: Result: $162

This means that 90% of all users that make on purchase, on average, spend less than $162.

Examples: average revenue per user

This section includes the configuration options for common custom numeric metrics related to average revenue per user. For all of the examples in this section, the randomization unit is "user."

Average revenue per user: only users who generated revenue

Expand Average revenue per user: only users who generated revenue

This metric sums purchase amounts per user who bought something, instead of calculating average amounts per purchase. This lets you find out about the purchasing behavior of users as a whole, ignoring whether the money they spent was all at once or spread between multiple purchases.

Event information:

• Metric type: Custom numeric
• Unit of measure: USD
• Success criteria: Higher than baseline

Metric analysis options:

• Unit aggregation method: Sum individual unit values
• Analysis method: Mean
• Units without events: Do not include units with no events in analysis

Here is what the metric setup looks like:

Here is an example of how the metric calculates its results:

Step 1	Step 2	Step 3
Users mde the following purchases: User A: $100, $300 User B: $10 User C: $7 User D: $0 (no purchases)	The metric calculates the sum for each user, excluding those who didn't purchase anything: User A: $400 User B: $10 User C: $7	The metric then calculates the average sum: Result: $139

Average revenue per user: all users

Expand Average revenue per user: all users

This metric sums purchase amounts per user, instead of calculating average amounts per purchase. This lets you find out about the purchasing behavior of users as a whole, ignoring whether the money they spent was all at once or spread between multiple purchases. This includes users who didn't purchase anything.

Event information:

• Metric type: Custom numeric
• Unit of measure: USD
• Success criteria: Higher than baseline

Metric analysis options:

• Unit aggregation method: Sum individual unit values
• Analysis method: Mean
• Units without events: Set the value for units with no events to zero

Here is what the metric setup looks like:

Here is an example of how the metric calculates its results:

Step 1	Step 2	Step 3
Users mde the following purchases: User A: $100, $300 User B: $10 User C: $7 User D: $0 (no purchases)	The metric calculates the sum for each user, including those who didn't purchase anything: User A: $400 User B: $10 User C: $7 User D: $0	The metric then calculates the average sum: Result: $104.25

Example: average number of purchases

This section includes the configuration options for common custom conversion/binary metrics related to average number of purchases. For the example in this section, the randomization unit is "user."

Average number of purchases: per user who purchased

Expand Average number of purchases: per user who purchased

You can use this metric to learn the average number of purchases for users who made a purchase.

Event information:

• Metric type: Custom conversion/binary
• Success criteria: Higher than baseline

Metric analysis options:

• Unit aggregation method: Sum individual unit values
• Analysis method: Mean

Here is what the metric setup looks like:

Here is an example of how the metric calculates its results:

Step 1	Step 2	Step 3
Users mde the following purchases: User A: 1 purchase, 1 purchase User B: 1 purchase User C: 1 purchase User D: 0 purchases	The metric calculates the sum for each user, excluding those who didn't purchase anything: User A: 2 User B: 1 User C: 1	The metric then calculates the average sum: Result: 1.33

Example: latency

This section includes the configuration options for common custom numeric metrics related to latency time. For the example in this section, the randomization unit is "request."

Latency excluding the slowest 1% of results

Expand Latency excluding the slowest 1% of results

This metric calculates latency, excluding the slowest 1% of requests that may skew the data. When using a percentile analysis method, LaunchDarkly automatically excludes units without events.

Event information:

• Metric type: Custom numeric
• Unit of measure: ms
• Success criteria: Lower than baseline

Metric analysis options:

• Unit aggregation method: Average individual unit values
• Analysis method: p99

Here is what the metric setup looks like:

Here is an example of how the metric calculates its results:

Step 1	Step 2
Requests had the following latency: Request A: 20ms Request B: 10ms Request C: 70ms Request D: 30ms	The metric then calculates the 99th percentile: Result: 70ms

This means that 99% of requests, on average, have a latency of less than 70ms.

Conclusion

The guide explained how to set up metrics to help answer common questions about your app's revenue and latency performance. For more examples of experiments you can run, read Example experiments.