Probability report
Read time: 6 minutes
Last edited: Sep 30, 2024
Overview
This topic explains how to read and use the probability report tab of an experiment.
The probability report tab includes:
- probability charts for each metric
- each variation's probability to be best
- the conversion rate or posterior mean, depending on the metric type
- the number of conversions or total value, depending on the metric type
- the number of exposures
Hover over each column's header to view more information about how LaunchDarkly calculated the results for that column. To learn more, read Analytic formulas for experiment variation means.
Probability charts
An experiment's probability chart provides a visual representation of the performance of each variation tested in the experiment:
Each experiment's probability chart is unique, and how you interpret the results depends on what metric you're measuring and the hypothesis of your experiment. The following sections provide general information about the x-axis and the y-axis to help you interpret experiment results.
To hide any of the variations from the probability chart, uncheck the box next to the variation's name:
Expand information about the x-axis
The horizontal x-axis displays the unit of the primary metric included in the experiment. For example, if the metric is measuring revenue, the unit might be dollars, or if the metric is measuring website latency, the unit might be milliseconds.
If the unit you're measuring on the x-axis is something you want to increase, such as revenue, account sign ups, and so on, then the farther to the right the curve is, the better. The variation with the curve farthest to the right means the unit the metric is measuring is highest for that variation.
If the unit you're measuring on the x-axis is something you want to decrease, such as website latency, then the farther to the left the curve is, the better. The variation with the curve farthest to the left means the unit the metric is measuring is lowest for that variation.
How wide a curve is on the x-axis determines the credible interval. Narrower curves mean the results of the variation fall within a smaller range of values, so you can be more confident in the likely results of that variation's performance.
In the example below, the green variation has a more precise credible interval than the purple variation:
To learn more, read Credible interval.
Expand information about the y-axis
The vertical y-axis measures probability. You can determine how probable it is that the metric will equal the number on the x-axis by how high the curve is.
In the example above, the green variation has a high probability that the metric will measure 0.4 for any given context. In other words, if someone encounters the green variation, there's a high probability that the metric will measure 0.4 for that person.
Probability to be best
Probability to be best is the likelihood that a variation had the biggest effect on a particular metric.
The variation with the highest probability to be best is highlighted above the probability report:
Funnel optimization experiments
In funnel optimization experiments, the probability report tab provides each variation's probability to be best for each step in the funnel, but the final metric in the funnel is the metric you should use to decide the winning variation for the experiment as a whole.
LaunchDarkly includes all end users that reach the last step in a funnel in the experiment's winning variation calculations, even if an end user skipped some steps in the funnel. For example, if your funnel metric group has four steps, and an end user takes step 1, skips step 2, then takes steps 3 and 4, the experiment still considers the end user to have completed the funnel and includes them in the calculations for the winning variation.
Conversion rate
The conversion rate displays for all conversion metrics. Examples of conversions include clicking on a button or entering information into a form.
Conversion metrics can be one of two types: count or binary.
Count conversion metrics
The value for each unit in a count conversion metric can be any positive value. The value equals the number of times the conversion occurred. For example, a value of 3 means the user clicked on a button three times.
The aggregated statistic for count conversion metrics is the average number of conversions across all units in the metric. For example, the average number of times users clicked on a button.
Count conversion metrics include:
- Clicked or tapped metrics using the Count option
- Custom conversion count metrics
- Page viewed metrics using the Count option
Binary conversion metrics
The value for each unit in a binary conversion metric can be either 1 or 0. A value of 1 means the conversion occurred, such as a user viewing a web page, or submitting a form. A value of 0 means no conversion occurred.
The aggregated statistic for binary conversion metrics is the percentage of units with at least one conversion. For example, the percentage of users who clicked at least once.
Binary conversion metrics include:
- Clicked or tapped metrics using the Occurrence option
- Custom conversion binary metrics
- Page viewed metrics using the Occurrence option
For funnel optimization experiments, the conversion rate includes all end users who completed the step, even if they didn't complete a previous step in the funnel. LaunchDarkly calculates the conversion rate for each step in the funnel by dividing the number of end users who completed that step by the total number of end users who started the funnel. LaunchDarkly considers all end users in the experiment for whom the SDK has sent a flag evaluation event as having started the funnel.
Posterior mean
The posterior mean displays only for numeric metrics. To learn more, read Custom numeric metrics.
The posterior mean is the variation's average numeric value that you should expect in this experiment, based on the data collected so far.
All of the data in the results table are based on a posterior distribution, which is the combination of the collected data and our prior beliefs about that data. To learn more about posterior distributions, read Frequentist and Bayesian modeling.
LaunchDarkly automatically performs checks on the results data, to make sure that actual context traffic matches the allocation you set. To learn more, read Understanding sample ratios.
Conversions, Total value, and Exposures
Depending on the metric type, the statistics details tab displays one of the following two columns containing the sum of unit values for the numerator of the metric:
- Conversions: the total number of conversions for a conversion metric
- Total value: the total value for a numeric metric
The statistics details tab also displays the exposures column, which is the total number of exposures, or experiment units, for the denominator of the metric.
The raw conversion rate is the number of conversions divided by the number of exposures. The raw mean is the total value divided by the number of exposures.
The raw conversion rate and raw mean may not equal the estimated conversion rate and estimated posterior mean shown in the "Conversion rate" and "Posterior mean" columns.
This can be due to:
- regularization through empirical Bayes priors, or
- covariate adjustment through CUPED (Controlled experiments Using Pre-Experiment Data)
You can also use the REST API: Get experiment results
To learn more about troubleshooting if your experiment hasn't received any metric events, read Experimentation Results page status: "This metric has never received an event for this iteration".