Understanding sample ratios
Read time: 5 minutes
Last edited: Dec 14, 2024
Overview
This topic explains what sample ratios are and how to detect a sample ratio mismatch (SRM).
In your experiments, the sample ratio is the ratio between end users receiving each of your experiment variations. An SRM is a mismatch between the number of users expected and the actual number of users included in each variation.
SRMs can occur if you are using JavaScript-based client-side SDKs. If LaunchDarkly alerts you of an SRM in one of your experiments, there are actions you can take to correct it.
Here's what an SRM alert looks like on an experiment's Results tab:
How to fix SRMs in JavaScript-based SDKs
SRMs are most commonly caused by outdated SDKs, event deduplication, and problematic application behavior.
Update your SDKs
Outdated SDKs are the most common cause of SRMs. If you observe an SRM and are using any of the LaunchDarkly JavaScript-based client-side SDKs, you may need to update your SDK.
If you are using JavaScript or React Web, confirm that you are using at least these versions:
- Version 2.23.0 or newer of the JavaScript SDK
- Version 2.28.0 or newer of the React Web SDK
These versions send the correct versions of events that LaunchDarkly needs for experimentation tracking. To learn more, read Experimentation and metric events.
A context is a generalized way of referring to the people, services, machines, or other resources that encounter feature flags in your product. To learn more, read Contexts.
Creating contexts and evaluating flags based on contexts is supported in the latest major versions of most of our SDKs. If you upgrade your JavaScript SDK or React Web SDK to version 3, those versions replace users with contexts. These new major versions also include the updates in versions 2.23.0 and 2.28.0, respectively, that address the most common causes for SRMs.
Event deduplication
Older versions of the JavaScript SDK, prior to version 2.23.0, deduplicate events by default. This can cause an SRM in the case where you evaluate the same flag before and after a call to identify.
Expand for an example of event deduplication with identify
If you add attributes to your user object after initialization and then call identify, event deduplication can cause a non-random loss of experimentation events.
Here's how this can happen:
- Your code initializes the JavaScript SDK.
- An end user encounters a flag in your application. The user does not match your experiment rule, and LaunchDarkly serves a flag variation to the user.
- Your application learns more about the end user. Your code adds attributes to the user object, without changing the user key, and calls
identify. - Because of the
identifycall, LaunchDarkly re-evaluates the flag for the same user. Based on the additional attributes, the user now matches your experiment rule and is eligible for the experiment. - The user is randomly assigned to the same flag variation that they were previously served.
If these steps happen within the SDK's deduplication window of five minutes, the user will not be counted as part of the experiment. If, in step 5, the user is randomly assigned to a different flag variation than they were previously served, then they will be counted correctly as part of the experiment. Because some users are being counted and some users are not, you will have an SRM.
Older versions of the React Web SDK, prior to version 2.28.0, have a similar issue. Furthermore, in the React Web SDK LDProvider calls allFlags after initializing the client. This can potentially exacerbate issues with event deduplication.
To fix this, we strongly recommend upgrading your SDK version. If you can't upgrade your SDK version, you can set the allowFrequentDuplicateEvents option to true.
Expand for allowFrequentDuplicateEvents code sample
Here's how to set the allowFrequentDuplicateEvents option in your JavaScript SDK to true:
The allowFrequentDuplicateEvents option has been deprecated in the JavaScript SDK version 2.23.0 and React Web SDK version 2.28.0. It has been removed in version 3 of the JavaScript and React Web SDKs.
Event loss due to application behavior
Some experimentation implementations can lose events disproportionally for JavaScript-based client-side SDKs. For example, if your experiment is an A/B test where the "A" does nothing and the "B" immediately redirects your end users to another page, it's possible that more events from the "B" variation will be lost.
The LaunchDarkly SDKs send events immediately when the page unloads. However, some browsers, such as Safari on iOS, do not respect asynchronous calls on unload. If LaunchDarkly cannot record all of the events for your experiment, then your experiment will have an SRM.
Check if one of your treatment variations causes users to leave the page more quickly than the control variation. If so, you may need to adjust your treatment variations to correct this behavior.
How SRMs occur
There are many ways an SRM can occur, but the most common is that the randomization method is incorrect.
Imagine you have an A/B test with 50% of users assigned to control variation A and 50% assigned to treatment variation B. You should expect that with a sufficient sample size, the split is roughly 50/50. If you have 10,000 users in your experiment iteration, you should expect around 5,000 users in variation A and 5,000 in variation B. Because the sampling is random, this split will not be exact. However, it should be close. You should not have 4,000 users in variation A and 6,000 users in variation B.
LaunchDarkly uses statistics to determine if the sample ratio is more different than what you should expect from a random sampling. If there's a 99% or greater probability that the units, or users, in the experiment are not arriving at the same rate as expected given how you've allocated the traffic, then LaunchDarkly displays a warning noting that your experiment results are almost certainly invalid.
For more troubleshooting guidance, read Troubleshooting SRM errors with experiments.