Understanding sample ratios

Overview

This topic explains what sample ratios are and how to detect a sample ratio mismatch (SRM).

In your experiments, the sample ratio is the ratio between end users receiving each of your experimentation treatments. An SRM is a mismatch between the proportion of users expected in your experimentation treatments and the actual number of users in each treatment.

SRMs can occur if you are using JavaScript-based client-side SDKs. If LaunchDarkly alerts you of an SRM in one of your experiments, there are actions you can take to correct it.

Here's what an SRM alert looks like on an experiment's Results tab:

How to fix SRMs in JavaScript-based SDKs

SRMs are most commonly caused by outdated SDKs, event deduplication, and problematic application behavior.

Update your SDKs

Outdated SDKs are the most common cause of SRMs. If you observe an SRM and are using any of the LaunchDarkly JavaScript-based client-side SDKs, you may need to update your SDK.

If you are using JavaScript or React Web, confirm that you are using at least these versions:

• Version 2.23.0 or newer of the JavaScript SDK
• Version 2.28.0 or newer of the React Web SDK

These versions send the correct versions of events that LaunchDarkly needs for experimentation tracking. To learn more, read Experimentation and metric events.

Event deduplication

Older versions of the JavaScript SDK, prior to version 2.23.0, deduplicate events by default. This can cause an SRM in the case where you evaluate the same flag before and after a call to identify.

Older versions of the React Web SDK, prior to version 2.28.0, have a similar issue. Furthermore, in the React Web SDK LDProvider calls allFlags after initializing the client. This can potentially exacerbate issues with event deduplication.

To fix this, we strongly recommend upgrading your SDK version. If you can't upgrade your SDK version, you can set the allowFrequentDuplicateEvents option to true.

const options = { allowFrequentDuplicateEvents: true };
const client = LDClient.initialize('client-side-id-123abc', user, options);
COPY

Event loss due to application behavior

Some experimentation implementations can lose events disproportionally for JavaScript-based client-side SDKs. For example, if your experiment is an A/B test where the "A" does nothing and the "B" immediately redirects your end users to another page, it's possible that more events from the "B" variation will be lost.

The LaunchDarkly SDKs send events immediately when the page unloads. However, some browsers, such as Safari on iOS, do not respect asynchronous calls on unload. If LaunchDarkly cannot record all of the events for your experiment, then your experiment will have an SRM.

Check to see if one of your experimentation treatments results in users leaving the page more quickly than the other treatments. You may need to adjust your experimentation treatments to account for this behavior.

How SRMs occur

There are many ways an SRM can occur, but the most common is that the randomization method is incorrect.

Imagine you have an A/B test with 50% of users assigned to treatment A and 50% assigned to treatment B, you should expect that with a sufficient sample size, the split is roughly 50/50. If you have 10,000 users in your experiment iteration, you should expect around 5,000 users in treatment A and 5,000 in treatment B. Because the sampling is random, this split will not be exact. However, it should be close. You should not have 4,000 users in treatment A and 6,000 users in treatment B.

LaunchDarkly uses statistics to determine if the sample ratio is more different than what you should expect from a random sampling. If there's a 99% or greater probability that the units, or users, in the experiment are not arriving at the same rate as expected given how you've allocated the traffic, then LaunchDarkly displays a warning noting that your experiment results are almost certainly invalid.