• Home
  • Integrations
  • SDKs
  • Guides
  • API docs
No results for ""
EXPAND ALL

EDIT ON GITHUB

Controlling experiment populations

Read time: 3 minutes
Last edited: Oct 21, 2021

Overview

This topic explains what an experiment population is, and how to use LaunchDarkly's features to show experiments only to a specific group of users.

Understanding experiment populations

You can get accurate results more quickly if you only expose some of your user base to experiments. You have the option to only include a subset of users in your experiments. This subset of users is referred to as your experiment population.

You can create a custom experiment population by selecting specific flag targeting rules to include in your active experiments.

For example, imagine you are testing alternate copy for your checkout button. You're currently targeting all your Canadian and US users with the 'true' variation for the alternate button, which shows your new copy, but you only want to run an experiment measuring click conversions for users in the United States. To accomplish this, you select the user targeting rule that affects US-based users and de-select the rule that targets users in Canada.

You may want to refine your experiment population for any of the following reasons:

  1. Running targeted experiments for a subset of your flag-targeted users.
  2. Excluding user groups whose events you do not need to measure. For example, users affected by 'Default' rules.
  3. Reducing the events volume an experiment generates in order to manage billing costs.
Can't see the "Configure event settings" options?

If you can't see the feature described below, your SDKs may be out of date. To solve this problem:

  1. Update your SDKs to the latest version.
  2. Contact support@launchdarkly.com to enable the feature.

Customizing experiment population

Experiment allocation

If you have experiment allocation enabled, customizing your experiment populations works differently than described below. To learn how to customize your experiments with experiment allocation enabled, read Allocating experiment traffic.

You can refine your user base into a specific experiment population by following the procedure below.

Customizing events for Data Export

To learn how to configure which user data LaunchDarkly sends to Data Export destinations, read Customizing which data LaunchDarkly exports.

To customize your experiment population:

  1. Identify which flag is associated with the experiment you wish to modify.
  2. Navigate to that flag's Settings tab.
  3. Click N of N targeting rules in the "Exporting events from" line. The "Configure event settings" screen opens:

A flag's Settings page with the targeting rules link called out.
A flag's Settings page with the targeting rules link called out.

  1. Click the Custom selection radio box to modify the users who can see an experiment. When you click Custom selection, you can choose which targeting rules you wish to include or exclude from an experiment. If you use controlled populations and choose Custom selection, the experiment excludes all users who return a PREREQUISITE_FAILED evaluation reason. To learn more about flag prerequisites, read Prerequisite checks.
  2. Select the checkboxes of rules you wish to include in the experiment. This refines the experiment's population down to the group you choose:

The "Configure event settings" menu with some targeting rules selected.
The "Configure event settings" menu with some targeting rules selected.

  1. Click Save Changes. A confirmation dialog appears.
  2. Type the name or key of the environment in the text box and click Confirm.

The modified list of users you created is now the experiment's population.

Experiment populations do not affect flag targeting rules

You can customize your experiment's population at will without your flag targeting rules changing. Flag targeting controls which variation of a flag a user sees. An experiment's population determines which of your users contribute to Experimentation results or Data Export streams. To learn more, read Targeting users.

Changing targeting rules during an experiment

If you update a feature flag's targeting rules while an experiment is running, a user who previously saw one variation may then see a different variation. Data for that user is recorded twice, because LaunchDarkly deduplicates user data only at the variation level, not the flag level. To prevent the experiment from recording user data twice, restart it after you make any changes to a flag's targeting rules. This clears all existing data from the experiment.

To restart the experiment:

  1. Find the feature flag associated with your experiment.
  2. Click on the flag's Experimentation tab.
  3. Click on the overflow menu in the relevant experiment.
  4. Click Reset data.

The data from the experiment is now cleared.