• Home
  • Integrations
  • SDKs
  • Guides
  • API docs
No results for ""
EXPAND ALL

EDIT ON GITHUB

Experimenting with feature flags

Read time: 8 minutes
Last edited: Jan 05, 2022

Overview

This guide explains how to run an experiment in LaunchDarkly. Experiments are useful for product managers, engineers, designers, or anyone who wants to validate their hypothesis about any part of their platform controlled by feature flags.

In this guide, you will:

  • Configure an SDK,
  • Create a multivariate flag,
  • Configure targeting rules,
  • Create a metric and connect it to a flag, which creates an experiment, and
  • Run and analyze the results of an experiment.
The experiment in this guide is for example use only

The experiments in this guide are examples only. They are intended to teach you the basic principles and process of conducting an experiment in LaunchDarkly. Do not replicate this experiment in production and expect rigorously produced results.

To complete an experiment that results in useful data, you should conduct more preparation than is outlined in this guide, including rigorous sampling practices.

To learn more about how to create an effective experiment, read Designing experiments in LaunchDarkly.

Prerequisites

To complete this guide, you must have the following prerequisites:

  • You must have completed the exercises in Rules and targeting. The exercises in this guide rely on concepts from that guide.
  • An active LaunchDarkly account with the ability to create a flag.
  • Basic knowledge of the Python and JavaScript programming languages. You need these to complete the exercises in this guide.
  • "Do not track" must be disabled in your browser for the experiment's click events to work. To confirm that "Do not track" is disabled, visit AllAboutDNT.com.
  • You should be familiar with the concept of experimentation for software products, and know what results you want to determine. To learn more, read Designing experiments in LaunchDarkly.
  • You must have Experimentation enabled in your LaunchDarkly account. To enable Experimentation, contact your LaunchDarkly account manager or email sales@launchdarkly.com.
  • Access to Glitch.
See also: How to use Glitch

Concepts

The following concepts will help you understand this guide.

Multivariate flags

A multivariate flag is a feature flag that is not boolean and has more than one possible variation. Multivariate flags allow you to use one flag to serve multiple variations of a feature simultaneously. To learn more, read Understanding multivariate flags.

Setting up the app

This app builds on the same app we use in the Rules and targeting guide. If you haven't completed that guide yet, read and complete those exercises first and then return to this guide.

The app below begins where the previous guide left off:

To begin, remix the Glitch project to create an editable copy of the app. Use the embedded Glitch app above, or click here to remix the app.

Setting up the LaunchDarkly project and environment

The LaunchDarkly Python SDK and client-side JavaScript SDK are installed in this app already. You must add your project's SDK key, which you'll do in the next section.

You can reuse the project and environment you created in Getting started with your first flag, or create a new one.

Here is an image of where to find your SDK keys:

The Account Settings page in LaunchDarkly with environment SDK keys called out.
The Account Settings page in LaunchDarkly with environment SDK keys called out.

Configuring the SDK key and client ID

Glitch lets you store environment variables, such as secrets or config variables, that your app can read from memory. This is a more secure practice than hard-coding these directly into your code.

Here is an image of a sample .env file:

An .env file
An .env file

To configure the SDK key and client ID:

  1. Find a file called .env in the Glitch sidebar.
  2. Copy the SDK key from the Test environment by clicking on the key.
  3. Assign the SDK key to the environment variable called LD_SDK_KEY.
  4. Copy the Client-side ID from the Test environment.
  5. Assign the Client-side ID to the environment variable called LD_CLIENT_SIDE_ID:
LD_SDK_KEY=<your-SDK-key>
LD_CLIENT_SIDE_ID=<your-client-side-id>

Creating a multivariate flag

Experiments can use boolean flags for A/B tests, or multivariate flags for A/B/n tests. Multivariate flags have multiple settings that let you define different variations of one feature. This exercise uses a multivariate flag.

Experiments use multivariate flags to test different versions of a feature. Settings can include strings, numbers, or JSON.

This exercise tests different prices to determine at what price students are more willing to purchase a trip. The trips are discounted to 25%, 50%, and 75% off their original price.

A 25% discount means a student would pay 75% of the original cost of the trip. The equation to calculate this for a $100 trip is 100 * .75. Similarly, to calculate a 50% discount for a $100 trip, enter 100 * .5

The application stores numbers like this in an array. You will use a multivariate flag's value to calculate the discount. To do this, multiply the values in the array by a constant .25 for 75% off, .5 for 50% off, and .75 for 25% off.

Each of those values corresponds to one variation of a feature flag. When you run an experiment, you compare how each variation of a flag produces different behavior in your users.

Creating the "Student Discount Pricing" feature flag

First, create a new flag in LaunchDarkly.

To create a new flag:

  1. Click Create Flag.
  2. Name your flag Student Discount Pricing.
  3. Enter a description of the flag to help account members understand what the flag is for.
  4. Click into the Tags dropdown and enter experiment. This creates a tag that lets you filter flags based on which flags have experiments running.
  5. Check the SDKs using Client-side ID checkbox.
  6. In the "Flag variations" dropdown choose Number.
  7. Enter a name and value for the variations as given below:
  • Variation 1 value: .25
  • Variation 1 name: 75% off
  • Variation 2 value: .5
  • Variation 2 name: 50% off
  • Variation 3 value: .75
  • Variation 3 name: 25% off
  • Variation 4 value: 1
  • Variation 4 name: No discount
The "Create a feature flag" screen for the Student Discount Pricing flag.
The "Create a feature flag" screen for the Student Discount Pricing flag.
  1. Click Save flag.

Targeting users

A successful experiment requires two or more segments to test against. How you segment your users varies based on your business.

Here are some possible ways to segment users:

  • Logged in as opposed to anonymous
  • By company
  • By geography
  • Randomly

In this experiment, you target users based on their email address. You want to offer pricing discounts based on whether a person is a student. Users with a .edu email address are presented with three different prices.

To target users with a .edu email address:

  1. Navigate to the Targeting menu for your new flag.
  2. In the "Target users who match these rules" section, click Add rules.
  3. Click the Experiment menu icon.
  4. Click Run an experiment on this rule.
  5. In the "Select an attribute" dropdown choose email.
  6. In the "Select an operator" dropdown choose ends with.
  7. In the "Enter some values" field enter .edu.
  8. In the last dropdown choose a percentage rollout.
  9. Enter percentage values for the four variations of your flag. The values must add up to 100%:
A flag's targeting rules configuration.
A flag's targeting rules configuration.
  1. Toggle flag targeting on.
  2. Click Save changes.

Adding the flag's code to the application

Next, update your app to display discounted pricing if a user logs in with a .edu email address.

To update your app:

  1. Navigate to your Glitch app.
  2. Find the server.py file in the sidebar and click to open it.
  3. Replace the pricing array line price = np.array([10000, 25000, 100000]) with the following:
discount = ld_client.variation('student-discount-pricing', user, 1)
price = np.array([10000, 25000, 100000]) * discount

Validating the flag's behavior

Before going on to set up your experiment, confirm the multivariate flag and targeting rules work.

To confirm the rules work:

  1. From the Glitch editing window, click View app. The Starlight Travel app appears.
  2. Click Log in:
Starlight Travel app login.
Starlight Travel app login.
  1. Enter a random email address ending in .edu.
  2. Click Log in.
  3. A discounted price appears for all tiers.
  4. Click your avatar in the upper right and select Log out.
  5. Log in again with a random email ending in .com.
  6. Standard pricing appears.
Disable "Do not track"

If you have enabled "Do not track," you will not be able to log in to the Glitch app. If you disable "Do not track" but the login still fails, try disabling ad blockers or popup blockers in your browser as well.

Creating a click metric

Metrics let you measure user behaviors affected by your flag. The three types of metrics are click, pageview, and custom metrics.

To learn more about metrics in experiments, read Creating experiments.

First, copy the URL for your Glitch app. To find it:

  1. From the Glitch console, click the Share button. A pop-up with a URL appears.
  2. Copy the Live site URL.
  3. Save the URL somewhere where it is easy to access.
The Share project pop-up in Glitch.
The Share project pop-up in Glitch.

Next, create a click metric to track the discount which results in the highest number of packages purchased:

  1. In LaunchDarkly, select Experiments from the sidebar menu.
  2. Navigate to the Metrics tab.
  3. Click Create metric. The "Create a new metric screen" appears.
  4. Name your metric "Student pricing tier".
  5. Enter a Description to indicate what the metric is tracking, such as “Track price chosen by students”.
  6. Choose Click from the "Event kind" dropdown.
  7. Enter a.btn in the "Click targets" field.
  8. In the "Target URL" dropdown, choose Substring Match.
  9. Paste in the URL for your Glitch app, which you found and saved in the previous procedure:
The "Create a new metric" screen.
The "Create a new metric" screen.

Connecting the metric to the flag

Next, connect the metric to the flag to create an experiment. To do this:

  1. In LaunchDarkly, click on the student-discount-pricing flag. The flag's dashboard appears.
  2. Click into the flag's Experimentation tab.
  3. Click Create an Experiment. The "Experiment setup" screen appears.
  4. Choose the metric you created from the "Add a new metric" dropdown.
  5. Select Variation 4, "No discount," as the baseline.
  6. Click Save Experiment Settings:
The Experiment setup screen.
The Experiment setup screen.

You return to the flag's Experiments tab and the experiment is ready for you to start it.

Starting your experiment

For this exercise, you will simulate traffic to your application. To do this:

  1. In the flag's Experiments tab, click Start recording.
  2. Open a new browser window.
  3. Load the Glitch application.
  4. Generate traffic by pressing !.
  5. Wait for the simulation to complete and to view the results in LaunchDarkly.
You may have to wait for Experimentation results

Data can take up to ten minutes to appear in LaunchDarkly after your experiment starts.

When you press !, 100 .com email addresses and 100 .edu email addresses try to access your app. None of the .com email addresses will receive a reduced price. The .edu addresses will randomly receive discounted prices. This information is reflected in the experiment's results.

Interpreting experiment results

When you start your experiment, LaunchDarkly runs a statistical test in the background. After LaunchDarkly has enough data, you receive a message on the results page letting you know that your results are statistically significant. This means that the measured differences between the variations are likely due to an actual underlying change in user behavior, rather than random chance.

LaunchDarkly uses two different types of statistical tests to report results. Both types of tests are well-known in the field of statistics:

Reading experiment data

The data your experiment collected is represented in a set of columns in the experiment's card.

Here is an image of the experiment's data:

An experiment's results page.
An experiment's results page.

The list below explains what each column means:

  • Variation: Which flag variation's data is shown in the row.
  • Conversions / unique users: How many unique users took action based on the variation they saw, relative to the total number of unique users who viewed the page. The denominator is the number of views matching the Glitch URL you specified when you created the click metric.
  • Conversion rate: The percentage of users who took action based on the variation they saw, relative to the total number of unique users who encountered that flag variation.
  • Confidence interval: A range of values between which the actual value for the conversion rate likely falls. The smaller the confidence interval, the more confident you can be about the prediction. For example, a confidence interval of 11%-13% is more reliable than a confidence interval of 10%-30%.
  • Change: The difference, in positive or negative values, the flag variation has from the baseline variation. Baseline flag variations say Baseline rather than showing a value.
  • P-value: The likelihood that differences in user behavior within an experiment are random, rather than a result of receiving different variations of a flag. LaunchDarkly uses the industry standard of a p-value of .05 or lower to consider results statistically significant. The lower the p-value, the more likely it is that the results of your experiment are statistically significant. For example, a p-value of .03 means there's a 97% chance differences in user behavior are due to the flag variations you tested, and there's a 3% chance any differences in user behavior were random.

Conclusion

In this guide you:

  • Created a multivariate flag
  • Configured a metric
  • Ran an experiment
  • Interpreted the results

We hope this helped you learn how to collect metrics from feature flags to make data-informed decisions about features.

Want to know more? Start a trial.

Your 14-day trial begins as soon as you sign up. Learn to use LaunchDarkly with the app's built-in tutorial. You'll see how easy it is to manage the whole feature lifecycle from concept to launch to control.

Want to try it out? Start a trial.