• HOME
  • INTEGRATIONS
  • SDKS
  • GUIDES
  • API DOCS
No results for ""
EXPAND ALL
CLOSE
launchdarkly.com

EDIT ON GITHUB

Experimenting with feature flags

Read time: 6 minutes
Last edited: Oct 13, 2020

Overview

This guide explains how to run an experiment in LaunchDarkly. Experiments are useful for product managers, engineers, designers, or anyone who wants to validate their hypothesis about any part of their platform controlled by feature flags.

In this guide, you will:

  • Install and configure a client-side SDK,
  • Create a multivariate flag,
  • Configure targeting rules,
  • Create a metric and connect it to a flag, which creates an experiment, and
  • Run and analyze the results of an experiment.
The experiment in this guide is for example use only

The experiments in this guide are examples only. They are intended to teach you the basic principles and process of conducting an experiment in LaunchDarkly. Do not replicate this experiment in production and expect rigorously produced results.

To complete an experiement that results in useful data, you should conduct more preparation than is outlined in this guide, including rigorous sampling practices.

To learn more about how to create an effective experiment, read Designing experiments in LaunchDarkly.

Prerequisites

To complete this guide, you must have the following prerequisites:

  • You must have completed the exercises in Rules and targeting. The exercises in this guide rely on concepts from that guide. To learn more about how LaunchDarkly's rules and targeting practices, read Rules and targeting.
  • An active LaunchDarkly account with the ability to create a flag.
  • Basic knowledge of the Python and Javascript programming languages. You need these to complete the exercises in this guide.
  • "Do not track" must be disabled in your browser for the experiment's click events to work. To confirm that "Do not track" is disabled, visit AllAboutDNT.com.
  • You should be familiar with the concept of experimentation for software products, and know what results you want to determine. To learn more, read Introduction to experimentation.
  • You must have Experimentation enabled in your LaunchDarkly account. To enable experimentation, contact your LaunchDarkly account manager or email sales@launchdarkly.com.
  • Access to Glitch.
See also: How to use Glitch

Concepts

The following concepts will help you understand this guide.

Multivariate flags

A multivariate flag is a feature flag that is not boolean and has more than one possible variation. Multivariate flags allow you to use one flag to serve multiple variations of a feature simultaneously.

Setting up the app

This app builds on the same app in the Rules and targeting guide. If you haven't completed that guide yet, read and complete those exercises first and then return to this guide.

The app below begins where the previous guide left off.

To begin, remix the Glitch project to create an editable copy of the app. Use the embedded Glitch app above, or click here to remix the app.

Setting up the LaunchDarkly project and environment

The LaunchDarkly Python SDK and client-side JavaScript SDK are installed in this app already. You must add your project's SDK key, which we'll do in the next section.

You can re-use the project and environment you created in Getting started with your first flag, or create a new one.

Project and environments
Project and environments

Configuring the SDK key and client ID

Glitch lets you store environment variables, such as secrets or config variables, that your app can read from memory. This is a more secure practice than hard-coding these directly into your code.

.env file
.env file

  1. Find a file called .env in the Glitch sidebar.
  2. Copy the SDK key from the Test environment by clicking on the key.
  3. Assign the SDK key to the environment variable called LD_SDK_KEY.
  4. Copy the Client-side ID from the Test environment.
  5. Assign the Client-side ID to the environment variable called LD_SDK_KEY.
1LD_SDK_KEY=<your-SDK-key>
2LD_CLIENT_SIDE_ID=<your-client-side-id>

Creating a multivariate flag

Experiments can use boolean flags for A/B tests, or multivariate flags for A/B/n tests. Multivariate flags have multiple settings that let you define different variations of one feature. This exercise uses a multivariate flag.

Experiments use multivariate flags to test different versions of a feature. Settings can include strings, numbers, JSON objects, or JSON arrays.

This exercise tests different prices to determine at what price students are more willing to purchase a trip. The trips are discounted to 25%, 50%, and 75% off their original price.

A 25% discount means a student would pay 75% of the original cost of the trip. The equation to calculate this for a $100 trip is 100 * .75. Similarly, to calculate a 50% discount for a $100 trip, you would enter 100 * .5

Our application stores numbers like this in an array. We will use a multivariate flag's value to calculate the discount. To do this, we multiply the values in the array by a constant .25 for 75% off, .5 for 50% off, and .75 for 25% off.

Each of those values corresponds to one variation of a feature flag. When we run an experiment, we compare how each variation of a flag produces different behavior in our users.

Creating the "Student Discount Pricing" feature flag

First, create a new flag in LaunchDarkly.

  1. Click + Flag.
  2. Name your flag "Student Discount Pricing" (omit the quotes around the phrase).
  3. Enter a description of the flag to help team members understand what the flag is for.
  4. Click into the Tags dropdown and enter “experiment”. This creates a tag that lets you filter flags based on which flags have experiments running.
  5. Under "Flag variations", choose Number.
  6. Enter a name and value for the variations as given below:
  • Variation 1 value: .25
  • Variation 1 name: 75% off
  • Variation 2 value: .5
  • Variation 2 name: 50% off
  • Variation 3 value: .75
  • Variation 3 name: 25% off
  • Variation 4 value: 1
  • Variation 4 name: No discount
  1. Click the 'Make this flag available to client-side SDKs' checkbox.
  2. Save your new flag.

Targeting users

A successful experiment requires two or more segments to test against. How you segment your users will vary based on your business.

Here are some possible ways to segment users:

  • Logged in as opposed to anonymous
  • By company
  • By geography
  • Randomly

In this experiment, we target users based on their email address. We want to offer pricing discounts based on whether a person is a student. Users with a .edu email address will be presented with three different prices.

  1. In LaunchDarkly, navigate to the Targeting menu for your new flag.
  2. Under the "Target users who match these rules" section, click Add a rule.
  3. Under "Select an attribute" choose email, ends with, and .edu.
  4. Select a percentage rollout. Enter percentage values for the four variations of your flag. The values must add up to 100%.
  5. Toggle flag targeting on.

Targeting rules configuration
Targeting rules configuration

Adding the flag's code to the application

Now you must update your app to display discounted pricing if a user logs in with a .edu email address.

To update your app:

  1. Navigate to your Glitch app.
  2. Find the server.py file in the sidebar and click to open it.
  3. Add or change the pricing array code. Comments in the file indicate where to add the code.
1discount = ld_client.variation('student-discount-pricing', user, 1)
2price = np.array([10000, 25000, 100000]) * discount

Validating the flag's behavior

Before going on to set up your experiment, confirm the multivariate flag and targeting rules work.

  1. From the Glitch editing window, click View app. The Space Camp app appears.
  2. Click log-in.

Space camp app login
Space camp app login

  1. Enter a random email ending in .edu.
  2. A discounted price of $10,000, $25,000, or $100,000 should appear.
  3. Log out.
  4. Log in again with a random email ending in .com
  5. Standard pricing should appear.
Remember to disable "Do not track"
If you have not disabled "Do not track," you will not be able to log in to the Glitch app. If "Do not track" is disabled, but the login still fails, try disabling ad blockers or popup blockers in your browser as well.

Creating a click metric

Metrics let you measure user behaviors affected by your flag. The three types of metrics are click, pageview, and custom metrics.

To learn more about metrics in experiments, read Creating experiments.

First, you must get the URL for your Glitch app. To find it:

  1. From the Glitch console, click the Share button. A pop-up with a URL appears.
  2. Select Live App from the pop-up.
  3. Copy the Live App URL.
  4. Save the URL somewhere where it is easy to access.

Here's how to create a click metric to track the discount which results in the highest number of packages purchased:

  1. Navigate to LaunchDarkly.
  2. Select Metrics from the sidebar menu.
  3. Click + Metric. The "Create a new metric screen" appears.
  4. Name your metric "Student pricing tier".
  5. Enter a Description to indicate what the metric is tracking. For example, you could enter “Track price chosen by students”.
  6. Choose Click from the "Event kind" dropdown.
  7. Enter a.btn in the CSS selector in the "Click targets" field.
  8. In the "Target URL" field, choose Substring Match.
  9. Paste in the URL for your Glitch app, which you found and saved in the previous procedure.

Connecting the metric to the flag

You must connect the metric to the flag to create an experiment.

  1. Navigate to LaunchDarkly.
  2. Click on the student-discount-pricing flag. The flag's dashboard appears.
  3. Click into the flag's Experiments tab.
  4. Click + Create an Experiment. The "Update experiment settings for all environments" menu appears.
  5. Choose the metric you created from the "Add a new metric" dropdown.
  6. Select Variation 4, "No discount," as the baseline.
  7. Click Save Experiment Settings. You return to the flag's Experiments tab.

You've created the experiment and it's ready to go. Now you have to start it.

Starting your experiment

For this exercise, we will simulate traffic to your application.

  1. In the flag's Experiments tab, click Start Recording.
  2. Open a new browser window.
  3. Load the Glitch application.
  4. Generate traffic by pressing !.
  5. Wait for the simulation to complete and to view the results in LaunchDarkly.
You may have to wait for experimentation results
Data can take up to 10 minutes to appear in LaunchDarkly after your experiment starts.

After you press !, 100 .com email addresses and 100 .edu email addresses will try to access your app. None of the .com email addresses will receive a reduced price. The .edu addresses will randomly receive discounted prices. This information is reflected in the experiment's results.

Conclusion

In this guide you:

  • Created a multi-variate flag
  • Configured metrics
  • Ran an experiment

We hope this helped you learn how to collect metrics from feature flags to make data-informed decisions about features.

Want to know more? Start a trial.

Your 30-day trial begins as soon as you sign up. Learn to use LaunchDarkly with the app's built-in quick start guide. You'll see how easy it is to manage the whole feature lifecycle from concept to launch to control.

Want to try it out? Start a trial.