Experimenting with feature flags
Read time: 6 minutes
Last edited: Aug 07, 2020
This guide explains how to run an experiment in LaunchDarkly. Experiments are useful for product managers, engineers, designers, or anyone who wants to validate their hypothesis about any part of their platform controlled by feature flags.
In this guide, you will:
- Install and configure a client-side SDK,
- Create a multivariate flag,
- Configure targeting rules,
- Create a metric and connect it to a flag, which creates an experiment, and
- Run and analyze the results of an experiment.
The experiments in this guide are examples only. They are intended to teach you the basic principles and process of conducting an experiment in LaunchDarkly. Do not replicate this experiment in production and expect rigorously produced results.
To complete an experiement that results in useful data, you should conduct more preparation than is outlined in this guide, including rigorous sampling practices.
To learn more about how to create an effective experiment, read Designing experiments in LaunchDarkly.
To complete this guide, you must have the following prerequisites:
- You must have completed the exercises in Rules and targeting. The exercises in this guide rely on concepts from that guide. To learn more about how LaunchDarkly's rules and targeting practices, read Rules and targeting.
- An active LaunchDarkly account with the ability to create a flag.
- "Do not track" must be disabled in your browser for the experiment's click events to work. To confirm that "Do not track" is disabled, visit AllAboutDNT.com.
- You should be familiar with the concept of experimentation for software products, and know what results you want to determine. To learn more, read Introduction to experimentation.
- You must have Experimentation enabled in your LaunchDarkly account. To enable experimentation, contact your LaunchDarkly account manager or email email@example.com.
- Access to Glitch.
The following concepts will help you understand this guide.
A multivariate flag is a feature flag that is not boolean and has more than one possible variation. Multivariate flags allow you to use one flag to serve multiple variations of a feature simultaneously.
This app builds on the same app in the Rules and targeting guide. If you haven't completed that guide yet, read and complete those exercises first and then return to this guide.
The app below begins where the previous guide left off.
To begin, remix the Glitch project to create an editable copy of the app. Use the embedded Glitch app above, or click here to remix the app.
You can re-use the project and environment you created in Getting started with your first flag, or create a new one.
Glitch lets you store environment variables, such as secrets or config variables, that your app can read from memory. This is a more secure practice than hard-coding these directly into your code.
- Find a file called
.envin the Glitch sidebar.
- Copy the SDK key from the Test environment by clicking on the key.
- Assign the SDK key to the environment variable called
- Copy the Client-side ID from the Test environment.
- Assign the Client-side ID to the environment variable called
Experiments can use boolean flags for A/B tests, or multivariate flags for A/B/n tests. Multivariate flags have multiple settings that let you define different variations of one feature. This exercise uses a multivariate flag.
Experiments use multivariate flags to test different versions of a feature. Settings can include strings, numbers, JSON objects, or JSON arrays.
This exercise tests different prices to determine at what price students are more willing to purchase a trip. The trips are discounted to 25%, 50%, and 75% off their original price.
A 25% discount means a student would pay 75% of the original cost of the trip. The equation to calculate this for a $100 trip is
100 * .75. Similarly, to calculate a 50% discount for a $100 trip, you would enter
100 * .5
Our application stores numbers like this in an array. We will use a multivariate flag's value to calculate the discount. To do this, we multiply the values in the array by a constant .25 for 75% off, .5 for 50% off, and .75 for 25% off.
Each of those values corresponds to one variation of a feature flag. When we run an experiment, we compare how each variation of a flag produces different behavior in our users.
First, create a new flag in LaunchDarkly.
- Click + Flag.
- Name your flag "Student Discount Pricing" (omit the quotes around the phrase).
- Enter a description of the flag to help team members understand what the flag is for.
- Click into the Tags dropdown and enter “experiment”. This creates a tag that lets you filter flags based on which flags have experiments running.
- Under "Flag variations", choose Number.
- Enter a name and value for the variations as given below:
- Variation 1 value: .25
- Variation 1 name: 75% off
- Variation 2 value: .5
- Variation 2 name: 50% off
- Variation 3 value: .75
- Variation 3 name: 25% off
- Variation 4 value: 1
- Variation 4 name: No discount
- Click the 'Make this flag available to client-side SDKs' checkbox.
- Save your new flag.
A successful experiment requires two or more segments to test against. How you segment your users will vary based on your business.
Here are some possible ways to segment users:
- Logged in as opposed to anonymous
- By company
- By geography
In this experiment, we target users based on their email address. We want to offer pricing discounts based on
whether a person is a student. Users with a
.edu email address will be presented
with three different prices.
- In LaunchDarkly, navigate to the Targeting menu for your new flag.
- Under the "Target users who match these rules" section, click Add a rule.
- Under "Select an attribute" choose
ends with, and
- Select a percentage rollout. Enter percentage values for the four variations of your flag. The values must add up to 100%.
- Toggle flag targeting on.
Now you must update your app to display discounted pricing if a user logs in with a .edu email address.
To update your app:
- Navigate to your Glitch app.
- Find the
server.pyfile in the sidebar and click to open it.
- Add or change the pricing array code. Comments in the file indicate where to add the code.
1discount = ld_client.variation('student-discount-pricing', user, 1)2price = np.array([10000, 25000, 100000]) * discount
Before going on to set up your experiment, confirm the multivariate flag and targeting rules work.
- From the Glitch editing window, click View app. The Space Camp app appears.
- Click log-in.
- Enter a random email ending in .edu.
- A discounted price of $10,000, $25,000, or $100,000 should appear.
- Log out.
- Log in again with a random email ending in .com
- Standard pricing should appear.
Metrics let you measure user behaviors affected by your flag. The three types of metrics are click, pageview, and custom metrics.
To learn more about metrics in experiments, read Creating experiments.
First, you must get the URL for your Glitch app. To find it:
- From the Glitch console, click the Share button. A pop-up with a URL appears.
- Select Live App from the pop-up.
- Copy the Live App URL.
- Save the URL somewhere where it is easy to access.
Here's how to create a click metric to track the discount which results in the highest number of packages purchased:
- Navigate to LaunchDarkly.
- Select Metrics from the sidebar menu.
- Click + Metric. The "Create a new metric screen" appears.
- Name your metric "Student pricing tier".
- Enter a Description to indicate what the metric is tracking. For example, you could enter “Track price chosen by students”.
- Choose Click from the "Event kind" dropdown.
a.btnin the CSS selector in the "Click targets" field.
- In the "Target URL" field, choose Substring Match.
- Paste in the URL for your Glitch app, which you found and saved in the previous procedure.
You must connect the metric to the flag to create an experiment.
- Navigate to LaunchDarkly.
- Click on the
student-discount-pricingflag. The flag's dashboard appears.
- Click into the flag's Experiments tab.
- Click + Create an Experiment. The "Update experiment settings for all environments" menu appears.
- Choose the metric you created from the "Add a new metric" dropdown.
- Select Variation 4, "No discount," as the baseline.
- Click Save Experiment Settings. You return to the flag's Experiments tab.
You've created the experiment and it's ready to go. Now you have to start it.
For this exercise, we will simulate traffic to your application.
- In the flag's Experiments tab, click Start Recording.
- Open a new browser window.
- Load the Glitch application.
- Generate traffic by pressing
- Wait for the simulation to complete and to view the results in LaunchDarkly.
After you press
!, 100 .com email addresses and 100 .edu email addresses will try to access your app. None of the .com email addresses will receive a reduced price. The .edu addresses will randomly receive
discounted prices. This information is reflected in the experiment's results.
In this guide you:
- Created a multi-variate flag
- Configured metrics
- Ran an experiment
We hope this helped you learn how to collect metrics from feature flags to make data-informed decisions about features.