• HOME
  • INTEGRATIONS
  • SDKS
  • GUIDES
  • API DOCS
No results for ""
EXPAND ALL
launchdarkly.com

EDIT ON GITHUB

Creating experiments

Read time: 7 minutes
Last edited: Oct 06, 2020

Overview

This topic explains how to set up and configure an experiment in LaunchDarkly. It introduces the concepts of metrics, explains different metric types, and explains how metrics interact with feature flags to create experiments.

In LaunchDarkly, the combination of metrics and flags is an experiment. Experiments let you measure the effect of flags on users by mapping them to the metrics your team cares about.

You can create an experiment from a flag's Experiments tab or from the Experiments dashboard. To learn more, read Creating experiments.

How experiments work

When a user performs a metric-tracked action in your application after they encounter a feature flag, the experiment logs an event.

For example, an experiment might show that users are more likely to convert by clicking "Checkout" when the Checkout button is a certain color. The metric you would track to measure that is when they click on the Checkout button. It would be connected to a flag serving four variations, each of which is a different color for the button.

Creating experiments

It takes three steps to create an experiment. You must:

  1. Create a metric, or choose an existing metric
  2. Connect the metric to feature flags, and
  3. Start recording your experiment

You can reuse existing metrics by connecting them to multiple flags, which allows you to compare how this metric fares across numerous flags. Similarly, a single flag can have more than one metric connected to it, which allows you to see how this flag performs against various measurements.

Find all the experiments in your environment by clicking Experiments in the sidebar menu.

The Experiments dashboard.
The Experiments dashboard.

After you set up your first metric and flag, you can create new experiments by clicking the + Experiment button.

Creating metrics

Metrics are the measure against which you evaluate flag events. You can use metrics to track all kinds of things, from how often users access a URL to how long that URL takes to load a page. Create metrics that align with your business goals and connect them to your team's flags to track the impact of different flag variations over time.

You can use different metrics to track experiment results in LaunchDarkly. They are:

  • Conversion metrics, which register events when a user takes an action based on a feature flag they encounter.
  • Numeric metrics, which measure numeric values against a baseline you set.

The following table explains different actions you can track with a metric:

Event kindDescription
ClickThis is a conversion metric. Tracks the clicks on a UI element. For example, how frequently a user clicks the Save button.
PageviewThis is a conversion metric. Tracks how many times a page is viewed. For example, how many times a blog post is viewed based on three different titles.
Custom (conversion)Tracks events for any arbitrary event. For example, whether a user search called a service or not.
Custom (numeric)This event kind is required for numeric metrics. Tracks increases or decreases in numeric value against a baseline you set. For example, how many items are in a user's cart when they check out of your online store.
We recommend the Javascript or React SDKs for some metrics
You must be using the Javascript or React SDK to use click and pageview metrics. If you do not have the Javascript or React SDK and would like to track clicks or pageviews, use a custom conversion metric.

Creating click metrics

Click metrics track how often a user interacts with an element of your product's UI.

For example, you might want to track how often a user clicks a Save button when it is a red or blue. You can create a metric that monitors clicks on the Save button as designated by a CSS selector located at a specific URL.

You can even monitor clicks on the button across different URLs by telling the metric to track different URLs based on substrings or simple matches. For example, you could track an event on every URL with /checkout-page/ in the URL path.

What's a CSS selector?

CSS selectors designate specific areas of your product UI, like buttons or images, that your users can interact with.

For example, some CSS selectors include:

  • #example-element-id,

  • .example-classname,

  • button.example-button, or .example-button, .example-button-alt.

To create a click metric:

  1. Navigate to the Metrics screen.
  2. Click New. The "Create a new metric" screen appears.
  3. Enter a human-readable Name for the metric.
  4. (Optional) Enter a Description.
  5. (Optional) Type or choose from available Tags.
  6. (Optional) Choose a Maintainer. By default, the person who creates the metric is set as the Maintainer.
  7. Choose Click from the Event kind dropdown.
  8. Enter one or more CSS selectors in the Click targets field.
  9. Specify a type of URL you want to track behavior on. There are four types of URL matching:
URL matching schemaDescriptionExample
Simple matchThis triggers an event when the user accesses a URL, or clicks on an element hosted at a URL, that matches the string you enter in the Target URL field or a formulation of that URL that included appended parameters. If you want events to record for URLs with substrings, specify each URL individually.You specify: example.com.
Views or clicks at example.com, and
example.com#signin trigger events.
You specify: example.com and
example.com/login.
Views or clicks on example.com and example.com/login trigger events.
Exact matchThis triggers an event when the user accesses a URL, or clicks on an element hosted at a URL, that exactly matches what you enter in the Target URL field. Identical base URLs with different substrings do not trigger events.You specify: example.com.
Only views or clicks at example.com trigger events.
Views and clicks at www.example.com/signup and
https://www.example.com do not trigger events.
Substring matchThis triggers an event when the user accesses a URL, or clicks on an element hosted at a URL with a specific string of text in it.You specify: your-substringexample.com/your-substring, home.com/username/your-substring/, launchdarkly.com/your-substring/settings Because all of these URLs contain your-substring, clicks on the CSS selectors you specify on each page trigger events.
Regular expressionThis triggers an event based on a regex expression you specify. This lets you designate more precise locations to trigger click and pageview events.You specify: example\.com\\/account\\/.*\\/billing This matches any string between account/ and /billing, such as http://example.com/account/12345/billing
  1. (Optional) Click Add Additional URL to track behavior on more than one URL.
  2. Click Save Metric.

Creating pageview metrics

Pageview metrics track how often users see an element on a page hosted at a specific URL.

For example, you might want to track how often customers view an item when its price is discounted different amounts.

To create a pageview metric:

  1. Follow steps 1-6 in Creating click metrics until you have configured the form up to the "Event information" section.
  2. Choose Page view from the Event kind dropdown.

The Event kind dropdown.
The Event kind dropdown.

  1. Specify a type of URL you want to track behavior on. To learn about the different options for URL matching, see the table in Creating click metrics.
  2. (Optional) Click Add Additional URL to track behavior on more than one URL.
  3. Click Save Metric.

Creating custom conversion metrics

Custom conversion metrics require you to create an event name and insert it into your application to track conversions for any event in your code. To add a custom conversion metric in LaunchDarkly, you must identify it with a code snippet embedded in your app.

When you're ready to add a custom event, put this snippet in your code:

1ldClient.track("Example event name", user, value);

The area of your code you should put custom conversion metric information into, and the type of information you should include, varies based on which SDKs you use.

Often, the information you should put in your code includes the event key, user object, data field, or numeric value fields. To learn more, read the documentation for your SDKs.

To create a custom conversion metric:

  1. Follow steps 1-6 in Creating click metrics until you have configured the form up to the "Event information" section.
  2. Choose Custom from the Event kind dropdown.

The Event kind dropdown.
The Event kind dropdown.

  1. Click to select the Conversion radio button
  2. Enter a human-readable Event name. This is the event name you will reference in your code snippet when you insert the metric information into your app.
  3. Click Save Metric.

The "Event information" section of the "Create a new metric" screen with a conversion metric configured.
The "Event information" section of the "Create a new metric" screen with a conversion metric configured.

Creating numeric metrics

Alternatively, you can use numeric metrics to track the changes in a value against a baseline value. Unlike conversion metrics, which track whether or not an event occurred, numeric metrics let you measure changes in value against a baseline flag variation you choose.

For example, you can use numeric metrics to track:

  • Page load time
  • The efficacy of various search algorithms
  • The number of items in a shopping cart at checkout

To add a custom numeric metric in LaunchDarkly, you must identify it with a code snippet embedded in your app.

When you're ready to add a custom event, put this snippet in your code:

1ldClient.track("Example event name", user, null, numericValue);
2
3/* The `user` parameter is ommitted in client-side SDKs */

Like conversion metrics, numeric metrics also require you to create an event name and insert it into your application's code to track metric data. The area of your code you should put custom conversion metric information into, and the type of information you should include, varies based on which SDKs you use.

Often, the information you should put in your code includes the event key, user object, data field, or numeric value fields. To learn more, read the documentation for your SDKs.

Can't see numeric metrics options?
If you can't see the feature described below, your SDKs may be out of date. To solve this problem, update your SDKs to the latest version, and contact Support to enable numeric metrics.

To create a numeric metric:

  1. Follow steps 1-6 in Creating click metrics until you have configured the form up to the "Event information" section.
  2. Enter a human-readable Event name. This is the event name you will reference in your code snippet when you insert the metric information into your app.
  3. Enter a Unit of measure to track.
  4. Choose a Success criteria from the dropdown.
  5. Click Save Metric.

The "Event information" section of the "Create a new metric" screen with a numeric metric configured.
The "Event information" section of the "Create a new metric" screen with a numeric metric configured.

Choosing success criteria for numeric metrics

When you designate a success criteria, you can choose Lower than baseline or Higher than baseline depending on what you want to track. For example, if you're trying to track page load time to measure the impact of your infrastructure changes, you might want a success criteria that is lower than the baseline if your baseline flag variation shows the original page load time and you want to measure decreases in page load time.

Connecting metrics to feature flags

After you've created a metric, you can connect it to one or more feature flags. If users load a URL, click an element, or otherwise participate in the behavior the metric tracks, LaunchDarkly sends an event to your experiment.

You can connect multiple metrics to one flag to run multiple experiments about that same flag. Alternatively, you can connect the same metric to multiple flags, which saves you from having to create identical metrics for multiple flags.

There are two ways to connect metrics to feature flags:

  • From the Experiments dashboard, and
  • From a flag's Experiments tab.

Creating an experiment from the Experiments dashboard

To connect a metric to a flag from the Experiments dashboard:

  1. Navigate to the Experiments dashboard.
  2. Click + Experiment.

The Experiments dashboard with the "+ Experiment" button called out.
The Experiments dashboard with the "+ Experiment" button called out.

  1. Choose a flag from the "Select flag" dropdown.
  2. (Optional) If the flag doesn't have a baseline variation, choose one from the "Select baseline" dropdown.
  3. Choose a metric from the "Connect metric" dropdown.
  4. Click Save Experiment Settings.

Your new experiment appears on the dashboard.

You're not done yet! After you create an experiment, you have to start it in every environment where you want it to run.

To learn more, read Starting your experiment.

Creating an experiment from a flag's Experiments tab

To connect a metric to a flag from the flag's Experiments tab:

  1. Navigate to the flag's Experiments tab.
  2. If the flag does not have other running experiments, click Create an Experiment. If it already has running experiments, click Manage Experiments. The "Update experiment settings for all environments" screen appears.

A flag's Experiments tab with the Manage Experiments button called out.
A flag's Experiments tab with the Manage Experiments button called out.

  1. Enter a metric name or choose a metric from the Add new metric dropdown. The metric you choose appears in the table below.
  2. Choose a Baseline from the dropdown.
  3. Click Save Experiment Settings. The new experiment appears in the flag's Experiments tab.

The "Update experiment settings for all environments" screen, configured to connect a metric to a flag.
The "Update experiment settings for all environments" screen, configured to connect a metric to a flag.

You're not done yet! After you create an experiment, you have to start it in every environment where you want it to run.

To learn more, read Starting your experiment.

Starting your experiment

After you've connected a metric to a flag, you can start your experiment whenever you want. All you need to do is press the Start Recording button in every environment where you want to record data.

To learn more about starting and pausing recording for experiments, read The experiment lifecycle.

Experiment settings are environment-specific

When you create an experiment, it appears in every environment in your project. However, the Start Recording button only impacts the experiment in one environment at a time. If you want to run the experiment in multiple environments, you must click Start Recording for each environment individually. For example, you might run an experiment in Staging to gather data internally before turning it on in Production to gather customer data.