• HOME
  • INTEGRATIONS
  • SDKS
  • GUIDES
  • API DOCS
No results for ""
EXPAND ALL

EDIT ON GITHUB

Experimentation

Read time: 4 minutes
Last edited: Jul 28, 2021

Overview

This topic explains the concepts and value of LaunchDarkly's Experimentation feature. Read on to learn about Experimentation, how to use it, and what it can do for you.

Experimentation is an add-on feature

Experimentation is available as an add-on for customers on a Pro or Enterprise plan. To learn more, read about our pricing. To add Experimentation to your plan, contact Sales.

What is Experimentation?

LaunchDarkly's Experimentation feature lets you validate the impact of features you roll out to your app or infrastructure. Measure whatever you need, from pageviews to clicks, page load time to infrastructure costs, and more.

By connecting metrics you create to flags in your LaunchDarkly environment, you can precisely measure the changes in your users' behavior based on what flags they see. This helps you make more informed decisions, so the features your development team ships align with your business objectives.

Monitor existing experiments from the Experimentation dashboard, accessible in the sidebar menu.

Flags + Metrics = Experiments
Flags + Metrics = Experiments

Why should I use Experimentation?

LaunchDarkly's flexible, role-agnostic platform means anyone can become an Experimentation expert, regardless of their role. Product managers can use experiments to measure the value of the features they ship, designers can use experiments to conduct multivariate testing on UI and UX components, and DevOps engineers can use it to test the efficacy of their infrastructure changes.

When experiments are tied to feature flags, you can make changes based on your findings quickly and easily.

If an experiment tells you a flag has positive impact, you can roll that flag out to your entire userbase to maximize its value. Alternatively, if you don't like the results you're getting from one flag, just switch it off and minimize its impact on your userbase.

Experimentation can do all of these things and more:

  • A/B/n testing (Multivariate testing)
  • Chart-based visualizations of your experiment results
  • Pause and resume experiments so you have granular control over when data is recorded
  • Display confidence intervals with results so you decide which variation you can trust to have the most impact
  • Expose specific groups of users to experiments, refining your testing population and getting you accurate results faster

How does it work?

Experiments track the impact of different flag variations against a baseline variation of the metric you want to measure. When a user encounters a feature flag in your application, LaunchDarkly registers an event. If that flag is currently connected to a metric, the experiment receives a data point. Experiment data is collected in the Experiments tab, which displays experiment data in near-real time.

After a flag variation has reached statistical significance, LaunchDarkly declares a "winning" flag variation for your experiment. A winning variation is the variation with the most positive deviation from the baseline. You may have multiple winning variations, depending on what you're trying to measure.

After you decide which flag variation has the impact you want, you can seamlessly roll that variation out to 100% of your users with LaunchDarkly's percentage rollouts feature. To learn more about percentage rollouts, read Targeting users.

Here is an image of active experiment data:

An active experiment.
An active experiment.

If you're satisfied with the results, you can pause an experiment after it declares a winning variation, or you can let it run indefinitely to continue gathering data.

To learn more about using experiments, read Managing experiments.

Experimentation requires certain SDK versions

This feature is supported on the following versions of our SDKs (and later versions, if applicable):

  • Go: 4.11.0

  • Java: 4.8.0

  • Node (server-side): 5.9.0

  • Node (client-side): 1.1.0

  • PHP: 3.6.0

  • Python: 6.10.0

  • Ruby: 5.6.0

  • .NET: 5.8.0

  • JavaScript: 2.14.0

  • React: 2.15.0

  • Xamarin: 1.1.0

  • Android: 2.9.0

  • Electron: 1.2.0

Best practices for using Experimentation

As you think about creating experiments for your features, keep these best practices in mind:

  • If a user in your experiment has multiple user keys in LaunchDarkly, use alias events. For example, if a user might access your experiment from both a logged in and logged out state, each state will generate its own user key. Use alias events to associate the two user keys, which prevents one user from generating duplicate events. To learn more, read Associating anonymous users with logged in users.
  • Use feature flags on every new feature you develop. This is a best practice, but it especially helps when you're running experiments in LaunchDarkly. By flagging every feature, you can quickly turn any aspect of your product into an experiment.
  • Consider experiments from day one. Create hypotheses in the planning stage of feature development, so you and your team are ready to run experiments as soon as your feature launches.
  • Define what you're measuring. Align with your team on which tangible metrics you're optimizing for, and what results constitute success.
  • Plan your experiments in relation to each other. If you're running multiple experiments simultaneously, make sure they don't collect similar or conflicting data.

You can use experiments to measure a variety of different outcomes. Some example experiments include:

  • Test the efficacy of different search implementations, such as elasticsearch versus SOLR versus Algolia
  • Track how features you ship are increasing or decreasing page load time
  • Calculate conversion rates by monitoring how frequently users click on various page elements

Pricing for experiments

Experiment events are charged on a per-event basis. Consider this when you're discussing your billing plan with LaunchDarkly. This feature is only available for Pro and Enterprise plans, so speak to your LaunchDarkly account team if you have questions.

You can minimize the cost of Experimentation by only running experiments on selected user groups.

The number of experiments per flag (i.e. having multiple metrics connected to a single flag), the user engagement with that flag, and the target groups included in the experiment can all affect the volume of events LaunchDarkly sends.

Experimentation includes a built-in feature that lets you refine the users included in an experiment based on the flag's targeting rules. To learn more, read Controlling experiment populations.

You may be billed twice for some events if you're also using LaunchDarkly's Data Export features. To learn more, read Data Export.

We want your feedback!

We're actively soliciting feedback and input on Experimentation. If you try it out and have something to tell us, we're eager to hear from you. Send us an email at feedback@launchdarkly.com.

Here are the topics in this category:

  • Creating experiments
  • Managing experiments