No results for ""
EXPAND ALL
  • Home
  • API docs

GIVE DOCS FEEDBACK

Using funnel optimization experiments

Read time: 10 minutes
Last edited: May 01, 2024
Experimentation is available for Pro and Enterprise plans

Experimentation is available as an add-on to customers on a Pro or Enterprise plan. To learn more, read about our pricing. To add Experimentation to your plan, contact Sales.

Overview

This guide provides an example of how to create and read the results of a funnel optimization experiment.

Funnel optimization experiments measure business-critical user flows and provide results specific to those product funnels. You can use funnel metric groups with funnel optimization experiments to measure the results of multi-step user journeys.

Funnel optimization experiments let your product and marketing teams safely and rapidly increase the performance of your funnels by:

  • measuring the performance of your main acquisition and engagement channels
  • identifying strong points and weak points in your funnel flow
  • testing changes to ensure your funnel's effectiveness improves over time

Prerequisites

To complete this guide, you must have the following prerequisites:

  • an understanding of your user funnel and the metrics you want to track at each step in the funnel
  • an active LaunchDarkly account with Experimentation enabled, and with permissions to create flags, metrics, and experiments

Concepts

In order to use this guide, you should understand the following concepts:

Metrics

Metrics measure audience behaviors in your app or product, and how those behaviors are affected by different flag variations in your experiment. You can use metrics to track a wide range of end-user behaviors. Custom conversion/binary metrics and click metrics are most often used with funnel optimization experiments. To learn more, read Choose a metric type.

Funnel experiments can only use metrics that use the "mean" analysis method. To learn more, read Analysis method.

User funnels

A "funnel" is a marketing model that describes a customer's journey through your purchasing cycle, typically from the awareness stage to the purchasing stage. LaunchDarkly's funnel optimization experiments use multiple metrics to track the performance of each of the steps in your funnel over time.

Example: new user sign-up flow

In this example, we'll demonstrate how to create an experiment that measures the effectiveness of two different versions of a sign-up form on increasing sign-ups for a product. The experiment will measure the effect on each step in the funnel and determine the most effective variation overall.

Creating this experiment requires six steps:

  1. Creating a feature flag to experiment on: You can create a new flag, or you may already have an existing flag, to run an experiment on. This flag should control the feature on your site you want to experiment on, and include the variations you want to test as well as a control variation. You can use a boolean flag if you have just one variation to test in addition to the control, or a multivariate flag if you have more than one.
  2. Creating the metrics for your metric group: Create a metric for each end-user action you want to track as part of your funnel. This might include actions like clicking a sign-up button, entering contact information, opting in to receive emails, and completing payment.
  3. Creating a funnel metric group: Funnel metric groups organize your metrics together in a set order. This ensures you're tracking end-user behavior in the correct sequence, and lets you reuse the same group in multiple experiments so you don't have re-set up your metrics each time.
  4. Building the experiment: Building the experiment connects the metric group to your flag and determines the audience you want to run the experiment on.
  5. Turning flag targeting on and starting an iteration: Your experiment will begin recording data after you turn on flag targeting and start an iteration. You can stop the iteration after you have received enough results to make a decision about the winning variation.
  6. Reading experiment results: The experiment's Results tab includes information about how each variation performed in each step of the funnel, as well as the winning variation overall.

These steps are explained in detail below.

Create the feature flag

Imagine you have noticed that in your sign up flow, a lot of users drop off at your payment step. You think it might be related to the user interface (UI), so you’re going to test two new versions of the payment screen.

To test these versions, first create a feature flag for your experiment with three variations: a payment option with a drop-down menu, with a radio button, and with the current UI which will serve as the control.

To create the flag:

  1. Navigate to the flags list.
  2. Click Create flag.
  3. Enter "Enable new sign-up menu options" for the Name.
  4. (Optional) Enter a Description of the flag.
  5. Choose the Experiment flag template.
The "Configuration" section of the "Create flag" page.
The "Configuration" section of the "Create flag" page.
  1. Select the "String" Flag variations type.
  2. For the first variation, update the Name to "Old flow."
  3. For the second variation, update the Name to "Dropdown select."
  4. For the third variation, update the Name to "Radio select."
  5. For the second variation, update the Name to "Dropdown select."
  6. For the third variation, update the Name to "Radio select."
  7. Click Create flag.

Here is what the Variations tab of the flag looks like:

The "Variations" tab of a flag for use in a funnel optimization experiment.
The "Variations" tab of a flag for use in a funnel optimization experiment.

To learn more about creating flags, read Creating new flags.

Create metrics

Next, create the metrics in your funnel.

In this example, your sign-up flow has four steps:

  1. The customer clicks your site's "Sign up" button
  2. They enter their personal information
  3. They add a payment method
  4. They click the "Pay now" button

This table explains the metrics you will create for this example:

NameMetric typeTarget typeRandomization unit
Sign-up button clickedClickExact matchuser
Sign-up info enteredCustom conversion/binaryn/auser
Sign-up payment addedCustom conversion/binaryn/auser
Sign-up completedClickExact matchuser

To create the first metric:

  1. Navigate to the Metrics list.
  2. Click Create metric. The "Create metric" dialog appears.
  3. Enter "Sign-up button clicked" as the Name.
  4. (Optional) Enter a Description.
  5. In the "Event information" section, select Click.
  6. Enter the CSS selector for your sign-up button in the Click targets field. In this example, the selector is .button.
  7. Select the "exact match" Target type.
  8. Enter the Target URL that corresponds to your sign-up button:
A click metric with a "user" randomization unit.
A click metric with a "user" randomization unit.

  1. Select Average individual unit values for the Unit aggregation method.
  2. Choose the "user" Randomization unit.
  3. Click Create metric.

To create the second metric:

  1. Enter "Sign-up info entered" as the Name.
  2. (Optional) Enter a Description.
  3. Select Conversion/binary.
  4. Enter an Event name such as info-entered. You will reference this event name in your code snippet when you insert the metric information into your app.
  5. Select Average individual unit values for the Unit aggregation method.
  6. Choose the "user" Randomization unit.
  7. Click Create metric.

To create the third metric, repeat the process for creating a conversion/binary metric, but name the metric "Sign-up payment added" and choose an event name like payment-entered.

To create the fourth metric, repeat the process for creating a click metric, but name the metric "Sign-up completed" with a target type of "Exact match."

To learn more, read Metrics.

Create a funnel metric group

Now that you have created your four metrics, you can combine them together into a funnel metric group.

To create your funnel metric group:

  1. Navigate to the Funnel groups tab from the Metrics list.
  2. Click Create metric group. The "Create metric group" dialog appears.
  3. Enter "New plan sign-up flow" for the Name.
  4. Enter "click sign up" as the first Step name.
  5. Choose the "Sign-up button clicked" metric from the Metric menu.
  6. Enter "Info entered" as the second Step name.
  7. Choose the "Sign-up info entered" metric from the Metric menu:
A funnel optimization group with two steps added.
A funnel optimization group with two steps added.
  1. Click Add metric and repeat steps 4-5 for the remaining two steps in your funnel.
  2. Click Create metric group.
Metric order is important

The order of your metrics within a funnel metric group must match the order of the steps your users will take within the funnel. If they do not match the order of your user flow, your experiment results will be invalid.

To learn more, read Metric groups.

Build the experiment

Next, it's time to put it all together to create a funnel optimization experiment.

To build the experiment:

  1. Navigate to the Experiments list.
  2. Click Create experiment.
  3. The "Experiment details" step opens. Enter "New user sign-up flow" for the Name.
  4. Enter a Hypothesis such as "Using a better sign-up UI will result in a higher number of overall sign ups for the product."
  5. Select Funnel optimization.
  6. Click Next. The "Choose randomization unit and attributes" step opens.
  7. Choose the "user" context kind from the Randomization unit menu.
  8. Click Next. The "Select metrics" step opens.
  9. Choose your "New plan sign-up flow" Funnel group:
The "Select metrics" step of a new funnel optimization experiment.
The "Select metrics" step of a new funnel optimization experiment.
  1. Click Next. The "Choose flag variations" step opens.
  2. Choose "Enable new sign-up menu options" from the Flag menu.
  3. Click Next. The "Set audience" step opens.
  4. Choose the default targeting rule to run the experiment on.
The "Set audience" step with the default rule chosen.
The "Set audience" step with the default rule chosen.
  1. Enter the percentage of traffic for each variation you want to include in the experiment. In this example, 10% will be in the control variation, 10% will be in the "Dropdown select" variation, and 10% will be in the "Radio select" variation.
  2. Select the control variation to serve to the remaining population.
  3. Click Finish. You are returned to the experiment's Design tab.

To learn more, read Creating experiments.

Turn flag targeting on and start an iteration

When you're ready, turn your flag's targeting on and begin recording results by clicking Start on the experiment's Design tab:

An experiment with the "Start" button called out.
An experiment with the "Start" button called out.

To learn more, read Turning flags on and off and Start experiment iterations.

Experiment results

After enough users have encountered your experiment, you can evaluate its performance on the experiment's Results tab.

The winning variation displays at the top of the probability report:

The winning variation on an experiment's "Results" tab.
The winning variation on an experiment's "Results" tab.

The "Radio select" variation is performing best for this funnel.

You can examine the results more closely in the probability report. Here, you can see that the "Radio select" variation has a conversion rate of about 18%:

The winning variation on an experiment's "Results" tab.
The winning variation on an experiment's "Results" tab.

However, you can also see that in the third "sign up payment added" step, the "Dropdown select" variation actually performed slightly better at 32% than the "Radio select" variation at 31%, but for some reason users aren’t completing the purchase as often. Still, LaunchDarkly declared the "Radio select" variation the winner because it had the highest conversion rate of all the variations in the funnel overall.

To learn more, read Winning variations.

You can also send data to Snowflake

Snowflake users who want to explore this data more in-depth can send all of the experiment data to a Snowflake database using the Snowflake integration. After your experiment data is in the same warehouse as your other data, you can easily build custom reports to answer more complicated product behavior questions. To learn more, read Snowflake.

Conclusion

In this guide you learned how to create a funnel optimization experiment to measure the success of a user funnel. You learned how to create the flag, how to set up metrics and a metric group, how to build the experiment, and how to interpret funnel optimization experiment results. To learn more about how you can use Experimentation to meet your business needs, contact contact Sales or your LaunchDarkly representative.

Want to know more? Start a trial.

Your 14-day trial begins as soon as you sign up. Get started in minutes using the in-app Quickstart. You'll discover how easy it is to release, monitor, and optimize your software.

Want to try it out? Start a trial.