Using funnel optimization experiments

Overview

This guide provides an example of how to create and read the results of a funnel optimization experiment.

Funnel optimization experiments measure business-critical user flows and provide results specific to those product funnels. You can use funnel metric groups with funnel optimization experiments to measure the results of multi-step user journeys.

Funnel optimization experiments let your product and marketing teams safely and rapidly increase the performance of your funnels by:

• measuring the performance of your main acquisition and engagement channels
• identifying strong points and weak points in your funnel flow
• testing changes to ensure your funnel's effectiveness improves over time

Prerequisites

To complete this guide, you must have the following prerequisites:

• an understanding of your user funnel and the metrics you want to track at each step in the funnel
• an active LaunchDarkly account with Experimentation enabled, and with permissions to create flags, metrics, and experiments

Concepts

In order to use this guide, you should understand the following concepts:

Metrics

Metrics measure audience behaviors in your app or product, and how those behaviors are affected by different flag variations in your experiment. You can use metrics to track a wide range of end-user behaviors. Custom conversion/binary metrics and click metrics are most often used with funnel optimization experiments. To learn more, read Choosing a metric type.

Funnel experiments can only use metrics that use the "mean" analysis method. To learn more, read Analysis method.

User funnels

A "funnel" is a marketing model that describes a customer's journey through your purchasing cycle, typically from the awareness stage to the purchasing stage. LaunchDarkly's funnel optimization experiments use multiple metrics to track the performance of each of the steps in your funnel over time.

Example: new user sign-up flow

In this example, we'll demonstrate how to create an experiment that measures the effectiveness of two different versions of a sign-up form on increasing sign-ups for a product. The experiment will measure the effect on each step in the funnel and determine the most effective variation overall.

Creating this experiment requires six steps:

1. Creating a feature flag to experiment on: You can create a new flag, or you may already have an existing flag, to run an experiment on. This flag should control the feature on your site you want to experiment on, and include the variations you want to test as well as a control variation. You can use a boolean flag if you have just one variation to test in addition to the control, or a multivariate flag if you have more than one.
2. Creating the metrics for your metric group: Create a metric for each end-user action you want to track as part of your funnel. This might include actions like clicking a sign-up button, entering contact information, opting in to receive emails, and completing payment.
3. Creating a funnel metric group: Funnel metric groups organize your metrics together in a set order. This ensures you're tracking end-user behavior in the correct sequence, and lets you reuse the same group in multiple experiments so you don't have re-set up your metrics each time.
4. Building the experiment: Building the experiment connects the metric group to your flag and determines the audience you want to run the experiment on.
5. Turning flag targeting on and starting an iteration: Your experiment will begin recording data after you turn on flag targeting and start an iteration. You can stop the iteration after you have received enough results to make a decision about the winning variation.
6. Reading experiment results: The experiment's Results tab includes information about how each variation performed in each step of the funnel, as well as the winning variation overall.

These steps are explained in detail below.

Creating the feature flag

Imagine you have noticed that in your sign up flow, a lot of users drop off at your payment step. You think it might be related to the user interface (UI), so you’re going to test two new versions of the payment screen.

To test these versions, first create a feature flag for your experiment with three variations: a payment option with a drop-down menu, with a radio button, and with the current UI which will serve as the control.

To create the flag:

1. Navigate to the flags list.
2. Click Create flag.
3. Enter a unique, human-readable Name.
4. (Optional) Update the flag Key. You'll use this key to reference the flag in your code. A suggested key auto-populates from the name you enter, but you can customize it if you wish.
5. (Optional) Enter a Description of the flag. A brief, human-readable description helps your account members understand what the flag is for.
6. (Optional) Update the Maintainer for the flag.
7. Choose the Experiment flag template in the Configuration section.
8. Select the "String" Flag variations type.
9. For the first variation, update the Name to "Old flow."
10. For the second variation, update the Name to "Dropdown select."
11. For the third variation, update the Name to "Radio select."
12. Click Create flag.

Here is what the Variations tab of the flag looks like:

To learn more about creating flags, read Create flags.

Creating metrics

Next, create the metrics in your funnel.

In this example, your sign-up flow has four steps:

1. The customer clicks your site's "Sign up" button
2. They enter their personal information
3. They add a payment method
4. They click the "Pay now" button

This table explains the metrics you will create for this example:

To create the first metric:

1. Navigate to the Metrics list.
2. Click Create metric. The "Create metric" panel appears.
3. Enter "Sign-up button clicked" as the Name.
4. (Optional) Enter a Description.
5. In the "Event information" section, select Click.
6. Enter the CSS selector for your sign-up button in the Click targets field. In this example, the selector is .button.
7. Select the "exact match" Target type.
8. Enter the Target URL that corresponds to your sign-up button:

9. Select "Average individual unit values" for the Unit aggregation method.
10. Choose the "user" Randomization unit.
11. Click Create metric.

To create the second metric:

1. Enter "Sign-up info entered" as the Name.
2. (Optional) Enter a Description.
3. Select Conversion/binary.
4. Enter an Event name such as info-entered. You will reference this event name in your code snippet when you insert the metric information into your app.
5. Select "Average individual unit values" for the Unit aggregation method.
6. Choose the "user" Randomization unit.
7. Click Create metric.

To create the third metric, repeat the process for creating a conversion/binary metric, but name the metric "Sign-up payment added" and choose an event name like payment-entered.

To create the fourth metric, repeat the process for creating a click metric, but name the metric "Sign-up completed" with a target type of "Exact match."

To learn more, read Metrics.

Creating a funnel metric group

Now that you have created your four metrics, you can combine them together into a funnel metric group.

To create your funnel metric group:

1. Navigate to the Funnel groups tab from the Metrics list.
2. Click Create metric group. The "Create metric group" dialog appears.
3. Enter "New plan sign-up flow" for the Name.
4. Enter "click sign up" as the first Step name.
5. Choose the "Sign-up button clicked" metric from the Metric menu.
6. Enter "Info entered" as the second Step name.
7. Choose the "Sign-up info entered" metric from the Metric menu:

8. Click Add metric and repeat steps 4-5 for the remaining two steps in your funnel.
9. Click Create metric group.

To learn more, read Metric groups.

Building the experiment

Next, it's time to put it all together to create a funnel optimization experiment.

To build the experiment:

1. Navigate to the Experiments list.
2. Click Create experiment.
3. The "Experiment details" step opens. Enter "New user sign-up flow" for the Name.
4. Enter a Hypothesis such as "Using a better sign-up UI will result in a higher number of overall sign ups for the product."
5. Select Funnel optimization.
6. Click Next. The "Choose randomization unit and attributes" step opens.
7. Choose the "user" context kind from the Randomization unit menu.
8. Click Next. The "Select metrics" step opens.
9. Choose your "New plan sign-up flow" Funnel group:

10. Click Next. The "Choose flag variations" step opens.
11. Choose "Enable new sign-up menu options" from the Flag menu.
12. Click Next. The "Set audience" step opens.
13. Choose the default targeting rule to run the experiment on.

15. Enter the percentage of traffic for each variation you want to include in the experiment. In this example, 10% will be in the control variation, 10% will be in the "Dropdown select" variation, and 10% will be in the "Radio select" variation.
16. Select the control variation to serve to the remaining population.
17. Click Finish. You are returned to the experiment's Design tab.

To learn more, read Creating experiments.

Turning flag targeting on and starting an iteration

When you're ready, turn your flag's targeting on and begin recording results by clicking Start on the experiment's Design tab:

To learn more, read Turning flags on and off and Starting experiment iterations.

Reading experiment results

After enough users have encountered your experiment, you can evaluate its performance on the experiment's Results tab.

The winning variation displays at the top of the probability report:

The "Radio select" variation is performing best for this funnel.

You can examine the results more closely in the probability report. Here, you can see that the "Radio select" variation has a conversion rate of about 18%:

However, you can also see that in the third "sign up payment added" step, the "Dropdown select" variation actually performed slightly better at 32% than the "Radio select" variation at 31%, but for some reason users aren’t completing the purchase as often. Still, LaunchDarkly declared the "Radio select" variation the winner because it had the highest conversion rate of all the variations in the funnel overall.

To learn more, read Winning variations.

Conclusion

In this guide you learned how to create a funnel optimization experiment to measure the success of a user funnel. You learned how to create the flag, how to set up metrics and a metric group, how to build the experiment, and how to interpret funnel optimization experiment results. To learn more about how you can use Experimentation to meet your business needs, contact contact Sales or your LaunchDarkly representative.