Using feature flags to mitigate risk in infrastructure migrations

Overview

This guide explains how you can use feature flags to mitigate risks associated with migrating infrastructure components, like databases or other services, from one provider to another.

Migration projects are common in software development. Every software team at some point migrates one infrastructure component to another. You may perform a migration to improve the scalability and performance of a database on a growing app, improve your caching layer, or transition from an in-house service to a cloud-based solution.

Infrastructure migrations often come with an ample amount of risk. For example, moving from one database to another requires careful planning and execution to maintain data integrity. These types of projects require a lot of work, but they don’t need to be risky. You can use feature flags to manage the transition from one infrastructure component to another.

Using feature flags to facilitate migrations limits or eliminates downtime, allowing you to perform the migration seamlessly. The best part is that if something goes wrong, or you're not getting the results you want, disabling the migration is as fast as toggling the flag off. You can revert a migration with little to no consequence to your end users.

Prerequisites

This technical guide explores a database migration strategy. To use it successfully, you should understand:

• How applications use a database to read and write data
• How to perform CRUD operations on databases

Benefits of using feature flags to manage migrations

When managing a migration, you should avoid a sudden switch-over from your old code to your new code. When your old code and new code are large and complex, it’s better to make the change gradual and easy to reverse. We recommend making both versions available at the same time, and gradually increasing the percentage or types of traffic sent to the new code.

The traditional method is to separate your old code and new code into different codebases or branches. This may require separate deployment clusters and differentiation at a network level. This can be challenging, as it depends on the network being easily segmentable, involves changing configurations in multiple places, and doesn’t have much room for granularity.

A more efficient method is to have both the old code and new code in the same service, with logic in the code deciding which to use for a particular request. This concentrates the decision logic in one place. It gives you more flexibility and granularity, though you may still need to depend on your network configuration in some cases depending on how much contextual info is sent in the request message.

The downside to this method is that every change to that logic requires a code change and deployment, which can mean downtime, especially if the deployment needs to be rolled back. A gradual migration may involve many phases as you validate the new code, expand its capabilities, fix bugs, deploy clusters, migrate consumers, and so on. Each of those phases requires small changes in that logic. This is where LaunchDarkly feature flags come in.

You can define sets of identifiers within targeting rules to eliminate the need for multiple deployments. To learn more, read Targeting with flags.

Some possibilities for targeting rules include:

• Only use the new code when a “test mode” parameter is provided.
• Only use the new code on servers in set A.
• Consumers in set A get the old code, consumers in set B get the new code. Otherwise fall back to a default.
• Only 1% of requests with given criteria use the new code. Later, set a schedule to gradually increase that percentage.
• There are multiple active versions of consumer C, so only use the new code if the semantic version number is greater than 2.5.2.
• All requests get the new code, except those where parameter P matches one a regular expression.
• If the “debug” parameter is set, or if consumers X and Y experience unintended behavior, set the log level to DEBUG.
• If the monitoring system hits an error threshold, use flag triggers turn off all targeting and switch back to the old code momentarily. To learn more, read Flag triggers.

All of these rules are definable through either the LaunchDarkly user interface (UI) or our REST API. None of them require any code change.

You can change these rules whenever you like, and your code will pick up the changes in milliseconds. If you realize there’s a problem and need to undo a change, you can quickly toggle the flag off with no deployment or restart required.

Migrating between databases with feature flags

The example below focuses on a database migration, but you can use a similar strategy for other infrastructure migrations as well.

For this example, let's start with a data model that is never updated, where events are written and later read, but never changed. Later in the guide, we'll discuss how to handle the more complicated data model with ongoing event updates.

In the next sections, we refer to two databases, Database A and Database B, or dbA and dbB. dbA is the original database from which you will migrate. dbB is the destination database to which you will migrate your data. It receives data from dbA.

Here is a diagram representing migrating databases with feature flags:

Preparing your Data Access Object (DAO)

You read and write events in dbA with Data Access Object (or DAO) code.

First, you must write a new DAO that uses the same interface as dbA, but can also read and write to dbB. If there are differences in the way you need to query the data in the new data store, you must address those requirements now.

Feature flagging your reads and writes

Now, you have two implementations of your DAO interface, one backed by dbA and the other by dbB.

Use a set of four feature flags to control reading and writing to each database independently (referred to below as events-dbA-read, events-dbB-read, events-dbA-write, and events-dbB-write).

For example, the function below will save an event to the appropriate database depending on the value of your feature flags (events-dbA-write and events-dbB-write):

func storeEvent(evt Event, account ld.User) {
  if ldclient.BoolVariation('events-dbA-write', account, true) {
   DBAEventsDao.createEvent(evt)
  }
  if ldclient.BoolVariation('events-dbB-write', account, true) {
   DBBEventsDao.createEvent(evt)
  }
}
COPY

If you follow the code above precisely, you can save the same event in both databases. This is deliberate. It lets us write events to the new data store while maintaining the integrity of the old store.

If you need to cancel the migration or roll back from dbB to dbA, we can do so with confidence because both databases have all of the events stored.

The code that reads the events is more complicated. Here, the strategy is to read from both data stores at the same time, compare the results, then return the results from dbA. If the results are different between the data stores, then we log it.

Here is an example:

func findEventById(id string, account ld.User) Event {
  // Check both 'read' flags
  shouldReadDBA := ldclient.BoolVariation('events-dbA-read', account, true)
  shouldReadDBB := ldclient.BoolVariation('events-dbB-read', account, true)
  if shouldReadDBA && shouldReadDBB { // compare results
    dbAEvt := DBAEventsDao.findEventById(id)
    dbBEvt := DBBEventsDao.findEventById(id)
    // In this example, we are just performing a simple deep-equality check.
    // In some more complex cases, you might want to perform some other
    // integrity check to compare the two results, and ensure they
    // are as you expect.
    if !reflect.DeepEqual(dbAEvt, dbBEvt) {
      logger.Error.Printf(
        "dbA and dbB events differ: dbA: %+v, dbB: %+v",
        dbAEvt,
        dbBEvt)
    }
    return dbAEvt
  } else if shouldReadDBB { // read from just dbB
    return DBBEventsDao.findEventById(id)
  } else { // read from just dbA
    return DBAEventsDao.findEventById(id)
  }
}
COPY

This implementation has a bias toward the incumbent data store, which in this example is dbA. It returns the value read from dbA when it reads from both stores, and it will also read from dbA any time the dbB flag is false.

Rolling it out

Now that you’ve wrapped your data access in feature flags, you can plan the rollout.

You can leave the read flag (events-dbB-read) turned off for everyone, and progressively roll out the write flag (events-dbB-write) to your user base. Watch performance metrics and error logs. If things perform as expected and you've verified the data is written correctly in dbB, roll the flag out to more contexts, until all events are being written to both databases.

When you are happy with the write performance and have confirmed that the data in dbB is correct, you can plan to migrate the missing data that's not yet in dbB. This data was written before the events-dbB-write flag was turned on. You can do this as a one-time copy with a script, ignoring any duplicate entries that came in after you enabled writes to dbB. Make sure to do a final check on the resulting data in dbB.

After your data is copied to dbB, you can start turning on reads to the new data store. Turn on reads for a small segment of your contexts and monitor the performance metrics and error logs. Specifically, pay attention for a “dbA and dbB events differ” message. That message does not indicate that something is wrong in production. Even if you receive that message, it hasn’t affected anyone else. They are still using the data from your old data store. If you receive that error message, you can fix any issues without impacting your end users and move forward.

To learn more about rollouts, read Percentage rollouts.

After you're comfortable with the performance metrics and error counts you're receiving with reads and writes, you can now turn off the read and write flags for dbA, remove all references to all four flags in your code, and you are left with just dbB.

Here is a diagram representing an example migration strategy:

Variations to consider

Depending on the complexity of your current data store and your volume of traffic, you may want to break your migration into even more steps. Here are some additional variations to consider:

• Create flags for critical processes. Rather than wrapping all of your reads and writes in just two flags, create additional pairs of read/write flags for specific, critical processes in your application. You can roll these out separately to further mitigate risk.
• Create flags for processes that are changing. Sometimes an infrastructure change necessitates a functional change. For example, if you are migrating between two different kinds of databases, you may have to rework some of your queries substantially. Wrapping these changes in a separate flag makes it easier to evaluate their performance and correctness independently of your other updates.
• Use flags for tuning. In addition to flags that control reading and writing to each database, you can add flags within your DAO to help you tune the implementation for your new system. For example, if you are moving to a highly distributed database, you might use an integer flag type to tune the sizes you use for bulk inserts. To learn more about multivariate flags, read Creating flag variations.

Migrating between schemas with feature flags

More frequently, you may find that you need to make changes to your database as your data requirements change. Some reasons you might need to migrate schemas include:

• storing new kinds of information
• adding a column to a table
• changing a primary key
• removing elements
• splitting fields

If your database has frequent read and write actions, it may make more sense to treat the migration as a database migration. You can create a secondary database containing the new schema, then migrate using the methods discussed in Migrating between databases with feature flags.

Less busy databases don't require the extra step. Instead, you can make the changes in place. Feature flags allow you to deliberately and carefully manage your changes within the database, greatly reducing the risk of data disruption. You don't have to make the changeover all at once. You can pause the migration as many times as you need to assess your progress and change direction if necessary.

Adding a new column to your database

Imagine you have been collecting five-digit ZIP codes as part of your client's addresses. You now want to begin collecting the longer ZIP+4 codes, putting the additional four digits in a new column. You need to add a new column to your database and instruct your application to begin reading from and writing to that column.

Here's how to approach the migration:

1. Plan feature flags for your code changes: Add your feature flags to the affected elements of your codebase, not your database. Wrap the read action and the write action in separate flags.
2. Update your application code to collect and retrieve ZIP+4 address information: When migrating schema, it's important to separate your database changes from your codebase changes. Begin by making changes to your code that reference the database change, and wrap the code changes in feature flags.
3. Make changes to your database: Add the ZIP+4 column to your database. For the moment, the column will sit empty. Your feature flags allowing the read and write of ZIP+4 data are still off.
4. Toggle on the write action feature flag: Your application begins populating the new ZIP+4 fields. You may want to retroactively fill in the missing ZIP+4 fields for addresses already in the database, or you may just want to collect the data for new addresses.
5. Toggle on the read action feature flag: How long you wait to toggle on this flag depends on your needs and processes. It may make sense to begin using the field right away, or you may want to wait until you have a critical mass of addresses with the ZIP+4 data before reading from the new column.
6. Remove your old code and feature flags: When you're confident the new schema is working, remove your old code and feature flags from your application. This prevents the build up of technical debt in your code.

Make the changes to your database backwards compatible, if possible. Maintaining backwards-compatibility makes it easier if you need to stop in the middle of your process and turn off a feature flag when something doesn't work as expected.

Conclusion

The techniques described in this guide are only some strategies for a fairly simple server-side based system. There are cases where these strategies may not be enough.

For example, your target data store may need to have a different schema, forcing your DAO to read and write that data differently. This is a likely scenario if you're moving from a document database to a relational database. In this case, your new DAO may require changes to the application that will also need to be flagged.

Additionally, if you have corresponding mobile and desktop apps that need updated code to access the new data store, you must plan those migrations as well, because client-side apps are often not version-consistent across user bases. This may require shipping new versions of your mobile and desktop apps and waiting for your customers to slowly upgrade to the new versions before attempting the migration.

There's no one-size-fits-all strategy for infrastructure migrations. This guide explores only a few strategies that can serve as a foundation to build on. Using feature flags for migration projects will increase your confidence, limit the risks, and allow you to make infrastructure migrations happen seamlessly.