AI model flags

Overview

This topic explains how you can use artificial intelligence (AI) model flags to roll out and test new large language models (LLMs) within your generative AI applications. You can use AI model flags to deliver new AI models and configurations for large language models.

About AI model flags

AI model flags, sometimes called LLM flags, are JSON flags. Each flag variation contains the following:

• A modelID field, which identifies the LLM in use. Over time LaunchDarkly will introduce new models to adjust to different use cases, training data parameters, vendors, and more.
• One or more configuration parameters, which are specific to the LLM. For example, many LLMs include parameters for temperature, max_token, top_p, or top_k. Each LLM provider uses different parameters.

Create AI model flags

To create an AI model flag:

1. Navigate to the Flags list.
2. Click Create flag. The "Create new flag" page appears.
3. Enter a unique, human-readable Name.
4. (Optional) Update the flag Key. You'll use this key to reference the flag in your code. A suggested key auto-populates from the name you enter, but you can customize it if you wish.
5. (Optional) Enter a Description of the flag. A brief, human-readable description helps your account members understand what the flag is for.
6. (Optional) Update the Maintainer for the flag.
7. (Optional) Check the Include flag in this project's release pipeline box. To learn more, read Release pipelines.
8. Choose the AI model template in the Configuration section:

9. Choose Yes or No to indicate whether this flag is temporary. AI model flags are usually permanent.
10. Select the JSON flag type.
11. In the "Variations" section, review each variation and update as needed. Each variation identifies the LLM, in the modelID field, and its configuration parameters.
12. (Optional) Update the default variations.
13. Choose one or more tags from the Tags menu.
14. Check the SDKs using Mobile Key and/or SDKs using client-side ID boxes to indicate which client-side SDKs you will use to evaluate this flag. If you are using a server-side SDK, leave these boxes unchecked.
15. Click Create flag.

To learn more, read Creating new flags.

Common use cases

In the most common use case, your application includes several AI models within its server-side API configuration. Your application uses a LaunchDarkly SDK to evaluate the AI model flag, and parses the flag variation returned. Then, it uses the LLM and configuration parameters from that flag variation as part of an API request to a specific LLM that is part of your application.

Applications working with AI models typically include a default model, and then have different variations of that model using different model types. For example, you might use an existing Anthropic Claude model for v1 instant, as well as a model variation that supports Claude 3 Haiku, and another variation that supports Claude 3 Sonnet. Each model variation corresponds to a flag variation in your AI model flag. Your application might use all model variations for testing purposes, to determine which one is most viable based on data and cost.

Organizations introduce new AI models frequently, so we expect that you will create new flag variations for your AI model flag to reflect this. For example, you may add new flag variations for additional versions of an existing configuration. You may also add new flag variations to introduce new models from different vendors or upgraded models from your existing vendor.

We recommend that you use experiments to measure the effectiveness of these models, alongside metrics such as user sentiment, token count, or content conversion. To learn more, read Experimentation.

Modify the AI model flag template

To modify your AI model flag template:

1. Click the project dropdown. The project menu appears:

2. Select Project settings.
3. Select Flags. The Flags settings list appears.
4. In the "Flag templates section, click pencil icon next to "AI model."
5. Choose Yes or No to mark all AI model flags as temporary or permanent.
6. In the "Variations" section, edit the flag variation Names.
7. In the "Default variations" section, choose which default variations to serve when the flag is toggled On or Off.
8. In the "Tags" section, type to add tags to each new flag by default.
9. Click Save.

Verify that the updated default template settings are correct by navigating to the Flags list and creating a new AI model flag.