Quickstart for AI configs

Overview

This topic explains how to get started with the LaunchDarkly AI configs product.

You can use AI configs to customize, test, and roll out new large language models (LLMs) within your generative AI applications. AI configs may be right for you if you want to:

• manage your model configuration and messages outside of your application code
• upgrade your app to the newest model version, then gradually release to your customers
• start using a new model provider, and progressively move your production traffic to the new provider
• compare the performance of different messages and models

With AI configs, both the messages and the model evaluation are specific to each end user, at runtime. You can update your messages, specific to each end user, without redeploying your application.

Working with AI configs is available to members with an Admin role, Owner role, or custom role that allows AI config actions.

Follow the steps below to incorporate AI configs into your app. If you'd prefer to learn from an example that's built on a specific generative AI application, read our guide on Managing AI model configuration outside of code with the Node.js AI SDK.

Step 1, in your app: Install an AI SDK

First, install one of the LaunchDarkly AI SDKs in your app:

Next, import the LaunchDarkly AI client in your app and initialize a single, shared instance of it:

The AI SDKs each require that you specify your SDK key to authorize your application to connect to a particular environment within LaunchDarkly. SDK keys are specific to each project and environment. They are available from the Environments list for each project in the LaunchDarkly UI.

Then, set up the context. Contexts are the people or resources who will encounter generated AI content in your application. The context attributes determine which variation of the AI config LaunchDarkly serves to the end user, based on the targeting rules in your AI config. If you are using template variables in the messages in your AI config's variations, the context attributes also fill in values for the template variables.

Here's how:

Step 2, in LaunchDarkly: Create an AI config

Next, create an AI config in LaunchDarkly:

1. Click Create and choose AI config.
2. In the "Create AI config" dialog, give your AI config a human-readable Name.
3. Click Create.

The empty Variations tab of your new AI config displays:

Then, create a variation. Every AI config has one or more variations, each of which includes your AI messages and model configuration.

Here's how:

1. In the create panel in the Variations tab, replace "Untitled variation" with a variation Name. You'll use this to refer to the variations when you set up targeting rules, below.
2. Click Select a model and choose the model to use. LaunchDarkly provides a list of common models.
3. Select a message role and enter the message for the variation. If you'd like to customize the message at runtime, use {{ example_variable }} or {{ ldctx.example_context_attribute }} within the message. The LaunchDarkly AI SDK will substitute the correct values when you customize the AI config from within your app.

• To learn more about how variables and context attributes are inserted into messages at runtime, read Customizing AI configs.

4. Click Save changes.

You can also select Import from playground in your variation to bring your model, model parameters, and messages from an external playground.

Here's an example of a completed variation:

Step 3, in LaunchDarkly: Set up targeting rules

Next, set up targeting rules for your AI config. These rules determine which of your customers receive a particular variation of your AI config.

To specify the AI config variation to use by default when the AI config is toggled on:

1. Select the Targeting tab for your AI config.
2. In the "Default rule" section, click Edit.
3. Configure the default rule to serve a specific variation.
4. Click Review and save.

Here is an example of a default rule:

At this point, the AI config is not yet toggled on. We recommend doing that after you've added code to your application to pull the customized messages and model configuration from LaunchDarkly. If you'd like to turn it on now, skip ahead to Step 5.

When you're ready to add more AI config variations, you can come back to this step and set up additional targeting rules. If you are familiar with LaunchDarkly's flag targeting, the process is very similar: with AI configs, you can target individuals or segments, or target contexts with custom rules. To learn how, read Target with AI configs.

Step 4, in your app: Customize the AI config, track metrics

In your code, use the config function from the LaunchDarkly AI SDK to customize the AI config. You need to call config each time you generate content from your AI model.

The config function returns the customized messages and model configuration along with a tracker instance for recording metrics. Customization means that any variables you include in the AI config variation's messages have their values set to the context attributes and variables you pass to the config function. Then, you can pass the customized messages directly to your AI. You should also set up a fallback value to use in case of an error.

Here's how:

Then, use one of the track[Model]Metrics or TrackRequest functions to record metrics from your AI model generation. LaunchDarkly provides specific functions for completions for several common AI model families, and an option to record this information yourself. Remember that you need to call config each time you generate content from your AI model.

Here's how:

Step 5, in LaunchDarkly: Enable the AI config

Now that your application is set up, return to the LaunchDarkly user interface and enable your AI config. On the Targeting tab, toggle the AI config On. Then click Review and save.

When an end user opens your application, they'll get the AI config variation you've defined, either in the default rule or in a custom targeting rule. Your app's AI content generation will use the model and messages from the AI config variation, and the messages will be customized for each end user.

Step 6, in LaunchDarkly: Monitor the AI config

Select the Monitoring tab for your AI config. As end users use your application, LaunchDarkly monitors the performance of your AI configs. Metrics are updated approximately every minute.

Learn more about AI configs

The following sections provide answers to common questions about working with AI configs.

Integration with AI providers

In the AI configs product, LaunchDarkly is not handling the integration to the AI provider. The LaunchDarkly AI SDKs provide your application with model configuration details, including customized messages, and model parameters such as temperature and tokens. It is your application's responsibility to pass this information to the AI provider.

The LaunchDarkly AI SDKs provide methods to help you track how your AI model generation is performing, and in some cases, these methods take the completion from common AI providers as a parameter. However, it is still your application's responsibility to call the AI provider. To learn more, read Tracking AI metrics.

Privacy and personally identifiable information (PII)

LaunchDarkly does not send any of the information you provide to any models, and does not use any of the information to fine tune any models.

You should follow your own organization's policies regarding if or when it may be acceptable to send end-user data either to LaunchDarkly or to an AI provider. To learn more, read AI configs and information privacy.

Availability of new models

When you create a new AI config variation, you must select a model from the provided list. LaunchDarkly updates this list regularly. To request a new model, click the Give feedback option and let us know what models you'd like to have included.

What's next

We're continuing to build out the AI configs product. As the AI configs early access expands, we're committed to adding the following improvements:

• Additional AI SDKs
• Support for approvals, so that account members have the option to require approval request for changes to an AI configs variations or targeting
• Support for viewing differences between AI config variations