Overview

Prompt Management provides a centralized registry for the prompts used by your LLM applications. Instead of hardcoding prompt templates in application code or configuration files, create, version, and update prompts through Agent Observability, then retrieve them at runtime.

Runtime retrieval is supported in Python through the ddtrace SDK. Prompt retrieval and Prompt Tracking are separate: LLMObs.get_prompt() can retrieve a managed prompt without enabling Agent Observability, but Agent Observability must be enabled to create LLM spans and associate prompt metadata with them.

After creating prompt versions, use Prompt Experimentation to compare them with an A/B test or deploy one progressively with a Guarded Rollout.

Prompt Management works alongside Prompt Tracking. When Agent Observability is enabled, managed prompts passed directly to supported, automatically instrumented LLM calls are associated with the resulting spans.

Prerequisites

• Python 3.9 or later.
• ddtrace>=4.13.0
• Your Datadog site and a Datadog API key. The API key is required for prompt retrieval even if traces are sent through the Datadog Agent.
• A Datadog application key with the llm_observability_read, feature_flag_config_read, and feature_flag_environment_config_read permissions to resolve prompts by environment. If you select an existing application key in Datadog, ensure that it has these permissions.
• To manage prompts through the API or Python SDK, the application key also requires the llm_observability_write and feature_flag_config_write permissions.

Install the SDK

Install or upgrade the latest ddtrace package in the Python environment used by your application:

pip install --upgrade ddtrace

Use a managed prompt in Python

Integrate Prompt Management with a coding agent

Integrate a managed prompt with a coding agent of your choice by pasting in the following prompt:

Follow the instructions at https://docs.datadoghq.com/llm_observability/instrument/agentic.md to integrate the Datadog managed prompt <PROMPT_ID> into this application for environment <DEPLOYMENT_ENVIRONMENT> and track its use in Agent Observability.

Prompt variables: <PROMPT_VARIABLES>

When configuring the environment, use the following values:

DD_SITE=
DD_ENV=<DEPLOYMENT_ENVIRONMENT>

Optionally, append selected Datadog credentials so the coding agent can configure and verify the integration in the same session:

Selected Datadog credentials:

DD_API_KEY=<DATADOG_API_KEY>
DD_APP_KEY=<DATADOG_APP_KEY>

Treat these values as secrets and handle them according to the linked guide. Do not repeat or expose them.

Note: Including the API and application keys in the prompt is optional and is not required for the coding agent to integrate Prompt Management. Include them only in a trusted coding-agent session.

After the integration is complete, run your application and trigger the modified LLM flow. Return to the prompt page to view usage; new prompt calls may take a minute to appear.

Configure prompt retrieval

Provide the Datadog site, credentials, and deployment environment through the configuration and secret-management workflow already used by your application. For example, use the application’s environment file, Docker Compose or Kubernetes configuration, deployment platform, or secret manager. At runtime, the following environment variables must be set before importing ddtrace:

export DD_SITE="<DATADOG_SITE>"
export DD_API_KEY="<DATADOG_API_KEY>"
export DD_APP_KEY="<DATADOG_APP_KEY>"
export DD_ENV="<DEPLOYMENT_ENVIRONMENT>"

DD_ENV selects the environment used to resolve the prompt version and must match an environment where the prompt is deployed.

Retrieve, format, and use a prompt

Preserve the prompt already used by your application as the fallback. The fallback keeps the application working if registry, environment-resolution, network, or server failures occur.

The following example retrieves and formats a chat prompt, then passes the formatted messages directly to OpenAI:

from ddtrace.llmobs import LLMObs
from openai import OpenAI

default_messages = [
    {"role": "system", "content": "You are a support agent for {{company}}."},
    {"role": "user", "content": "{{question}}"},
]

variables = {
    "company": "Acme Inc.",
    "question": "How do I reset my password?",
}

prompt = LLMObs.get_prompt(
    "customer-support-greeting",
    fallback=default_messages,
)
messages = prompt.format(**variables)

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
)

prompt.format() returns a string for a text prompt and a list of messages for a chat prompt. Pass the formatted value to the corresponding text or messages parameter of your LLM provider call.

If retrieval fails and no fallback is provided, get_prompt() raises a ValueError. A fallback does not replace authentication: DD_API_KEY is always required, and DD_APP_KEY is also required when DD_ENV is set.

Without prompt composition, combine prompts in application code or manage the final provider-facing prompt as a single prompt. To include one managed prompt in another, see Reuse prompts with composition. Prompt composition is in Preview.

Select a version

Without DD_ENV, get_prompt() retrieves the latest prompt version:

prompt = LLMObs.get_prompt("customer-support-greeting")

With DD_ENV, get_prompt() resolves the prompt version for that environment. This requires DD_APP_KEY with the read permissions listed in Prerequisites.

To retrieve an exact numeric version independently of DD_ENV, pass version:

prompt = LLMObs.get_prompt("customer-support-greeting", version=2)

The version argument takes precedence over environment resolution.

Track prompt usage

To associate a managed prompt with an LLM span, enable Agent Observability and run the application with automatic instrumentation through its existing execution workflow.

If the application receives its configuration before the Python process starts, use ddtrace-run. For example, the equivalent shell command is:

DD_SITE="<DATADOG_SITE>" \
DD_API_KEY="<DATADOG_API_KEY>" \
DD_APP_KEY="<DATADOG_APP_KEY>" \
DD_ENV="<DEPLOYMENT_ENVIRONMENT>" \
DD_SERVICE="<SERVICE_NAME>" \
DD_LLMOBS_ENABLED=1 \
ddtrace-run python app.py

If the application loads its configuration in Python, load the configuration first, then import ddtrace.auto before importing the LLM provider or other application modules:

from dotenv import load_dotenv

load_dotenv()

import ddtrace.auto

from ddtrace.llmobs import LLMObs
from openai import OpenAI

Run this setup with the application’s normal Python command, such as python app.py. Do not also use ddtrace-run; it initializes ddtrace before the application can load its configuration.

If the application does not send data through a Datadog Agent, also set DD_LLMOBS_AGENTLESS_ENABLED=1.

For a supported automatically instrumented provider, pass the value returned by prompt.format() directly to the provider call, as shown in Retrieve, format, and use a prompt. This automatically associates the managed prompt with the resulting span.

Copying, rebuilding, or converting the formatted value can discard its prompt-tracking metadata. For example, concatenating a managed system prompt with a user question creates a new string without that metadata. Use LLMObs.annotation_context() to associate the managed prompt with the resulting LLM span:

prompt = LLMObs.get_prompt(
    "customer-support-system-prompt",
    fallback="You are a helpful support agent writing for a {{audience}} audience.",
)
variables = {"audience": audience}
system_prompt = prompt.format(**variables)
combined_prompt = f"{system_prompt}\n\nUser question: {question}"

with LLMObs.annotation_context(
    prompt=prompt.to_annotation_dict(**variables),
):
    response = client.responses.create(
        model="gpt-4o",
        input=combined_prompt,
    )

Pass the same variables to to_annotation_dict() that you pass to format() so that the tracked prompt includes the values used for that call.

annotation_context() associates metadata with an LLM span created inside the context; it does not create the span. For providers that are not automatically instrumented, first manually instrument the LLM call to create an LLM span. An explicit annotation_context() takes precedence over automatic prompt tracking. See Prompt Tracking for more information.

Create and manage prompts

Create prompts and publish new versions in the Prompts UI, through the Python SDK, or through the API.

Create a prompt

Promote a tracked prompt

To promote a prompt already tracked in Agent Observability to a managed prompt, navigate to the Prompts page, open the prompt, and click Register. You can then update the prompt in the UI and retrieve it at runtime.

In the UI from scratch

Navigate to the Prompts page and click + New Prompt.

In the Prompt Editor:

1. Add one or more messages and assign each a role: System, User, or Assistant.
2. Use {{variable_name}} syntax in any message to add dynamic content.
3. Optional: If you have access to the message placeholders Preview, click Add Message Placeholder. See the Insert messages at runtime section.
4. Optional: Click Run to test the prompt with sample values.
5. Click Save Prompt to open the save dialog.

Structure the prompt so the user query and context are injected as variables:

In the save dialog:

Click Create Prompt to save the prompt to the registry.

Update, list, and delete prompts

In the UI

Open a prompt in the Prompts page to:

• Create a new version: Click Edit and update the messages in the Prompt Editor.
• Deploy a version to another environment: Select a version and update its Deployment environments.
• Delete a prompt: Select Delete from the prompt’s options menu. This removes the prompt and its version history from the registry.

Use the Python SDK

Use LLMObs.create_prompt() to create a prompt and deploy its first version to one or more environments. The env_ids values are Feature Flags environment IDs, which you can obtain from the List environments API:

from ddtrace.llmobs import LLMObs

chat_template = [
    {"role": "system", "content": "You are a support agent for {{company}}."},
    {"role": "user", "content": "{{question}}"},
]

created_prompt = LLMObs.create_prompt(
    "customer-support-greeting",
    chat_template,
    env_ids=["<FEATURE_FLAG_ENVIRONMENT_ID>"],
)

To publish and deploy another version, use LLMObs.create_prompt_version():

created_version = LLMObs.create_prompt_version(
    "customer-support-greeting",
    updated_chat_template,
    env_ids=["<FEATURE_FLAG_ENVIRONMENT_ID>"],
)

Treat prompt creation, versioning, and deployment as setup operations. Do not perform them during application startup or from a request path. At runtime, retrieve deployed prompts with LLMObs.get_prompt().

These methods require the API and application key permissions listed in Prerequisites.

Use LLMObs.list_prompts() and LLMObs.list_prompt_versions() to inspect managed prompts, LLMObs.update_prompt() and LLMObs.update_prompt_version() to update metadata or deployments, and LLMObs.delete_prompt() to delete a prompt and all of its versions.

Use the API

Use the Prompt Management API to create, retrieve, update, and delete prompts and prompt versions. See the Agent Observability API reference for endpoint schemas, request media types, and examples.

Insert messages at runtime

Message placeholders insert conversation history or tool interactions into a saved prompt at runtime. A text variable, such as {{question}}, replaces text inside a message. A message placeholder inserts a list of complete messages.

The prompt version stores the placeholder’s name and position, not the messages you pass at runtime. To move the history within the prompt, publish a new prompt version. You do not need to change application code.

Preview SDK access: Contact Datadog Support or your Customer Success Manager for the SDK version to use for your language.

Define the placeholder

In the Prompt Editor, click Add Message Placeholder and enter a name, such as history. Use the up and down arrow buttons to move the placeholder between messages. Use a different name from any text variable in the prompt. The editor shows the compatible SDK requirement before you save.

To create the same prompt with the Python SDK, add an item with "type": "placeholder" where the history belongs:

from ddtrace.llmobs import LLMObs

LLMObs.create_prompt(
    "support-assistant",
    [
        {"role": "system", "content": "You are a concise assistant for {{plan}} customers."},
        {"type": "placeholder", "name": "history"},
        {"role": "user", "content": "{{question}}"},
    ],
    env_ids=["<FEATURE_FLAG_ENVIRONMENT_ID>"],
)

The placeholder name, history, is the key your application uses to pass messages at runtime. It is not a message role. For setup requirements and environment IDs, see the Use the Python SDK section.

Supply runtime values

Retrieve the prompt, then pass the history list with the text variables. The SDK inserts the history messages in order at the placeholder’s position.

prompt = LLMObs.get_prompt("support-assistant")

variables = {
    "plan": "enterprise",
    "question": "Can I export the report?",
    "history": [
        {"role": "user", "content": "Where are reports located?"},
        {"role": "assistant", "content": "Under Analytics."},
    ],
}
messages = prompt.format(**variables)

prompt, err := llmobs.GetPrompt(ctx, "support-assistant")
if err != nil {
	return err
}

variables := map[string]any{
	"plan":     "enterprise",
	"question": "Can I export the report?",
	"history": []map[string]any{
		{"role": "user", "content": "Where are reports located?"},
		{"role": "assistant", "content": "Under Analytics."},
	},
}
rendered, err := prompt.Format(variables)
if err != nil {
	return err
}
messages := rendered.Messages

const prompt = await tracer.llmobs.getPrompt('support-assistant')

const variables = {
  plan: 'enterprise',
  question: 'Can I export the report?',
  history: [
    { role: 'user', content: 'Where are reports located?' },
    { role: 'assistant', content: 'Under Analytics.' }
  ]
}
const messages = prompt.format(variables)

The result contains four messages, with no placeholder item:

[
  {"role": "system", "content": "You are a concise assistant for enterprise customers."},
  {"role": "user", "content": "Where are reports located?"},
  {"role": "assistant", "content": "Under Analytics."},
  {"role": "user", "content": "Can I export the report?"}
]

Pass the formatted messages to your model provider. If an inserted message contains {{variable}} syntax, the SDK leaves it as literal text.

Use the same prompt tracking workflow as for other managed prompts. Prompt metadata preserves the placeholder definition rather than its runtime values; expanded messages follow the existing input-capture and privacy settings.

Include tool interactions

A placeholder can also insert tool calls and tool responses. Use your model provider’s message format. For example, in the OpenAI Chat Completions format, set each tool response’s tool_call_id to the id of its tool call:

variables["history"] = [
    {
        "role": "assistant",
        "tool_calls": [{
            "id": "call_1",
            "type": "function",
            "function": {"name": "get_plan", "arguments": "{}"},
        }],
    },
    {"role": "tool", "tool_call_id": "call_1", "content": "enterprise"},
]
messages = prompt.format(**variables)

The SDK inserts these messages unchanged. Your application executes the tool and supplies its response.

Message placeholder requirements and limits

• Pass a list for every placeholder. An empty list ([]) inserts no messages.
• Inserted messages can contain text, assistant tool calls, or tool responses. The SDK does not execute tools or validate provider-specific fields.
• If a prompt uses the same placeholder name more than once, each occurrence inserts the same list. Different placeholder names can receive different lists.
• Nested placeholders and multimodal content, such as images or audio, are not supported.

Version prompt configuration

Store settings alongside your prompt so you can update and roll back both as one version. Use configuration for:

• Model settings, such as model and temperature.
• Structured output schemas, such as response_format.
• Tool definitions, such as tools and tool_choice.

Configuration is a JSON object whose fields you define. Your application reads and applies these settings; Datadog does not automatically apply them to Playground runs or model calls. Do not store secrets in configuration.

Add configuration

1. On the Prompts page, click New Prompt and write your template.

2. Click Save Prompt. Enter a prompt ID, expand Configuration (optional), and add settings:

   {
     "model": "<MODEL_NAME>",
     "temperature": 0.2
   }
   

3. Replace <MODEL_NAME> with a model that supports these settings, then click Create prompt.

The editor requires a valid JSON object. Its example text is a placeholder, not a saved configuration.

Update configuration

1. Open a prompt version and select the Configuration tab.
2. Click Update configuration and edit the settings.
3. Click Save version. To inspect the diff before saving, click Review changes first.

This creates a version without overwriting the original. Use Compare to inspect configuration changes.

Deploy the version to an environment when it is ready. Applications retrieving that environment receive its selected template and configuration together. To roll back both, deploy an earlier version. Saving alone does not change the version an environment serves.

Use configuration in your application

Retrieve the prompt deployed to your application’s environment, then pass its configuration to your model client.

Preview SDK access: Contact Datadog Support or your Customer Success Manager for the SDK version to use for your language.

These examples use a prompt named summarizer with the configuration shown above.

from ddtrace.llmobs import LLMObs

prompt = LLMObs.get_prompt("summarizer")
config = prompt.config

model = config["model"]
temperature = config.get("temperature", 0.2)

const prompt = await tracer.llmobs.prompts.getPrompt('summarizer')
const config = prompt.config

const model = config.model
const temperature = config.temperature ?? 0.2

prompt, err := llmobs.GetPrompt(ctx, "summarizer")
if err != nil {
    return err
}
config := prompt.Config()

model := config["model"].(string)
temperature, ok := config["temperature"].(float64)
if !ok {
    temperature = 0.2
}

API authoring: You can also create prompts and versions with the Prompt Management API. Omitting config creates an empty configuration for a new prompt or inherits the latest configuration for a new version. Send {} to clear it.

Reuse prompts with composition

Prompt composition lets one prompt include another, so you can reuse shared instructions without copying them. For example, a support assistant and a billing assistant can include the same response policy. You can include another prompt in two ways:

• Chat messages: Include some or all messages from a chat prompt.
• Text: Insert a text prompt’s content inside a message.

Each include points to one exact version. Publishing a new version of the included prompt doesn’t change prompts that already include it.

Include chat messages

In the UI

The following example adds a shared response policy to a support assistant prompt.

1. Save a prompt with the ID response-policy and one System message: Answer concisely. If you do not know the answer, say so.
2. On the Prompts page, click New Prompt. In the Prompt Editor, click Include Prompt, select response-policy version 1, and click Add prompt.
3. After the included prompt, add a User message containing {{question}}. If the editor added empty messages, remove them.
4. Click Save, enter support-assistant-composed as the prompt ID, and click Create prompt.

Your prompt now contains:

System: Answer concisely. If you do not know the answer, say so.
User: {{question}}

Your application retrieves and formats the prompt as usual. The retrieved prompt already contains the included messages, so you don’t need to fetch response-policy separately.

By default, an include adds every message from the included prompt, in order. To include only some messages, reorder them, or repeat one, click Included Prompt in the Prompt Editor and select Customize messages. Customizing doesn’t change the included prompt.

With the API

Use an include object in template.messages to reference a specific version of a chat prompt. This example assumes a chat prompt response-policy with a version 1. To create a prompt that includes it, send this JSON body to POST /api/v2/llm-obs/v1/prompts:

{
  "data": {
    "type": "prompt-templates",
    "attributes": {
      "prompt_id": "support-assistant-composed",
      "template": {
        "messages": [
          { "include": { "prompt_id": "response-policy", "version": 1 } },
          { "role": "user", "content": "{{question}}" }
        ]
      }
    }
  }
}

For authentication and message-selection options, see Create an Agent Observability prompt.

Include text in a message

To reuse a phrase instead of complete messages, click Include Prompt, select a text prompt and version, and click Insert text. The reference is added to the end of the last editable message. Move it where you need it.

For example, if response-style version 1 contains Answer concisely., write:

{{>response-style version=1}} Answer {{question}}.

The resolved template is:

Answer concisely. Answer {{question}}.

The reference resolves to the included text exactly, with no added spaces or line breaks. Always specify a version. Without one, {{>response-style}} stays literal text, not an include. When creating a prompt without Preview access, inline references remain literal text. Previously saved composed versions remain usable.

In API requests, use the same syntax in a text template or a chat message’s content.

Review and update includes

On a saved version, Prompt Template shows the references you authored. Resolved Prompt shows the expanded messages, before runtime variables are filled in.

When a shared policy changes, use its Used By tab to find prompts that reference it. Open a consuming prompt, replace the include with the new source version, then test, save, and deploy the updated prompt. Existing versions keep their original content, even if the source is later deleted.

Advanced usage

Serve multiple versions from one environment

Prompt Management builds on Datadog’s Feature Flags product. Each environment resolves get_prompt() calls to a default version, and can also serve a different version to calls that match a targeting rule.

For example, roll out an unstable prompt version to a subset of users in production with a targeting rule, while everyone else keeps getting the stable version:

## DD_ENV=production
prompt = LLMObs.get_prompt("my-prompt")               # resolves to the stable version
prompt = LLMObs.get_prompt("my-prompt", tag="unstable") # resolves to the unstable version

To configure this:

1. On the prompt’s version list, hover over an environment and click Targeting Rules.

2. Click Add Targeting Rule.

3. Define the rule filter. For example, match calls to get_prompt() that pass the attribute tag=unstable, and set the resulting variant to the unstable prompt version.

4. Save the rule. Calls with tag=unstable resolve to the matched version; all other calls fall back to the default version.

Pass the attributes referenced by your targeting rules as keyword arguments to get_prompt(). Calls that don’t pass a matching attribute continue to resolve to the environment’s default version.

To retrieve an exact version regardless of any targeting rule, pass version as described in Select a version.

Further reading

Additional helpful documentation, links, and articles:

Prompt TrackingDOCUMENTATION PlaygroundDOCUMENTATION Agent Observability SDKDOCUMENTATION