Overview

Prompt Tracking links prompt templates and versions to LLM calls. Prompt Tracking works alongside Agent Observability’s traces, spans, and Playground, as well as Prompt Management, which provides a centralized registry for creating and versioning prompts.

Prompt Tracking enables you to:

• See all prompts used by your LLM application or agent, with call volume and latency over time
• Compare prompts or versions by calls, latency, tokens used, and cost
• See detailed information about a prompt: review its version history, view a text diff, and jump to traces using a specific version
• Filter Trace Explorer by prompt name, ID, or version to isolate impacted requests
• Reproduce a run by populating Agent Observability Playground with the exact template and variables from any span

Set up Prompt Tracking

When Agent Observability is enabled, prompts fetched from the Prompt Management registry with LLMObs.get_prompt() are tracked automatically if the value returned by prompt.format() is passed directly to a supported, automatically instrumented LLM call. If the formatted value is copied or transformed, use LLMObs.annotation_context() as described in the Prompt Management documentation. The following setup options apply to prompts defined outside the registry.

With structured prompt metadata

To use Prompt Tracking, you can submit structured prompt metadata (ID, optional version, template, variables).

Agent Observability Python SDK

If you are using the Agent Observability Python SDK (dd-trace v3.16.0+), attach prompt metadata to the LLM span using the prompt argument or helper. See the Agent Observability Python SDK documentation.

Agent Observability Node.js SDK

If you are using the Agent Observability Node.js SDK (dd-trace v5.83.0+), attach prompt metadata to the LLM span using the prompt option. See the Agent Observability Node.js SDK documentation.

Agent Observability API

If you are using the Agent Observability API intake, submit prompt metadata to the Spans API endpoint. See the Agent Observability HTTP API reference documentation.

OpenTelemetry instrumentation

If you are using OpenTelemetry instrumentation, you can attach prompt metadata to your LLM spans by setting the _dd.ml_obs.prompt_tracking attribute with a JSON string containing your prompt information.

Set the attribute on any LLM span:

import json

span.set_attribute("_dd.ml_obs.prompt_tracking", json.dumps({
    "name": "greeting-prompt",
    "version": "v1",
    "template": "Hello {{name}}, tell me about {{topic}}",
    "variables": {"name": "Alice", "topic": "weather"}
}))

span.setAttribute("_dd.ml_obs.prompt_tracking", JSON.stringify({
    name: "greeting-prompt",
    version: "v1",
    template: "Hello {{name}}, tell me about {{topic}}",
    variables: { name: "Alice", topic: "weather" }
}));

span.SetAttributes(attribute.String("_dd.ml_obs.prompt_tracking",
    `{"name":"greeting-prompt","version":"v1","template":"Hello {{name}}, tell me about {{topic}}","variables":{"name":"Alice","topic":"weather"}}`,
))

The following fields are supported in the prompt tracking JSON:

With LangChain templates

If you are using LangChain prompt templates, Datadog automatically captures prompt metadata without code changes. IDs are derived from module or template names. To override these IDs, see Agent Observability Auto-instrumentation: LangChain.

Use Prompt Tracking in Agent Observability

View your app in Agent Observability and select Prompts on the left. The Prompts view features the following information:

• Prompt Call Count: A timeseries chart displaying calls per prompt (or per version) over time
• Recent Prompt Updates: Information about recent prompt updates, including time of last update, call count, average latency, and average tokens per call
• Most Tokens Used: Prompts ranked by total (input or output) tokens
• Highest Latency Prompts: Prompts ranked by average duration

Click on a prompt to open a detailed side-panel view that features information about version activity and various metrics. You can also see a diff view of two versions, open Trace Explorer pre-filtered to spans that use a selected version, or start a Playground session pre-populated with the selected version’s template and variables.

You can use the Agent Observability Trace Explorer to locate requests by prompt usage. You can use a prompt’s name, ID, and version as facets for both trace-level and span-level search. Click any LLM span to see the prompt that generated it.

Further Reading

Documentation, liens et articles supplémentaires utiles:

Track, compare, and optimize your LLM prompts with Datadog LLM ObservabilityBLOG Investigate with LLM ObservabilityLEARNING CENTER