---
title: Agent Tracking
description: >-
  Use Agent Tracking to monitor your agents, their versions, and what each
  version costs.
breadcrumbs: >-
  Docs > Agent Observability > Agent Observability Instrumentation > Agent
  Tracking
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Agent Tracking

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

## Overview{% #overview %}

Agent Tracking groups the spans in your application by the agent that produced them, so you can monitor each agent, compare its versions, and see how its configuration changed between versions.

Agent Tracking enables you to:

- See all agents in your application, with call volume, latency, tokens, estimated cost, and error rate
- Find your most expensive and slowest agents
- Break down an agent's activity by version
- Compare two versions side by side, including a diff of their instructions, tools, and model
- Jump to the spans for an agent or a specific version

{% image
   source="https://docs.dd-static.net/images/llm_observability/monitoring/agent-tracking-hero.a4509bbf53b3d2730af5480d281ee63e.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/monitoring/agent-tracking-hero.a4509bbf53b3d2730af5480d281ee63e.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Agents section of an app's Overview in Agent Observability, showing Agent Usage, Recent agents, Most Expensive Agents, and Highest Latency Agents." /%}

Agent Tracking is built on three pieces of data:

| Data              | What it does                                                                        | How it is set                            |
| ----------------- | ----------------------------------------------------------------------------------- | ---------------------------------------- |
| Agent attribution | Links each LLM, tool, and embedding span to the agent responsible for it            | Automatic                                |
| Agent version     | Identifies which version of an agent produced a trace                               | Manual                                   |
| Agent manifest    | Describes the agent's configuration: instructions, tools, model, and model settings | Automatic for some frameworks, or manual |

## Set up Agent Tracking{% #set-up-agent-tracking %}

### Agent attribution{% #agent-attribution %}

Agent attribution requires no code changes. When a span starts, the SDK finds its nearest `agent` ancestor and records that agent's name and span ID on the span under `meta.agent_attribution`. Attribution propagates across services through distributed tracing headers.

Spans with no `agent` ancestor are not attributed.

#### Supported SDK versions{% #supported-sdk-versions %}

| SDK     | Minimum version   |
| ------- | ----------------- |
| Python  | 4.13.0            |
| Node.js | 5.121.0 or 6.10.0 |
| Java    | 1.66.0            |

### Agent version{% #agent-version %}

Set a version on your agent span. The version is set as an `agent_version` tag on the agent span and on the spans started under it in the same process. Agent Tracking uses the tag on the agent's LLM and embedding spans to break down tokens and cost by version.

{% tab title="Python" %}
Requires `ddtrace` v4.14.0+.

**Note**: On versions before v4.17.0, child spans are not tagged automatically. Tag them with `LLMObs.annotation_context(tags={"agent_version": "v2"})` to break down tokens and cost by version.

```python
from ddtrace.llmobs import LLMObs

with LLMObs.agent(name="travel_planner", version="v2"):
    ...
```

You can also pass `agent={"version": "v2"}` to `LLMObs.annotate()` or `LLMObs.annotation_context()`.
{% /tab %}

{% tab title="Node.js" %}
Requires `dd-trace` v5.134.0+ or v6.23.0+.

```javascript
const { llmobs } = require('dd-trace')

llmobs.trace({ kind: 'agent', name: 'travel_planner', version: 'v2' }, () => {
  // ...
})
```

The version must not contain a comma. You can also declare it with `llmobs.annotate({ agent: { version: 'v2' } })` or `llmobs.annotationContext({ agent: { version: 'v2' } }, fn)`.
{% /tab %}

{% tab title="Java" %}
Requires `dd-trace-java` v1.66.0+.

```java
LLMObsSpan agentSpan = LLMObs.startAgentSpan("travel_planner", null, null, "v2");
```

The version propagates only when passed to `startAgentSpan`. Setting the `agent_version` tag afterward with `setTag` only tags that span.
{% /tab %}

### Agent manifest{% #agent-manifest %}

The agent manifest describes how an agent is configured. Agent Tracking uses it to show what changed between versions.

#### Automatic{% #automatic %}

The Python SDK reports a manifest automatically for agents built with supported frameworks. Each integration reports the manifest fields that its framework makes available, so the fields can differ by framework. Supported frameworks include PydanticAI, OpenAI Agents SDK, CrewAI, LangGraph, Google ADK, and Claude Agent SDK.

#### Manual{% #manual %}

{% tab title="Python" %}
Requires `ddtrace` v4.16.0+. Pass an `agent` dictionary to `LLMObs.annotate()` on an agent span, or to `LLMObs.annotation_context()` to apply it to every agent span in a block:

```python
from ddtrace.llmobs import LLMObs

with LLMObs.agent(name="travel_planner") as span:
    LLMObs.annotate(
        span=span,
        agent={
            "version": "v2",
            "instructions": "You help users plan trips.",
            "model": "gpt-4o",
            "model_settings": {"temperature": 0.2},
            "tools": [
                {
                    "name": "search_flights",
                    "description": "Search for flights between two cities.",
                    "parameters": {"origin": {"type": "string", "required": True}},
                },
            ],
        },
    )
```

{% /tab %}

{% tab title="Node.js" %}
Requires `dd-trace` v5.132.0+ or v6.21.0+.

```javascript
llmobs.trace({ kind: 'agent', name: 'travel_planner' }, () => {
  llmobs.annotate({
    agent: {
      version: 'v2',
      instructions: 'You help users plan trips.',
      model: 'gpt-4o',
      modelSettings: { temperature: 0.2 },
      tools: [{ name: 'search_flights', description: 'Search for flights between two cities.' }]
    }
  })
})
```

{% /tab %}

{% tab title="Java" %}
Requires `dd-trace-java` v1.67.0+.

```java
agentSpan.annotateAgentManifest(
    LLMObs.AgentManifest.builder()
        .instructions("You help users plan trips.")
        .model("gpt-4o")
        .tools(List.of(LLMObs.AgentTool.from("search_flights")))
        .build());
```

{% /tab %}

All manifest fields are optional. Keys that are not in the following table are dropped, and each later annotation updates only the fields that it sets.

| Field            | Description                                                                                                       |
| ---------------- | ----------------------------------------------------------------------------------------------------------------- |
| `name`           | The agent's name. Defaults to the agent span's name.                                                              |
| `instructions`   | The system instructions the agent runs with.                                                                      |
| `model`          | The model the agent is configured to call.                                                                        |
| `model_settings` | Inference parameters such as `temperature`, `max_tokens`, and `top_p`. Provider-specific keys are dropped.        |
| `tools`          | The tools the agent can call. Each tool requires a `name` and accepts an optional `description` and `parameters`. |

{% image
   source="https://docs.dd-static.net/images/llm_observability/monitoring/agent-version-manifest.6a2cf1694feab0ed84d01c011793a541.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/monitoring/agent-version-manifest.6a2cf1694feab0ed84d01c011793a541.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="A single agent version in the agent side panel, showing its tools along with usage, cost, and latency charts." /%}

## Use Agent Tracking in Agent Observability{% #use-agent-tracking-in-agent-observability %}

View your app in Agent Observability and open the Overview. The Agents section contains:

- Agent Usage: A timeseries chart displaying calls per agent over time
- Recent agents: Each agent's last seen time, number of versions, p95 latency, tokens, estimated cost, span count, and error rate
- Most Expensive Agents: Agents ranked by estimated cost
- Highest Latency Agents: Agents ranked by p95 duration

Tokens and cost are summed from the LLM and embedding spans attributed to each agent.

Click an agent to open a side panel with metrics broken down by version. Select a version to see its instructions and manifest.

{% image
   source="https://docs.dd-static.net/images/llm_observability/monitoring/agent-version-details.2d0b130c07313247acb392ae7ef02239.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/monitoring/agent-version-details.2d0b130c07313247acb392ae7ef02239.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Agent side panel showing usage by version and a table of versions with spans, tokens, cost, latency, and error rate." /%}

Click Compare to diff two versions side by side, including their instructions, tools, and model.

{% image
   source="https://docs.dd-static.net/images/llm_observability/monitoring/agent-version-compare.9b9b091e3dba08ed6d568bbc55161a94.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/monitoring/agent-version-compare.9b9b091e3dba08ed6d568bbc55161a94.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="Two agent versions compared side by side, with a diff of their instructions and usage by version." /%}

Click View Spans to open [Trace Explorer](https://app.datadoghq.com/llm/traces) filtered to that agent or version.

## Agent Tracking limitations{% #agent-tracking-limitations %}

- **Nearest agent only**: A span is attributed to its nearest agent ancestor. When one agent calls another, the LLM calls made by the inner agent count toward the inner agent only, not the outer one.
- **Agents are identified by name**: Two agents with the same name are combined, and renaming an agent starts a new history.
- **Version propagation is in-process**: `agent_version` does not propagate across services.
- **Version comparison**: The diff covers instructions, tools, and model. Other manifest fields are displayed but not compared.

## Further Reading{% #further-reading %}

Additional helpful documentation, links, and articles:

- [Prompt Tracking](https://docs.datadoghq.com/llm_observability/instrument/prompt_tracking.md)
- [Agent Monitoring](https://docs.datadoghq.com/llm_observability/guide/agent_monitoring.md)
