Overview

Datadog provides a set of Claude Code skills that bring Agent Observability analysis directly into your development workflow. Rather than navigating dashboards manually, you can invoke these skills from a Claude Code session to classify sessions, diagnose failures, compare experiments, generate Python experiment code, and bootstrap evaluators — all against your live production data.

The skills produce structured, actionable output — RCA reports with before/after fix proposals, generated evaluator code, experiment comparisons — that you can pass directly to a coding agent to apply fixes to your application. When Claude Code has access to your codebase, it can search for the relevant system prompt, tool definitions, or routing logic and propose specific diffs without leaving the session.

Setup

Prerequisites

• Claude Code installed and authenticated
• At least one LLM application instrumented with Agent Observability and producing traces
• A data backend: either the Datadog MCP server or the pup CLI

Install the skills

The skills are published in the agent-skills repository. Install them with the following command:

npx skills add datadog-labs/agent-skills/agent-observability --full-depth -y

The skills are available in any Claude Code session after installing.

Datadog MCP server

To use the Datadog MCP server option, connect the Agent Observability MCP server to your Claude Code session:

claude mcp add --scope user --transport http datadog-llmo-mcp \
  '?toolsets=llmobs,core'

All skills detect the MCP server automatically at startup and use it throughout.

Option B: pup CLI

If you prefer not to use the MCP server, the skills also run over pup, Datadog’s internal CLI. Install pup and authenticate:

pup auth login

Each skill detects at startup whether the MCP server is available; if not, it checks for pup and switches to pup mode automatically. You can also force pup mode explicitly by passing --backend pup to any skill invocation.

In pup mode, all Datadog API calls are made through pup llm-obs subcommands instead of MCP tools. The output and workflow are identical.

Skills

Classify sessions and traces

/agent-observability-session-classify assesses whether a user’s intent was satisfied in a given interaction. It operates in three modes depending on what you provide:

The skill pulls from up to three signal sources, and accuracy improves the more data it has access to:

• Agent Observability traces — the full span tree, conversation content, tool call results, and eval judge verdicts. Always available.
• RUM behavioral signals — page views, custom actions, dwell time, and explicit feedback events that confirm or contradict what the trace shows. Available when RUM is instrumented for your app.
• Audit Trail — server-confirmed write events (dashboards created, monitors modified, notebooks deleted) that prove whether the assistant’s actions actually landed. Most authoritative signal when the session involved asset creation or editing.

The skill returns a compact yes / partial / no verdict with a one-sentence reason by default. Add verbose: true for a full markdown report.

Examples:

/agent-observability-session-classify session_id=abc-123
/agent-observability-session-classify trace_id=def-456
/agent-observability-session-classify ml_app=my-chatbot --timeframe now-7d

Diagnose failures with root cause analysis

/agent-observability-trace-rca walks the span tree of failing traces to identify why your LLM application is producing poor results. It selects the best analysis mode based on available signals: LLM-judge eval verdicts (strongest signal), runtime errors, or structural anomalies such as latency outliers and agent-loop decisions.

The skill samples failing spans, groups them into a failure taxonomy, and compiles a structured RCA report with root cause categories, supporting evidence, and concrete fix proposals. Each fix includes the actual text or code from the trace — system prompt excerpts, tool argument shapes, routing logic — with a BEFORE / AFTER showing exactly what to change.

When Claude Code has access to your codebase, the skill searches for the relevant source files and proposes diffs you can apply immediately. For system prompt deficiencies, tool misuse, or routing errors, this means going from diagnosis to a pull request without leaving the session.

Examples:

/agent-observability-trace-rca ml_app=my-chatbot
/agent-observability-trace-rca ml_app=my-chatbot eval_name=faithfulness --timeframe now-24h

Analyze and compare experiments

/agent-observability-experiment-analyzer retrieves experiment results and surfaces what changed between a candidate and a baseline. It works for a single experiment (exploratory analysis) or a pair (comparative analysis).

The skill highlights which metrics improved or regressed, which event categories shifted, and where the candidate underperformed — so you can make a confident promotion decision.

Examples

/agent-observability-experiment-analyzer experiment_id=exp-123
/agent-observability-experiment-analyzer experiment_id=exp-456 baseline_id=exp-123

Generate experiment code with the Python SDK

/agent-observability-experiment-py-bootstrap emits a self-contained .py script or Jupyter .ipynb notebook that uses the ddtrace.llmobs SDK and matches the canonical reference notebooks.

The dataset can be a local DatasetRecordRaw[] JSON (inlined into the file), a CSV (loaded at runtime via LLMObs.create_dataset_from_csv), an existing Datadog dataset by name (LLMObs.pull_dataset), or — by default — a small inline 3-record sample.

All flags below are optional. Invoke /agent-observability-experiment-py-bootstrap with no arguments and the skill prompts for what it needs and emits a runnable file against the default 3-record sample.

Examples

/agent-observability-experiment-py-bootstrap --purpose "validate output accuracy"
/agent-observability-experiment-py-bootstrap --purpose "test tool selection on ambiguous queries" --dataset ./data/qa.json
/agent-observability-experiment-py-bootstrap --dataset-name qa_v3 --project-name customer-qa
/agent-observability-experiment-py-bootstrap --task-source mymodule.handlers:respond --evaluator-style remote
/agent-observability-experiment-py-bootstrap --placeholder-task --format ipynb

Bootstrap evaluators from trace data

/agent-observability-eval-bootstrap analyzes production traces from an ml_app (or an RCA report already in context) and proposes a suite of evaluators that would catch the observed failure modes. It outputs one of four artifacts:

The first three modes share the same evaluator-proposal workflow and differ only in how the suite is materialized. The fourth mode (--emit-dataset) is independent — it samples root spans for the ml_app (filtered to @status:ok), extracts input_data and expected_output per record, runs a PII scrub on string values, and writes a JSON file you can publish to Datadog as a dataset and then run an experiment against. Per-record tags are auto-normalized (bare strings wrapped as tag:<value>) so Dataset.append() does not reject the record. The expected_output field is documented as the production behavior baseline, not ground truth — useful for regression-style experiments (does my refactor change observed outputs?) before being promoted to a labelled gold set.

Examples

/agent-observability-eval-bootstrap ml_app=my-chatbot
/agent-observability-eval-bootstrap ml_app=my-chatbot --publish
/agent-observability-eval-bootstrap ml_app=my-chatbot --data-only
/agent-observability-eval-bootstrap ml_app=my-chatbot --emit-dataset ./datasets/my_chatbot_seed.json --trace-limit 25

Run the end-to-end pipeline

/agent-observability-eval-pipeline chains the agent-observability sub-skills into a single supervised, narrated workflow that walks from production traces through evaluators, datasets, experiments, and analysis. Every phase has the same envelope — a banner that names the entity being produced, a pedagogy block explaining its purpose, the action (a sub-skill call or a small executable step), and a checkpoint that waits for your confirmation. It is the recommended starting point when you have no existing evaluators or experiments and want a deterministic walkthrough.

Phase 1: Classify ml_app traces      → agent-observability-session-classify (ml_app mode)
Phase 2: Root cause analysis         → agent-observability-trace-rca
Phase 3: Bootstrap evaluators        → agent-observability-eval-bootstrap
Phase 4: Create + publish dataset    → agent-observability-eval-bootstrap --emit-dataset + LLMObs.create_dataset(records=...)
Phase 5: Generate + run experiment   → agent-observability-experiment-py-bootstrap + python <generated_file>
                                       (with an in-phase review beat between codegen and run)
Phase 6: Analyze experiment          → agent-observability-experiment-analyzer

Each phase has a canonical short name — the same value accepted by --start-at and --stop-after. Use these names whenever you need to refer to a single phase unambiguously (e.g. in scripts, in chat with teammates, or in support tickets):

Phases 4 and 5 are the only two that execute code on your machine; the rest are read-only or write generated files to --output-dir. The classic three-phase eval-pipeline behavior (classify → RCA → bootstrap evaluators only) is preserved by passing --stop-after eval-bootstrap. Phase 5 pauses between codegen and execution so you can review the generated experiment file before any provider tokens are spent — type run to execute, edit to pause and adjust, or stop to exit cleanly.

Enter and exit at any phase. The pipeline persists each phase’s primary output (classification summary, RCA report, evaluator suite, dataset, published dataset name, experiment file, experiment run, analyzer report) to <output-dir>/state/0N-<name>.{md,json} before each checkpoint renders. This means:

• stop at any checkpoint (or --stop-after <phase> from the top) ends the run cleanly, leaving a re-enterable artifact on disk.
• --start-at <phase> loads every prior phase’s state file (or accepts an override flag if you supplied one) and skips directly to the named phase. You can resume hours or days later, or jump straight to “just re-analyze this experiment” without re-running anything earlier.

Checkpoint vocabulary at every phase: continue advances, stop exits cleanly, redo re-runs the current phase (with optional adjustment notes appended), back steps backward one phase. Any other input is treated as adjustment.

Only <ml_app> is required. Every flag below is optional — the skill picks sensible defaults for each one. The minimal invocation is /agent-observability-eval-pipeline <ml_app>; the rest of the table is for when you want to override a default, resume mid-flow, or pin a specific output location.

Examples

# Full six-phase walkthrough for a brand new ml_app
/agent-observability-eval-pipeline my-chatbot --project-name my-chatbot

# Organize the dataset and experiment under a specific Datadog project
# (the project is created lazily — no need to pre-create it in the UI)
/agent-observability-eval-pipeline my-chatbot --project-name customer-qa-eval

# Classic three-phase eval-pipeline behavior — preserves backward compatibility
/agent-observability-eval-pipeline my-chatbot --stop-after eval-bootstrap

# Resume from where a previous run stopped
/agent-observability-eval-pipeline my-chatbot --start-at experiment

# Re-analyze a previous experiment run without re-running it
/agent-observability-eval-pipeline my-chatbot --start-at analyze --experiment-id <UUID>

# Run a single phase in isolation
/agent-observability-eval-pipeline my-chatbot --start-at dataset --stop-after dataset

Project name — if --project-name is omitted, the skill auto-derives it from your codebase (in order: pyproject.toml → setup.cfg → setup.py → package.json → cwd basename), falling back to experiment-sdk-default. The resolved name is surfaced in the Precheck output before any phase runs, so you can confirm or override it without re-invoking. The Datadog project itself is created lazily by LLMObs.enable(project_name=...) when Phase 4 publishes the dataset — you never need to pre-create it in the UI.

Typical workflow

If you are new to evaluating an LLM application, the recommended flow is:

1. Run the pipeline to walk from production traces all the way through evaluators, a seed dataset, an experiment, and analysis:

   /agent-observability-eval-pipeline <ml_app> --project-name <project>
   

   To stop at the classic evaluator-only output (no dataset or experiment), pass --stop-after eval-bootstrap. To resume a previous run, pass --start-at <phase> — the pipeline reloads prior state from <output-dir>/state/ and continues from there.

2. Apply fixes. The RCA report produced in Phase 2 includes specific before/after fix proposals grounded in trace evidence. Pass the report to a coding agent (or act on it directly) to fix system prompts, tool definitions, or routing logic in your codebase.

3. Run an offline experiment using the generated evaluators against a labeled dataset to validate their quality before enabling them in production. See the Evaluation Developer Guide.

4. Publish online evaluators once the evaluators are validated. Running /agent-observability-eval-bootstrap with --publish creates online LLM-judge evaluators in Datadog that run automatically on your production traces in real time — no code changes required:

   /agent-observability-eval-bootstrap <ml_app> --publish
   

5. Monitor and iterate. As your app evolves, re-run /agent-observability-trace-rca and /agent-observability-eval-bootstrap to catch new failure modes and keep your evaluator suite current.

Additional helpful documentation, links, and articles:

Agent Observability EvaluationsDOCUMENTATION LLM ExperimentsDOCUMENTATION Evaluation Developer Guide: Build custom evaluatorsGUIDE Improve AI agent quality with Bits EvalsBLOG datadog-labs/agent-skillsGITHUB