# Agent Observability

> This index is a subsection of the documentation. The canonical entry point for reading the docs is at the root: https://docs.datadoghq.com/llms.txt
> If you got here by guessing the path to the subsection's index, double back to the root llms.txt to ensure you access the latest content structure.

- [Agent Observability](https://docs.datadoghq.com/llm_observability.md): Overview of Agent Observability, a platform for monitoring, troubleshooting, and improving LLM applications and AI agents.
- [Build with AI](https://docs.datadoghq.com/llm_observability/build_with_ai.md): Use the Datadog MCP server, CLI, and Claude Code skills to build and analyze LLM applications from your development environment.
- [Configure Agent Observability](https://docs.datadoghq.com/llm_observability/configure.md): Configure prompt management and automation rules for Agent Observability.
- [Data Governance](https://docs.datadoghq.com/llm_observability/data_governance.md): Control access to sensitive Agent Observability data with data access controls and RBAC, redact data with span processors, and learn how long Agent Observability retains each type of data.
- [Agent Observability Guides](https://docs.datadoghq.com/llm_observability/guide.md)
- [Use Agent Observability to Improve Applications](https://docs.datadoghq.com/llm_observability/improve.md): Iterate on your LLM applications and agents with the playground, datasets, and experiments.
- [Agent Observability Instrumentation](https://docs.datadoghq.com/llm_observability/instrument.md): Overview of instrumentation options for Agent Observability, including SDK-based and API-based approaches for Python, Node.js, and Java.
- [Investigate](https://docs.datadoghq.com/llm_observability/investigate.md): How to explore more of your application in Agent Observability.
- [Lapdog](https://docs.datadoghq.com/llm_observability/lapdog.md): Run an Agent Observability dashboard locally to inspect coding-agent and application traces in your browser without a Datadog account.
- [Quickstart](https://docs.datadoghq.com/llm_observability/quickstart.md): Get started with Agent Observability by instrumenting a Python, Node.js, or Java LLM application using the Agent Observability SDK.

## Agent Observability Guides

- [Advanced Experiment Runs](https://docs.datadoghq.com/llm_observability/guide/advanced_experiment_runs.md): Run experiments multiple times to account for model variability on a subset of your dataset, and automate experiment execution in CI/CD pipelines.
- [Agent Monitoring](https://docs.datadoghq.com/llm_observability/guide/agent_monitoring.md): Monitor, troubleshoot, and improve agentic applications built on OpenAI Agents SDK, LangGraph, or CrewAI using Agent Observability.
- [Analyze Your Experiments Results](https://docs.datadoghq.com/llm_observability/guide/analyzing_experiment_results.md): How to analyze Agent Observability Experiments results.
- [Datadog-CrewAI integration for Agent Observability](https://docs.datadoghq.com/llm_observability/guide/crewai_guide.md): Learn how to integrate CrewAI with Agent Observability using auto-instrumentation to monitor and trace your CrewAI agent runs.
- [Experiments API](https://docs.datadoghq.com/llm_observability/guide/experiments_api.md): Reference the Agent Observability Experiments HTTP API.
- [Set Up and Use Agent Observability Experiments](https://docs.datadoghq.com/llm_observability/guide/experiments.md): How to set up Agent Observability Experiments and start running experiments.
- [MCP Clients](https://docs.datadoghq.com/llm_observability/guide/monitor_mcp_client.md): Learn how to instrument and monitor MCP clients with Agent Observability.
- [Tracing Proxy Services](https://docs.datadoghq.com/llm_observability/guide/monitor_proxy_services.md): Learn how to use Agent Observability to trace LLM calls through proxy or gateway services as part of a complete end-to-end trace.
- [Instrument a Next.js Application for Agent Observability](https://docs.datadoghq.com/llm_observability/guide/nextjs_guide.md): Learn how to set up Agent Observability in a Next.js application using auto-instrumentation to monitor and trace your LLM calls.
- [Trace an LLM Application in AWS Lambda](https://docs.datadoghq.com/llm_observability/guide/trace-an-llm-application-in-aws-lambda.md): Instrument an existing AWS Lambda function with Agent Observability by using the Datadog Extension and language layers.

## Agent Observability Instrumentation

- [Correlating Agent Observability and APM](https://docs.datadoghq.com/llm_observability/instrument/agent_observability_and_apm.md): Learn how to navigate between Agent Observability spans and APM spans so you can gain insights into LLM-specific operations and your broader application ecosystem.
- [Agentic Instrumentation](https://docs.datadoghq.com/llm_observability/instrument/agentic.md)
- [HTTP API Reference](https://docs.datadoghq.com/llm_observability/instrument/api.md): Reference documentation for the Agent Observability HTTP API, used to send LLM traces and spans to Datadog from applications in any language.
- [Automatic Instrumentation for Agent Observability](https://docs.datadoghq.com/llm_observability/instrument/auto_instrumentation.md): Enable automatic tracing for LLM frameworks and libraries in your Python or Node.js application using Agent Observability integrations.
- [Multimodal Support](https://docs.datadoghq.com/llm_observability/instrument/multimodal.md): Attach audio and images to LLM spans in Agent Observability and render them inline in the trace view, automatically or through manual instrumentation.
- [OpenTelemetry Instrumentation](https://docs.datadoghq.com/llm_observability/instrument/otel_instrumentation.md): Instrument LLM applications with OpenTelemetry using GenAI or OpenInference semantic conventions and send traces to Agent Observability without the Datadog SDK.
- [Prompt Tracking](https://docs.datadoghq.com/llm_observability/instrument/prompt_tracking.md): Use Prompt Tracking to track your prompt templates and versions.
- [Agent Observability SDK Reference](https://docs.datadoghq.com/llm_observability/instrument/sdk.md): Reference documentation for the Agent Observability SDKs for Python, Node.js, Java, and Go (experimental), covering automatic and manual instrumentation.

### Agentic Instrumentation

- [Java Application Agentic Instrumentation](https://docs.datadoghq.com/llm_observability/instrument/agentic/java.md)
- [Node.js Application Agentic Instrumentation](https://docs.datadoghq.com/llm_observability/instrument/agentic/nodejs.md)
- [Prompt Management Agentic Integration](https://docs.datadoghq.com/llm_observability/instrument/agentic/prompt_management.md)
- [Python Application Agentic Instrumentation](https://docs.datadoghq.com/llm_observability/instrument/agentic/python.md)

## Build with AI

- [Analyze LLM Applications with Claude Code Skills](https://docs.datadoghq.com/llm_observability/build_with_ai/claude_code_skills.md): Use Datadog's Claude Code skills to classify sessions, diagnose failures, compare experiments, generate Python experiment code, and bootstrap evaluators against your live production data.
- [Agent Observability MCP and Skills](https://docs.datadoghq.com/llm_observability/build_with_ai/mcp_server.md): Connect AI agents to your Agent Observability traces and experiments using the Datadog MCP Server.

## Configure Agent Observability

- [Automation Rules](https://docs.datadoghq.com/llm_observability/configure/automation_rules.md): Route Agent Observability traces into annotation queues or datasets automatically using filter-based rules.
- [Prompt Experimentation](https://docs.datadoghq.com/llm_observability/configure/prompt_experimentation.md): Compare managed prompt versions with A/B tests and deploy versions with Guarded Rollouts.
- [Prompt Management](https://docs.datadoghq.com/llm_observability/configure/prompt_management.md): Create, version, and retrieve managed prompts in Python, Go, and JavaScript applications with Prompt Management.

## Investigate

- [Annotation Queues](https://docs.datadoghq.com/llm_observability/investigate/annotation_queues.md): Enable systematic human review of LLM traces to identify failure modes, validate automated evaluations, and build golden datasets.
- [Cost](https://docs.datadoghq.com/llm_observability/investigate/cost.md): Monitor your LLM tokens and costs.
- [Evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations.md): Learn how to configure Evaluators for your LLM application.
- [Export API](https://docs.datadoghq.com/llm_observability/investigate/export_api.md): Reference for the Agent Observability Export API, which provides endpoints to search and retrieve span data for external evaluations or offline storage.
- [Insights](https://docs.datadoghq.com/llm_observability/investigate/insights.md): Agent Observability Insights identifies recurring cost and reliability problems in existing traces and recommends fixes.
- [Agent Observability Metrics](https://docs.datadoghq.com/llm_observability/investigate/metrics.md): Learn about useful metrics you can generate from Agent Observability data.
- [Patterns](https://docs.datadoghq.com/llm_observability/investigate/patterns.md): Discover and analyze production traffic patterns in your agent with automated topic clustering.
- [Querying spans and traces](https://docs.datadoghq.com/llm_observability/investigate/querying.md): Learn how to query Agent Observability spans and traces in the Trace Explorer, including how to search by attribute, tags, and trace-level properties.

### Evaluations

- [Evaluator compatibility](https://docs.datadoghq.com/llm_observability/investigate/evaluations/compatibility.md): Learn about the compatibility requirements for evaluators.
- [End-User Feedback](https://docs.datadoghq.com/llm_observability/investigate/evaluations/end_user_feedback.md): Submit end-user feedback to Agent Observability and connect it to spans, traces, sessions, or external entities.
- [Evaluator Developer Guide](https://docs.datadoghq.com/llm_observability/investigate/evaluations/evaluation_developer_guide.md): Learn how to build custom evaluators using the Agent Observability SDK.
- [External Evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations/external_evaluations.md): Submit custom evaluations to Agent Observability using the Python SDK or the Agent Observability API to track response quality.
- [Language Mismatch](https://docs.datadoghq.com/llm_observability/investigate/evaluations/language_mismatch.md): Learn about Datadog's Language Mismatch evaluator.
- [Custom LLM-as-a-Judge Evaluators](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations.md): How to create custom LLM-as-a-judge evaluators, and how to use their evaluation results across Agent Observability.
- [Managed Evaluators](https://docs.datadoghq.com/llm_observability/investigate/evaluations/managed_evaluations.md): Learn how to configure managed evaluators for your LLM applications.

#### Custom LLM-as-a-Judge Evaluators

- [Connect your LLM provider account](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/connect_to_account.md): How to connect to your LLM provider account to support judge LLM based evaluations
- [Prompt Templating](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/prompt_templating.md): Reference for the templating used in custom LLM-as-a-judge evaluator prompts—variables, array operators, span and trace filters, session paths, and resolution rules.
- [Session-Level Evaluators](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/session_level_evaluations.md): Run a custom LLM-as-a-judge across an entire user session, with examples of when to use session scope over trace or span scope.
- [LLM-as-a-Judge Evaluator Templates](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/template_evaluations.md): Learn how to create LLM-as-a-Judge evaluators from templates for your LLM applications.
- [Trace-Level Evaluators](https://docs.datadoghq.com/llm_observability/investigate/evaluations/llm_as_a_judge_evaluations/trace_level_evaluations.md): Run a custom LLM-as-a-judge across an entire trace, with examples of when to use trace scope over span scope.

#### External Evaluations

- [DeepEval Evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations/external_evaluations/deepeval.md): Use DeepEval evaluators with Agent Observability Experiments.
- [NeMo Evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations/external_evaluations/nemo.md): Learn how to submit NVIDIA NeMo Evaluator model evaluation scores to Agent Observability to monitor benchmarking results alongside LLM traces.
- [Pydantic Evaluations](https://docs.datadoghq.com/llm_observability/investigate/evaluations/external_evaluations/pydantic.md): Use Pydantic evaluators with Agent Observability Experiments.

#### Managed Evaluators

- [Sensitive Data Scanner](https://docs.datadoghq.com/llm_observability/investigate/evaluations/managed_evaluations/security_and_safety_evaluations.md): Learn how to configure managed evaluators for your LLM applications.

## Quickstart

- [Agent Observability Terms and Concepts](https://docs.datadoghq.com/llm_observability/quickstart/terms.md): Reference guide for Agent Observability key terms and concepts, including spans, traces, and evaluations.

## Use Agent Observability to Improve Applications

- [Data Access Control in Agent Observability](https://docs.datadoghq.com/llm_observability/improve/access_control.md): Use Data Access Control to restrict Agent Observability projects, and the evaluator configs for an application, to specific teams or roles.
- [Datasets](https://docs.datadoghq.com/llm_observability/improve/datasets.md): Using datasets in Agent Observability Experiments, including how to create, retrieve, and manage datasets, as well as information about versioning.
- [Experiments](https://docs.datadoghq.com/llm_observability/improve/experiments.md): An overview of Agent Observability Experiments feature.
- [Playground](https://docs.datadoghq.com/llm_observability/improve/playground.md): Use the Agent Observability Playground to test prompts interactively against any connected model provider and run experiments with evaluators from a browser-based interface.