Datadog’s LLM Observability Python SDK provides integrations that automatically trace and annotate calls to LLM frameworks and libraries. Without changing your code, you can get out-of-the-box traces and observability for calls that your LLM application makes to the following frameworks:

In addition to capturing latency and errors, the integrations capture the input parameters, input and output messages, and token usage (when available) of each traced call.

Enabling and disabling integrations

All integrations are enabled by default.

To disable all integrations, use the in-code SDK setup and specify integrations_enabled=False.

To only enable specific integrations:

1. Use the in-code SDK setup, specifying integrations_enabled=False.
2. Manually enable the integration with ddtrace.patch() at the top of the entrypoint file of your LLM application:

from ddtrace import patch
from ddtrace.llmobs import LLMObs

LLMObs.enable(integrations_enabled=False, ...)
patch(<INTEGRATION_NAME_IN_LOWERCASE>=True)

Note: Use botocore as the name of the AWS Bedrock integration when manually enabling.

OpenAI

The OpenAI integration provides automatic tracing for the OpenAI Python SDK’s completion and chat completion endpoints.

Traced methods

The OpenAI integration instruments the following methods, including streamed calls:

• Completions:
  • OpenAI().completions.create()
  • AsyncOpenAI().completions.create()
• Chat completions:
  • OpenAI().chat.completions.create()
  • AsyncOpenAI().chat.completions.create()

LangChain

The LangChain integration provides automatic tracing for the LangChain Python SDK’s LLM, chat model, and chain calls.

Traced methods

The LangChain integration instruments the following methods:

• LLMs:
  • llm.invoke(), llm.ainvoke()
• Chat models
  • chat_model.invoke(), chat_model.ainvoke()
• Chains/LCEL
  • chain.invoke(), chain.ainvoke()
  • chain.batch(), chain.abatch()

Note: The LangChain integration does not yet support tracing streamed calls.

AWS Bedrock

The AWS Bedrock integration provides automatic tracing for the AWS Bedrock Runtime Python SDK’s chat model calls (using Boto3/Botocore).

Traced methods

The AWS Bedrock integration instruments the following methods:

• Chat messages:
  • InvokeModel
• Streamed chat messages:
  • InvokeModelWithResponseStream

Note: The AWS Bedrock integration does not yet support tracing embedding calls.

Anthropic

The Anthropic integration provides automatic tracing for the Anthropic Python SDK’s chat message calls.

Traced methods

The Anthropic integration instruments the following methods:

• Chat messages (including streamed calls):
  • Anthropic().messages.create(), AsyncAnthropic().messages.create()
• Streamed chat messages:
  • Anthropic().messages.stream(), AsyncAnthropic().messages.stream()