--- title: Run an LLM inference description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: Docs > API Reference > LLM Observability --- # Run an LLM inference{% #run-an-llm-inference %} Copy pageCopied {% tab title="v2" %} **Note**: This endpoint is in Preview and is subject to change. If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/). | Datadog site | API endpoint | | ----------------- | ------------------------------------------------------------------------------------------------------ | | ap1.datadoghq.com | POST https://api.ap1.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | ap2.datadoghq.com | POST https://api.ap2.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | app.datadoghq.eu | POST https://api.datadoghq.eu/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | app.ddog-gov.com | POST https://api.ddog-gov.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | us2.ddog-gov.com | POST https://api.us2.ddog-gov.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | app.datadoghq.com | POST https://api.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | us3.datadoghq.com | POST https://api.us3.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | | us5.datadoghq.com | POST https://api.us5.datadoghq.com/api/v2/llm-obs/v1/integrations/{integration}/{account_id}/inference | ### Overview Run an LLM inference request through the specified integration and account, returning the model response and token usage. ### Arguments #### Path Parameters | Name | Type | Description | | ----------------------------- | ------ | ---------------------------------- | | integration [*required*] | string | The name of the LLM integration. | | account_id [*required*] | string | The ID of the integration account. | ### Request #### Body Data (required) Inference request parameters. {% tab title="Model" %} | Parent field | Field | Type | Description | | --------------------- | ---------------------------- | -------- | --------------------------------------------------------------------------------------------------------------- | | | anthropic_metadata | object | Anthropic-specific metadata for an inference request. | | anthropic_metadata | effort | enum | The effort level for Anthropic inference. Allowed enum values: `low,medium,high,max` | | anthropic_metadata | thinking | object | Configuration for Anthropic extended thinking feature. | | thinking | budget_tokens | int64 | Maximum token budget for extended thinking. Required when type is `enabled`. | | thinking | type [*required*] | enum | The thinking mode for Anthropic extended thinking. Allowed enum values: `enabled,disabled,adaptive` | | | azure_openai_metadata | object | Azure OpenAI-specific metadata for an integration account or inference request. | | azure_openai_metadata | deployment_id | string | The Azure OpenAI deployment ID. | | azure_openai_metadata | model_version | string | The model version deployed in Azure. | | azure_openai_metadata | resource_name | string | The Azure OpenAI resource name. | | | bedrock_metadata | object | Amazon Bedrock-specific metadata for an inference request. | | bedrock_metadata | region | string | The AWS region for the Bedrock request. | | | frequency_penalty | double | Penalty for token frequency to reduce repetition. | | | json_schema | string | JSON schema for structured output, if supported by the model. | | | max_completion_tokens | int64 | Maximum number of completion tokens to generate (alternative to max_tokens for some providers). | | | max_tokens | int64 | Maximum number of tokens to generate. | | | messages [*required*] | [object] | List of messages in an inference conversation. | | messages | content | string | Plain text content of the message. | | messages | contents | [object] | List of structured content blocks in a message. | | contents | type [*required*] | string | The content block type. | | contents | value [*required*] | object | The typed value of a message content block. | | value | text | string | Plain text content. | | value | tool_call | object | A tool call made during LLM inference. | | tool_call | arguments | object | The arguments passed to the tool. | | tool_call | name | string | The name of the tool being called. | | tool_call | tool_id | string | Unique identifier for the tool call. | | tool_call | type | string | The type of tool call. | | value | tool_call_result | object | The result returned by a tool call during LLM inference. | | tool_call_result | name | string | The name of the tool that produced this result. | | tool_call_result | result | string | The result content returned by the tool. | | tool_call_result | tool_id | string | Identifier matching the corresponding tool call. | | tool_call_result | type | string | The type of tool result. | | messages | id | string | Unique identifier for the message. | | messages | role | string | The role of the message author. | | messages | tool_calls | [object] | List of tool calls in a message. | | tool_calls | arguments | object | The arguments passed to the tool. | | tool_calls | name | string | The name of the tool being called. | | tool_calls | tool_id | string | Unique identifier for the tool call. | | tool_calls | type | string | The type of tool call. | | messages | tool_results | [object] | List of tool results in a message. | | tool_results | name | string | The name of the tool that produced this result. | | tool_results | result | string | The result content returned by the tool. | | tool_results | tool_id | string | Identifier matching the corresponding tool call. | | tool_results | type | string | The type of tool result. | | | model_id [*required*] | string | The model identifier to use for inference. | | | openai_metadata | object | OpenAI-specific metadata for an inference request. | | openai_metadata | reasoning_effort | enum | The reasoning effort level for OpenAI models that support it. Allowed enum values: `none,low,medium,high,xhigh` | | openai_metadata | reasoning_summary | enum | The verbosity of the reasoning summary. Allowed enum values: `auto,concise,detailed` | | | presence_penalty | double | Penalty for token presence to encourage topic diversity. | | | temperature | double | Sampling temperature between 0 and 2. Higher values produce more random output. | | | tools | [object] | List of tools available to the model. | | tools | function [*required*] | object | A function definition for a tool available to the model. | | function | description | string | A description of what the function does. | | function | name [*required*] | string | The name of the function. | | function | parameters [*required*] | object | JSON schema describing the function parameters. | | tools | type [*required*] | string | The type of tool. | | | top_k | int64 | Top-K sampling parameter. | | | top_p | double | Nucleus sampling probability mass. | | | vertex_ai_metadata | object | Vertex AI-specific metadata for an integration account or inference request. | | vertex_ai_metadata | location | string | The Vertex AI region. | | vertex_ai_metadata | project | string | The Google Cloud project ID. | | vertex_ai_metadata | project_ids | [string] | List of Google Cloud project IDs available to the service account. | {% /tab %} {% tab title="Example" %} ```json { "anthropic_metadata": { "effort": "medium", "thinking": { "budget_tokens": 1024, "type": "enabled" } }, "azure_openai_metadata": { "deployment_id": "my-gpt4-deployment", "model_version": "0613", "resource_name": "my-azure-resource" }, "bedrock_metadata": { "region": "us-east-1" }, "frequency_penalty": 0, "json_schema": "{\"type\":\"object\",\"properties\":{\"answer\":{\"type\":\"string\"}}}", "max_completion_tokens": 1024, "max_tokens": 1024, "messages": [ { "content": "What is the capital of France?", "contents": [ { "type": "text", "value": { "text": "Hello, how can I help you?", "tool_call": { "arguments": { "location": "San Francisco" }, "name": "get_weather", "tool_id": "call_abc123", "type": "function" }, "tool_call_result": { "name": "get_weather", "result": "The weather in San Francisco is 68°F and sunny.", "tool_id": "call_abc123", "type": "function" } } } ], "id": "msg_001", "role": "user", "tool_calls": [ { "arguments": { "location": "San Francisco" }, "name": "get_weather", "tool_id": "call_abc123", "type": "function" } ], "tool_results": [ { "name": "get_weather", "result": "The weather in San Francisco is 68°F and sunny.", "tool_id": "call_abc123", "type": "function" } ] } ], "model_id": "gpt-4o", "openai_metadata": { "reasoning_effort": "medium", "reasoning_summary": "auto" }, "presence_penalty": 0, "temperature": 0.7, "tools": [ { "function": { "description": "Get the current weather for a location.", "name": "get_weather", "parameters": { "properties": { "location": { "type": "string" } }, "type": "object" } }, "type": "function" } ], "top_k": 50, "top_p": 1, "vertex_ai_metadata": { "location": "us-central1", "project": "my-gcp-project", "project_ids": [ "my-gcp-project" ] } } ``` {% /tab %} ### Response {% tab title="200" %} OK {% tab title="Model" %} The result of an LLM inference request, including input parameters and the model response. | Parent field | Field | Type | Description | | --------------------- | --------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------- | | | anthropic_metadata | object | Anthropic-specific metadata for an inference request. | | anthropic_metadata | effort | enum | The effort level for Anthropic inference. Allowed enum values: `low,medium,high,max` | | anthropic_metadata | thinking | object | Configuration for Anthropic extended thinking feature. | | thinking | budget_tokens | int64 | Maximum token budget for extended thinking. Required when type is `enabled`. | | thinking | type [*required*] | enum | The thinking mode for Anthropic extended thinking. Allowed enum values: `enabled,disabled,adaptive` | | | azure_openai_metadata | object | Azure OpenAI-specific metadata for an integration account or inference request. | | azure_openai_metadata | deployment_id | string | The Azure OpenAI deployment ID. | | azure_openai_metadata | model_version | string | The model version deployed in Azure. | | azure_openai_metadata | resource_name | string | The Azure OpenAI resource name. | | | bedrock_metadata | object | Amazon Bedrock-specific metadata for an inference request. | | bedrock_metadata | region | string | The AWS region for the Bedrock request. | | | error_response | object | Error details returned when an inference provider returns an error. | | error_response | message [*required*] | string | A human-readable description of the error. | | error_response | type [*required*] | string | The provider-specific error type. | | | frequency_penalty | double | Frequency penalty that was applied. | | | json_schema | string | JSON schema that was applied for structured output. | | | max_completion_tokens | int64 | Maximum number of completion tokens that were configured. | | | max_tokens | int64 | Maximum number of tokens that were configured. | | | messages [*required*] | [object] | List of messages in an inference conversation. | | messages | content | string | Plain text content of the message. | | messages | contents | [object] | List of structured content blocks in a message. | | contents | type [*required*] | string | The content block type. | | contents | value [*required*] | object | The typed value of a message content block. | | value | text | string | Plain text content. | | value | tool_call | object | A tool call made during LLM inference. | | tool_call | arguments | object | The arguments passed to the tool. | | tool_call | name | string | The name of the tool being called. | | tool_call | tool_id | string | Unique identifier for the tool call. | | tool_call | type | string | The type of tool call. | | value | tool_call_result | object | The result returned by a tool call during LLM inference. | | tool_call_result | name | string | The name of the tool that produced this result. | | tool_call_result | result | string | The result content returned by the tool. | | tool_call_result | tool_id | string | Identifier matching the corresponding tool call. | | tool_call_result | type | string | The type of tool result. | | messages | id | string | Unique identifier for the message. | | messages | role | string | The role of the message author. | | messages | tool_calls | [object] | List of tool calls in a message. | | tool_calls | arguments | object | The arguments passed to the tool. | | tool_calls | name | string | The name of the tool being called. | | tool_calls | tool_id | string | Unique identifier for the tool call. | | tool_calls | type | string | The type of tool call. | | messages | tool_results | [object] | List of tool results in a message. | | tool_results | name | string | The name of the tool that produced this result. | | tool_results | result | string | The result content returned by the tool. | | tool_results | tool_id | string | Identifier matching the corresponding tool call. | | tool_results | type | string | The type of tool result. | | | model_id [*required*] | string | The model identifier used for inference. | | | openai_metadata | object | OpenAI-specific metadata for an inference request. | | openai_metadata | reasoning_effort | enum | The reasoning effort level for OpenAI models that support it. Allowed enum values: `none,low,medium,high,xhigh` | | openai_metadata | reasoning_summary | enum | The verbosity of the reasoning summary. Allowed enum values: `auto,concise,detailed` | | | presence_penalty | double | Presence penalty that was applied. | | | response [*required*] | object | The output of a completed LLM inference call. | | response | assessment [*required*] | string | An optional assessment of the inference output quality. | | response | content [*required*] | string | The text content of the model response. | | response | finish_reason [*required*] | string | The reason the model stopped generating tokens. | | response | inference_codes [*required*] | [object] | List of generated code snippets for the inference configuration. | | inference_codes | code [*required*] | string | The generated code content. | | inference_codes | id [*required*] | string | Unique identifier for the code snippet. | | inference_codes | type [*required*] | string | The programming language or SDK type of the code snippet. | | response | input_tokens [*required*] | int64 | Number of input tokens consumed. | | response | internal_reasoning | object | The model's internal reasoning or thinking output, if available. | | internal_reasoning | reasoning_tokens | int64 | Number of tokens used for internal reasoning. | | internal_reasoning | text [*required*] | string | The reasoning text produced by the model. | | response | latency [*required*] | int64 | Request latency in milliseconds. | | response | output_tokens [*required*] | int64 | Number of output tokens generated. | | response | tools [*required*] | [object] | List of tools available to the model. | | tools | function [*required*] | object | A function definition for a tool available to the model. | | function | description | string | A description of what the function does. | | function | name [*required*] | string | The name of the function. | | function | parameters [*required*] | object | JSON schema describing the function parameters. | | tools | type [*required*] | string | The type of tool. | | response | total_tokens [*required*] | int64 | Total tokens used (input plus output). | | | temperature | double | Sampling temperature that was used. | | | tools | [object] | List of tools available to the model. | | tools | function [*required*] | object | A function definition for a tool available to the model. | | function | description | string | A description of what the function does. | | function | name [*required*] | string | The name of the function. | | function | parameters [*required*] | object | JSON schema describing the function parameters. | | tools | type [*required*] | string | The type of tool. | | | top_k | int64 | Top-K sampling parameter that was used. | | | top_p | double | Nucleus sampling parameter that was used. | | | vertex_ai_metadata | object | Vertex AI-specific metadata for an integration account or inference request. | | vertex_ai_metadata | location | string | The Vertex AI region. | | vertex_ai_metadata | project | string | The Google Cloud project ID. | | vertex_ai_metadata | project_ids | [string] | List of Google Cloud project IDs available to the service account. | {% /tab %} {% tab title="Example" %} ```json { "anthropic_metadata": { "effort": "medium", "thinking": { "budget_tokens": 1024, "type": "enabled" } }, "azure_openai_metadata": { "deployment_id": "my-gpt4-deployment", "model_version": "0613", "resource_name": "my-azure-resource" }, "bedrock_metadata": { "region": "us-east-1" }, "error_response": { "message": "The model does not exist.", "type": "invalid_request_error" }, "frequency_penalty": 0, "json_schema": "{\"type\":\"object\",\"properties\":{\"answer\":{\"type\":\"string\"}}}", "max_completion_tokens": 1024, "max_tokens": 1024, "messages": [ { "content": "What is the capital of France?", "contents": [ { "type": "text", "value": { "text": "Hello, how can I help you?", "tool_call": { "arguments": { "location": "San Francisco" }, "name": "get_weather", "tool_id": "call_abc123", "type": "function" }, "tool_call_result": { "name": "get_weather", "result": "The weather in San Francisco is 68°F and sunny.", "tool_id": "call_abc123", "type": "function" } } } ], "id": "msg_001", "role": "user", "tool_calls": [ { "arguments": { "location": "San Francisco" }, "name": "get_weather", "tool_id": "call_abc123", "type": "function" } ], "tool_results": [ { "name": "get_weather", "result": "The weather in San Francisco is 68°F and sunny.", "tool_id": "call_abc123", "type": "function" } ] } ], "model_id": "gpt-4o", "openai_metadata": { "reasoning_effort": "medium", "reasoning_summary": "auto" }, "presence_penalty": 0, "response": { "assessment": "pass", "content": "The capital of France is Paris.", "finish_reason": "stop", "inference_codes": [ { "code": "import openai\nclient = openai.OpenAI()\n...", "id": "code-python-001", "type": "python" } ], "input_tokens": 15, "internal_reasoning": { "reasoning_tokens": 256, "text": "Let me think about this step by step..." }, "latency": 843, "output_tokens": 10, "tools": [ { "function": { "description": "Get the current weather for a location.", "name": "get_weather", "parameters": { "properties": { "location": { "type": "string" } }, "type": "object" } }, "type": "function" } ], "total_tokens": 25 }, "temperature": 0.7, "tools": [ { "function": { "description": "Get the current weather for a location.", "name": "get_weather", "parameters": { "properties": { "location": { "type": "string" } }, "type": "object" } }, "type": "function" } ], "top_k": 50, "top_p": 1, "vertex_ai_metadata": { "location": "us-central1", "project": "my-gcp-project", "project_ids": [ "my-gcp-project" ] } } ``` {% /tab %} {% /tab %} {% tab title="400" %} Bad Request {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="401" %} Unauthorized {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="403" %} Forbidden {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} {% tab title="429" %} Too many requests {% tab title="Model" %} API error response. | Field | Type | Description | | ------------------------ | -------- | ----------------- | | errors [*required*] | [string] | A list of errors. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ "Bad Request" ] } ``` {% /tab %} {% /tab %} {% tab title="500" %} Internal Server Error {% tab title="Model" %} API error response. | Parent field | Field | Type | Description | | ------------ | ------------------------ | -------- | ------------------------------------------------------------------------------- | | | errors [*required*] | [object] | A list of errors. | | errors | detail | string | A human-readable explanation specific to this occurrence of the error. | | errors | meta | object | Non-standard meta-information about the error | | errors | source | object | References to the source of the error. | | source | header | string | A string indicating the name of a single request header which caused the error. | | source | parameter | string | A string indicating which URI query parameter caused the error. | | source | pointer | string | A JSON pointer to the value in the request document that caused the error. | | errors | status | string | Status code of the response. | | errors | title | string | Short human-readable summary of the error. | {% /tab %} {% tab title="Example" %} ```json { "errors": [ { "detail": "Missing required attribute in body", "meta": {}, "source": { "header": "Authorization", "parameter": "limit", "pointer": "/data/attributes/title" }, "status": "400", "title": "Bad Request" } ] } ``` {% /tab %} {% /tab %} ### Code Example ##### \## default # \# Path parameters export integration="openai" export account_id="account-abc123" \# Curl command curl -X POST "https://api.datadoghq.com/api/v2/llm-obs/v1/integrations/${integration}/${account_id}/inference" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "max_tokens": 256, "messages": [ { "content": "What is the capital of France?", "role": "user" } ], "model_id": "gpt-4o", "temperature": 0.7 } EOF {% /tab %}