OpenAI

Supported OS Linux Windows Mac OS

Overview

Get cost estimation, prompt and completion sampling, error tracking, performance metrics, and more out of OpenAI account-level, Python, and Node.js library requests using Datadog metrics, APM, and logs.

Setup

Note: This setup method does not collect openai.api.usage.* metrics. To collect these metrics, also follow the API key setup instructions.

Installation

Web: Get Account-level Usage and Cost Metrics

Note: This setup method only collects openai.api.usage* metrics, and if you enable OpenAI in Cloud Cost Management, you will also get cost metrics, no additional permissions or setup required. Use the agent setup below for additional metrics.

  1. Login to your OpenAI Account.
  2. Navigate to View API Keys under account settings.
  3. Click the Create a new secret key button.
  4. Copy the created API Key to your clipboard.
  5. Navigate to the configuration tab inside Datadog OpenAI integration tile.
  6. Enter an account name and OpenAI API key copied above in the accounts configuration.
  7. If you use Cloud Cost Management and enable collecting cost data, it will be visible in Cloud Cost Management within 24 hours. (collected data)

APM: Get Usage Metrics for Python and Node.js Applications

  1. Enable APM and StatsD in your Datadog Agent. For example, in Docker:
docker run -d
  --cgroupns host \
  --pid host \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  -e DD_API_KEY=<DATADOG_API_KEY> \
  -p 127.0.0.1:8126:8126/tcp \
  -p 127.0.0.1:8125:8125/udp \
  -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
  -e DD_APM_ENABLED=true \
  gcr.io/datadoghq/agent:latest
  1. Install the Datadog APM Python library.
pip install ddtrace
  1. Prefix your OpenAI Python application command with ddtrace-run and the following environment variables as shown below:
DD_SERVICE="my-service" DD_ENV="staging" ddtrace-run python <your-app>.py

Notes:

  • Non-US1 customers must set DD_SITE on the application command to the correct Datadog site parameter as specified in the table in the Datadog Site page (for example, datadoghq.eu for EU1 customers).

  • If the Agent is using a non-default hostname or port, be sure to also set DD_AGENT_HOST, DD_TRACE_AGENT_PORT, or DD_DOGSTATSD_PORT.

See the APM Python library documentation for more advanced usage.

Configuration

See the APM Python library documentation for all the available configuration options.

Log Prompt & Completion Sampling

To enable log prompt and completion sampling, set the DD_OPENAI_LOGS_ENABLED="true" environment variable. By default, 10% of traced requests will emit logs containing the prompts and completions.

To adjust the log sample rate, see the APM library documentation.

Note: Logs submission requires DD_API_KEY to be specified when running ddtrace-run.

Validation

Validate that the APM Python library can communicate with your Agent using:

ddtrace-run --info

You should see the following output:

    Agent error: None

Debug Logging

Pass the --debug flag to ddtrace-run to enable debug logging.

ddtrace-run --debug

This displays any errors sending data:

ERROR:ddtrace.internal.writer.writer:failed to send, dropping 1 traces to intake at http://localhost:8126/v0.5/traces after 3 retries ([Errno 61] Connection refused)
WARNING:ddtrace.vendor.dogstatsd:Error submitting packet: [Errno 61] Connection refused, dropping the packet and closing the socket
DEBUG:ddtrace.contrib.openai._logging.py:sent 2 logs to 'http-intake.logs.datadoghq.com'

Note: This setup method does not collect openai.api.usage.* metrics. To collect these metrics, also follow the API key setup instructions.

Installation

  1. Enable APM and StatsD in your Datadog Agent. For example, in Docker:
docker run -d
  --cgroupns host \
  --pid host \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  -e DD_API_KEY=<DATADOG_API_KEY> \
  -p 127.0.0.1:8126:8126/tcp \
  -p 127.0.0.1:8125:8125/udp \
  -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
  -e DD_APM_ENABLED=true \
  gcr.io/datadoghq/agent:latest
  1. Install the Datadog APM Node.js library.
npm install dd-trace
  1. Inject the library into your OpenAI Node.js application.
DD_TRACE_DEBUG=1 DD_TRACE_BEAUTIFUL_LOGS=1 DD_SERVICE="my-service" \
  DD_ENV="staging" DD_API_KEY=<DATADOG_API_KEY> \
  NODE_OPTIONS='-r dd-trace/init' node app.js

Note: If the Agent is using a non-default hostname or port, you must also set DD_AGENT_HOST, DD_TRACE_AGENT_PORT, or DD_DOGSTATSD_PORT.

See the APM Node.js OpenAI documentation for more advanced usage.

Configuration

See the APM Node.js library documentation for all the available configuration options.

Log prompt and completion sampling

To enable log prompt and completion sampling, set the DD_OPENAI_LOGS_ENABLED=1 environment variable. By default, 10% of traced requests emit logs containing the prompts and completions.

To adjust the log sample rate, see the APM library documentation.

Note: Logs submission requires DD_API_KEY to be specified.

Validation

Validate that the APM Node.js library can communicate with your Agent by examining the debugging output from the application process. Within the section titled “Encoding payload,” you should see an entry with a name field and a correlating value of openai.request. See below for a truncated example of this output:

{
  "name": "openai.request",
  "resource": "listModels",
  "meta": {
    "component": "openai",
    "span.kind": "client",
    "openai.api_base": "https://api.openai.com/v1",
    "openai.request.endpoint": "/v1/models",
    "openai.request.method": "GET",
    "language": "javascript"
  },
  "metrics": {
    "openai.response.count": 106
  },
  "service": "my-service",
  "type": "openai"
}

Data Collected

Metrics

openai.api.usage.n_context_tokens_total
(gauge)
Total number of context tokens used (all-time)
Shown as token
openai.api.usage.n_generated_tokens_total
(gauge)
Total number of generated response tokens (all-time)
Shown as token
openai.api.usage.n_requests
(count)
Total number of requests
Shown as request
openai.organization.ratelimit.requests.remaining
(gauge)
Number of requests remaining in the rate limit.
Shown as request
openai.organization.ratelimit.tokens.remaining
(gauge)
Number of tokens remaining in the rate limit.
Shown as token
openai.ratelimit.requests
(gauge)
Number of requests in the rate limit.
Shown as request
openai.ratelimit.tokens
(gauge)
Number of tokens in the rate limit.
Shown as token
openai.request.duration
(gauge)
Request duration distribution.
Shown as nanosecond
openai.request.error
(count)
Number of errors.
Shown as error
openai.tokens.completion
(gauge)
Number of tokens used in the completion of a response from OpenAI.
Shown as token
openai.tokens.prompt
(gauge)
Number of tokens used in the prompt of a request to OpenAI.
Shown as token
openai.tokens.total
(gauge)
Total number of tokens used in a request to OpenAI.
Shown as token

Events

The OpenAI integration does not include any events.

Service Checks

The OpenAI integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.

Further Reading

Additional helpful documentation, links, and articles: