---
title: Connect your LLM provider account
description: >-
  How to connect to your LLM provider account to support judge LLM based
  evaluations
breadcrumbs: >-
  Docs > Agent Observability > Investigate > Evaluations > Custom LLM-as-a-Judge
  Evaluators > Connect your LLM provider account
---

> For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt).

# Connect your LLM provider account

{% callout %}
# Important note for users on the following Datadog sites: app.ddog-gov.com, us2.ddog-gov.com

{% alert level="danger" %}
This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site.md). ({% placeholder "user-datadog-site-name" /%}).
{% /alert %}

{% /callout %}

Configure the LLM provider you would like to use for bring-your-own-key (BYOK) evaluations. You only have to complete this step once.

{% tab title="OpenAI" %}

{% alert level="danger" %}
If you are subject to HIPAA, you are responsible for ensuring that you connect only to an OpenAI account that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.
{% /alert %}

Connect your OpenAI account to Agent Observability with your OpenAI API key.

1. In Datadog, navigate to [Agent Observability > Settings > Integrations](https://app.datadoghq.com/llm/settings/integrations).
1. Select Connect on the OpenAI tile.
1. Follow the instructions on the tile.
   - Provide your OpenAI API key. Ensure that this key has write permission for model capabilities.
1. Enable Use this API key to evaluate your LLM applications.
1. Agent Observability requires that the `complete/chat` API endpoint be available for the selected model. See [OpenAI's model overview page](https://developers.openai.com/api/docs/models) for details about which models support this endpoint.

{% image
   source="https://docs.dd-static.net/images/llm_observability/configuration/openai-tile.20f9a9ec33deec8cd3b7a031d38c258b.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/configuration/openai-tile.20f9a9ec33deec8cd3b7a031d38c258b.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The OpenAI configuration tile in Agent Observability. Lists instructions for configuring OpenAI and providing your OpenAI API key." /%}

Agent Observability does not support [data residency](https://platform.openai.com/docs/guides/your-data#which-models-and-features-are-eligible-for-data-residency) for OpenAI.
{% /tab %}

{% tab title="Azure OpenAI" %}

{% alert level="danger" %}
If you are subject to HIPAA, you are responsible for ensuring that you connect only to an Azure OpenAI account that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.
{% /alert %}

Connect your Azure OpenAI account to Agent Observability with your OpenAI API key. Datadog strongly recommends using the `GPT-4o mini` model for evaluations. The selected model version must support [structured output](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/structured-outputs), and the Chat Completions API must be available. See a [full list of compatible models](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/models-sold-directly-by-azure?tabs=global-standard-aoai%2Cglobal-standard&pivots=azure-openai).

1. In Datadog, navigate to [Agent Observability > Settings > Integrations](https://app.datadoghq.com/llm/settings/integrations).
1. Select Connect on the Azure OpenAI tile.
1. Follow the instructions on the tile.
   - Provide your Azure OpenAI API key. Ensure that this key has write permission for model capabilities.
   - Provide the Resource Name, Deployment ID, and API version to complete integration.

{% image
   source="https://docs.dd-static.net/images/llm_observability/configuration/azure-openai-tile.e9eda3d31c231e453daef891671ec93c.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/configuration/azure-openai-tile.e9eda3d31c231e453daef891671ec93c.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Azure OpenAI configuration tile in Agent Observability. Lists instructions for configuring Azure OpenAI and providing your API Key, Resource Name, Deployment ID, and API Version." /%}

{% /tab %}

{% tab title="Anthropic" %}

{% alert level="danger" %}
If you are subject to HIPAA, you are responsible for ensuring that you connect only to an Anthropic account that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.
{% /alert %}

Connect your Anthropic account to Agent Observability with your Anthropic API key.

1. In Datadog, navigate to [Agent Observability > Settings > Integrations](https://app.datadoghq.com/llm/settings/integrations).
1. Select Connect on the Anthropic tile.
1. Follow the instructions on the tile.
   - Provide your Anthropic API key. Ensure that this key has write permission for model capabilities.

{% image
   source="https://docs.dd-static.net/images/llm_observability/configuration/anthropic-tile.d9769847394ba0351ed9ae950c4dd041.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/configuration/anthropic-tile.d9769847394ba0351ed9ae950c4dd041.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Anthropic configuration tile in Agent Observability. Lists instructions for configuring Anthropic and providing your Anthropic API key." /%}

{% /tab %}

{% tab title="Amazon Bedrock" %}

{% alert level="danger" %}
If you are subject to HIPAA, you are responsible for ensuring that you connect only to an Amazon Bedrock account that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.
{% /alert %}

Connect your Amazon Bedrock account to Agent Observability with your AWS Account.

1. In Datadog, navigate to [Agent Observability > Settings > Integrations](https://app.datadoghq.com/llm/settings/integrations).

1. Select Connect on the Amazon Bedrock tile.

1. Follow the instructions on the tile.

   {% image
      source="https://docs.dd-static.net/images/llm_observability/configuration/amazon-bedrock-tile.afc83f52f13c6311f781caccb2201aaa.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/configuration/amazon-bedrock-tile.afc83f52f13c6311f781caccb2201aaa.png?auto=format&fit=max&w=850&dpr=2 2x"
      alt="The Amazon Bedrock configuration tile in Agent Observability. Lists instructions for configuring Amazon Bedrock." /%}

1. Be sure to configure the Invoke models from Amazon Bedrock role to run evaluations. More details about the InvokeModel action can be found in the [Amazon Bedrock API reference documentation](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html).

   {% image
      source="https://docs.dd-static.net/images/llm_observability/configuration/amazon-bedrock-tile-step-2.97f602085810fadf2d237d5d62a2f1a3.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/configuration/amazon-bedrock-tile-step-2.97f602085810fadf2d237d5d62a2f1a3.png?auto=format&fit=max&w=850&dpr=2 2x"
      alt="The second step in configuring Amazon Bedrock requiring users to add permissions to the integration account." /%}

{% /tab %}

{% tab title="GCP Vertex AI" %}

{% alert level="danger" %}
If you are subject to HIPAA, you are responsible for ensuring that you connect only to a Google Cloud Platform account that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.
{% /alert %}

Connect Vertex AI to Agent Observability with your Google Cloud Platform account.

1. In Datadog, navigate to [Agent Observability > Settings > Integrations](https://app.datadoghq.com/llm/settings/integrations).
1. On the Google Cloud Vertex AI tile, click Connect to add a new GCP account, or click Configure next to where your existing accounts are listed to begin the onboarding process.
   - You will see all GCP accounts connected to Datadog in this page. However, you must still go through the onboarding process for an account to use it in Agent Observability.
1. Follow the onboarding instructions to configure your account.
   - Add the [Vertex AI User](https://docs.cloud.google.com/vertex-ai/docs/general/access-control#aiplatform.user) role to your account and enable the [Vertex AI API](https://console.cloud.google.com/apis/library/aiplatform.googleapis.com).

{% image
   source="https://docs.dd-static.net/images/llm_observability/configuration/vertex-ai-pint.9174c851c6ea22afe8475df5339ccfa0.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/llm_observability/configuration/vertex-ai-pint.9174c851c6ea22afe8475df5339ccfa0.png?auto=format&fit=max&w=850&dpr=2 2x"
   alt="The Vertex AI onboarding workflow. Follow steps to configure your GCP service account with the right Vertex AI permissions for use with Agent Observability." /%}

**Note**: When you run evaluations, the location selector offers single-region, multi-region, and global options. For details on each option, see [Google's Vertex AI locations documentation](https://docs.cloud.google.com/gemini-enterprise-agent-platform/resources/locations).
{% /tab %}

{% tab title="Custom Provider" %}

{% alert level="danger" %}
If you are subject to HIPAA, you are responsible for ensuring that you only connect to a custom provider that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.
{% /alert %}

Your custom provider endpoint must be compatible with the [OpenAI API specification](https://platform.openai.com/docs/api-reference/introduction).

Connect your custom provider to Agent Observability with your base URL, a way to authenticate, and optional headers. Datadog can authenticate with an API key, or with short-lived OAuth 2.0 access tokens from your identity provider.

1. In Datadog, navigate to [Agent Observability > Settings > Integrations](https://app.datadoghq.com/llm/settings/integrations).
1. Select the Custom Provider for Agent Observability tile.
1. Click the Configure tab, then click New to create a provider.
1. Follow the instructions on the tile.
   - Enter a name for your provider.
   - Select your provider.
   - Provide your base URL.
   - Choose an authentication method and provide its credentials, as described below.
   - Optionally, add any headers.

#### Authenticate with an API key{% #authenticate-with-an-api-key %}

Select API key and provide your API key. Datadog sends it to your provider as `Authorization: Bearer <API_KEY>`.

#### Authenticate with OAuth 2.0 client credentials{% #authenticate-with-oauth-20-client-credentials %}

{% alert level="info" %}
OAuth 2.0 client credentials authentication is in Preview. Contact your Datadog representative to request access.
{% /alert %}

Some providers, such as AI gateways, accept OAuth 2.0 access tokens from your identity provider (IdP). For these providers, Datadog can obtain short-lived tokens with the [client credentials grant](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4) instead of using a static API key. Datadog authenticates to your IdP's token endpoint with a client ID and client secret. It caches the access token until shortly before it expires, and sends it to your provider as `Authorization: Bearer <ACCESS_TOKEN>`. The client secret is sent only to your IdP's token URL, never to your provider.

Select OAuth 2.0 client credentials and provide:

- Token URL: Your IdP's token endpoint. Must use HTTPS.
- Client ID and Client secret: The credentials of the application that represents Datadog in your IdP.
- Scope (optional): Space-separated scopes that your provider requires.
- Audience (optional): Only for IdPs that require an `audience` parameter.

When you save the provider, Datadog requests an access token and calls your provider's `/models` endpoint to verify the connection.

To rotate the client secret, generate a new secret in your IdP, update it in Datadog, and then deactivate the old secret. To revoke Datadog's access, deactivate the application in your IdP. Access tokens that were already issued remain valid until they expire, so keep token lifetimes short.

##### Okta example{% #okta-example %}

The client credentials grant on an Okta custom authorization server requires an Okta plan that supports it.

1. In Okta, go to Security > API > Authorization Servers and use a custom authorization server whose audience matches what your provider validates. Your token URL is `<ISSUER>/v1/token`.
1. On the authorization server's Scopes tab, add a scope for provider access, such as `gateway.invoke`. Okta requires at least one scope for client credentials requests.
1. Go to Applications > Create App Integration, select API Services, and select Use Okta-generated client ID.
   - Under Client Credentials, set Client authentication to Client secret.
   - Under General Settings, turn off Require Demonstrating Proof of Possession (DPoP) header in token requests.
   - Copy the client ID and client secret.
1. On the authorization server's Access Policies tab, add a policy assigned to the application, with a rule that allows the Client Credentials grant type and the scope from step 2.
1. In Datadog, enter the token URL, client ID, client secret, and scope. Leave Audience empty.

{% /tab %}

If your LLM provider restricts IP addresses, you can obtain the required IP ranges by visiting [Datadog's IP ranges documentation](https://docs.datadoghq.com/api/latest/ip-ranges.md), selecting your `Datadog Site`, pasting the `GET` URL into your browser, and copying the `webhooks` section.