Connect your LLM provider account

This product is not supported for your selected Datadog site. ().

Connect your LLM provider account

Configure the LLM provider you would like to use for bring-your-own-key (BYOK) evaluations. You only have to complete this step once.

    If you are subject to HIPAA, you are responsible for ensuring that you connect only to an OpenAI account that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.

    Connect your OpenAI account to LLM Observability with your OpenAI API key. LLM Observability uses the GPT-4o mini model for evaluations.

    1. In Datadog, navigate to LLM Observability > Settings > Integrations.
    2. Select Connect on the OpenAI tile.
    3. Follow the instructions on the tile.
      • Provide your OpenAI API key. Ensure that this key has write permission for model capabilities.
    4. Enable Use this API key to evaluate your LLM applications.
    The OpenAI configuration tile in LLM Observability. Lists instructions for configuring OpenAI and providing your OpenAI API key.

    LLM Observability does not support data residency for OpenAI.

    If you are subject to HIPAA, you are responsible for ensuring that you connect only to an Azure OpenAI account that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.

    Connect your Azure OpenAI account to LLM Observability with your OpenAI API key. Datadog strongly recommends using the GPT-4o mini model for evaluations. The selected model version must support structured output.

    1. In Datadog, navigate to LLM Observability > Settings > Integrations.
    2. Select Connect on the Azure OpenAI tile.
    3. Follow the instructions on the tile.
      • Provide your Azure OpenAI API key. Ensure that this key has write permission for model capabilities.
      • Provide the Resource Name, Deployment ID, and API version to complete integration.
    The Azure OpenAI configuration tile in LLM Observability. Lists instructions for configuring Azure OpenAI and providing your API Key, Resource Name, Deployment ID, and API Version.
    If you are subject to HIPAA, you are responsible for ensuring that you connect only to an Anthropic account that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.

    Connect your Anthropic account to LLM Observability with your Anthropic API key. LLM Observability uses the Haiku model for evaluations.

    1. In Datadog, navigate to LLM Observability > Settings > Integrations.
    2. Select Connect on the Anthropic tile.
    3. Follow the instructions on the tile.
      • Provide your Anthropic API key. Ensure that this key has write permission for model capabilities.
    The Anthropic configuration tile in LLM Observability. Lists instructions for configuring Anthropic and providing your Anthropic API key.
    If you are subject to HIPAA, you are responsible for ensuring that you connect only to an Amazon Bedrock account that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.

    Connect your Amazon Bedrock account to LLM Observability with your AWS Account. LLM Observability uses the Haiku model for evaluations.

    1. In Datadog, navigate to LLM Observability > Settings > Integrations.

    2. Select Connect on the Amazon Bedrock tile.

    3. Follow the instructions on the tile.

      The Amazon Bedrock configuration tile in LLM Observability. Lists instructions for configuring Amazon Bedrock.

    4. Be sure to configure the Invoke models from Amazon Bedrock role to run evaluations. More details about the InvokeModel action can be found in the Amazon Bedrock API reference documentation.

      The second step in configuring Amazon Bedrock requiring users to add permissions to the integration account.
    If you are subject to HIPAA, you are responsible for ensuring that you connect only to a Google Cloud Platform account that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.

    Connect Vertex AI to LLM Observability with your Google Cloud Platform account. LLM Observability uses the gemini-2.5-flash model for evaluations.

    1. In Datadog, navigate to LLM Observability > Settings > Integrations.
    2. On the Google Cloud Vertex AI tile, click Connect to add a new GCP account, or click Configure next to where your existing accounts are listed to begin the onboarding process.
      • You will see all GCP accounts connected to Datadog in this page. However, you must still go through the onboarding process for an account to use it in LLM Observability.
    3. Follow the onboarding instructions to configure your account.
    The Vertex AI onboarding workflow. Follow steps to configure your GCP service account with the right Vertex AI permissions for use with LLM Observability.
    If you are subject to HIPAA, you are responsible for ensuring that you only connect to an AI Gateway that is subject to a business associate agreement (BAA) and meets all requirements for HIPAA compliance.

    Your AI Gateway must be compatible with the OpenAI API specification.

    Connect your AI Gateway to LLM Observability with your base URL, API key, and headers.

    1. In Datadog, navigate to LLM Observability > Settings > Integrations.
    2. Click the Configure tab, then click New to create a new gateway.
    3. Follow the instructions on the tile.
      • Provide a name for your gateway.
      • Select your provider.
      • Provide your base URL.
      • Provide your API key and optionally any headers.
    The AI Gateway configuration tile in LLM Observability. Lists instructions for configuring an ai gateway

    If your LLM provider restricts IP addresses, you can obtain the required IP ranges by visiting Datadog’s IP ranges documentation, selecting your Datadog Site, pasting the GET URL into your browser, and copying the webhooks section.