Dual Shipping

Dual shipping can impact billing if you are sending data to multiple Datadog organizations. For more information about the impact of this configuration, contact Datadog Support.

Overview

This guide provides examples of Agent configurations for dual shipping different types of data (for example, APM, logs, Cluster Agent metrics) to multiple Datadog organizations and sites. For more information about Datadog sites, see Getting Started with Datadog sites.

Note: Use Observability Pipelines if you want to dual ship logs or split log traffic across different logging vendors, cloud storages, or SIEM providers.

For a full list of network traffic destinations, see Network Traffic.

Metrics and service checks

You can add the YAML configuration to your datadog.yaml or launch the Agent with the appropriate environment variables.

YAML configuration

Requires Agent version >= 6.17 or 7.17.

In datadog.yaml:

additional_endpoints:
  "https://app.":
  - apikey2
  - apikey3
  "https://app.<DD_SITE>": # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.eu).
  - apikey4

Environment variable configuration

Requires Agent version >= 6.18 or 7.18.

DD_ADDITIONAL_ENDPOINTS='{\"https://app.\": [\"apikey2\", \"apikey3\"], \"https://app.<DD_SITE>\": [\"apikey4\"]}' # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.eu).

APM

YAML configuration

Requires Agent version >= 6.7.0.

In datadog.yaml:

apm_config:
  [...]
  additional_endpoints:
    "https://trace.agent.":
    - apikey2
    - apikey3
    "https://trace.agent.<DD_SITE>": # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.eu).
    - apikey4

Environment variable configuration

Requires Agent version >= 6.19 or 7.19.

DD_APM_ADDITIONAL_ENDPOINTS='{\"https://trace.agent.\": [\"apikey2\", \"apikey3\"], \"https://trace.agent.<DD_SITE>\": [\"apikey4\"]}' # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.eu).

Continuous Profiler

YAML configuration

Requires Agent version >= 6.7.0.

In datadog.yaml:

apm_config:
  [...]
  profiling_additional_endpoints:
    "https://intake.profile./api/v2/profile":
    - apikey2
    - apikey3
    "https://intake.profile.<DD_SITE>/api/v2/profile": # Replace "<DD_SITE>" with your Datadog site parameter (for example, datadoghq.eu).
    - apikey4

Environment variable configuration

Requires Agent version >= 6.19 or 7.19.

DD_APM_PROFILING_ADDITIONAL_ENDPOINTS='{\"https://intake.profile./api/v2/profile\": [\"apikey2\", \"apikey3\"], \"https://intake.profile.<DD_SITE>/api/v2/profile\": [\"apikey4\"]}' # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.eu).

Note: Uploads to additional endpoints for the Continuous Profiler product are done through best-effort delivery.

  • The main endpoint has the highest priority. Uploads to additional endpoints are only handled after uploads to the main endpoint have completed successfully.
  • Responses from additional endpoints are not forwarded back to the profiler. Any errors during delivery to additional endpoints are logged in the Agent error logs.

Live Processes

YAML configuration

Requires Agent version >= 6.4.0.

In datadog.yaml:

process_config:
  [...]
  additional_endpoints:
    "https://process.":
    - apikey2
    - apikey3
    "https://process.<DD_SITE>": # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.eu).
    - apikey4

Environment variable configuration

Requires Agent version >= 6.20 or 7.20.

DD_PROCESS_ADDITIONAL_ENDPOINTS='{\"https://process.\": [\"apikey2\", \"apikey3\"], \"https://process.<DD_SITE>\": [\"apikey4\"]}' # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.eu).

Cluster Agent metrics

Configure the Agent to send Cluster Agent metrics, such as Kubernetes State Metrics Core, to additional endpoints.

HELM configuration

In Datadog values.yaml:

clusterAgent:
  env:
    - name: DD_ADDITIONAL_ENDPOINTS
      value: '{"https://app.": ["apikey2"]}'

Cluster Agent metrics provider

To ensure autoscaling is resilient to failure, configure the Cluster Agent to run your metric queries for the HPA against your multiple Datadog regions with dual-shipped data. Configure the Datadog Cluster Agent manifest with several endpoints:

cluster-agent-deployment.yaml

external_metrics_provider:
  endpoints:
  - api_key: <DATADOG_API_KEY>
    app_key: <DATADOG_APP_KEY>
    url: https://app.<DD_SITE>
  - api_key: <DATADOG_API_KEY>
    app_key: <DATADOG_APP_KEY>
    url: https://app.<DD_SITE>

Orchestrator

HELM configuration

In Datadog values.yaml:

agents:
  customAgentConfig:
    process_config:
      additional_endpoints:
        "https://process.":
        - apikey2
    orchestrator_explorer:
      orchestrator_additional_endpoints:
        "https://orchestrator.":
        - apikey3

clusterAgent:
...
  datadog_cluster_yaml:
    orchestrator_explorer:
      orchestrator_additional_endpoints:
        "https://orchestrator.<DD_SITE>": # Replace <DD_SITE> with your Datadog site parameter (for example, ddog-gov.com).
        - apikey4

Environment variable configuration

DD_ORCHESTRATOR_EXPLORER_ORCHESTRATOR_ADDITIONAL_ENDPOINTS='{\"https://orchestrator.\": [\"apikey2\", \"apikey3\"], \"https://orchestrator.<DD_SITE>\": [\"apikey4\"]}' # Replace <DD_SITE> with your Datadog site parameter (for example, ddog-gov.com).

CI Visibility

YAML configuration

Requires Agent >= 6.38 or 7.38.

In datadog.yaml:

evp_proxy_config:
  [...]
  additional_endpoints:
    "https://<VERSION>-app.agent.":
    - apikey2
    - apikey3
    "https://<VERSION>-app.agent.<DD_SITE>":  # Replace <VERSION> and <DD_SITE> with your Agent version and Datadog site parameter (for example, 7-38-0 and datadoghq.eu).
    - apikey4

Environment variable configuration

DD_EVP_PROXY_CONFIG_ADDITIONAL_ENDPOINTS='{\"https://<VERSION>-app.agent.\": [\"apikey2\", \"apikey3\"], \"https://<VERSION>-app.agent.<DD_SITE>\": [\"apikey4\"]}'  # Replace <VERSION> and <DD_SITE> with your Agent version and Datadog site parameter (for example, 7-38-0 and datadoghq.eu).

Logs

Use the Agent if you want to dual ship logs to multiple Datadog organizations. Use Observability Pipelines if you want to send logs to Datadog and external destinations.

TCP requires Agent version >= 6.6.
HTTPS requires Agent version >= 6.13.

YAML configuration

In datadog.yaml:

logs_config:
  force_use_http: true
  additional_endpoints:
  - api_key: "apiKey2"
    Host: "agent-http-intake.logs."
    Port: 443
    is_reliable: true

Environment variable configuration

Requires Agent >= 6.18 or 7.18.

DD_LOGS_CONFIG_FORCE_USE_HTTP=true
DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS="[{\"api_key\": \"apiKey2\", \"Host\": \"agent-http-intake.logs.\", \"Port\": 443, \"is_reliable\": true}]"

When setting up additional endpoints, you must explicitly set use_http to tell the Agent which transport to use. The same transport configuration is shared among all additional endpoints.

The is_reliable setting (First available in Agent 7.34.0) tells the Agent to treat this endpoint with the same priority as the primary endpoint. The primary endpoint is always reliable. This ensures that data is not missed if a destination becomes unavailable.

For example, if you’re sending data to the main endpoint and an additional endpoint with is_reliable: true, and one endpoint becomes unavailable, data continues to flow to the other endpoint. If both endpoints become unavailable, the Agent stops reading and sending data until at least one endpoint recovers. This ensures all data makes it to at least one reliable endpoint.

The is_reliable setting defaults to true in Agent 7.37.0+. Unreliable endpoints only send data if at least one reliable endpoint is available. You may define multiple additional endpoints with a mixed usage of is_reliable values. Datadog recommends that you use the default is_reliable setting.

You can add the YAML configuration to your datadog.yaml or launch the Agent with the appropriate environment variables.

Database Monitoring

YAML configuration

Requires Agent >= 6.29 or 7.29.

In datadog.yaml:

database_monitoring:
  samples:
    force_use_http: true
    additional_endpoints:
    - api_key: "apiKey2"
      Host: "dbm-metrics-intake."
      Port: 443
      is_reliable: true
  activity:
    force_use_http: true
    additional_endpoints:
    - api_key: "apiKey2"
      Host: "dbquery-intake."
      Port: 443
      is_reliable: true
  metrics:
    force_use_http: true
    additional_endpoints:
    - api_key: "apiKey2"
      Host: "dbm-metrics-intake."
      Port: 443
      is_reliable: true

Environment variable configuration

DD_DATABASE_MONITORING_SAMPLES_USE_HTTP=true
DD_DATABASE_MONITORING_SAMPLES_ADDITIONAL_ENDPOINTS="[{\"api_key\": \"apiKey2\", \"Host\": \"dbm-metrics-intake.\", \"Port\": 443, \"is_reliable\": true}]"
DD_DATABASE_MONITORING_ACTIVITY_USE_HTTP=true
DD_DATABASE_MONITORING_ACTIVITY_ADDITIONAL_ENDPOINTS="[{\"api_key\": \"apiKey2\", \"Host\": \"dbquery-intake.\", \"Port\": 443, \"is_reliable\": true}]"
DD_DATABASE_MONITORING_METRICS_USE_HTTP=true
DD_DATABASE_MONITORING_METRICS_ADDITIONAL_ENDPOINTS="[{\"api_key\": \"apiKey2\", \"Host\": \"dbm-metrics-intake.\", \"Port\": 443, \"is_reliable\": true}]"

When setting up additional endpoints, you must explicitly set use_http to tell the Agent which transport to use. The same transport configuration is shared among all additional endpoints.

The is_reliable setting (First available in Agent 7.34.0) tells the Agent to treat this endpoint with the same priority as the primary endpoint. The primary endpoint is always reliable. This ensures that data is not missed if a destination becomes unavailable.

For example, if you’re sending data to the main endpoint and an additional endpoint with is_reliable: true, and one endpoint becomes unavailable, data continues to flow to the other endpoint. If both endpoints become unavailable, the Agent stops reading and sending data until at least one endpoint recovers. This ensures all data makes it to at least one reliable endpoint.

The is_reliable setting defaults to true in Agent 7.37.0+. Unreliable endpoints only send data if at least one reliable endpoint is available. You may define multiple additional endpoints with a mixed usage of is_reliable values. Datadog recommends that you use the default is_reliable setting.

You can add the YAML configuration to your datadog.yaml or launch the Agent with the appropriate environment variables.

Network Devices

YAML configuration

Requires Agent >= 6.29 or 7.29.

In datadog.yaml:

network_devices:
  metadata:
    force_use_http: true
    additional_endpoints:
    - api_key: "apiKey2"
      Host: "ndm-intake."
      Port: 443
      is_reliable: true
  snmp_traps:
    forwarder:
      force_use_http: true
      additional_endpoints:
      - api_key: "apiKey2"
        Host: "ndm-intake."
        Port: 443
        is_reliable: true
  netflow:
    forwarder:
      force_use_http: true
      additional_endpoints:
      - api_key: "apiKey2"
        Host: "ndm-intake."
        Port: 443
        is_reliable: true

Environment variable configuration

DD_NETWORK_DEVICES_METADATA_USE_HTTP=true
DD_NETWORK_DEVICES_METADATA_ADDITIONAL_ENDPOINTS="[{\"api_key\": \"apiKey2\", \"Host\": \"ndm-intake.\", \"Port\": 443, \"is_reliable\": true}]"

When setting up additional endpoints, you must explicitly set use_http to tell the Agent which transport to use. The same transport configuration is shared among all additional endpoints.

The is_reliable setting (First available in Agent 7.34.0) tells the Agent to treat this endpoint with the same priority as the primary endpoint. The primary endpoint is always reliable. This ensures that data is not missed if a destination becomes unavailable.

For example, if you’re sending data to the main endpoint and an additional endpoint with is_reliable: true, and one endpoint becomes unavailable, data continues to flow to the other endpoint. If both endpoints become unavailable, the Agent stops reading and sending data until at least one endpoint recovers. This ensures all data makes it to at least one reliable endpoint.

The is_reliable setting defaults to true in Agent 7.37.0+. Unreliable endpoints only send data if at least one reliable endpoint is available. You may define multiple additional endpoints with a mixed usage of is_reliable values. Datadog recommends that you use the default is_reliable setting.

You can add the YAML configuration to your datadog.yaml or launch the Agent with the appropriate environment variables.

Network Path

YAML configuration

Requires Agent >= 6.55 or 7.55.

In datadog.yaml:

network_path:
  forwarder:
    use_http: true
    additional_endpoints:
    - api_key: "apiKey2"
      Host: "netpath-intake."
      Port: 443
      is_reliable: true

Environment variable configuration

DD_NETWORK_PATH_FORWARDER_USE_HTTP=true
DD_NETWORK_PATH_FORWARDER_ADDITIONAL_ENDPOINTS="[{\"api_key\": \"apiKey2\", \"Host\": \"netpath-intake.\", \"Port\": 443, \"is_reliable\": true}]"

When setting up additional endpoints, you must explicitly set use_http to tell the Agent which transport to use. The same transport configuration is shared among all additional endpoints.

The is_reliable setting (First available in Agent 7.34.0) tells the Agent to treat this endpoint with the same priority as the primary endpoint. The primary endpoint is always reliable. This ensures that data is not missed if a destination becomes unavailable.

For example, if you’re sending data to the main endpoint and an additional endpoint with is_reliable: true, and one endpoint becomes unavailable, data continues to flow to the other endpoint. If both endpoints become unavailable, the Agent stops reading and sending data until at least one endpoint recovers. This ensures all data makes it to at least one reliable endpoint.

The is_reliable setting defaults to true in Agent 7.37.0+. Unreliable endpoints only send data if at least one reliable endpoint is available. You may define multiple additional endpoints with a mixed usage of is_reliable values. Datadog recommends that you use the default is_reliable setting.

You can add the YAML configuration to your datadog.yaml or launch the Agent with the appropriate environment variables.

Cloud Security Misconfigurations

YAML configuration

In datadog.yaml:

compliance_config:
  endpoints:
    force_use_http: true
    additional_endpoints:
    - api_key: "apiKey2"
      Host: "https://<VERSION>-app.agent."  # Replace <VERSION> with your Datadog Agent version (for example, 7-34-0).
      Port: 443
      is_reliable: true

Environment variable configuration

DD_COMPLIANCE_CONFIG_ENDPOINTS_USE_HTTP=true
DD_COMPLIANCE_CONFIG_ENDPOINTS_ADDITIONAL_ENDPOINTS="[{\"api_key\": \"apiKey2\", \"Host\": \"https://<VERSION>-app.agent.\", \"Port\": 443, \"is_reliable\": true}]"  # Replace <VERSION> with your Datadog Agent version (for example, 7-34-0).

When setting up additional endpoints, you must explicitly set use_http to tell the Agent which transport to use. The same transport configuration is shared among all additional endpoints.

The is_reliable setting (First available in Agent 7.34.0) tells the Agent to treat this endpoint with the same priority as the primary endpoint. The primary endpoint is always reliable. This ensures that data is not missed if a destination becomes unavailable.

For example, if you’re sending data to the main endpoint and an additional endpoint with is_reliable: true, and one endpoint becomes unavailable, data continues to flow to the other endpoint. If both endpoints become unavailable, the Agent stops reading and sending data until at least one endpoint recovers. This ensures all data makes it to at least one reliable endpoint.

The is_reliable setting defaults to true in Agent 7.37.0+. Unreliable endpoints only send data if at least one reliable endpoint is available. You may define multiple additional endpoints with a mixed usage of is_reliable values. Datadog recommends that you use the default is_reliable setting.

You can add the YAML configuration to your datadog.yaml or launch the Agent with the appropriate environment variables.

Workload Protection

YAML configuration

In datadog.yaml:

runtime_security_config:
  endpoints:
    force_use_http: true
    additional_endpoints:
    - api_key: "apiKey2"
      Host: "https://<VERSION>-app.agent."  # Replace <VERSION> with your Datadog Agent version (for example, 7-34-0).
      Port: 443
      is_reliable: true

Environment variable configuration

DD_RUNTIME_SECURITY_CONFIG_ENDPOINTS_USE_HTTP=true
DD_RUNTIME_SECURITY_CONFIG_ENDPOINTS_ADDITIONAL_ENDPOINTS="[{\"api_key\": \"apiKey2\", \"Host\": \"https://<VERSION>-app.agent.\", \"Port\": 443, \"is_reliable\": true}]"  # Replace <VERSION> with your Datadog Agent version (for example, 7-34-0).

When setting up additional endpoints, you must explicitly set use_http to tell the Agent which transport to use. The same transport configuration is shared among all additional endpoints.

The is_reliable setting (First available in Agent 7.34.0) tells the Agent to treat this endpoint with the same priority as the primary endpoint. The primary endpoint is always reliable. This ensures that data is not missed if a destination becomes unavailable.

For example, if you’re sending data to the main endpoint and an additional endpoint with is_reliable: true, and one endpoint becomes unavailable, data continues to flow to the other endpoint. If both endpoints become unavailable, the Agent stops reading and sending data until at least one endpoint recovers. This ensures all data makes it to at least one reliable endpoint.

The is_reliable setting defaults to true in Agent 7.37.0+. Unreliable endpoints only send data if at least one reliable endpoint is available. You may define multiple additional endpoints with a mixed usage of is_reliable values. Datadog recommends that you use the default is_reliable setting.

You can add the YAML configuration to your datadog.yaml or launch the Agent with the appropriate environment variables.

Dual shipping in Kubernetes

    If you’re using the Datadog Agent Helm chart, you can configure these settings with a configmap. In the values.yaml, set useConfigMap: true and add the relevant settings to customAgentConfig.

    # agents.useConfigMap -- Configures a configmap to provide the agent configuration. Use this in combination with the `agents.customAgentConfig` parameter.
  useConfigMap:  true

  # agents.customAgentConfig -- Specify custom contents for the datadog agent config (datadog.yaml)
  ## ref: https://docs.datadoghq.com/agent/configuration/agent-configuration-files/?tab=agentv6
  ## ref: https://github.com/DataDog/datadog-agent/blob/main/pkg/config/config_template.yaml
  ## Note the `agents.useConfigMap` needs to be set to `true` for this parameter to be taken into account.
  customAgentConfig:
    additional_endpoints:
      "https://app.<DD_SITE>":  # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.com).
      - apikey2
      - apikey3
      "https://app.<DD_SITE>":  # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.eu).
      - apikey4

    logs_config:
      force_use_http: true
      additional_endpoints:
      - api_key: "apiKey2"
        Host: "agent-http-intake.logs.<DD_SITE>" # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.com).
        Port: 443
        is_reliable: true

    To avoid exposing your API key(s) in clear text inside the ConfigMap, you can also use the environment variable configuration and reference a Kubernetes secret. Here is an example to send metrics to an additional region:

    1. Create a Kubernetes secret with your environment variable configuration value from this guide:
      kubectl create -n <DATADOG AGENT NAMESPACE> secret generic dual-shipping --from-literal metrics='{"https://app.<DD_SITE>": ["apikey4"]}' # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.eu).
    2. Use the Helm chart parameters datadog.env or datadog.envFrom to reference this secret in your configuration:
      datadog:
  [...]
  env:
  - name: DD_ADDITIONAL_ENDPOINTS
    valueFrom:
      secretKeyRef:
        name: dual-shipping
        key: metrics

    If you’re using the Datadog Agent operator, you can set the [key].customConfigurations.[key].configData override key to set these settings. The example below replaces the datadog.yaml configuration file of the node Agent to send metrics and logs to additional regions.

    apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  override:
    nodeAgent:
      customConfigurations:
        datadog.yaml:
          ## Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.com (US1) for `apikey2` and `apikey3`, and datadoghq.eu (EU) for `apikey4`).
          configData: |-
            additional_endpoints:
              "https://app.<DD_SITE>":  
              - apikey2
              - apikey3
              "https://app.<DD_SITE>":  
              - apikey4
            logs_config:
              force_use_http: true
              additional_endpoints:
              - api_key: "apiKey2"
                Host: "agent-http-intake.logs.<DD_SITE>"
                Port: 443
                is_reliable: true

    To avoid exposing your API key(s) in clear text inside the ConfigMap, you can also use the environment variable configuration and reference a Kubernetes secret. Here is an example to send metrics to an additional region:

    1. Create a Kubernetes secret with your environment variable configuration value from this guide:
      kubectl create -n <DATADOG AGENT NAMESPACE> secret generic dual-shipping --from-literal metrics='{"https://app.<DD_SITE>": ["apikey4"]}'  # Replace <DD_SITE> with your Datadog site parameter (for example, datadoghq.eu).
    2. Use the [key].env parameter to reference this secret in your configuration:
      apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  override:
    nodeAgent:
      env:
      - name: DD_ADDITIONAL_ENDPOINTS
        valueFrom:
          secretKeyRef:
            name: dual-shipping
            key: metrics

    Further reading

    Additional helpful documentation, links, and articles:

    Centralize and govern your OpenTelemetry pipeline with the DDOT gatewayBLOG more more Network TrafficGUIDE more more Send logs to external destinations with Observability PipelinesDOCUMENTATION more more