Connect to Datadog over AWS PrivateLink

Connect to Datadog over AWS PrivateLink

Datadog PrivateLink does not support the Datadog for Government site.

This guide walks you through how to configure AWS PrivateLink for use with Datadog.

Overview

The overall process consists of configuring an internal endpoint in your VPC that local Datadog Agents can send data to. Your VPC endpoint is then peered with the endpoint within Datadog’s VPC.

Setup

Datadog exposes AWS PrivateLink endpoints in us-east-1.

However, to route traffic to Datadog’s PrivateLink offering in us-east-1 from other regions, use inter-region Amazon VPC peering. Inter-region VPC peering enables you to establish connections between VPCs across different AWS regions. This allows VPC resources in different regions to communicate with each other using private IP addresses. For more details, see Amazon VPC peering.

  1. Connect the AWS Console to region us-east-1 and create a VPC endpoint.
  1. Select Find service by name.
  2. Fill the Service Name text box according to which service you want to establish AWS PrivateLink for:
Datadog PrivateLink service name Private DNS name
Logs (Agent HTTP intake) com.amazonaws.vpce.us-east-1.vpce-svc-025a56b9187ac1f63 agent-http-intake.logs.datadoghq.com
Logs (User HTTP intake) com.amazonaws.vpce.us-east-1.vpce-svc-0e36256cb6172439d http-intake.logs.datadoghq.com
API com.amazonaws.vpce.us-east-1.vpce-svc-064ea718f8d0ead77 api.datadoghq.com
Metrics com.amazonaws.vpce.us-east-1.vpce-svc-09a8006e245d1e7b8 metrics.agent.datadoghq.com
Containers com.amazonaws.vpce.us-east-1.vpce-svc-0ad5fb9e71f85fe99 orchestrator.datadoghq.com
Process com.amazonaws.vpce.us-east-1.vpce-svc-0ed1f789ac6b0bde1 process.datadoghq.com
Traces com.amazonaws.vpce.us-east-1.vpce-svc-0355bb1880dfa09c2 trace.agent.datadoghq.com
  1. Click Verify. If this does not return Service name found, reach out to Datadog support.
  2. Choose the VPC and subnets that should be peered with the Datadog VPC service endpoint.
  3. Make sure that for Enable DNS name, Enable for this endpoint is checked:
  1. Choose the security group of your choice to control what can send traffic to this VPC endpoint.

    Note: The security group must accept inbound traffic on TCP port 443.

  2. Click Create endpoint at the bottom of the screen. If successful, the following is displayed:

  1. Click on the VPC endpoint ID to check its status.
  2. Wait for the status to move from Pending to Available. This can take up to 10 minutes. Once it shows Available, you can use AWS PrivateLink.
  1. If you are collecting logs data, ensure your Agent is configured to send logs over HTTPS. If the data is not already there, add the following to the Agent datadog.yaml configuration file:

    logs_config:
        use_http: true
    

    If you are using the container Agent, set the following environment variable instead:

    DD_LOGS_CONFIG_USE_HTTP=true
    

    This configuration is required when sending logs to Datadog with AWS PrivateLink and the Datadog Agent, and is not required for the Lambda Extension. For more details, see Agent log collection.

  2. Restart your Agent to send data to Datadog through AWS PrivateLink.

Amazon VPC peering

  1. Connect the AWS Console to region us-east-1 and create a VPC endpoint.
  1. Select Find service by name.
  2. Fill the Service Name text box according to the service you want to establish AWS PrivateLink for:
Datadog PrivateLink service name
Metrics com.amazonaws.vpce.us-east-1.vpce-svc-09a8006e245d1e7b8
Logs (Agent HTTP intake) com.amazonaws.vpce.us-east-1.vpce-svc-025a56b9187ac1f63
Logs (User HTTP intake) com.amazonaws.vpce.us-east-1.vpce-svc-0e36256cb6172439d
API com.amazonaws.vpce.us-east-1.vpce-svc-064ea718f8d0ead77
Process com.amazonaws.vpce.us-east-1.vpce-svc-0ed1f789ac6b0bde1
Traces com.amazonaws.vpce.us-east-1.vpce-svc-0355bb1880dfa09c2
Containers com.amazonaws.vpce.us-east-1.vpce-svc-0ad5fb9e71f85fe99
  1. Click Verify. If this does not return Service name found, reach out to Datadog support.

  2. Next, choose the VPC and subnets that should be peered with the Datadog VPC service endpoint. Do not select Enable DNS name as VPC peering requires DNS to be manually configured.

  3. Choose the security group of your choice to control what can send traffic to this VPC endpoint.

    Note: The security group must accept inbound traffic on TCP port 443.

  4. Hit Create endpoint at the bottom of the screen. If successful, the following is displayed:

  1. Click on the VPC endpoint ID to check its status.
  2. Wait for the status to move from Pending to Available. This can take up to 10 minutes.

Amazon Route53

  1. Create a Route53 private hosted zone for each service you have created an AWS PrivateLink endpoint for. Attach the private hosted zone to the VPC in us-east-1.

Use the list below to map service and DNS name to different parts of Datadog:

Datadog PrivateLink service name Private DNS name
Metrics com.amazonaws.vpce.us-east-1.vpce-svc-09a8006e245d1e7b8 metrics.agent.datadoghq.com
Logs (Agent HTTP intake) com.amazonaws.vpce.us-east-1.vpce-svc-025a56b9187ac1f63 agent-http-intake.logs.datadoghq.com
Logs (User HTTP intake) com.amazonaws.vpce.us-east-1.vpce-svc-0e36256cb6172439d http-intake.logs.datadoghq.com
API com.amazonaws.vpce.us-east-1.vpce-svc-064ea718f8d0ead77 api.datadoghq.com
Process com.amazonaws.vpce.us-east-1.vpce-svc-0ed1f789ac6b0bde1 process.datadoghq.com
Traces com.amazonaws.vpce.us-east-1.vpce-svc-0355bb1880dfa09c2 trace.agent.datadoghq.com
Containers com.amazonaws.vpce.us-east-1.vpce-svc-0ad5fb9e71f85fe99 orchestrator.datadoghq.com

You can also find this information by interrogating the AWS API, DescribeVpcEndpointServices, or by using the following CLI command: aws ec2 describe-vpc-endpoint-services --service-names <service-name>.

For example, in the case of the Datadog metrics endpoint:

aws ec2 describe-vpc-endpoint-services --service-names com.amazonaws.vpce.us-east-1.vpce-svc-09a8006e245d1e7b8 | jq '.ServiceDetails[0].PrivateDnsName'

This returns metrics.agent.datadoghq.com, the private hosted zone name that you need in order to create and associate with the VPC which the Agent traffic originates in. Overriding this record grabs all existing dynamic Agent-versioned hostnames.

  1. Within each new Route53 private hosted zone, create an A record with the same name. Toggle the Alias option, then under Route traffic to, choose Alias to VPC endpoint, us-east-1, and enter the DNS name of the VPC endpoint associated with the DNS name.

    Note: To retrieve your DNS name, see the View endpoint service private DNS name configuration documentation.

  1. Configure VPC peering and routing between the VPC in us-east-1 that contains the Datadog PrivateLink endpoints and the VPC in the region where the Datadog Agents run.

  2. If the VPCs are in different AWS accounts, the VPC containing the Datadog Agent must be authorized to associate with the Route53 private hosted zones before continuing. Create a VPC association authorization for each Route53 private hosted zone using the region and VPC ID of the VPC where the Datadog Agent runs. This option is not available in the AWS Console. It must be configured using the AWS CLI, SDK, or API.

  3. Edit the Route53 hosted zone to add the non-us-east-1 VPC.

  1. VPCs that have the Private Hosted Zone (PHZ) attached need to have certain settings toggled on, specifically enableDnsHostnames and enableDnsSupport in the VPCs that the PHZ is associated with. See Considerations when working with a private hosted zone.

  2. Restart the Agent to send data to Datadog through AWS PrivateLink.

Troubleshooting DNS resolution and connectivity

The DNS names should resolve to IP addresses contained within the CIDR block of the VPC in us-east-1, and connections to port 443 should succeed.

If DNS is resolving to public IP addresses, then the Route53 zone has not been associated with the VPC in the alternate region, or the A record does not exist.

If DNS resolves correctly, but connections to port 443 are failing, then VPC peering or routing may be misconfigured, or port 443 may not be allowed outbound to the CIDR block of the VPC in us-east-1.

The VPCs with Private Hosted Zone (PHZ) attached need to have a couple of settings toggled on. Specifically, enableDnsHostnames and enableDnsSupport need to be turned on in the VPCs that the PHZ is associated with. See Amazon VPC settings.

Datadog Agent

  1. If you are collecting logs data, ensure your Agent is configured to send logs over HTTPS. If the data is not already there, add the following to the Agent datadog.yaml configuration file:

    logs_config:
        use_http: true
    

    If you are using the container Agent, set the following environment variable instead:

    DD_LOGS_CONFIG_USE_HTTP=true
    

    This configuration is required when sending logs to Datadog with AWS PrivateLink and the Datadog Agent, and is not required for the Lambda Extension. For more details, see Agent log collection.

  2. Restart the Agent.

Further Reading