Amazon ECS on EC2

Agent Check

Overview

Amazon ECS on EC2 is a highly scalable, high performance container management service for Docker containers running on EC2 instances.

This page covers AWS ECS setup with Datadog Container Agent v6. For other setups, see:

• Datadog Container Agent v5 setup for AWS ECS
• Datadog Host Agent setup with Autodiscovery

Setup

To monitor your ECS containers and tasks with Datadog, run the Agent as a container on every EC2 instance in your ECS cluster. As detailed below, there are a few setup steps:

1. Add an ECS Task
2. Create or Modify your IAM Policy
3. Schedule the Datadog Agent as a Daemon Service

If you don’t have a working EC2 Container Service cluster configured, review the Getting Started section in the ECS documentation.

Metric collection

Create an ECS Task

This task launches the Datadog container. When you need to modify the configuration, update this task definition as described further down in this guide. If you’re using APM, DogStatsD, or logs, set the appropriate flags in the task definition:

• If you are using APM, set portMappings so your downstream containers can ship traces to the Agent service. APM uses port 8126 and TCP to receive traces, so set this as a hostPort in the task’s definition. Note that in order to enable trace collection from other containers, you must ensure that the DD_APM_NON_LOCAL_TRAFFIC environment variable is set to true. Learn more about APM with containers.
• If you are using DogStatsD, set a hostPort of 8125 as UDP in the task’s definition. Note that in order to enable DogStatsD metrics collection from other containers, you must ensure the DD_DOGSTATSD_NON_LOCAL_TRAFFIC environment variable is set to true.
• If you are using logs, refer to the dedicated Log collection documentation.

Double check the security group settings on your EC2 instances. Make sure these ports are not open to the public. Datadog uses the private IP to route to the Agent from the containers.

You may either configure the task using the AWS CLI tools or using the Amazon Web Console.

"healthCheck": {
  "retries": 3,
  "command": ["CMD-SHELL","agent health"],
  "timeout": 5,
  "interval": 30,
  "startPeriod": 15
}

Create or Modify your IAM Policy

Add the following permissions to your Datadog IAM policy to collect Amazon ECS metrics. For more information on ECS policies, review the documentation on the AWS website.

Run the Agent as a Daemon Service

Ideally you want the Datadog Agent to load on one container on each EC2 instance. The easiest way to achieve this is to run the Datadog Agent as a Daemon Service.

Schedule a Daemon Service in AWS using Datadog’s ECS Task

1. Log in to the AWS console and navigate to the ECS Clusters section. Click into your cluster you run the Agent on.
2. Create a new service by clicking the Create button under Services.
3. For launch type, select EC2 then the task definition created previously.
4. For service type, select DAEMON, and enter a Service name. Click Next.
5. Since the Service runs once on each instance, you won’t need a load balancer. Select None. Click Next.
6. Daemon services don’t need Auto Scaling, so click Next Step, and then Create Service.

Dynamic detection and monitoring of running services

Datadog’s Autodiscovery can be used in conjunction with ECS and Docker to automatically discovery and monitor running tasks in your environment.

AWSVPC Mode

For Agent v6.10+, awsvpc mode is supported for both applicative containers and the Agent container, provided:

1. For the apps and the Agent in awsvpc mode, security groups must be set to allow:

   • The Agent’s security group to reach the applicative containers on relevant ports.
   • The Agent’s security group to reach the host instances on TCP port 51678. The ECS Agent container must either run in host network mode (default) or have a port binding on the host.

2. For apps in awsvpc mode and the Agent in bridge mode, security groups must be set to allow the host instances security group to reach the applicative containers on relevant ports.

Resource tag collection

To collect ECS resource tags:

1. Verify your Amazon ECS container instances are associated to an IAM role. This can be done when creating a new cluster with the ECS cluster creation wizard or in the launch configuration used by an autoscaling group.
2. Update the IAM role used by your Amazon ECS container instances with: ecs:ListTagsForResource.

3. Update your datadog-agent-ecs.json file (datadog-agent-ecs1.json if you are using an original Amazon Linux AMI) to enable resource tag collection by adding the following environment variable:

   {
     "name": "DD_ECS_COLLECT_RESOURCE_TAGS_EC2",
     "value": "true"
   }

Notes

• Ensure your the IAM role is associated with your Amazon ECS container instances and not the underlying EC2 instance.
• ECS resource tags can be collected from EC2 instances, but not from AWS Fargate.
• This feature requires Datadog Agent v6.17+ or v7.17+.
• The Agent supports ECS tag collection from the tasks, services, and container instances ECS resources.

Log collection

To collect all logs written by running applications in your ECS containers and send it to your Datadog application:

Activate Log integrations

The source attribute is used to identify the integration to use for each container. Override it directly in your containers labels to start using log integrations. Read Datadog’s Autodiscovery guide for logs in order to learn more about this process.

Process collection

To collect processes information for all your containers and send it to Datadog:

Trace collection

After installing the Datadog Agent, enable trace collection (optional):

1. Set the following parameters in the task definition for the datadog/agent container:

   • Port mapping: host / container port 8126, Protocol tcp:

     containerDefinitions": [
     {
     "name": "datadog-agent",
     "image": "datadog/agent:latest",
     "cpu": 10,
     "memory": 256,
     "essential": true,
     "portMappings": [
     {
       "hostPort": 8126,
       "protocol": "tcp",
       "containerPort": 8126
     }
     ],
     ...

   • For Agent v7.17 or lower, add the following environment variables:

     ...
       "environment": [
         ...
       {
         "name": "DD_APM_ENABLED",
         "value": "true"
       },
       {
         "name": "DD_APM_NON_LOCAL_TRAFFIC",
         "value": "true"
       },
       ...
       ]
     ...

   See all environment variables available for Agent trace collection.

2. Assign the private IP address for each underlying instance your containers are running on in your application container to the DD_AGENT_HOST environment variable. This allows your application traces to be shipped to the Agent. The Amazon’s EC2 metadata endpoint allows discovery of the private IP address. To get the private IP address for each host, curl the following URL:

   curl http://169.254.169.254/latest/meta-data/local-ipv4

   and set the result as your Trace Agent Hostname environment variable for each application container shipping to APM:

   os.environ['DD_AGENT_HOST'] = <EC2_PRIVATE_IP>

   In cases where variables on your ECS application are set at launch time, you must set the hostname as an environment variable with DD_AGENT_HOST. Otherwise, you can set the hostname in your application’s source code for Python, Javascript, or Ruby. For Java and .NET you can set the hostname in the ECS task. For example:

   • Dropdown

   import requests
   from ddtrace import tracer
   
   
   def get_aws_ip():
     r = requests.get('http://169.254.169.254/latest/meta-data/local-ipv4')
     return r.text
   
   tracer.configure(hostname=get_aws_ip())

   For more examples of how to set the Agent hostname in other languages, refer to the change agent hostname documentation.

   const tracer = require('dd-trace');
   const request = require('request');
   
   request('http://169.254.169.254/latest/meta-data/local-ipv4', function(
       error,
       resp,
       body
   ) {
       tracer.init({ hostname: body });
   });
   

   For more examples of how to set the Agent hostname in other languages, refer to the change agent hostname documentation.

   require 'ddtrace'
   require 'net/http'
   
   Datadog.configure do |c|
     c.tracer hostname: Net::HTTP.get(URI('http://169.254.169.254/latest/meta-data/local-ipv4'))
   end

   package main
   
   import (
       "net/http"
       "io/ioutil"
       "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
   )
   
   resp, err := http.Get("http://169.254.169.254/latest/meta-data/local-ipv4")
           bodyBytes, err := ioutil.ReadAll(resp.Body)
           host := string(bodyBytes)
      if err == nil {
           //set the output of the curl command to the DD_Agent_host env
           os.Setenv("DD_AGENT_HOST", host)
            // tell the trace agent the host setting
           tracer.Start(tracer.WithAgentAddr(host))
           defer tracer.Stop()

   Copy this script into the entryPoint field of your ECS task definition, updating the values with your application jar and argument flags.

   "entryPoint": [
     "sh",
     "-c",
     "export DD_AGENT_HOST=$(curl http://169.254.169.254/latest/meta-data/local-ipv4); java -javaagent:/app/dd-java-agent.jar <APPLICATION_ARG_FLAGS> -jar <APPLICATION_JAR_FILE/WAR_FILE>"
   ]

   For more examples of how to set the Agent hostname in other languages, refer to the change agent hostname documentation.

   "entryPoint": [
     "sh",
     "-c",
     "export DD_AGENT_HOST=$(curl http://169.254.169.254/latest/meta-data/local-ipv4); dotnet ${APP_PATH}"
   ]

Network Performance Monitoring collection (Linux only)

1. Follow the above instructions to install the Datadog Agent.
   • If you are installing for the first time, there is a datadog-agent-ecs.json file available, datadog-agent-sysprobe-ecs.json (datadog-agent-sysprobe-ecs1.json if you are using an original Amazon Linux AMI), for use with the above instructions. Note that initial NPM setup requires the CLI, as you cannot add linuxParameters in the AWS UI.

2. If you already have a task definition, update your datadog-agent-ecs.json file (datadog-agent-ecs1.json if you are using an original Amazon Linux AMI) with the following configuration:

   {
   "containerDefinitions": [
    (...)
      "mountPoints": [
        (...)
        {
          "containerPath": "/sys/kernel/debug",
          "sourceVolume": "debug"
        },
        (...)
      ],
      "environment": [
        (...)
        {
          "name": "DD_SYSTEM_PROBE_ENABLED",
          "value": "true"
        }
      ],
      "linuxParameters": {
       "capabilities": {
         "add": [
           "SYS_ADMIN",
           "SYS_RESOURCE",
           "SYS_PTRACE",
           "NET_ADMIN"
         ]
       }
     },
   ],
   "requiresCompatibilities": [
   "EC2"
   ],
   "volumes": [
    (...)
    {
     "host": {
       "sourcePath": "/sys/kernel/debug"
     },
     "name": "debug"
    },
    (...)
   ],
   "family": "datadog-agent-task"
   }

Data Collected

Metrics

Each of the metrics retrieved from AWS is assigned the same tags that appear in the AWS console, including but not limited to host name, security-groups, and more.

Note: Metrics prefixed with ecs.containerinsights.* come from the AWS CloudWatch agent.

Events

To reduce noise, the AWS ECS integration is automatically whitelisted to include only events that contain the following words: drain, error, fail, insufficient memory, pending, reboot, terminate. See example events below:

To remove the whitelist and receive all events from your Datadog AWS ECS integration, reach out to Datadog support.

Service Checks

• aws.ecs.agent_connected: Returns CRITICAL if the Agent cannot connect, otherwise OK.

Troubleshooting

Need help? Contact Datadog support.

Further Reading

Additional helpful documentation, links, and articles:

Key ECS metrics to monitorBLOG ECS Fargate IntegrationDOCUMENTATION