Datadog-Elastic Beanstalk Integration

Overview

AWS Elastic Beanstalk is an easy-to-use service for deploying and scaling web applications and services developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker on familiar servers such as Apache, Nginx, Passenger, and IIS.

Setup

Installation

If you haven’t already, set up the Amazon Web Services integration first.

To receive Elastic Beanstalk metrics, you must enable the Enhanced Health Reporting feature for your environment, and configure your environment to publish enhanced health metrics to CloudWatch.

Note: This will increase your CloudWatch custom metric charges.

Configuration

Monitor Elastic Beanstalk environments with the Datadog agent container

If you use Docker containers in a Beanstalk environment, and want to monitor your Docker usage in this environment, the containerized Datadog agent is a tool of choice.

Read on to understand how to configure your Beanstalk environment to integrate the Datadog agent container.

Task definition

To run docker environments with multiple containers per instance, Elastic Beanstalk relies on Amazon EC2 Container Service (ECS). For this reason you need to describe the containers you want to deploy the ECS-way. Elastic Beanstalk allows you to do so through a file called Dockerrun.aws.json.

A Dockerrun.aws.json file is an Elastic Beanstalk–specific JSON file that describes how to deploy a set of Docker containers as an Elastic Beanstalk application. You can use this file for a multicontainer Docker environment.

Dockerrun.aws.json describes the containers to deploy to each container instance in the environment as well as the data volumes to create on the host instance for the containers to mount.

A Dockerrun.aws.json file can be used on its own or zipped up with additional source code in a single archive. Source code that is archived with a Dockerrun.aws.json is deployed to container instances and accessible in the /var/app/current/ directory. Use the volumes section of the config to provide mount points for the containers running on the instance, and the mountPoints section of the embedded container definitions to mount them from the containers.

The following snippet illustrates a Dockerrun.aws.json declaring the Datadog agent. Update it with the definition of your containers to define your own Dockerrun.aws.json. This file can be zipped with additional content to send to the instances which run the described containers, and shipped to Beanstalk. For more info about the syntax of this file you can refer to the Beanstalk documentation.

{
  "AWSEBDockerrunVersion": 2,
  "volumes": [
    {
       "name": "docker_sock",
       "host": {
            "sourcePath": "/var/run/docker.sock"
      }
    },
    {
       "name": "proc",
       "host": {
            "sourcePath": "/proc/"
      }
    },
    {
       "name": "cgroup",
       "host": {
            "sourcePath": "/cgroup/"
      }
    }
  ],
  "containerDefinitions": [
    {
      "name": "dd-agent",
      "image": "datadog/docker-dd-agent:latest",
      "environment": [
            {
              "name": "API_KEY",
              "value": "your_api_key"
            },
            {
              "name": "TAGS",
              "value": "simple-tag-0, tag-key-1:tag-value-1"
            }
      ],
      "memory": 128,
      "mountPoints": [
        {
          "sourceVolume": "docker_sock",
          "containerPath": "/var/run/docker.sock",
          "readOnly": false
        },
        {
          "sourceVolume": "proc",
          "containerPath": "/host/proc",
          "readOnly": true
        },
        {
          "sourceVolume": "cgroup",
          "containerPath": "/host/sys/fs/cgroup",
          "readOnly": true
        }
      ]
    }
  ]
}

Creating the environment

Once the container definition is ready, the last step is to ship it to Beanstalk. This step is explained in the multicontainer Docker tutorial of the Elastic Beanstalk documentation.

Data Collected

Metrics

aws.elasticbeanstalk.application_latency_p_1_0
(gauge)
The average time to complete the fastest 10 percent of requests.
shown as second
aws.elasticbeanstalk.application_latency_p_5_0
(gauge)
The average time to complete the fastest 50 percent of requests.
shown as second
aws.elasticbeanstalk.application_latency_p_7_5
(gauge)
The average time to complete the fastest 75 percent of requests.
shown as second
aws.elasticbeanstalk.application_latency_p_8_5
(gauge)
The average time to complete the fastest 85 percent of requests.
shown as second
aws.elasticbeanstalk.application_latency_p_9_0
(gauge)
The average time to complete the fastest 90 percent of requests.
shown as second
aws.elasticbeanstalk.application_latency_p_9_5
(gauge)
The average time to complete the fastest 95 percent of requests.
shown as second
aws.elasticbeanstalk.application_latency_p_9_9
(gauge)
The average time to complete the fastest 99 percent of requests.
shown as second
aws.elasticbeanstalk.application_latency_p_9_9_9
(gauge)
The average time to complete the fastest 99.9 percent of requests.
shown as second
aws.elasticbeanstalk.application_requests_2xx
(count)
The number of requests that completed with a 2XX status code.
shown as request
aws.elasticbeanstalk.application_requests_3xx
(count)
The number of requests that completed with a 3XX status code.
shown as request
aws.elasticbeanstalk.application_requests_4xx
(count)
The number of requests that completed with a 4XX status code.
shown as request
aws.elasticbeanstalk.application_requests_5xx
(count)
The number of requests that completed with a 5XX status code.
shown as request
aws.elasticbeanstalk.application_requests_total
(count)
The number of requests completed by the instance or environment.
shown as request
aws.elasticbeanstalk.cpuidle
(gauge)
[Instance] The percentage of time the CPU was in the idle state in the last minute.
shown as percent
aws.elasticbeanstalk.cpuiowait
(gauge)
[Instance] The percentage of time the CPU was in the iowait state in the last minute.
shown as percent
aws.elasticbeanstalk.cpuirq
(gauge)
[Instance] The percentage of time the CPU was in the interrupt request state in the last minute.
shown as percent
aws.elasticbeanstalk.cpunice
(gauge)
[Instance] The percentage of time the CPU was in the nice state in the last minute.
shown as percent
aws.elasticbeanstalk.cpusoftirq
(gauge)
[Instance] The percentage of time the CPU was in the soft interrupt request state in the last minute.
shown as percent
aws.elasticbeanstalk.cpusystem
(gauge)
[Instance] The percentage of time the CPU was in the system state in the last minute.
shown as percent
aws.elasticbeanstalk.cpuuser
(gauge)
[Instance] The percentage of time the CPU was in the user state in the last minute.
shown as percent
aws.elasticbeanstalk.environment_health
(gauge)
[Environment] The health status of the environment. The possible values are 0 (OK) 1 (Info) 5 (Unknown) 10 (No data) 15 (Warning) 20 (Degraded) and 25 (Severe).
shown as
aws.elasticbeanstalk.instance_health
(gauge)
[Instance] The health status of the instance.
shown as instance
aws.elasticbeanstalk.instances_degraded
(count)
[Environment] The number of instances with Degraded health status.
shown as instance
aws.elasticbeanstalk.instances_info
(count)
[Environment] The number of instances with Info health status.
shown as instance
aws.elasticbeanstalk.instances_no_data
(count)
[Environment] The number of instances with no health status data.
shown as instance
aws.elasticbeanstalk.instances_ok
(count)
[Environment] The number of instances with OK health status.
shown as instance
aws.elasticbeanstalk.instances_pending
(count)
[Environment] The number of instances with Pending health status.
shown as instance
aws.elasticbeanstalk.instances_severe
(count)
[Environment] The number of instances with Severe health status.
shown as instance
aws.elasticbeanstalk.instances_unknown
(count)
[Environment] The number of instances with Unknown health status.
shown as instance
aws.elasticbeanstalk.instances_warning
(count)
[Environment] The number of instances with Warning health status.
shown as instance
aws.elasticbeanstalk.load_average_1min
(gauge)
[Instance] The average CPU load over the last minute.
shown as
aws.elasticbeanstalk.load_average_5min
(gauge)
[Instance] The average CPU load over the last five minutes.
shown as
aws.elasticbeanstalk.root_filesystem_util
(gauge)
[Instance] The percentage of disk space in use.
shown as percent

Each of the metrics retrieved from AWS will be assigned the same tags that appear in the AWS console, including but not limited to host name, security-groups, and more.