Amazon ECS Managed Instances

Docs > Containers > Amazon ECS > Amazon ECS Managed Instances

Datadog Container Monitoring enables visibility into applications running on Amazon ECS Managed Instances.

How it works

To monitor your ECS Managed Instances with Datadog, run the Datadog Agent as a daemon task. A daemon task allows ECS to automatically deploy to each host as managed instances join your cluster. This enables full visibility into all containers on the instance without requiring changes to your application task definitions.

Setup

This setup requires Datadog Agent 7.77.0+.

The following instructions assume that you have configured an ECS Managed Instances cluster. See the Amazon ECS Managed Instances documentation for creating a cluster.

Create a daemon task definition file
Register the daemon task definition
Create the daemon

Create a daemon task definition file

This ECS daemon task definition launches the Datadog Agent container with the necessary configurations.

Download datadog-agent-ecs-managed-instances-daemon.json. This file provides minimal configuration for core infrastructure monitoring. For more sample task definition files with various features enabled, see the Set up additional Agent features section on this page.
Modify the task definition file:
- Set the DD_API_KEY environment variable by replacing <YOUR_DATADOG_API_KEY> with the Datadog API key for your account. Alternatively, you can also supply the ARN of a secret stored in AWS Secrets Manager.
- Set the DD_SITE environment variable to your Datadog site. Your site is:

(Optional) To add an Agent health check, add the following to your ECS task definition:

"healthCheck": {
  "retries": 3,
  "command": ["CMD-SHELL","agent health"],
  "timeout": 5,
  "interval": 30,
  "startPeriod": 15
}

Register the daemon task definition

Log in to your AWS Console and navigate to the Elastic Container Service section.
Select Task Definitions in the navigation pane. On the Create new task definition menu, select Create new task definition with JSON.
In the JSON editor box, paste the contents of your daemon task definition file.
Select Create.

Use the AWS CLI to execute the following command:

aws ecs register-daemon-task-definition --cli-input-json file://<path to datadog-agent-ecs-managed-instances-daemon.json>

Create the daemon

The daemon is attached to a capacity provider and deploys automatically to each managed instance in that capacity provider.

Log in to your AWS Web Console and navigate to the Elastic Container Service section.
Choose the cluster to run the Datadog Agent on.
On the Daemons tab, click Create.
For Daemon Task Definition, select the task definition created in the previous steps.
Enter a Daemon name.
For Capacity Provider, choose the capacity provider tied to the cluster.
Fill in the rest of the optional fields based on your preference.
Click Create.

Use the AWS CLI to execute the following command:

aws ecs create-daemon \
--daemon-name <DAEMON_NAME> \
--daemon-task-definition-arn <DAEMON_TASK_DEFINITION_ARN> \
--capacity-provider-arns <CAPACITY_PROVIDER_ARN>

Set up additional Datadog Agent features

Metrics collection

To enable integrations, add Docker label annotations to your application containers in the ECS task definition.

Add an integration

Update the task definition

Log in to your AWS Web Console and navigate to the ECS section.
Choose the cluster the Datadog Agent is running on.
Click the Tasks tab, then click the Task definition name containing the Datadog Agent container.
Click the Create new revision button.
Select the application container you want to monitor and click Edit.
Under Docker labels, add the following:

Key	Value
com.datadoghq.ad.instances	`[{"host": "%%host%%", "port": <PORT_NUMBER>}]`
com.datadoghq.ad.check_names	`["<CHECK_NAME>"]`
com.datadoghq.ad.init_configs	`[{}]`

Click the Update button, then click the Create button.

Add dockerLabels to your application container in the task definition JSON, then register the new revision using the AWS CLI:

{
  "containerDefinitions": [
    {
      "name": "<APP_CONTAINER_NAME>",
      "dockerLabels": {
        "com.datadoghq.ad.instances": "[{\"host\": \"%%host%%\", \"port\": <PORT_NUMBER>}]",
        "com.datadoghq.ad.check_names": "[\"<CHECK_NAME>\"]",
        "com.datadoghq.ad.init_configs": "[{}]"
      }
    }
  ]
}

aws ecs register-task-definition --cli-input-json file://<path-to-task-definition.json>

Update the service

Within the cluster, click the Services tab, then click the Service Name.
Click the Update button.
For Task Definition, choose the latest Revision from the dropdown menu.
Click the Update Service button.

Use the AWS CLI to update the service with the new task definition revision:

aws ecs update-service --cluster <CLUSTER_NAME> \
--service <SERVICE_NAME> \
--task-definition <TASK_DEFINITION_ARN>

Examples

Use the following table to enter the Docker labels with the AWS Web Console for a Redis container:

Key	Value
com.datadoghq.ad.instances	`[{"host": "%%host%%", "port": 6379}]`
com.datadoghq.ad.check_names	`["redisdb"]`
com.datadoghq.ad.init_configs	`[{}]`

Use the following JSON under containerDefinitions to configure a Redis container with Docker labels through the AWS CLI:

{
  "name": "redis",
  "image": "redis:latest",
  "essential": true,
  "dockerLabels": {
    "com.datadoghq.ad.instances": "[{\"host\": \"%%host%%\", \"port\": 6379}]",
    "com.datadoghq.ad.check_names": "[\"redisdb\"]",
    "com.datadoghq.ad.init_configs": "[{}]"
  }
}

Alternative: Mount a configuration file

To provide a custom integration configuration, you can also mount a configuration file directly onto the Datadog Agent container.

Create the following file structure:

|- datadog
  |- Dockerfile
  |- conf.d
    |-redis.yaml

The redis.yaml file contains the configurations for the Redis integration.

redis.yaml

ad_identifiers:
  - redis

init_config:

instances:
    - host: %%host%%
      port: 6379

The Dockerfile is used to build a Datadog Agent image and include the redis.yaml file at the correct location:

Dockerfile

FROM public.ecr.aws/datadog/agent:latest

COPY conf.d/ /etc/datadog-agent/conf.d/

After the image is built and pushed to an image registry, reference the custom image in the ECS task definition:

{
    "containerDefinitions": [
        {
            "image": "<registry-domain>/<namespace-or-account>/<repository>:<tag>",
            "name": "datadog-agent",
            ...
        }
    ],
    ...
}

Trace collection (APM)

Instrument your application based on your setup:

Language
Java
Python
Ruby
Go
Node.js
PHP
C++
.NET Core
.NET Framework

UDS

The recommended method is Unix Domain Socket (UDS). The daemon Agent exposes its socket on the host filesystem, which application tasks access by mounting the same host path.

Consult the sample datadog-agent-ecs-managed-instances-daemon-apm.json file for a complete daemon task definition.

Update the daemon task definition to add the dd-sockets host volume and mount:

{
    "containerDefinitions": [
        {
            "name": "datadog-agent",
            ...
            "mountPoints": [
                ...
                {
                    "containerPath": "/var/run/datadog",
                    "readOnly": false,
                    "sourceVolume": "dd-sockets"
                }
            ]
        }
    ],
    "volumes": [
        ...
        {
            "host": {
                "sourcePath": "/var/run/datadog"
            },
            "name": "dd-sockets"
        }
    ]
}

The Agent maintains socket files at /var/run/datadog/apm.socket and /var/run/datadog/dsd.socket.

Update each application task definition to mount the same host path and configure the tracer:

{
    "containerDefinitions": [
        {
            "name": "<APP_CONTAINER_NAME>",
            ...
            "environment": [
                {
                    "name": "DD_TRACE_AGENT_URL",
                    "value": "unix:///var/run/datadog/apm.socket"
                }
            ],
            "mountPoints": [
                {
                    "containerPath": "/var/run/datadog",
                    "readOnly": true,
                    "sourceVolume": "dd-sockets"
                }
            ]
        }
    ],
    "volumes": [
        {
            "host": {
                "sourcePath": "/var/run/datadog"
            },
            "name": "dd-sockets"
        }
    ]
}

Log collection

Container log collection through the agent is not supported in daemon mode on ECS Managed Instances.

To collect container logs, use one of the following alternatives:

AWS FireLens: Configure a Fluent Bit sidecar in your application task definitions. See the ECS Fargate log collection documentation for setup instructions.
awslogs log driver: Route logs to Amazon CloudWatch Logs and forward them to Datadog using the CloudWatch Logs Lambda forwarder or a subscription filter.

Process collection

To collect Live Process information for all your containers and send it to Datadog, update the daemon task definition with the DD_PROCESS_AGENT_ENABLED environment variable:

{
    "containerDefinitions": [
        {
            "name": "datadog-agent",
            ...
            "environment": [
                ...
                {
                    "name": "DD_PROCESS_AGENT_ENABLED",
                    "value": "true"
                }
            ]
        }
    ]
}

Cloud Network Monitoring

This feature is only available for Linux.

Consult the sample datadog-agent-ecs-managed-instances-daemon-sysprobe.json file for a complete task definition.

Update your existing daemon task definition to include the following configuration:

{
    "containerDefinitions": [
        {
            "name": "datadog-agent",
            ...
            "environment": [
                ...
                {
                    "name": "DD_SYSTEM_PROBE_NETWORK_ENABLED",
                    "value": "true"
                }
            ],
            "linuxParameters": {
                "capabilities": {
                    "add": [
                        "SYS_ADMIN",
                        "SYS_RESOURCE",
                        "SYS_PTRACE",
                        "NET_ADMIN",
                        "NET_BROADCAST",
                        "NET_RAW",
                        "IPC_LOCK",
                        "CHOWN"
                    ]
                }
            },
            "mountPoints": [
                ...
                {
                    "containerPath": "/sys/kernel/debug",
                    "sourceVolume": "debug",
                    "readOnly": false
                }
            ]
        }
    ],
    "volumes": [
        ...
        {
            "name": "debug",
            "host": {
                "sourcePath": "/sys/kernel/debug"
            }
        }
    ]
}

For more information, see the Cloud Network Monitoring documentation.

Migrate from sidecar to daemon setup

Follow these steps to migrate from the legacy sidecar deployment to the daemon deployment.

1. Set up the daemon

Follow the Setup instructions on this page to create and register a daemon task definition, then create the daemon on your cluster.

2. Remove the Datadog Agent sidecar from application task definitions

For each application task definition that includes a datadog-agent sidecar container, remove the datadog-agent entry from containerDefinitions.

3. Update APM configuration

The daemon Agent exposes its trace socket on the host filesystem rather than through a shared in-task volume. Update each application task definition to use the host path.

Remove the shared empty volume ("host": {}) that was used in the sidecar setup and replace it with a host path volume:

{
    "containerDefinitions": [
        {
            "name": "<APP_CONTAINER_NAME>",
            "environment": [
                {
                    "name": "DD_TRACE_AGENT_URL",
                    "value": "unix:///var/run/datadog/apm.socket"
                }
            ],
            "mountPoints": [
                {
                    "containerPath": "/var/run/datadog",
                    "readOnly": true,
                    "sourceVolume": "dd-sockets"
                }
            ]
        }
    ],
    "volumes": [
        {
            "host": {
                "sourcePath": "/var/run/datadog"
            },
            "name": "dd-sockets"
        }
    ]
}

If you were previously collecting traces over UDP (using the sidecar’s localhost address), switch to UDS using the configuration above.

4. Update process collection

The sidecar setup used pidMode: task to enable process collection. The daemon setup uses the DD_PROCESS_CONFIG_PROCESS_COLLECTION_ENABLED environment variable instead.

Remove the pidMode parameter from your application task definitions. Then confirm the daemon task definition includes:

{
    "name": "DD_PROCESS_CONFIG_PROCESS_COLLECTION_ENABLED",
    "value": "true"
}

5. Register and deploy the updated task definitions

After completing the changes above, register each updated application task definition and redeploy the associated services.

In the AWS Console, navigate to Elastic Container Service > Task Definitions.
Select your application task definition and click Create new revision with JSON.
Apply your changes and click Create.
Navigate to the service using this task definition, click Update, select the new revision, and click Update Service.

aws ecs register-task-definition --cli-input-json file://<path-to-updated-task-definition.json>
aws ecs update-service --cluster <CLUSTER_NAME> --service <SERVICE_NAME> --task-definition <NEW_TASK_DEFINITION_ARN>

Sidecar setup (legacy)

Daemon scheduling is the recommended deployment method for ECS Managed Instances. New deployments should use the daemon setup instead.

In the sidecar deployment model, the Datadog Agent runs as an additional container within each application task definition.

Create the task definition file

Download datadog-agent-ecs-managed-instances-sidecar.json. This file provides minimal configuration for core infrastructure monitoring.
Modify the task definition file:
- Set the DD_API_KEY environment variable by replacing <YOUR_API_KEY> with the Datadog API key for your account. Alternatively, you can also supply the ARN of a secret stored in AWS Secrets Manager.
- Set the DD_SITE environment variable to your Datadog site. Your site is:

Register the task definition

Log in to your AWS Console and navigate to the Elastic Container Service section.
Select Task Definitions in the navigation pane. On the Create new task definition menu, select Create new task definition with JSON.
In the JSON editor box, paste the contents of your task definition file.
Select Create.

Use the AWS CLI to execute the following command:

aws ecs register-task-definition --cli-input-json file://<path to datadog-agent-ecs-managed-instances-sidecar.json>

Run the task as a replica service

Log in to your AWS Web Console and navigate to the Elastic Container Service section.
Choose the cluster to run the Datadog Agent on.
On the Services tab, click Create.
For Task Definition, select the task created in the previous steps.
Enter a Service name.
For Launch type, choose Capacity Provider and select the Managed Instance capacity provider tied to the cluster.
For Number of tasks, enter 1. Click Next step.
Fill in the rest of the optional fields based on your preference.
Click Create service.

Use the AWS CLI to execute the following command:

aws ecs create-service --cluster <CLUSTER_NAME> \
--service-name <SERVICE_NAME> \
--task-definition <TASK_DEFINITION_ARN> \
--desired-count 1

Trace collection (APM)

UDP

To collect traces over UDP, do not set DD_AGENT_HOST. Keep the default localhost value.

UDS

To collect traces over UDS:

Add an empty volume onto the task definition using the volumes parameter.
Mount the volume onto the agent and application container using the mountPoints parameter.
Configure the environmental variable DD_DOGSTATSD_SOCKET on the application container and set it to /var/run/datadog/dsd.socket.

Example: Container definitions that configure collecting traces over UDS

{
    "containerDefinitions": [
        {
            "image": "datadog/agent:latest",
            "mountPoints": [
                {
                    "containerPath": "/var/run/datadog",
                    "readOnly": false,
                    "sourceVolume": "dd-sockets"
                }
            ],
            "name": "datadog-agent",
            ...
        },
        {
            "environment": [
                {
                    "name": "DD_DOGSTATSD_SOCKET",
                    "value": "/var/run/datadog/dsd.socket"
                }
            ],
            "mountPoints": [
                {
                    "containerPath": "/var/run/datadog",
                    "readOnly": false,
                    "sourceVolume": "dd-sockets"
                }
            ],
            "name": "app",
            ...
        }
    ],
    "volumes": [
        {
            "host": {},
            "name": "dd-sockets"
        }
    ]
}

Process collection

You can monitor processes in ECS Managed Instances in Datadog by using the Live Processes page. To enable process collection, add the PidMode parameter in the task definition and set it to task as follows: