Basic Agent Autodiscovery
Datadog の調査レポート: サーバーレスの状態 レポート: サーバーレスの状態

Basic Agent Autodiscovery

このページは日本語には対応しておりません。随時翻訳に取り組んでいます。翻訳に関してご質問やご意見ございましたら、お気軽にご連絡ください。

Overview

When you are monitoring a containerized infrastructure, one challenge that arises is that containers can shift from host to host. The dynamic nature of containerized systems makes them difficult to manually monitor.

To solve this issue, you can use Datadog’s Autodiscovery feature to automatically identify the services running on a specific container and gather data from those services. Whenever a container starts, the Datadog Agent identifies which services are running on this new container, looks for the corresponding monitoring configuration, and starts to collect metrics.

Autodiscovery lets you define configuration templates for Agent checks and specify which containers each checks should apply to.

The Agent watches for events like container creation, destruction, starts, and stops. The Agent then enables, disables, and regenerates static check configurations on such events. As the Agent inspects each running container, it checks if the container matches any of the Autodiscovery container identifiers from any loaded templates. For each match, the Agent generates a static check configuration by substituting the Template Variables with the matching container’s specific values. Then it enables the check using the static configuration.

How It Works

In the figure above, there is a host node with three pods, including a Redis pod and an Agent pod. The Kubelet, which schedules containers, runs as a binary on this node, and exposes the endpoints /metrics and /pods. Every 10 seconds, the Agent queries /pods and finds the Redis spec. It can also see information about the Redis pod itself.

The Redis spec in this example includes the following annotations:

labels:
  tags.datadoghq.com/redis.env: "prod"
  tags.datadoghq.com/redis.service: "my-redis"
  tags.datadoghq.com/redis.version: "6.0.3"
annotations:
  ad.datadoghq.com/redis.check_names: '["redisdb"]'
  ad.datadoghq.com/redis.init_configs: '[{}]'
  ad.datadoghq.com/redis.instances: |
    [
      {
        "host": "%%host%%",
        "port":"6379",
        "password":"%%env_REDIS_PASSWORD%%"
      }
    ]
  ad.datadoghq.com/redis.logs: '[{"source":"redis"}]'

In the example above, the tags.datadoghq.com labels set the env, service, and even version as tags for all logs and metrics emitted for the Redis pod. These standard labels are part of Unified Service Tagging. As a best practice, Datadog recommends using unified service tagging when configuring tags and environment variables.

check_names includes the names of the check to run, and init_configs contains some configuration parameters, such as minimum collection interval. Each item in instances represents the configuration to run for one instance of a check. Note that in this example, %%host%% is a template variable that is dynamically populated with your container’s IP.

From this, the Agent generates a static check configuration.

Setup

Setting up Autodiscovery for your infrastructure requires the following two steps:

  1. Enable Autodiscovery for your Datadog Agent.
  2. Create integration-specific configuration templates for each service you wish to monitor. Note that Datadog provides auto-configuration templates for some common containerized services, including Apache and Redis.

Enable Autodiscovery

With the Agent on a host

Add the following configuration block in the datadog.yaml configuration file.

listeners:
  - name: docker
config_providers:
  - name: docker
    polling: true

Add the following configuration block in the datadog.yaml configuration file.

listeners:
  - name: kubelet
config_providers:
  - name: kubelet
    polling: true
  # needed to support legacy docker label config templates
  - name: docker
    polling: true

ECS Fargate cannot be monitored with the Datadog Agent running as a binary on a host.

With the Agent as a container

To automatically enable Autodiscovery over Docker containers, mount /var/run/docker.sock into the Containerized Agent. On Windows, mount \\.\pipe\docker_engine.

Autodiscovery is enabled by default on Kubernetes.

To verify this, ensure the following environment variable is set:

KUBERNETES=true

To enable Autodiscovery over containers within Kubernetes, add the following environment variable when starting the containerized Agent:

ECS_FARGATE=true

Integration Templates

Once Autodiscovery is enabled, the Datadog Agent automatically attempts Autodiscovery for a number of services, including Apache and Redis, based on default Autodiscovery configuration files.

You can define an integration template in multiple forms: as Kubernetes pod annotations, Docker labels, a configuration file mounted within the Agent, a ConfigMap, and key-value stores.

In the following example, the tags.datadoghq.com Kubernetes labels are used to tag pod’s data with env, service, and version.

The Redis integration template is defined through Kubernetes pod annotations. It contains a custom password parameter and tags all its logs with the correct source.

apiVersion: v1
kind: Pod
metadata:
  ## name of your Pod
  name: my-redis
  labels:
    ## set standard labels for unified service tagging
    tags.datadoghq.com/redis.env: prod
    tags.datadoghq.com/redis.service: my-redis
    tags.datadoghq.com/redis.version: "6.0.3"
  annotations:
    ## names of check; matches name in integrations_core repo
    ad.datadoghq.com/redis.check_names: '["redisdb"]'
    ## some configs, like minimum collection interval
    ad.datadoghq.com/redis.init_configs: '[{}]'
    ad.datadoghq.com/redis.instances: |
      [
        ## config to run for one instance of check
        {
          "host": "%%host%%",
          "port":"6379",
          "password":"%%env_REDIS_PASSWORD%%"
        }
      ]
    ## setup for logs collection
    ad.datadoghq.com/redis.logs: '[{"source":"redis"}]'
spec:
  containers:
    - name: redis
      image: httpd
      ports:
        - containerPort: 80

To use Autodiscovery with other services, define templates for the services you wish to monitor. See the Autodiscovery Integration Templates documentation for further details.

Further Reading