Endpoint Checks with Autodiscovery

Endpoint check dispatching is enabled in the Operator deployment of the Cluster Agent by using the features.clusterChecks.enabled configuration key:

kind: DatadogAgent
apiVersion: datadoghq.com/v2alpha1
metadata:
  name: datadog
spec:
  features:
    clusterChecks:
      enabled: true

This configuration enables both cluster check and endpoint check dispatching between the Cluster Agent and the Agents.

Endpoint check dispatching is enabled by default in the Helm deployment of the Cluster Agent through the datadog.clusterChecks.enabled configuration key:

datadog:
  clusterChecks:
    enabled: true
  # (...)
clusterAgent:
  enabled: true
  # (...)

This configuration enables both cluster check and endpoint Check dispatching between the Cluster Agent and the Agents.

Cluster Agent setup

Enable the kube_endpoints configuration provider and listener on the Datadog Cluster Agent. Set the DD_EXTRA_CONFIG_PROVIDERS and DD_EXTRA_LISTENERS environment variables:

DD_EXTRA_CONFIG_PROVIDERS="kube_endpoints"
DD_EXTRA_LISTENERS="kube_endpoints"

Note: If the monitored endpoints are not backed by Pods, you must enable Cluster Checks. Add the kube_services configuration provider and listener:

DD_EXTRA_CONFIG_PROVIDERS="kube_endpoints kube_services"
DD_EXTRA_LISTENERS="kube_endpoints kube_services"

Restart the Agent to apply the configuration change.

Agent setup

Enable the endpointschecks configuration providers on the node-based Agent. This can be done in two ways:

• By setting the DD_EXTRA_CONFIG_PROVIDERS environment variable. This takes a space-separated string if you have multiple values:

  DD_EXTRA_CONFIG_PROVIDERS="endpointschecks"
  

• Or adding it to the datadog.yaml configuration file:

  config_providers:
      - name: endpointschecks
        polling: true
  

Note: If the monitored endpoints are not backed by Pods, you must enable cluster checks. This can be done by adding the clusterchecks configuration provider:

DD_EXTRA_CONFIG_PROVIDERS="endpointschecks clusterchecks"

Restart the Agent to apply the configuration change.

Use the clusterAgent.confd field to define your check configuration:

#(...)
clusterAgent:
  confd:
    <INTEGRATION_NAME>.yaml: |-
      advanced_ad_identifiers:
        - kube_endpoints:
            name: "<ENDPOINTS_NAME>"
            namespace: "<ENDPOINTS_NAMESPACE>"
      cluster_check: true
      init_config:
      instances:
        - url: "http://%%host%%"
          name: "<EXAMPLE_NAME>"      

Mount a /conf.d/http_check.yaml file in the Cluster Agent container with the following content:

advanced_ad_identifiers:
  - kube_endpoints:
      name: "<ENDPOINTS_NAME>"
      namespace: "<ENDPOINTS_NAMESPACE>"
cluster_check: true
init_config:
instances:
  - url: "http://%%host%%"
    name: "<EXAMPLE_NAME>"

Note: AD Annotations v2 was introduced in Datadog Agent 7.36 to simplify integration configuration. For previous versions of the Datadog Agent, use AD Annotations v1.

The syntax for annotating services is similar to that for annotating Kubernetes Pods:

ad.datadoghq.com/endpoints.checks: |
  {
    "<INTEGRATION_NAME>": {
      "init_config": <INIT_CONFIG>,
      "instances": [<INSTANCE_CONFIG>]
    }
  }  
ad.datadoghq.com/endpoints.logs: '[<LOGS_CONFIG>]'

This syntax supports a %%host%% template variable which is replaced by the IP of each endpoint. The kube_namespace, kube_service, and kube_endpoint_ip tags are automatically added to the instances.

Note: Custom endpoints log configuration is only supported during Docker socket log collection, and not Kubernetes log file collection.

Example: HTTP check on an NGINX-backed service with an NGINX check on the service’s endpoints

This service is associated with the Pods of the nginx deployment. Based on this configuration:

• An nginx-based endpoint check is dispatched for each NGINX Pod backing this service. This check is run by Agents on the same respective nodes as the NGINX Pods (using the Pod IP as %%host%%).
• An http_check-based cluster check is dispatched to a single Agent in the cluster. This check uses the IP of the service as %%host%%, automatically getting load balanced to the respective endpoints.
• The checks are dispatched with the tags env:prod, service:my-nginx, and version:1.19.0, corresponding to Unified Service Tagging labels.

apiVersion: v1
kind: Service
metadata:
    name: nginx
    labels:
        app: nginx
        tags.datadoghq.com/env: "prod"
        tags.datadoghq.com/service: "my-nginx"
        tags.datadoghq.com/version: "1.19.0"
    annotations:
      ad.datadoghq.com/service.checks: |
        {
          "http_check": {
            "init_config": {},
            "instances": [
              {
                "url": "http://%%host%%",
                "name": "My Nginx",
                "timeout": 1
              }
            ]
          }
        }        
      ad.datadoghq.com/endpoints.checks: |
        {
          "nginx": {
            "init_config": {},
            "instances": [
              {
                "name": "My Nginx Service Endpoints",
                "nginx_status_url": "http://%%host%%:%%port%%/status/"
              }
            ]
          }
        }        
      ad.datadoghq.com/endpoints.logs: '[{"source":"nginx","service":"webapp"}]'
spec:
    ports:
        - port: 80
          protocol: TCP
    selector:
        app: nginx

The syntax for annotating services is similar to that for annotating Kubernetes Pods:

ad.datadoghq.com/endpoints.check_names: '[<INTEGRATION_NAME>]'
ad.datadoghq.com/endpoints.init_configs: '[<INIT_CONFIG>]'
ad.datadoghq.com/endpoints.instances: '[<INSTANCE_CONFIG>]'
ad.datadoghq.com/endpoints.logs: '[<LOGS_CONFIG>]'

This syntax supports a %%host%% template variable which is replaced by the IP of each endpoint. The kube_namespace, kube_service, and kube_endpoint_ip tags are automatically added to the instances.

Note: Custom endpoints log configuration is only supported during Docker socket log collection, and not Kubernetes log file collection.

Example: HTTP check on an NGINX-backed service with an NGINX check on the service’s endpoints

This service is associated with the Pods of the nginx deployment. Based on this configuration:

• An nginx-based endpoint check is dispatched for each NGINX Pod backing this service. This check is run by Agents on the same respective nodes as the NGINX Pods (using the Pod IP as %%host%%).
• An http_check-based cluster check is dispatched to a single Agent in the cluster. This check uses the IP of the service as %%host%%, automatically getting load balanced to the respective endpoints.
• The checks are dispatched with the tags env:prod, service:my-nginx, and version:1.19.0, corresponding to Unified Service Tagging labels.

apiVersion: v1
kind: Service
metadata:
    name: nginx
    labels:
        app: nginx
        tags.datadoghq.com/env: "prod"
        tags.datadoghq.com/service: "my-nginx"
        tags.datadoghq.com/version: "1.19.0"
    annotations:
        ad.datadoghq.com/endpoints.check_names: '["nginx"]'
        ad.datadoghq.com/endpoints.init_configs: '[{}]'
        ad.datadoghq.com/endpoints.instances: |
            [
              {
                "name": "My Nginx Service Endpoints",
                "nginx_status_url": "http://%%host%%:%%port%%/nginx_status"
              }
            ]            
        ad.datadoghq.com/service.check_names: '["http_check"]'
        ad.datadoghq.com/service.init_configs: '[{}]'
        ad.datadoghq.com/service.instances: |
            [
              {
                "name": "My Nginx Service",
                "url": "http://%%host%%"
              }
            ]            
        ad.datadoghq.com/endpoints.logs: '[{"source":"nginx","service":"webapp"}]'
spec:
    ports:
        - port: 80
          protocol: TCP
    selector:
        app: nginx