Autodiscovery with JMX
Datadog's Research Report: The State of Serverless Report: The State of Serverless

Autodiscovery with JMX

JMX-based integration configurations used by the Datadog Agent can be too big to fit within Autodiscovery labels/annotations. Use Autodiscovery Container Identifiers.

The Datadog-Kafka integration example below leverages JMX to collect metrics and the Datadog Agent to send them to Datadog.

Agent preparation

  1. Enable the Kafka integration by renaming conf.yaml.example into conf.yaml in the Kafka integration directory: /etc/datadog-agent/conf.d/kafka.d

  2. Replace the parameters values from conf.yaml to fit the Agent Autodiscovery logic: The configuration files have host parameter values by default. If you are using the Agent with Autodiscovery, use the Autodiscovery Template Variables instead. In the following example, change the host parameter value from localhost to %%host%%:

      instances:
    
          ## @param host - string - required
          ## Kafka host to connect to.
          #
         - host: %%host%%
    
      # (...)
  3. To specify to the Agent that you want to apply this configuration file to your Kafka containers, configure an ad_identifiers parameter at the beginning of your conf.yaml file:

      ad_identifiers:
        - CUSTOM_AD_IDENTIFIER
    
      instances:
    
          ## @param host - string - required
          ## Kafka host to connect to.
          #
         - host: %%host%%
    
      # (...)

    Note: The example above uses a custom ad_identifers value, but you can specify the container short image as ad_identifiers if needed.

  4. Enable Autodiscovery for your Agent.

  1. Get the configuration files conf.yaml and metrics.yaml associated to the Datadog-Kafka integration. They can be found in your integrations configuration directory. Find below the list of JMX based integration with their associated files:

    IntegrationMetrics fileConfiguration file
    ActiveMqmetrics.yamlconf.yaml.example
    Cassandrametrics.yamlconf.yaml.example
    Hivemetrics.yamlconf.yaml.example
    Jboss Wildflymetrics.yamlconf.yaml.example
    Kafkametrics.yamlconf.yaml.example
    Solrmetrics.yamlconf.yaml.example
    Prestometrics.yamlconf.yaml.example
    Tomcatmetrics.yamlconf.yaml.example
  2. Rename conf.yaml.example into conf.yaml.

  3. Replace the parameter values from conf.yaml to fit the Agent Autodiscovery logic: The configuration files have host parameter values by default. If you are using the Agent with Autodiscovery, use the Autodiscovery Template Variables instead. In the following example, change the host parameter value from localhost to %%host%%:

      instances:
    
          ## @param host - string - required
          ## Kafka host to connect to.
          #
         - host: %%host%%
    
      # (...)
  4. To specify to the Agent that you want to apply this configuration file to your Kafka containers, configure an ad_identifiers parameter at the beginning of your conf.yaml file:

      ad_identifiers:
        - CUSTOM_AD_IDENTIFIER
    
      instances:
    
          ## @param host - string - required
          ## Kafka host to connect to.
          #
         - host: %%host%%
    
      # (...)

    Note: The example above uses a custom ad_identifers value, but you can specify the container short image as ad_identifiers if needed.

  5. After enabling Autodiscovery for your Agent, mount those configuration files (conf.yaml and metrics.yaml) in your Agent in the conf.d/kafka.d/ folder.

  6. (Optional) - If you can’t mount those files in the Agent container (like on AWS ECS), you should re-build the Agent docker image with those two configuration files in it:

    FROM datadog/agent:latest
    COPY <PATH_JMX_CONF_FILE> conf.d/kafka.d/
    COPY <PATH_JMX_METRICS_FILE> conf.d/kafka.d/

    Then use this new custom image as the regular containerized Agent.

Container Preparation

Once the Agent is configured and running, use the com.datadoghq.ad.check.id: CUSTOM_AD_IDENTIFIER label/annotations for your Kafka container to apply the check configuration through Autodiscovery:

apiVersion: v1
kind: Pod
# (...)
metadata:
  name: '<POD_NAME>'
  annotations:
    ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check.id: 'CUSTOM_AD_IDENTIFIER'
    # (...)
spec:
  containers:
    - name: '<CONTAINER_IDENTIFIER>'
# (...)

Note:

  • To apply a specific configuration to a given container, Autodiscovery identifies containers by name, not by image. It tries to match <CONTAINER_IDENTIFIER> to .spec.containers[0].name, not .spec.containers[0].image
  • If you define your Kubernetes pods directly with kind: Pod, add each pod’s annotations directly under its metadata section. If you define pods indirectly with replication controllers, ReplicaSets, or deployments, add pod annotations under .spec.template.metadata.

Dockerfile:

LABEL "com.datadoghq.ad.check.id"= 'CUSTOM_AD_IDENTIFIER'

docker-compose.yaml:

labels:
  com.datadoghq.ad.check.id: CUSTOM_AD_IDENTIFIER

docker run command:

-l com.datadoghq.ad.check.id= 'CUSTOM_AD_IDENTIFIER'

Docker Swarm:

When using Swarm mode for Docker Cloud, labels must be applied to the image:

version: "1.0"
services:
...
  project:
    image: '<IMAGE_NAME>'
    labels:
      com.datadoghq.ad.check.id: CUSTOM_AD_IDENTIFIER

Further Reading