Autodiscovery with JMX
Security Monitoring is now available Security Monitoring is now available

Autodiscovery with JMX

Leverage integrations autodiscovery annotations or use Autodiscovery Container Identifiers to collect your JMX-applications metrics from your pods in Kubernetes. Autodisocovery annotations is the recommended way to configure your Datadog-JMX integration, if the set of configuration parameters is too long to fit in annotations, use the Autodiscovery Container Identifiers method.

Autodiscovery annotations

The autodiscovery annotations logic consists in applying the JMX check configuration elements, through annotations, to your pod in order to allow the Agent to “automatically discover” them and configure its JMX check accordingly:

  1. Launch the Agent in your Kubernetes cluster with the datadog/agent:latest-jmx image name instead of the regular datadog/agent:latest.

  2. Apply the Autodiscovery annotations to the containers containing your JMX-application:

    apiVersion: v1
    kind: Pod
    metadata:
        name: <POD_NAME>
        annotations:
            ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check_names: >-
              '["<INTEGRATION_NAME>"]'
            ad.datadoghq.com/<CONTAINER_IDENTIFIER>.init_configs: >-
              '[{"is_jmx": true, "collect_default_metrics": true}]'
            ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances: >-
              '[{"host": "%%host%%","port":"<JMX_PORT>"}]'
            ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs: >-
              '[{"source":"<INTEGRATION_NAME>","service":"<INTEGRATION_NAME>"}]'
        # (...)
    
    spec:
        containers:
            - name: '<CONTAINER_IDENTIFIER>'
            # (...)
              env:
              - name: POD_IP
                valueFrom:
                  fieldRef:
                    fieldPath: status.podIP
    
              - name: JAVA_OPTS
                value: >-
                  -Xms256m -Xmx6144m
                  -Dcom.sun.management.jmxremote
                  -Dcom.sun.management.jmxremote.authenticate=false
                  -Dcom.sun.management.jmxremote.ssl=false
                  -Dcom.sun.management.jmxremote.local.only=false
                  -Dcom.sun.management.jmxremote.port=<JMX_PORT>
                  -Dcom.sun.management.jmxremote.rmi.port=<JMX_PORT>
                  -Djava.rmi.server.hostname=$(POD_IP)

    The JAVA_OPTS environment variable needs to be created, so that your JMX server allows the agent to connect to the RMI registry.

    Note:

    • <JMX_PORT> references the port that exposes JMX metrics.
    • In the example above, the connection to the RMI registry is not in SSL if you want to use SSL, use "rmi_registry_ssl": true in the ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances annotation and remove the corresponding Dcom.sun.management.jmxremote from JAVA_OPTS.

The list of JMX-ready integrations name <INTEGRATION_NAME> are:

For instance if you have a tomcat running exposing its JMX metrics on port 9012 you would do:

apiVersion: v1
kind: Pod
metadata:
    name: tomcat-test
    annotations:
        ad.datadoghq.com/tomcat.check_names: >-
          '["tomcat"]'
        ad.datadoghq.com/tomcat.init_configs: >-
          '[{"is_jmx": true, "collect_default_metrics": true}]'
        ad.datadoghq.com/tomcat.instances: >-
          '[{"host": "%%host%%","port":"9012"}]'
        ad.datadoghq.com/tomcat.logs: >-
          '[{"source":"Tomcat","service":"Tomcat"}]'

spec:
    containers:
        - name: tomcat
          image: tomcat:8.0
          imagePullPolicy: Always
          ports:
              - containerPort: 9012
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP

            - name: JAVA_OPTS
              value: >-
                -Xms256m -Xmx6144m
                -Dcom.sun.management.jmxremote
                -Dcom.sun.management.jmxremote.authenticate=false
                -Dcom.sun.management.jmxremote.ssl=false
                -Dcom.sun.management.jmxremote.local.only=false
                -Dcom.sun.management.jmxremote.port=9012
                -Dcom.sun.management.jmxremote.rmi.port=9012
                -Djava.rmi.server.hostname=$(POD_IP)

Autodiscovery Container Identifiers

If you need to pass a more complex configuration for your Datadog-JMX integration, leverage Autodiscovery Container Identifiers to pass custom integration configuration file or custom metrics.yaml file.

Agent preparation

Choose wether your Agent is running as a container in your cluster, or on your host directly:

If your Agent is running in your cluster and you want to autodiscover your container to collect JMX metrics:

  1. Make sure to run the Agent image the datadog/agent:latest-jmx instead of the regular datadog/agent:latest image.

  2. Get the configuration files conf.yaml and metrics.yaml associated to your integration. Find below the list of Datadog-JMX based integration with their associated files:

    Integration NameMetrics fileConfiguration file
    activemqmetrics.yamlconf.yaml.example
    cassandrametrics.yamlconf.yaml.example
    confluent_platformmetrics.yamlconf.yaml.example
    hivemetrics.yamlconf.yaml.example
    jboss_wildflymetrics.yamlconf.yaml.example
    kafkametrics.yamlconf.yaml.example
    solrmetrics.yamlconf.yaml.example
    prestometrics.yamlconf.yaml.example
    tomcatmetrics.yamlconf.yaml.example
  3. Rename the conf.yaml.example file into conf.yaml.

  4. Replace the parameter values from conf.yaml to fit the Agent Autodiscovery logic. The configuration files have host parameter values by default, use the Autodiscovery Template Variables logic instead. In the following Tomcat check example, the host parameter value is changed from localhost to %%host%%:

    init_config:
        ## @param is_jmx - boolean - required
        ## Whether or not this file is a configuration for a JMX integration.
        #
        is_jmx: true
    
        ## @param collect_default_metrics - boolean - required
        ## Whether or not the check should collect all default metrics.
        #
        collect_default_metrics: true
    
    instances:
        ## @param host - string - required
        ## Tomcat JMX hostname to connect to.
        #
        - host: '%%host%%'
    
          ## @param port - integer - required
          ## Tomcat JMX port to connect to.
          #
          port: 9012
  5. To specify to the Agent that you want to apply this configuration file to your application container, configure an ad_identifiers parameter at the beginning of your conf.yaml file:

    ad_identifiers:
        - '<CUSTOM_AD_IDENTIFIER>'
    
    init_config:
        # (...)
    instances:
        # (...)

    Note: The example above uses a custom ad_identifers value, but you can specify the container short image as ad_identifiers if needed.

  6. Mount those configuration files (conf.yaml and metrics.yaml) in your Agent in the conf.d/<INTEGRATION_NAME>.d/ folder.

  7. (Optional) - If you can’t mount those files in the Agent container (like on AWS ECS), you should re-build the Agent docker image with those two configuration files in it:

    FROM datadog/agent:latest-jmx
    COPY <PATH_JMX_CONF_FILE> conf.d/tomcat.d/
    COPY <PATH_JMX_METRICS_FILE> conf.d/tomcat.d/

    Then use this new custom image as the regular containerized Agent.

If your Agent is running on a host and you want to autodiscover your container to collect JMX metrics:

  1. Enable autodiscovery for your Agent.

  2. Enable the JMX integration you want to use by renaming the corresponding conf.yaml.example file into conf.yaml in the Agent integration directory. For instance for tomcat you would rename /etc/datadog-agent/conf.d/tomcat.d/conf.yaml.example into: /etc/datadog-agent/conf.d/tomcat.d/conf.yaml

  3. Replace the parameters values from conf.yaml file to fit the Agent Autodiscovery logic. The configuration files have host parameter values by default, use the Autodiscovery Template Variables instead. In the following Tomcat configuration example, the host parameter value is changed from localhost to %%host%%:

    init_config:
        ## @param is_jmx - boolean - required
        ## Whether or not this file is a configuration for a JMX integration.
        #
        is_jmx: true
    
        ## @param collect_default_metrics - boolean - required
        ## Whether or not the check should collect all default metrics.
        #
        collect_default_metrics: true
    
    instances:
        ## @param host - string - required
        ## Tomcat JMX hostname to connect to.
        #
        - host: '%%host%%'
    
          ## @param port - integer - required
          ## Tomcat JMX port to connect to.
          #
          port: 9012
  4. To specify to the Agent that you want to apply this configuration file to your application containers, configure an ad_identifiers parameter at the beginning of your conf.yaml file:

    ad_identifiers:
        - '<CUSTOM_AD_IDENTIFIER>'
    
    init_config:
        # (...)
    instances:
        # (...)

    Note: The example above uses a custom ad_identifers value, but you can specify the container short image as ad_identifiers if needed.

  5. Restart your Agent

Container Preparation

Once the Agent is configured and running, use the com.datadoghq.ad.check.id:"<CUSTOM_AD_IDENTIFIER>" label/annotations for your application container to apply the check configuration through Autodiscovery:

apiVersion: v1
kind: Pod
# (...)
metadata:
    name: '<POD_NAME>'
    annotations:
        ad.datadoghq.com/<CONTAINER_IDENTIFIER>.check.id: '<CUSTOM_AD_IDENTIFIER>'
        # (...)
spec:
    containers:
        - name: '<CONTAINER_IDENTIFIER>'
          # (...)
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP

            - name: JAVA_OPTS
              value: >-
                -Xms256m -Xmx6144m
                -Dcom.sun.management.jmxremote
                -Dcom.sun.management.jmxremote.authenticate=false
                -Dcom.sun.management.jmxremote.ssl=false
                -Dcom.sun.management.jmxremote.local.only=false
                -Dcom.sun.management.jmxremote.port=<JMX_PORT>
                -Dcom.sun.management.jmxremote.rmi.port=<JMX_PORT>
                -Djava.rmi.server.hostname=$(POD_IP)
# (...)

Note:

  • To apply a specific configuration to a given container, Autodiscovery identifies containers by name, not by image. It tries to match <CONTAINER_IDENTIFIER> to .spec.containers[0].name, not .spec.containers[0].image
  • If you define your Kubernetes pods directly with kind: Pod, add each pod’s annotations directly under its metadata section. If you define pods indirectly with replication controllers, ReplicaSets, or deployments, add pod annotations under .spec.template.metadata.
  • The JAVA_OPTS environment variable needs to be created, so that your JMX server allows the agent to connect to the RMI registry.
  • <JMX_PORT> references the port that exposes JMX metrics.
  • In the example above, the connection to the RMI registry is not in SSL if you want to use SSL, use "rmi_registry_ssl": true in the ad.datadoghq.com/<CONTAINER_IDENTIFIER>.instances annotation and remove the corresponding Dcom.sun.management.jmxremote from JAVA_OPTS.

Dockerfile:

LABEL "com.datadoghq.ad.check.id"= '<CUSTOM_AD_IDENTIFIER>'

docker-compose.yaml:

labels:
    com.datadoghq.ad.check.id: '<CUSTOM_AD_IDENTIFIER>'

docker run command:

-l com.datadoghq.ad.check.id= '<CUSTOM_AD_IDENTIFIER>'

Docker Swarm:

When using Swarm mode for Docker Cloud, labels must be applied to the image:

version: '1.0'
services:
# ...
project:
    image: '<IMAGE_NAME>'
    labels:
        com.datadoghq.ad.check.id: '<CUSTOM_AD_IDENTIFIER>'

Note: If the Agent and your JMX container are on the same network bridge, you need to instantiate your JMX server with -Djava.rmi.server.hostname=<CONTAINER_NAME>" where <CONTAINER_NAME> is your JMX-application container name.

Further Reading