In containerized environments there are a few differences in how the Agent connects to the JMX server. Autodiscovery features make it possible to dynamically setup these integrations. Use Datadog’s JMX based integrations to collect JMX applications metrics from your pods in Kubernetes.

If you are using the Java tracer for your applications, you can alternatively take advantage of the Java runtime metrics feature to send these metrics to the Agent.

Installation

Use a JMX-enabled Agent

JMX utilities are not installed in the Agent by default. To set up a JMX integration, append -jmx to your Agent’s image tag. For example, registry.datadoghq.com/agent:latest-jmx.

If you are using Datadog Operator or Helm, the following configurations append -jmx to your Agent’s image tag:

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  #(...)
  override:
    nodeAgent:
      image:
        jmxEnabled: true

agents:
  image:
    tagSuffix: jmx

Configuration

Use one of the following methods:

• Autodiscovery annotations (recommended)
• Autodiscovery configuration files: for heavy customization of configuration parameters

Autodiscovery annotations

In this method, a JMX check configuration is applied using annotations on your Java-based Pods. This allows the Agent to automatically configure the JMX check when a new container starts. Ensure these annotations are on the created Pod, and not on the object (Deployment, DaemonSet, etc.) creating the Pod.

Use the following template for Autodiscovery annotations:

apiVersion: v1
kind: Pod
metadata:
  name: <POD_NAME>
  annotations:
    ad.datadoghq.com/<CONTAINER_NAME>.checks: |
      {
        "<INTEGRATION_NAME>": {
          "init_config": {
            "is_jmx": true,
            "collect_default_metrics": true
          },
          "instances": [{
            "host": "%%host%%",
            "port": "<JMX_PORT>"
          }]
        }
      }
    # (...)
spec:
  containers:
    - name: '<CONTAINER_NAME>'
      # (...)
      env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: JAVA_OPTS
          value: >-
            -Dcom.sun.management.jmxremote
            -Dcom.sun.management.jmxremote.authenticate=false
            -Dcom.sun.management.jmxremote.ssl=false
            -Dcom.sun.management.jmxremote.local.only=false
            -Dcom.sun.management.jmxremote.port=<JMX_PORT>
            -Dcom.sun.management.jmxremote.rmi.port=<JMX_PORT>
            -Djava.rmi.server.hostname=$(POD_IP)

In this example:

• <POD_NAME> is the name of your pod.
• <CONTAINER_NAME> matches the desired container within your pod.
• <INTEGRATION_NAME> is the name of the desired JMX integration. See the list of available JMX integrations.
• Set <JMX_PORT> as desired, as long as it matches between the annotations and JAVA_OPTS.

With this configuration, the Datadog Agent discovers this pod and makes a request to the JMX server relative to the %%host%% Autodiscovery template variable—this request resolves to the IP address of the discovered pod. This is why java.rmi.server.hostname is set to the POD_IP address previously populated with the Kubernetes downward API.

Note: The JAVA_OPTS environment variable is commonly used in Java-based container images as a startup parameter (for example, java $JAVA_OPTS -jar app.jar). If you are using a custom application, or if your application does not follow this pattern, set these system properties manually.

Example annotation: Tomcat

The following configuration runs the Tomcat JMX integration against port 9012:

apiVersion: v1
kind: Pod
metadata:
  name: tomcat-test
  annotations:
    ad.datadoghq.com/tomcat.checks: |
      {
        "tomcat": {
          "init_config": {
            "is_jmx": true,
            "collect_default_metrics": true
          },
          "instances": [{
            "host": "%%host%%",
            "port": "9012"
          }]
        }
      }
spec:
  containers:
    - name: tomcat
      image: tomcat:8.0
      imagePullPolicy: Always
      ports:
        - name: jmx-metrics
          containerPort: 9012
      env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: JAVA_OPTS
          value: >-
            -Dcom.sun.management.jmxremote
            -Dcom.sun.management.jmxremote.authenticate=false
            -Dcom.sun.management.jmxremote.ssl=false
            -Dcom.sun.management.jmxremote.local.only=false
            -Dcom.sun.management.jmxremote.port=9012
            -Dcom.sun.management.jmxremote.rmi.port=9012
            -Djava.rmi.server.hostname=$(POD_IP)

Custom metric annotation template

If you need to collect additional metrics from these integrations, add them to the init_config section:

ad.datadoghq.com/<CONTAINER_NAME>.checks: |
  {
    "<INTEGRATION_NAME>": {
      "init_config": {
        "is_jmx": true,
        "collect_default_metrics": true,
        "conf": [{
          "include": {
            "domain": "java.lang",
            "type": "OperatingSystem",
            "attribute": {
               "FreePhysicalMemorySize": {
                 "metric_type": "gauge",
                 "alias": "jvm.free_physical_memory"
               } 
            }
          }
        }]
      },
      "instances": [{
        "host": "%%host%%",
        "port": "<JMX_PORT>"
      }]
    }
  }

See the JMX integration documentation for more information about the formatting for these metrics.

Autodiscovery configuration files

If you need to pass a more complex custom configuration for your Datadog-JMX integration, you can use Autodiscovery Container Identifiers to pass custom integration configuration files as well as a custom metrics.yaml file.

1. Compose configuration file

When using this method, the Agent needs a configuration file and an optional metrics.yaml file for the metrics to collect. These files can either be mounted into the Agent pod or built into the container image.

The configuration file naming convention is to first identify your desired integration name from the prerequisite steps of available integrations. Once this is determined, the Agent needs a configuration file named relative to that integration—or within that integration’s config directory.

For example, for the Tomcat integration, create either:

• /etc/datadog-agent/conf.d/tomcat.yaml, or
• /etc/datadog-agent/conf.d/tomcat.d/conf.yaml

If you are using a custom metrics.yaml file, include it in the integration’s config directory. (For example: /etc/datadog-agent/conf.d/tomcat.d/metrics.yaml.)

This configuration file should include ad_identifiers:

ad_identifiers:
  - <CONTAINER_IMAGE>

init_config:
  is_jmx: true
  conf:
    <METRICS_TO_COLLECT>

instances:
  - host: "%%host%%"
    port: "<JMX_PORT>"

Replace <CONTAINER_IMAGE> with the short image name of your desired container. For example, the container image gcr.io/CompanyName/my-app:latest has a short image name of my-app. As the Datadog Agent discovers that container, it sets up the JMX configuration as described in this file.

You can alternatively reference and specify custom identifiers to your containers if you do not want to base this on the short image name.

Like Kubernetes annotations, configuration files can use Autodiscovery template variables. In this case, the host configuration uses %%host%% to resolve to the IP address of the discovered container.

See the JMX integration documentation (as well as the example configurations for the pre-provided integrations) for more information about structuring your init_config and instances configuration for the <METRICS_TO_COLLECT>.

2. Mount configuration file

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  #(...)
  override:
    nodeAgent:
      image:
        jmxEnabled: true
      extraConfd:
        configDataMap:
          <INTEGRATION_NAME>.yaml: |-
            ad_identifiers:
              - <CONTAINER_IMAGE>

            init_config:
              is_jmx: true

            instances:
              - host: "%%host%%"
                port: "<JMX_PORT>"

datadog:
  confd:
    <INTEGRATION_NAME>.yaml: |
      ad_identifiers:
        - <CONTAINER_IMAGE>

      init_config:
        is_jmx: true

      instances:
        - host: "%%host%%"
          port: "<JMX_PORT>"

FROM gcr.io/datadoghq/agent:latest-jmx
COPY <PATH_JMX_CONF_FILE> conf.d/tomcat.d/
COPY <PATH_JMX_METRICS_FILE> conf.d/tomcat.d/

3. Expose JMX server

Set up the JMX server in a way that allows the Agent to access it:

spec:
  containers:
    - # (...)
      env:
      - name: POD_IP
        valueFrom:
          fieldRef:
            fieldPath: status.podIP
      - name: JAVA_OPTS
        value: >-
          -Dcom.sun.management.jmxremote
          -Dcom.sun.management.jmxremote.authenticate=false
          -Dcom.sun.management.jmxremote.ssl=false
          -Dcom.sun.management.jmxremote.local.only=false
          -Dcom.sun.management.jmxremote.port=<JMX_PORT>
          -Dcom.sun.management.jmxremote.rmi.port=<JMX_PORT>
          -Djava.rmi.server.hostname=$(POD_IP)   

Secure the JMX port

The configuration examples above disable JMX authentication and SSL (-Dcom.sun.management.jmxremote.authenticate=false and -Dcom.sun.management.jmxremote.ssl=false). Without additional safeguards, any pod in the cluster can connect to the JMX port and issue remote method invocations. To reduce this risk, use one or both of the following approaches.

Restrict network access with a Kubernetes NetworkPolicy

Apply a NetworkPolicy that allows ingress on the JMX port only from the Datadog Agent pods. Apply this approach without changing your JMX or Agent configuration. It is compatible with any CNI plugin that supports NetworkPolicy (for example, Calico or Cilium).

The Datadog Agent DaemonSet applies the label app.kubernetes.io/component: agent to its pods. Use this label as the ingress source selector:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: restrict-jmx-to-datadog-agent
  namespace: <APP_NAMESPACE>
spec:
  podSelector:
    matchLabels:
      app: <JAVA_APP_LABEL>
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/component: agent
      ports:
        - protocol: TCP
          port: <JMX_PORT>

Replace <APP_NAMESPACE> with the namespace of your Java application, <JAVA_APP_LABEL> with the label that identifies your Java pods, and <JMX_PORT> with the JMX port you configured.

If the Datadog Agent runs in a different namespace from your application, add a namespaceSelector to the ingress rule:

    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: <AGENT_NAMESPACE>
          podSelector:
            matchLabels:
              app.kubernetes.io/component: agent
      ports:
        - protocol: TCP
          port: <JMX_PORT>

Replace <AGENT_NAMESPACE> with the namespace where the Datadog Agent is deployed.

Enable JMX password authentication

To require credentials for JMX connections (an additional layer of defense), you can enable JMX authentication. The Datadog Agent JMX check supports user and password parameters.

Step 1 - Create a Kubernetes Secret with JMX credentials

Create a JMX password file and access file, then store them in a Kubernetes Secret:

# Create the password file (format: <ROLE> <PASSWORD>)
echo 'monitorRole <YOUR_JMX_PASSWORD>' > jmxremote.password
chmod 400 jmxremote.password

# Create the access file (format: <ROLE> readonly|readwrite)
echo 'monitorRole readonly' > jmxremote.access
chmod 400 jmxremote.access

# Create the Kubernetes Secret
kubectl create secret generic jmx-credentials \
  --from-file=jmxremote.password \
  --from-file=jmxremote.access \
  -n <APP_NAMESPACE>

Step 2 - Configure the Java application

Mount the Secret and update the JVM flags to enable authentication:

spec:
  containers:
    - name: '<CONTAINER_NAME>'
      env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: JAVA_OPTS
          value: >-
            -Dcom.sun.management.jmxremote
            -Dcom.sun.management.jmxremote.authenticate=true
            -Dcom.sun.management.jmxremote.password.file=/etc/jmx/jmxremote.password
            -Dcom.sun.management.jmxremote.access.file=/etc/jmx/jmxremote.access
            -Dcom.sun.management.jmxremote.ssl=false
            -Dcom.sun.management.jmxremote.local.only=false
            -Dcom.sun.management.jmxremote.port=<JMX_PORT>
            -Dcom.sun.management.jmxremote.rmi.port=<JMX_PORT>
            -Djava.rmi.server.hostname=$(POD_IP)
      volumeMounts:
        - name: jmx-credentials
          mountPath: /etc/jmx
          readOnly: true
  volumes:
    - name: jmx-credentials
      secret:
        secretName: jmx-credentials
        defaultMode: 0400

Step 3 - Pass credentials in the Autodiscovery annotation

Add user and password to the JMX check instance. Use the %%env_<ENV_VAR>%% template variable to reference the password from an environment variable in the Agent container rather than hardcoding it in the annotation:

ad.datadoghq.com/<CONTAINER_NAME>.checks: |
  {
    "<INTEGRATION_NAME>": {
      "init_config": {
        "is_jmx": true,
        "collect_default_metrics": true
      },
      "instances": [{
        "host": "%%host%%",
        "port": "<JMX_PORT>",
        "user": "monitorRole",
        "password": "%%env_JMX_PASSWORD%%"
      }]
    }
  }

Make JMX_PASSWORD available in the Agent container by injecting it from the same Kubernetes Secret:

# In your Datadog Agent DaemonSet or Helm values
env:
  - name: JMX_PASSWORD
    valueFrom:
      secretKeyRef:
        name: jmx-credentials
        key: jmxremote.password

For the full list of security-related parameters, including SSL and client certificate options (trust_store_path, key_store_path, rmi_registry_ssl), see the JMX integration configuration reference.

Available JMX integrations

The Datadog Agent comes with several JMX integrations pre-configured.

Each integration in the above table has a metrics.yaml file predefined to match the expected pattern of the returned JMX metrics per application. Use the listed integration names as <INTEGRATION_NAME> in your Autodiscovery annotations or configuration files.

Alternatively use jmx as your <INTEGRATION_NAME> to set up a basic JMX integration and collect the default jvm.* metrics only.

Further Reading

Additional helpful documentation, links, and articles:

Create and load an Autodiscovery integration templateDOCUMENTATION Match a container with the corresponding integration templateDOCUMENTATION Manage which container to include in the Agent AutodiscoveryDOCUMENTATION Dynamically assign and collect tags from your applicationDOCUMENTATION