Custom Metrics Server
Datadog の調査レポート: サーバーレスの状態 レポート: サーバーレスの状態

Custom Metrics Server



The Horizontal Pod Autoscaling feature was introduced in Kubernetes v1.2 to allow autoscaling off of basic metrics like CPU, but it requires a resource called metrics-server to run alongside your application. As of Kubernetes v1.6, it is possible to autoscale off of custom metrics. Custom metrics are user defined and are collected from within the cluster. As of Kubernetes v1.10, support for external metrics was introduced to autoscale off of any metric from outside the cluster that is collected for you by Datadog.

The custom and external metric providers, as opposed to the metrics server, are resources that have to be implemented and registered by the user.

As of v1.0.0, the Custom Metrics Server in the Datadog Cluster Agent implements the External Metrics Provider interface for external metrics. This documentation page explains how to set it up and how to autoscale your Kubernetes workload based off of your Datadog metrics.


  1. Running Kubernetes >v1.10 in order to be able to register the External Metrics Provider resource against the API server.
  2. Having the aggregation layer enabled. Refer to the Kubernetes aggregation layer configuration documentation.

Set up the Cluster Agent external metric server

Custom metrics server

To enable the Custom Metrics Server, follow the instructions to set up the Datadog Cluster Agent, but when editing the deployment manifest of the Datadog Cluster Agent in Step 3 - Create the Cluster Agent and its service, follow those extra steps:

  1. Set DD_EXTERNAL_METRICS_PROVIDER_ENABLED to true in the deployment of the Datadog Cluster Agent.
  2. Configure the <DD_APP_KEY> as well as the <DD_API_KEY> in the deployment of the Datadog Cluster Agent with the Datadog API and application keys for your account.
  3. Optional: set DATADOG_HOST to if you are using an EU account.

Register the External Metrics Provider

Once the Datadog Cluster Agent is up and running:

  1. Create the datadog-custom-metrics-server service, exposing the port 443 with the following custom-metric-server.yaml manifest:

      kind: Service
      apiVersion: v1
        name: datadog-custom-metrics-server
          app: datadog-cluster-agent
        - protocol: TCP
          port: 443
          targetPort: 443

    Apply this change by running kubectl apply -f custom-metric-server.yaml

  2. Download the rbac-hpa.yaml RBAC rules file.

  3. Register the Cluster Agent as an External Metrics Provider via the service, exposing the port 443 by applying the RBAC rules from above:

    kubectl apply -f rbac-hpa.yaml

Run the HPA

Once you have the Datadog Cluster Agent running and the service registered, create an HPA manifest and specify type: External for your metrics in order to notify the Datadog Cluster Agent to pull the metrics from Datadog:

    - type: External
      metricName: "<METRIC_NAME>"

Example: An HPA manifest to autoscale off an NGINX deployment based off of the Datadog metric:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
  name: nginxext
  minReplicas: 1
  maxReplicas: 3
    apiVersion: apps/v1
    kind: Deployment
    name: nginx
  - type: External
            kube_container_name: nginx
      targetAverageValue: 9

Note in this manifest that: * The HPA is configured to autoscale the deployment called nginx. * The maximum number of replicas created is 5, and the minimum is 1. * The metric used is, and the scope is kube_container_name: nginx. This metric format corresponds to the Datadog one.

Every 30 seconds, Kubernetes queries the Datadog Cluster Agent to get the value of this metric and autoscales proportionally if necessary. For advanced use cases, it is possible to have several metrics in the same HPA. As you can see in the Kubernetes horizontal pod autoscaling documentation, the largest of the proposed values is the one chosen.

Note: Running multiple Cluster Agents will raise API usage. The Datadog Cluster Agent completes 120 calls per hour for approximately 45 HPA objects in Kubernetes. Running more than 45 HPAs will increase the number of calls when fetching metrics from within the same org.

Further Reading