The Horizontal Pod Autoscaling feature was introduced in Kubernetes v1.2 to allow autoscaling off of basic metrics like
CPU, but it requires a resource called
metrics-server to run alongside your application. As of Kubernetes v1.6, it is possible to autoscale off of custom metrics.
Custom metrics are user defined and are collected from within the cluster. As of Kubernetes v1.10, support for external metrics was introduced to autoscale off of any metric from outside the cluster that is collected for you by Datadog.
The custom and external metric providers, as opposed to the metrics server, are resources that have to be implemented and registered by the user.
As of v1.0.0, the Custom Metrics Server in the Datadog Cluster Agent implements the External Metrics Provider interface for external metrics. This documentation page explains how to set it up and how to autoscale your Kubernetes workload based off of your Datadog metrics.
To enable the Custom Metrics Server, follow the instructions to set up the Datadog Cluster Agent, but when editing the deployment manifest of the Datadog Cluster Agent in Step 3 - Create the Cluster Agent and its service, follow those extra steps:
truein the deployment of the Datadog Cluster Agent.
<DD_APP_KEY>as well as the
<DD_API_KEY>in the deployment of the Datadog Cluster Agent with the Datadog API and application keys for your account.
https://app.datadoghq.euif you are using an EU account.
Once the Datadog Cluster Agent is up and running:
datadog-custom-metrics-server service, exposing the port
443 with the following
kind: Service apiVersion: v1 metadata: name: datadog-custom-metrics-server spec: selector: app: datadog-cluster-agent ports: - protocol: TCP port: 443 targetPort: 443
Apply this change by running
kubectl apply -f custom-metric-server.yaml
rbac-hpa.yaml RBAC rules file.
Register the Cluster Agent as an External Metrics Provider via the service, exposing the port
443 by applying the RBAC rules from above:
kubectl apply -f rbac-hpa.yaml
Once you have the Datadog Cluster Agent running and the service registered, create an HPA manifest and specify
type: External for your metrics in order to notify the Datadog Cluster Agent to pull the metrics from Datadog:
spec: metric: - type: External external: metricName: "<METRIC_NAME>" metricSelector: matchLabels: "<TAG_KEY>:<TAG_VALUE>"
Example: An HPA manifest to autoscale off an NGINX deployment based off of the
nginx.net.request_per_s Datadog metric:
apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: nginxext spec: minReplicas: 1 maxReplicas: 3 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: nginx metrics: - type: External external: metricName: nginx.net.request_per_s metricSelector: matchLabels: kube_container_name: nginx targetAverageValue: 9
Note in this manifest that:
* The HPA is configured to autoscale the deployment called
* The maximum number of replicas created is
5, and the minimum is
* The metric used is
nginx.net.request_per_s, and the scope is
kube_container_name: nginx. This metric format corresponds to the Datadog one.
Every 30 seconds, Kubernetes queries the Datadog Cluster Agent to get the value of this metric and autoscales proportionally if necessary. For advanced use cases, it is possible to have several metrics in the same HPA. As you can see in the Kubernetes horizontal pod autoscaling documentation, the largest of the proposed values is the one chosen.
Note: Running multiple Cluster Agents will raise API usage. The Datadog Cluster Agent completes 120 calls per hour for approximately 45 HPA objects in Kubernetes. Running more than 45 HPAs will increase the number of calls when fetching metrics from within the same org.