Custom Metrics Server

このページは英語では対応しておりません。随時翻訳に取り組んでいます。翻訳に関してご質問やご意見ございましたら、お気軽にご連絡ください。

Introduction

The Horizontal Pod Autoscaling feature was introduced in Kubernetes v1.2 to allow autoscaling off of basic metrics like CPU, but it requires a resource called metrics-server to run alongside your application. As of Kubernetes v1.6, it is possible to autoscale off of custom metrics. Custom metrics are user defined and are collected from within the cluster. As of Kubernetes v1.10, support for external metrics was introduced to autoscale off of any metric from outside the cluster that is collected for you by Datadog.

The custom and external metric providers, as opposed to the metrics server, are resources that have to be implemented and registered by the user.

As of v1.0.0, the Custom Metrics Server in the Datadog Cluster Agent implements the External Metrics Provider interface for external metrics. This documentation page explains how to set it up and how to autoscale your Kubernetes workload based off of your Datadog metrics.

Requirements

Running Kubernetes >v1.10 in order to be able to register the External Metrics Provider resource against the API server.
Having the aggregation layer enabled. Refer to the Kubernetes aggregation layer configuration documentation.

Set up the Cluster Agent external metric server

Custom metrics server

To enable the Custom Metrics Server, follow the instructions to set up the Datadog Cluster Agent, but when editing the deployment manifest of the Datadog Cluster Agent in Step 3 - Create the Cluster Agent and its service, follow those extra steps:

Set DD_EXTERNAL_METRICS_PROVIDER_ENABLED to true in the deployment of the Datadog Cluster Agent.
Configure the <DD_APP_KEY> as well as the <DD_API_KEY> in the deployment of the Datadog Cluster Agent with the Datadog API and application keys for your account.
Optional: set DATADOG_HOST to https://app.datadoghq.eu if you are using an EU account.

Register the External Metrics Provider

Once the Datadog Cluster Agent is up and running:

Create the datadog-custom-metrics-server service, exposing the port 443 with the following custom-metric-server.yaml manifest:

  kind: Service
  apiVersion: v1
  metadata:
    name: datadog-custom-metrics-server
  spec:
    selector:
      app: datadog-cluster-agent
    ports:
    - protocol: TCP
      port: 443
      targetPort: 443

Apply this change by running kubectl apply -f custom-metric-server.yaml

Download the rbac-hpa.yaml RBAC rules file.
Register the Cluster Agent as an External Metrics Provider via the service, exposing the port 443 by applying the RBAC rules from above:
```
kubectl apply -f rbac-hpa.yaml
```

Run the HPA

Once you have the Datadog Cluster Agent running and the service registered, create an HPA manifest and specify type: External for your metrics in order to notify the Datadog Cluster Agent to pull the metrics from Datadog:

spec:
  metric:
    - type: External
      external:
      metricName: "<METRIC_NAME>"
      metricSelector:
        matchLabels:
          "<TAG_KEY>:<TAG_VALUE>"

Example: An HPA manifest to autoscale off an NGINX deployment based off of the nginx.net.request_per_s Datadog metric:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: nginxext
spec:
  minReplicas: 1
  maxReplicas: 3
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx
  metrics:
  - type: External
    external:
      metricName: nginx.net.request_per_s
      metricSelector:
        matchLabels:
            kube_container_name: nginx
      targetAverageValue: 9

Note in this manifest that: * The HPA is configured to autoscale the deployment called nginx. * The maximum number of replicas created is 5, and the minimum is 1. * The metric used is nginx.net.request_per_s, and the scope is kube_container_name: nginx. This metric format corresponds to the Datadog one.

Every 30 seconds, Kubernetes queries the Datadog Cluster Agent to get the value of this metric and autoscales proportionally if necessary. For advanced use cases, it is possible to have several metrics in the same HPA. As you can see in the Kubernetes horizontal pod autoscaling documentation, the largest of the proposed values is the one chosen.