Overview

The datadog/cloudprem Helm chart creates Horizontal Pod Autoscalers (HPAs) for BYOC (Bring Your Own Cloud) Logs indexers and standalone compactors when you enable them. HPAs are disabled by default, and each component is configured independently.

Before you begin

Before you enable autoscaling, you need:

• A BYOC Logs deployment installed with the datadog/cloudprem Helm chart.
• Chart version 0.4.6 or later for standalone compactor autoscaling.
• Kubernetes Metrics Server, or another metrics API implementation, installed in the cluster.
• Enough node capacity for the maximum number of indexer and compactor pods.
• CPU requests configured for autoscaled workloads.

CPU-based HPA calculations use the pod CPU request. Indexers get CPU requests from indexer.podSize or indexer.resources.requests.cpu. For standalone compactors, configure compactor.resources.requests.cpu.

Enable indexer autoscaling

To enable the indexer HPA, set indexer.autoscaling.enabled to true:

indexer:
  autoscaling:
    enabled: true

When you enable indexer autoscaling, the HPA controls the number of indexer pods and ignores indexer.replicaCount.

Default indexer HPA settings:

Enable compactor autoscaling

To enable the compactor HPA, enable standalone compactors and set compactor.autoscaling.enabled to true:

enableStandaloneCompactors: true

compactor:
  autoscaling:
    enabled: true

The chart creates the compactor HPA only when you set both enableStandaloneCompactors and compactor.autoscaling.enabled to true. When you enable compactor autoscaling, the HPA controls the number of compactor pods and ignores compactor.replicaCount.

Default compactor HPA settings:

Override the defaults

Set minReplicas and maxReplicas alongside enabled to size the scaling range for your workload. Use the Cluster Sizing guide to pick a maximum that your node capacity supports:

indexer:
  autoscaling:
    enabled: true
    minReplicas: 4
    maxReplicas: 20

Apply the configuration

Add the autoscaling values to your BYOC Logs values file, then upgrade your release:

helm upgrade <RELEASE_NAME> datadog/cloudprem \
  --namespace <NAMESPACE_NAME> \
  --values datadog-values.yaml

Verify the HPAs

List HPAs in the BYOC Logs namespace:

kubectl get hpa -n <NAMESPACE_NAME>

Describe an HPA to check metrics and recent scaling events:

kubectl describe hpa <RELEASE_NAME>-indexer -n <NAMESPACE_NAME>
kubectl describe hpa <RELEASE_NAME>-compactor -n <NAMESPACE_NAME>

<RELEASE_NAME>-indexer and <RELEASE_NAME>-compactor are the default HPA names created by the chart. If you set nameOverride or fullnameOverride, use the resulting names instead.

If kubectl get hpa shows <unknown> in the TARGETS column, the HPA cannot read CPU metrics. Check that the metrics API is running and that the target pods have CPU requests.

Further reading

Additional helpful documentation, links, and articles:

Cluster SizingDOCUMENTATION Monitor BYOC LogsDOCUMENTATION CloudPrem Helm chart default valuesSOURCE