--- title: Managing DatadogPodAutoscaler with ArgoCD description: >- Deploy and manage DatadogPodAutoscaler custom resources for Kubernetes autoscaling using ArgoCD and GitOps breadcrumbs: >- Docs > Containers > Containers Guides > Managing DatadogPodAutoscaler with ArgoCD --- # Managing DatadogPodAutoscaler with ArgoCD ## Overview{% #overview %} The DatadogPodAutoscaler (DPA) is a Kubernetes custom resource definition (CRD) that enables autoscaling of Kubernetes workloads using [Datadog Kubernetes Autoscaling (DKA)](https://docs.datadoghq.com/containers/monitoring/autoscaling.md). This guide demonstrates how to manage DatadogPodAutoscaler resources using ArgoCD and GitOps principles to deploy an autoscaling configuration. ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. It monitors Git repositories containing Kubernetes manifests and keeps your cluster synchronized with the desired state defined in Git. This approach provides version control, audit trails, and automated deployment of your autoscaling infrastructure. ## Prerequisites{% #prerequisites %} Before you begin, ensure you have the following: - **Kubernetes cluster**: A working Kubernetes cluster (1.20 or later) with access using `kubectl` - **ArgoCD installed**: ArgoCD deployed in your cluster and accessible via CLI or UI - **Datadog API credentials**: Valid Datadog API key and application key - **Git repository**: A Git repository to store your manifests ## Project structure{% #project-structure %} This guide uses the App of Apps pattern with ArgoCD sync waves to ensure proper dependency creation and deployment order. ```gdscript3 . ├── argocd/ │ ├── root-app.yaml # App of Apps controller │ └── apps/ │ ├── datadog-operator.yaml # ArgoCD Application for Operator │ ├── datadog-agent.yaml # ArgoCD Application for Agent │ └── nginx-dka-demo.yaml # ArgoCD Application for workload ├── manifests/ │ └── stage2-agent/ │ └── datadog-agent.yaml # DatadogAgent custom resource └── charts/ └── nginx-dka-demo/ ├── Chart.yaml ├── values.yaml └── templates/ ├── deployment.yaml └── pod-autoscaler.yaml ``` ## Deployment stages{% #deployment-stages %} A multi-stage deployment approach is essential when working with Kubernetes custom resource definitions (CRDs) and ArgoCD. This ordered approach is necessary to ensure that you create and install the dependencies required for each stage in the process. Kubernetes CRDs must be installed in the cluster before you can create custom resources that use them. The DatadogPodAutoscaler CRD is created when you install the Datadog Operator in Stage 1. ArgoCD needs these CRDs to be present before it can successfully sync resources that depend on them. ArgoCD uses **sync waves** to control the deployment order through annotations. Sync waves are executed in ascending order (lower numbers first), and ArgoCD waits for all resources in a wave to be healthy before proceeding to the next wave. 1. **Stage 1 (Wave 0)**: Datadog Operator using Helm (creates CRDs) 1. **Stage 2 (Wave 1)**: Datadog Agent configured for Datadog Kubernetes Autoscaling - DatadogAgent custom resource with Autoscaling requirements enabled 1. **Stage 3 (Wave 2)**: Application workload with DatadogPodAutoscaler - NGINX deployment in demo namespace - DatadogPodAutoscaler resource for autoscaling the NGINX deployment ## Set up configuration files{% #set-up-configuration-files %} First, create a Git repository. You need to update all `repoURL` references in the ArgoCD Application manifests to point to your repository, as ArgoCD pulls manifests from Git. Set up the following configuration files for each stage in the process. ### Stage 1: Root Application (App of Apps){% #stage-1-root-application-app-of-apps %} The root Application is the App of Apps controller that manages all child Applications. In the `argocd/root-app.yaml` file: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: dka-root namespace: argocd spec: project: default source: repoURL: https://example.com/YOUR-USERNAME/dka-argocd-example targetRevision: HEAD path: argocd/apps destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: automated: prune: true selfHeal: true ``` ### Stage 2: Datadog Operator application{% #stage-2-datadog-operator-application %} This ArgoCD application deploys the Datadog Operator using Helm, which creates the necessary CRDs. In the `argocd/apps/datadog-operator.yaml` file: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: datadog-operator namespace: argocd annotations: argocd.argoproj.io/sync-wave: "0" spec: project: default source: repoURL: https://helm.datadoghq.com targetRevision: 2.18.1 chart: datadog-operator destination: server: https://kubernetes.default.svc namespace: datadog syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true - ServerSideApply=true ``` ### Stage 3: Datadog Agent application{% #stage-3-datadog-agent-application %} This ArgoCD application deploys the DatadogAgent custom resource. In the `argocd/apps/datadog-agent.yaml` file: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: datadog-agent namespace: argocd annotations: argocd.argoproj.io/sync-wave: "1" spec: project: default source: repoURL: https://example.com/YOUR-USERNAME/dka-argocd-example targetRevision: HEAD path: manifests/stage2-agent destination: server: https://kubernetes.default.svc namespace: datadog syncPolicy: automated: prune: true selfHeal: true ``` Create the DatadogAgent custom resource manifest: In the `manifests/stage2-agent/datadog-agent.yaml` file: ```yaml apiVersion: datadoghq.com/v2alpha1 kind: DatadogAgent metadata: name: datadog namespace: datadog spec: features: autoscaling: workload: enabled: true eventCollection: unbundleEvents: true global: site: datadoghq.com credentials: apiSecret: secretName: datadog-secret keyName: api-key appSecret: secretName: datadog-secret keyName: app-key clusterName: minikube-dka-demo kubelet: tlsVerify: false override: clusterAgent: env: - name: DD_AUTOSCALING_FAILOVER_ENABLED value: "true" nodeAgent: env: - name: DD_AUTOSCALING_FAILOVER_ENABLED value: "true" ``` ### Stage 4: NGINX application with DatadogPodAutoscaler{% #stage-4-nginx-application-with-datadogpodautoscaler %} This ArgoCD application deploys an NGINX workload with a DatadogPodAutoscaler using a Helm chart. In the `argocd/apps/nginx-dka-demo.yaml` file: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: nginx-dka-demo namespace: argocd annotations: argocd.argoproj.io/sync-wave: "2" spec: project: default source: repoURL: https:/example.com/YOUR-USERNAME/dka-argocd-example targetRevision: HEAD path: charts/nginx-dka-demo destination: server: https://kubernetes.default.svc namespace: nginx-dka-demo syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ignoreDifferences: - group: apps kind: Deployment name: nginx-dka-demo namespace: nginx-dka-demo managedFieldsManagers: - datadog-cluster-agent ``` Create the Helm chart for the NGINX application: In the `charts/nginx-dka-demo/Chart.yaml` file: ```yaml apiVersion: v2 name: nginx-dka-demo description: NGINX demo application with DatadogPodAutoscaler type: application version: 0.1.0 appVersion: "1.0" ``` In the `charts/nginx-dka-demo/values.yaml` file: ```yaml replicaCount: 3 image: repository: nginx tag: latest pullPolicy: IfNotPresent resources: limits: cpu: 500m memory: 512Mi requests: cpu: 250m memory: 256Mi service: type: ClusterIP port: 80 autoscaler: enabled: true minReplicas: 3 maxReplicas: 100 targetCPUUtilization: 70 scaleUp: rules: - type: Percent value: 50 periodSeconds: 120 stabilizationWindowSeconds: 600 strategy: Max scaleDown: rules: - type: Percent value: 20 periodSeconds: 1200 stabilizationWindowSeconds: 600 strategy: Max ``` In the `charts/nginx-dka-demo/templates/deployment.yaml` file: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: nginx-dka-demo labels: app: nginx spec: replicas: {{ .Values.replicaCount }} selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" imagePullPolicy: {{ .Values.image.pullPolicy }} ports: - containerPort: 80 resources: {{- toYaml .Values.resources | nindent 10 }} --- apiVersion: v1 kind: Service metadata: name: nginx-service namespace: nginx-dka-demo spec: selector: app: nginx ports: - port: {{ .Values.service.port }} targetPort: 80 type: {{ .Values.service.type }} ``` In the `charts/nginx-dka-demo/templates/pod-autoscaler.yaml` file: ```yaml {{- if .Values.autoscaler.enabled }} apiVersion: datadoghq.com/v1alpha2 kind: DatadogPodAutoscaler metadata: name: nginx namespace: nginx-dka-demo spec: targetRef: apiVersion: apps/v1 kind: Deployment name: nginx owner: Local constraints: minReplicas: {{ .Values.autoscaler.minReplicas }} maxReplicas: {{ .Values.autoscaler.maxReplicas }} containers: - name: nginx enabled: true objectives: - type: PodResource podResource: name: cpu value: type: Utilization utilization: {{ .Values.autoscaler.targetCPUUtilization }} applyPolicy: mode: Apply update: strategy: Auto scaleUp: strategy: {{ .Values.autoscaler.scaleUp.strategy }} stabilizationWindowSeconds: {{ .Values.autoscaler.scaleUp.stabilizationWindowSeconds }} rules: {{- toYaml .Values.autoscaler.scaleUp.rules | nindent 6 }} scaleDown: strategy: {{ .Values.autoscaler.scaleDown.strategy }} stabilizationWindowSeconds: {{ .Values.autoscaler.scaleDown.stabilizationWindowSeconds }} rules: {{- toYaml .Values.autoscaler.scaleDown.rules | nindent 6 }} {{- end }} ``` ## Deployment instructions{% #deployment-instructions %} After you have set up the configuration files and pushed them to your Git repository, follow these steps to deploy the components using ArgoCD. ### Create Datadog secret{% #create-datadog-secret %} Create a Kubernetes secret with your Datadog API and application keys in the `datadog` namespace: ```bash kubectl create namespace datadog kubectl create secret generic datadog-secret \ --from-literal=api-key=YOUR_API_KEY \ --from-literal=app-key=YOUR_APP_KEY \ -n datadog ``` ### Deploy root application{% #deploy-root-application %} Deploy the root application, which manages all child applications using the App of Apps pattern: ```bash kubectl apply -f argocd/root-app.yaml ``` ArgoCD now monitors your Git repository and automatically deploys all applications in the correct order based on sync waves. ### Enable autoscaling on the cluster in Datadog{% #enable-autoscaling-on-the-cluster-in-datadog %} Navigate to the [autoscaling settings page](https://app.datadoghq.com/orchestration/scaling/settings) in the Datadog UI to enable autoscaling for the cluster. ### Verify sync wave progression{% #verify-sync-wave-progression %} Watch ArgoCD applications synchronize in order: ```bash kubectl get applications -n argocd ``` You should see all applications appear and synchronize in wave order: `datadog-operator` (wave 0), then `datadog-agent` (wave 1), and `nginx-dka-demo` (wave 2). ### Validate deployment{% #validate-deployment %} Verify that the Datadog Operator and CRDs are deployed: ```bash kubectl get crd | grep datadoghq kubectl get pods -n datadog ``` You should see the Datadog CRDs created and the `datadog-operator` pod running. Verify that the Datadog Agent is deployed: ```bash kubectl get datadogagent -n datadog ``` You should see the DatadogAgent custom resource created in the `Running` state. Also verify that the Datadog Agent and `datadog-cluster-agent` pods are running: ```bash kubectl get pods -n datadog ``` Check DatadogPodAutoscaler status: ```bash kubectl get datadogpodautoscaler -n nginx-dka-demo kubectl describe datadogpodautoscaler nginx -n nginx-dka-demo ``` Congratulations, you have a workload managed by the Datadog Kubernetes Autoscaler using GitOps! ## Cleanup{% #cleanup %} To remove all resources, delete the root application. This cascades deletion to all child applications: ```bash kubectl delete application dka-root -n argocd ``` Alternatively, delete applications individually in reverse order: ```bash kubectl delete application nginx-dka-demo -n argocd kubectl delete application datadog-agent -n argocd kubectl delete application datadog-operator -n argocd ``` Delete the Datadog secret: ```bash kubectl delete secret datadog-secret -n datadog ``` ## Troubleshooting{% #troubleshooting %} ### ArgoCD sync failures{% #argocd-sync-failures %} Check application status and sync errors: ```bash kubectl describe application datadog-operator -n argocd kubectl describe application datadog-agent -n argocd kubectl describe application nginx-dka-demo -n argocd ``` View ArgoCD application Controller logs: ```bash kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller ``` ### CRD availability issues{% #crd-availability-issues %} If ArgoCD fails to sync because CRDs are not recognized, verify the Datadog Operator deployed successfully in wave 0: ```bash kubectl get crd | grep datadoghq kubectl get pods -n datadog ``` The sync wave annotations ensure proper ordering, but you can manually refresh the application: ```bash argocd app sync datadog-agent ``` ### Secret configuration problems{% #secret-configuration-problems %} Verify the Datadog secret exists and contains the correct keys: ```bash kubectl get secret datadog-secret -n datadog kubectl describe secret datadog-secret -n datadog ``` The secret must contain `api-key` and `app-key` fields. ### DatadogPodAutoscaler events{% #datadogpodautoscaler-events %} Check DatadogPodAutoscaler events for scaling decisions and errors: ```bash kubectl get events -n nginx-dka-demo --sort-by='.lastTimestamp' ``` ### Cluster Agent logs{% #cluster-agent-logs %} Check Cluster Agent logs for autoscaling-related messages: ```bash kubectl logs -n datadog -l agent.datadoghq.com/component=cluster-agent ``` ## Further reading{% #further-reading %} - [Kubernetes autoscaling guide: determine which solution is right for your use case](https://www.datadoghq.com/blog/kubernetes-autoscaling-datadog/) - [Datadog Kubernetes Autoscaling](https://docs.datadoghq.com/containers/monitoring/autoscaling.md) - [Datadog Cluster Agent](https://docs.datadoghq.com/agent/cluster_agent.md) - [ArgoCD Documentation](https://argo-cd.readthedocs.io/) - [ArgoCD Sync Waves](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/)