Overview

Argo CD is a declarative GitOps continuous delivery (CD) tool for Kubernetes. It follows the GitOps pattern by using Git repositories to define the desired application state, and automates the deployment of applications in specified target environments.

Datadog CD Visibility integrates with Argo CD by using Argo CD Notifications. Argo CD notifications consists of two main components:

1. Triggers, which define when to send a notification.
2. Templates, which define what to send in a notification.

Setup

For more information on how to set up Argo CD notifications using webhooks, see the official Argo CD guide.

The first step is to create the service containing the Datadog intake URL and the Datadog API Key:

1. Add your Datadog API Key in the argocd-notifications-secret secret with the dd-api-key key. See the Argo CD guide for information on modifying the argocd-notifications-secret.
2. Add a notification service in the argocd-notifications-cm config map with the following format:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  service.webhook.cd-visibility-webhook: |
    url: https://webhook-intake./api/v2/webhook
    headers:
    - name: "DD-CD-PROVIDER-ARGOCD"
      value: "true"
    - name: "DD-API-KEY"
      value: $dd-api-key
    - name: "Content-Type"
      value: "application/json"    

cd-visibility-webhook is the name of the notification service, and $dd-api-key is a reference to the API Key stored in the argocd-notifications-secret secret.

The second step is to add the template in the same argocd-notifications-cm config map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  template.cd-visibility-template: |
    webhook:
      cd-visibility-webhook:
        method: POST
        body: |
            {
              "app": {{toJson .app}},
              "context": {{toJson .context}},
              "service_type": {{toJson .serviceType}},
              "recipient": {{toJson .recipient}},
              "commit_metadata": {{toJson (call .repo.GetCommitMetadata .app.status.operationState.syncResult.revision)}}
            }    

cd-visibility-template is the name of the template, and cd-visibility-webhook is a reference to the service created above.

The third step is to add the trigger, again in the same argocd-notifications-cm config map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  trigger.cd-visibility-trigger: |
    - when: app.status.operationState.phase in ['Succeeded', 'Failed', 'Error'] and app.status.health.status in ['Healthy', 'Degraded']
      send: [cd-visibility-template]
    - when: app.status.operationState.phase == 'Running' and app.status.health.status in ['Healthy', 'Degraded']
      send: [cd-visibility-template]    

cd-visibility-trigger is the name of the trigger, and cd-visibility-template is a reference to the template created above.

After the notification service, trigger, and template have been added to the config map, you can subscribe any of your Argo CD applications to the integration. Modify the annotations of the Argo CD application by either using the Argo CD UI or modifying the application definition with the following annotations:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    notifications.argoproj.io/subscribe.cd-visibility-trigger.cd-visibility-webhook: ""
    dd_env: <YOUR_ENV>
    dd_service: <YOUR_SERVICE>

There are three annotations:

1. The notifications annotation subscribes the Argo CD application to the notification setup created above.
2. The dd_env annotation configures the environment of the application. Replace YOUR_ENV above with the environment to which this application is deploying (for example: staging or prod). If you don’t set this annotation, the environment defaults to none.
3. The dd_service annotation configures the service of the application. Replace YOUR_SERVICE above with the service that the Argo CD application is deploying (for example: transaction-service). When this annotation is used, the service name is added to all the deployment executions generated from the application. Moreover, if your service is registered in Service Catalog, the team name is also added to all the deployment executions. If your Argo CD application is configured to deploy more than one service, see Tag an Argo CD application deploying multiple services.

See the Argo CD official guide for more details on applications subscriptions.

After this final step is completed, you can start monitoring your Argo CD deployments in Datadog.

Adding custom tags to deployment executions

You can optionally add custom tags to the deployment executions generated from Argo CD applications deployments. These tags can be used to filter, group, and aggregate deployment executions in Datadog. To add custom tags, add the dd_customtags annotation to your Argo CD application annotations and set the value to a comma-separated list of tags, structured as key:value pairs. For example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    notifications.argoproj.io/subscribe.cd-visibility-trigger.cd-visibility-webhook: ""
    dd_env: <YOUR_ENV>
    dd_customtags: "region:us1-east, team:backend"

Tag an Argo CD application deploying multiple services

If your Argo CD application deploys more than one service, Datadog can automatically infer the services deployed from an application sync. Datadog infers the services based on the Kubernetes resources that were modified.

To enable automatic service tagging, you need to monitor your Kubernetes infrastructure using the Datadog Agent and your Kubernetes resources should have the following labels:

• tags.datadoghq.com/service (required): specifies the Datadog service of this resource. For more information, see Unified Service Tagging.
• team (optional): specifies the Datadog team of this resource. If this label is omitted, the team is automatically retrieved from Service Catalog based on the service label.

Only the Kubernetes resources with the following kinds are eligible: Deployment, ReplicaSet, StatefulSet, Service, DaemonSet, Pod, Job, and CronJob.

Add the following annotations to your Argo CD application:

• dd_multiservice: true. This annotation specifies whether Datadog automatically infers the services deployed in a sync based on the changed Kubernetes resources.
• dd_k8s_cluster: set to the name of the Kubernetes cluster that the Argo CD application deploys to. The name must match the name reported in the Datadog Kubernetes product.

For example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    notifications.argoproj.io/subscribe.cd-visibility-trigger.cd-visibility-webhook: ""
    dd_env: <YOUR_ENV>
    dd_multiservice: true
    dd_k8s_cluster: example-cluster

Visualize deployments in Datadog

The Deployments and Executions pages populate with data after a deployment is executed. For more information, see Search and Manage and CD Visibility Explorer.

Correlate deployments with CI pipelines

By default, the Git metadata reported in deployment events is associated with the repository that Argo CD monitors. However, a common setup is to:

• Have an application repository, storing the source code, and a configuration repository, storing the Kubernetes manifests. Then, configure Argo CD to monitor the configuration repository, as outlined in the Argo CD Best Practices page.
• When a change occurs in the application repository, perform an automated commit that updates the configuration repository (for example, changing the current image of a Kubernetes resource).

The following diagram represents an example of this kind of setup:

The datadog-ci deployment correlate command can be used to correlate one or more configuration repository commits with an application repository commit. When an Argo CD deployment occurs, the configuration commit information in the deployment event is replaced by the related application repository commit, if any. There are two possible ways to perform the correlation using the command: automatic and manual setup. Both methods require version 2.44.0 or higher of the datadog-ci CLI.

- job: JobToUpdateConfigurationRepository
  run: |
    # Update the configuration files
    ...
    git commit
    # Correlate the deployment with the CI pipeline
    export DD_BETA_COMMANDS_ENABLED=1
    datadog-ci deployment correlate --provider argocd
    git push    

- job: JobToUpdateConfigurationRepository
  run: |
    # Correlate the deployment with the CI pipeline
    export DD_BETA_COMMANDS_ENABLED=1
    datadog-ci deployment correlate --provider argocd --config-repo <CONFIG_REPO_URL> --config-shas <COMMIT_SHA>    

Note: Even if a single repository is used to store both the source code and the Kubernetes manifest, you still need to run this command to correctly associate deployments and CI pipelines.

Validation

If the command has been correctly run, deployments contain Git metadata from the application repository instead of the configuration repository. Also, the deployment executions view now contains a new Pipeline tab representing the related CI pipeline trace.

Troubleshooting

If notifications are not sent, examine the logs of the argocd-notification-controller pod. The controller logs when it is sending a notification (for example: Sending notification ...) and when it fails to notify a recipient (for example: Failed to notify recipient ...). For additional troubleshooting scenarios, see the official Argo CD documentation.

Further reading

Additional helpful documentation, links, and articles:

Learn about Deployment VisibilityDOCUMENTATION Learn how to query and visualize deployment executionsDOCUMENTATION