Monitor Argo CD Deployments
This page is not yet available in Spanish. We are working on its translation.
If you have any questions or feedback about our current translation project,
feel free to reach out to us!CD Visibility is not available in the selected site () at this time.
Join the Preview!
CD Visibility for Argo CD is in Preview. If you're interested in this feature, complete the form to request access.
Request AccessOverview
Argo CD is a declarative GitOps continuous delivery (CD) tool for Kubernetes. It follows the GitOps pattern by using Git repositories to define the desired application state, and automates the deployment of applications in specified target environments.
Datadog CD Visibility integrates with Argo CD by using Argo CD Notifications. Argo CD notifications consists of two main components:
- Triggers, which define when to send a notification.
- Templates, which define what to send in a notification.
Minimal Setup
The setup below uses the Webhook notification service of Argo CD to send notifications to Datadog.
First, add your Datadog API Key in the argocd-notifications-secret
secret with the dd-api-key
key. See the Argo CD guide for information on modifying the argocd-notifications-secret
.
Then, modify the argocd-notifications-cm
ConfigMap to create the notification service, template, and trigger to send notifications to Datadog:
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-notifications-cm
data:
service.webhook.cd-visibility-webhook: |
url: https://webhook-intake.
/api/v2/webhook
headers:
- name: "DD-CD-PROVIDER-ARGOCD"
value: "true"
- name: "DD-API-KEY"
value: $dd-api-key
- name: "Content-Type"
value: "application/json"
template.cd-visibility-template: |
webhook:
cd-visibility-webhook:
method: POST
body: |
{
"app": {{toJson .app}},
"context": {{toJson .context}},
"service_type": {{toJson .serviceType}},
"recipient": {{toJson .recipient}},
"commit_metadata": {{toJson (call .repo.GetCommitMetadata .app.status.operationState.syncResult.revision)}}
}
trigger.cd-visibility-trigger: |
- when: app.status.operationState.phase in ['Succeeded', 'Failed', 'Error'] and app.status.health.status in ['Healthy', 'Degraded']
send: [cd-visibility-template]
- when: app.status.operationState.phase == 'Running' and app.status.health.status in ['Healthy', 'Degraded']
send: [cd-visibility-template]
The following resources have been added:
- The
cd-visibility-webhook
service targets the Datadog intake and configures the correct headers for the request. The DD-API-KEY
header references the dd-api-key
entry added previously in the argocd-notifications-secret
. - The
cd-visibility-template
defines what to send in the request for the cd-visibility-webhook
service. - The
cd-visibility-trigger
defines when to send the notification, and it references the cd-visibility-template
.
The call to populate the commit_metadata
field is not required. The field is used to enrich the payload with Git information.
If your Argo CD application source is not a defined commit SHA (for example, if you are using Helm repositories), adjust the body by removing that line and the comma in the previous line.
After the notification service, trigger, and template have been added to the config map, you can subscribe any of your Argo CD applications to the integration.
Modify the annotations of the Argo CD application by either using the Argo CD UI or modifying the application definition with the following annotations:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
annotations:
notifications.argoproj.io/subscribe.cd-visibility-trigger.cd-visibility-webhook: ""
dd_env: <YOUR_ENV>
dd_service: <YOUR_SERVICE>
dd_customtags: "region:us1-east, team:backend"
From the above snippet:
- The notifications annotation subscribes the Argo CD application to the notification setup created above. See the Argo CD official guide for more details on applications subscriptions.
- You can use the
dd_env
annotation to configure the environment of the application. Replace YOUR_ENV
above with the environment
to which this application is deploying (for example: staging
or prod
). If you don’t set this annotation,
the environment defaults to none
. - You can use the
dd_service
annotation to configure the service of the application. Replace YOUR_SERVICE
above with the service
that the Argo CD application is deploying (for example: transaction-service
). When this annotation is used, the service
name is added to all the deployment executions generated from the application. Moreover, if your service is
registered in Service Catalog, the team name is also added to all the deployment executions. If your Argo CD
application is configured to deploy more than one service, see Tag an Argo CD application deploying multiple services. - You can use the
dd_customtags
annotation to optionally add custom tags to the deployment executions generated for this Argo CD application.
The value should be set to a comma-separated list of tags, structured as key:value
pairs.
After you have subscribed your Argo CD application by adding the annotations above, new deployments of the application will start to appear in Datadog.
The Recommended Setup section below contains recommended actions to improve the monitoring reported in CD Visibility.
Recommended Setup
Change duration to wait for resources health
The duration reported in deployment events matches the sync duration in Argo CD. However, the sync duration generally represents the time spent by Argo CD to sync the Git repository state and the Kubernetes cluster state.
This means that what happens after the sync (for example, the time spent by the Kubernetes resources to start up) is not included in the duration.
To change the duration reported to wait until the configured resources have started up and reached a healthy state, add a new no-op resource monitored by your Argo CD application, with a PostSync Hook annotation.
The PostSync Hook will run after all the resources have reached a Healthy state, and the Argo CD sync will wait on the PostSync Hook result to update the application status as Healthy.
Below is represented an example of a PostSync Hook Job that runs a simple echo
command.
apiVersion: batch/v1
kind: Job
metadata:
name: cdvisibility-postsync-job # Ensures that the Argo CD sync duration waits for resources health
annotations:
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
containers:
- name: noop-echo
image: alpine:latest
command: ["echo", "all the sync resources have reached a healthy state"]
restartPolicy: Never
backoffLimit: 0
Correlate deployments with CI pipelines
By default, the Git metadata reported in deployment events is associated with the repository that Argo CD monitors. However, a common setup is to:
- Have an application repository, storing the source code, and a configuration repository, storing the Kubernetes manifests. Then, configure Argo CD to monitor the configuration repository, as outlined in the Argo CD Best Practices page.
- When a change occurs in the application repository, perform an automated commit that updates the configuration repository (for example, changing the current image of a Kubernetes resource).
The following diagram represents an example of this kind of setup:
The datadog-ci deployment correlate
command can be used to correlate one or more configuration repository commits with an application repository commit. When an Argo CD deployment occurs, the configuration commit information in the deployment event is replaced by the related application repository commit, if any. There are two possible ways to perform the correlation using the command: automatic and manual setup. Both methods require version 2.44.0
or higher of the datadog-ci
CLI.
In this method, the command automatically infers the current application commit (that is, the commit of the pipeline that the command is running in) and the configuration repository commits based on the current Git environment. For this to work properly, the command needs to be run between committing the changes and pushing them to the configuration repository:
- job: JobToUpdateConfigurationRepository
run: |
# Update the configuration files
...
git commit
# Correlate the deployment with the CI pipeline
export DD_BETA_COMMANDS_ENABLED=1
datadog-ci deployment correlate --provider argocd
git push
If the automatic setup is too limited for your use case, you can provide the configuration repository URL and SHAs manually:
- job: JobToUpdateConfigurationRepository
run: |
# Correlate the deployment with the CI pipeline
export DD_BETA_COMMANDS_ENABLED=1
datadog-ci deployment correlate --provider argocd --config-repo <CONFIG_REPO_URL> --config-shas <COMMIT_SHA>
You can omit the --config-repo
option if the CI is checked out to the configuration repository. See command syntax for additional details.
Note: Even if a single repository is used to store both the source code and the Kubernetes manifest, you still need to run this command to correctly associate deployments and CI pipelines.
Validation
If the command has been correctly run, deployments contain Git metadata from the application repository instead of the configuration repository. Also, the deployment executions view now contains a new Pipeline tab representing the related CI pipeline trace.
Tag an Argo CD application deploying multiple services
If your Argo CD application deploys more than one service, Datadog can automatically infer the services deployed from an application sync. Datadog infers the services based on the Kubernetes resources that were modified.
To enable automatic service tagging, you need to monitor your Kubernetes infrastructure using the Datadog Agent and your Kubernetes resources should have the following labels:
tags.datadoghq.com/service
(required): specifies the Datadog service of this resource. For more information, see Unified Service Tagging.team
(optional): specifies the Datadog team of this resource. If this label is omitted, the team is automatically retrieved from Service Catalog based on the service label.
Only the Kubernetes resources with the following kinds are eligible: Deployment
, ReplicaSet
, StatefulSet
, Service
, DaemonSet
, Pod
, Job
, and CronJob
.
Add the following annotations to your Argo CD application:
dd_multiservice
: true
. This annotation specifies whether Datadog automatically infers the services deployed in a sync based on the changed Kubernetes resources.dd_k8s_cluster
: set to the name of the Kubernetes cluster that the Argo CD application deploys to. The name must match the name reported in the Datadog Kubernetes product.
For example:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
annotations:
notifications.argoproj.io/subscribe.cd-visibility-trigger.cd-visibility-webhook: ""
dd_env: <YOUR_ENV>
dd_multiservice: true
dd_k8s_cluster: example-cluster
Visualize deployments in Datadog
The Deployments and Executions pages populate with data after a deployment has finished. For more information, see Search and Manage and CD Visibility Explorer.
Troubleshooting
If notifications are not sent, examine the logs of the argocd-notification-controller
pod. The controller logs when it is sending a notification (for example: Sending notification ...
) and when it fails to notify a recipient
(for example: Failed to notify recipient ...
). For additional troubleshooting scenarios, see the official Argo CD documentation.
Further reading
Más enlaces, artículos y documentación útiles: