Impact of the CircleCI Security Incident on the Datadog Agent

Page changelog
DateDescription
13 Jan 2023Initial publish
16 Jan 2023Updated rpm_check script v1.1.0, edits for clarity
17 Jan 2023Updated rpm_check script v1.2.0, clearer identify and remedy steps
3 Feb 2023Clarify which Agents 5 versions are signed with the affected key
Summary: Check your RPM-based Linux hosts (RHEL, CentOS, Rocky Linux, AlmaLinux, Amazon Linux, SUSE/SLES, Fedora) to find and fix any that trust the key with fingerprint 60A389A44A0C32BAE3C03F0B069B56F54172A230.

On January 4th, 2023, Datadog was notified by CircleCI that they were investigating a security incident that may have led to leaking of stored secrets. Datadog identified a single secret stored in CircleCI that could theoretically be misused by a potential attacker, an old RPM GNU Privacy Guard (GPG) private signing key and its passphrase. This page provides information about the implications of the potential leak, actions you should take on your hosts, and the measures Datadog is taking to mitigate any risks to our customers.

Note: As of January 16th, 2023, Datadog has no indication that the key was actually leaked or misused, but we are still taking and advising the following actions out of an abundance of caution.

The affected key

The impacted RPM GPG signing key has the fingerprint 60A389A44A0C32BAE3C03F0B069B56F54172A230, and is accessible in our signing keys location. This key was historically used to sign:

  • Agent 5 releases up to and including v5.32.8 and Agent 6 releases up to and including v6.13.0 (datadog-agent package)
  • Standalone DogStatsD 6 releases and Standalone DogStatsD 7 releases up to and including v7.23.1 (datadog-dogstatsd package)
Note: Official Datadog repositories were not compromised. The signing key, if actually leaked, could be used to construct an RPM package that looks like it is from Datadog, but it would not be enough to place such a package in our official package repositories. To be effective, a hypothetical attacker with the affected key would need to be able to upload the constructed RPM package to a repository used by your hosts.

Finding affected hosts

The incident can affect hosts running RPM-based Linux distributions, including RHEL, CentOS, Rocky Linux, AlmaLinux, Amazon Linux, SUSE/SLES, and Fedora. Hosts running other operating systems such as macOS, Windows, Debian, and Ubuntu, and Container Agents are not affected.

Before you start, if you have a large infrastructure, prioritize your search for hosts that trust the key. Because of the variety of ways packages can be installed and updated over time, Datadog recommends you check all RPM-based Linux hosts with a Datadog package installed. To help you prioritize which hosts to check first, consider the following guidance. The following scenarios are highly likely to be affected:

  • Agent v5 or v6
  • Standalone DogStatsD v6 or v7.23.2 and earlier

On RPM-based Linux hosts, the following scenarios are unlikely to be affected, but still worth checking your hosts:

  • Agent v7
  • Standalone DogStatsD v7.23.2+.

The following scenarios are not affected:

  • The Agent was installed with Datadog packages on macOS, Windows, Debian, or Ubuntu.
  • The host uses the Container Agent.

Check each host to see if it trusts the affected key from either the RPM database or the Datadog repo file, or both:

  1. Check the RPM database by running the following command:

    $ rpm -q gpg-pubkey-4172a230-55dd14f6

    The host trusts the key and requires action if the command runs successfully and prints gpg-pubkey-4172a230-55dd14f6.The host does not trust the key in the RPM database if the command fails, exiting with a non-0 exit code and outputting a message like package gpg-pubkey-4172a230-55dd14f6 is not installed.

  2. Check the Datadog repo file. The default location for the file is:

    • RHEL, CentOS, Rocky Linux, AlmaLinux, Amazon Linux, and Fedora: /etc/yum.repos.d/datadog.repo
    • OpenSUSE and SLES: /etc/zypp/repos.d/datadog.repo

    If the repo file mentions DATADOG_RPM_KEY.public under the gpgkey entry as shown in the following examples, the host trusts the affected key and requires action:

    • https://s3.amazonaws.com/public-signing-keys/DATADOG_RPM_KEY.public
    • https://keys.datadoghq.com/DATADOG_RPM_KEY.public
    • https://s3.amazonaws.com/yum.datadoghq.com/DATADOG_RPM_KEY.public
    • https://yum.datadoghq.com/DATADOG_RPM_KEY.public

If either one or both of these is identified as trusting the key, take the actions listed in the following sections to secure the affected host. If neither of these is identified as trusting the key, no further action is needed.

Securing the affected hosts

Ensure that your hosts stop trusting the affected key. If the previous steps indicated that a host trusts the key, follow these steps:

  1. If you use configuration automation tools or plugins, such as the Ansible Datadog role, update them to the latest version listed in What Datadog is doing.

    Remaining on older versions of these automation tools or plugins might reverse your remediation efforts. If you can’t yet update to the new fixed versions, add the manual key deletion steps (step 3 and 4) to your automation tool runbooks, and ensure these run after the Datadog tools and plugins in your runbook order.

  2. For hosts that are set up using the official Datadog install scripts, run the latest version of the install script to untrust the key and provision the updated repo files.

  3. If running rpm -q gpg-pubkey-4172a230-55dd14f6 still detects the key, delete the key from the RPM database and stop trusting it by running the following command:

    $ sudo rpm --erase gpg-pubkey-4172a230-55dd14f6

  4. If DATADOG_RPM_KEY.public is still listed in the repo file, delete the key by removing the gpgkey line that ends with DATADOG_RPM_KEY.public. If this is the only gpgkey entry in your repo file, replace it with https://keys.datadoghq.com/DATADOG_RPM_KEY_CURRENT.public. Read more in the section Implications of no longer trusting the affected key.

  5. Out of an abundance of caution, verify that Datadog built the packages signed by the affected key on the affected hosts by running this script:

    $ curl -o /tmp/rpm_check.sh https://docs.datadoghq.com/resources/sh/rpm_check.sh && chmod +x /tmp/rpm_check.sh
$ /tmp/rpm_check.sh

    This script verifies that any installed Datadog packages signed by the affected key were built by Datadog and searches for any packages signed by the affected key that weren’t built by Datadog.

    If the output contains lines that start with [ ERROR ] report these to Datadog Support along with the full script output.

Implications of no longer trusting the affected key

If the host in question uses Agent 7, there is no implication. Agent 7 packages were never signed with the affected key.

Hosts can no longer install:

  • Agents earlier than v6.14.0. Upgrade to at least v6.14.0 or Agent v7.
  • Standalone DogStatsD v6 or Standalone DogStatsD earlier than v7.24.0 (datadog-dogstatsd packages). Upgrade to Standalone DogStatsD 7.24.0+.
  • Agents earlier than v5.32.8. Install Agent v5.32.9+ or upgrade to v6.14.0+ or Agent v7.

What Datadog is doing

We released a new Agent 5 RPM for CentOS/RHEL, 5.32.9-1, signed with the current RPM signing key, C6559B690CA882F023BDF3F63F4D1729FD4BF915. The RPM is available through the Agent 5 RPM repository.

We released new versions of Agent installation methods to ensure they make hosts safe by explicitly removing the affected key from the RPM database and the Datadog repo file: