Overview

The Datadog Agent helps you securely manage your secrets by integrating with the following secrets management solutions:

• AWS Secrets Manager
• AWS SSM
• Azure KeyVault
• HashiCorp Vault
• File JSON
• File YAML

Instead of hardcoding sensitive values like API keys or passwords in plaintext within configuration files, the Agent can retrieve them dynamically at runtime. To reference a secret in your configuration, use the ENC[<secret_id>] notation. The secret is fetched and loaded in memory but is never written to disk or sent to the Datadog backend.

Note: You cannot use the ENC[] syntax in secret_* settings like secret_backend_command.

Options for retrieving secrets

Option 1: Using native Agent support for fetching secrets

Note: This option is not available for FIPS-enabled Agents at this time.

Starting in Agent version 7.70, the Datadog Agent natively supports several secret management solutions. Two new settings have been introduced to datadog.yaml: secret_backend_type and secret_backend_config.

secret_backend_type is used to specify which secret management solution to use, and secret_backend_config holds additional configuration relevant to that solution.

# datadog.yaml

secret_backend_type: <backend_type>
secret_backend_config:
  <KEY_1>: <VALUE_1>

More specific setup instructions depend on the backend type used. Refer to the appropriate section bellow for further information:

# datadog.yaml
secret_backend_type: aws.secrets
secret_backend_config:
  aws_session:
    aws_region: {regionName}

{
    "prodApiKey": "datadog api key to use",
    "anotherSecret1": "value2",
    "anotherSecret2": "value3",
}

api_key: ENC[My-Secrets;prodApiKey]

secret_backend_type: aws.secrets
secret_backend_config:
  aws_session:
    aws_region: us-east-1

/DatadogAgent/Production/ApiKey = <your_api_key>
/DatadogAgent/Production/ParameterKey2 = ParameterStringValue2
/DatadogAgent/Production/ParameterKey3 = ParameterStringValue3

# datadog.yaml
secret_backend_type: aws.ssm
secret_backend_config:
  aws_session:
    aws_region: us-east-1

api_key: "ENC[/DatadogAgent/Production/ApiKey]"
property1: "ENC[/DatadogAgent/Production/ParameterKey1]"
property2: "ENC[/DatadogAgent/Production/ParameterKey2]"

# datadog.yaml
secret_backend_type: azure.keyvault
secret_backend_config:
  keyvaulturl: {keyVaultURL}

# datadog.yaml

api_key: "ENC[secretKeyNameInKeyVault]"

path "<your mount path>/<additional subpath>" {
  capabilities = ["read"]
}

path "<your_mount_path>/data/<additional_subpath>" {
  capabilities = ["read"]
}

/*
Datadog needs access to mount information to check the Secrets Engine version
number. If access isn't granted, version 1 is assumed.
*/
path "sys/mounts" {
  capabilities = ["read"]
}

/DatadogAgent/Production/apikey: (SecureString) "<your_api_key>"

# datadog.yaml
api_key: "ENC[/Datadog/Production;apikey]" 

secret_backend_type: hashicorp.vault
secret_backend_config:
  vault_address: http://myvaultaddress.net
  vault_session:
    vault_auth_type: aws
    vault_aws_role: Name-of-IAM-role-attached-to-machine
    aws_region: us-east-1 // this field is optional, and will default to us-east-1 if not set

{
  "datadog_api_key": "your_api_key"
}

# datadog.yaml
api_key: "ENC[datadog_api_key]"

secret_backend_type: file.json
secret_backend_config:
  file_path: /path/to/secret.json

datadog_api_key: your api key

# datadog.yaml
api_key: "ENC[datadog_api_key]"
secret_backend_type: file.yaml
secret_backend_config:
  file_path: /path/to/secret.yaml

Option 2: Using the built-in Script for Kubernetes and Docker

For containerized environments, the Datadog Agent’s container images include a built-in script /readsecret_multiple_providers.sh starting with version v7.32.0. This script supports reading secrets from:

• Files: using ENC[file@/path/to/file]
• Kubernetes Secrets: using ENC[k8s_secret@namespace/secret-name/key]

datadog:
  [...]
  secretBackend:
    command: "/readsecret_multiple_providers.sh"

DD_SECRET_BACKEND_COMMAND=/readsecret_multiple_providers.sh

Example: Reading from mounted files

Kubernetes supports exposing Secrets as files inside a pod that the Agent can read to resolve secrets.

In Kubernetes, you can mount a Secret as a volume like this:

  containers:
    - name: agent
      #(...)
      volumeMounts:
        - name: secret-volume
          mountPath: /etc/secret-volume
  #(...)
  volumes:
    - name: secret-volume
      secret:
        secretName: test-secret

You can then reference the secret like this:

password: ENC[file@/etc/secret-volume/password]

Notes:

• The Secret must exist in the same namespace as the pod it is being mounted in.
• The script is able to access all subfolders, including the sensitive /var/run/secrets/kubernetes.io/serviceaccount/token. As such, Datadog recommends using a dedicated folder instead of /var/run/secrets.

Docker swarm secrets are mounted in the /run/secrets folder. For example, the Docker secret db_prod_passsword is located in /run/secrets/db_prod_password in the Agent container. This would be referenced in the configuration with ENC[file@/run/secrets/db_prod_password].

Example: Reading a Kubernetes secret across namespaces

If you want the Agent to read a Secret from a different namespace, use the k8s_secret@ prefix. For example:

password: ENC[k8s_secret@database/database-secret/password]

In this case, you must manually configure RBAC to allow the Agent’s Service Account to read the Secret:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: datadog-secret-reader
  namespace: database
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["database-secret"]
    verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: datadog-read-secrets
  namespace: database
subjects:
  - kind: ServiceAccount
    name: datadog-agent
    apiGroup: ""
    namespace: default
roleRef:
  kind: Role
  name: datadog-secret-reader
  apiGroup: ""

This Role gives access to the Secret: database-secret in the Namespace: database. The RoleBinding links up this permission to the ServiceAccount: datadog-agent in the Namespace: default. This needs to be manually added to your cluster with respect to your resources deployed.

Option 3: Creating a custom executable

To retrieve secrets, the Agent uses an external executable that you provide. The executable is used when new secrets are discovered and are cached for the lifecycle of the Agent. If you need to update or rotate a secret, you must restart the Agent to reload it.

This allow you to use any secret management solution and gives you full control on how the Agent accesses secrets.

The Agent sends to this executable a JSON payload over standard input containing a list of secret handles to resolve. Then, your executable fetches each secret and return them in a JSON format through a standard output.

The following example shows what the Agent sends to your executable on STDIN:

{
  "version": "1.0",
  "secrets": ["secret1", "secret2"]
}

• version (string): The format version.
• secrets (list of strings): Each string is a handle for a secret to fetch.

The executable responds through the following STDOUT output:

{
  "secret1": {"value": "decrypted_value", "error": null},
  "secret2": {"value": null, "error": "could not fetch the secret"}
}

• value (string): The secret value to be used in the configurations. This can be null in the case of an error.
• error (string): An error message or null.

If a secret fails to be resolved (either by returning a non-zero exit code or a non-null error), the related configuration is ignored by the Agent.

Never output sensitive information on stderr. If the binary exits with a different status code than 0, the Agent logs the standard error output of your executable for troubleshooting.

You can also build your own secret retrieval executable using any language. The only requirement is that it follows the input/output format described previously.

Here is a Go example that returns dummy secrets:

package main

import (
  "encoding/json"
  "fmt"
  "io/ioutil"
  "os"
)

type secretsPayload struct {
  Secrets []string `json:secrets`
  Version int      `json:version`
}

func main() {
  data, err := ioutil.ReadAll(os.Stdin)

  if err != nil {
    fmt.Fprintf(os.Stderr, "Could not read from stdin: %s", err)
    os.Exit(1)
  }
  secrets := secretsPayload{}
  json.Unmarshal(data, &secrets)

  res := map[string]map[string]string{}
  for _, handle := range secrets.Secrets {
    res[handle] = map[string]string{
      "value": "decrypted_" + handle,
    }
  }

  output, err := json.Marshal(res)
  if err != nil {
    fmt.Fprintf(os.Stderr, "could not serialize res: %s", err)
    os.Exit(1)
  }
  fmt.Printf(string(output))
}

This transforms your configuration:

instances:
  - server: db_prod
    user: ENC[db_prod_user]
    password: ENC[db_prod_password]

Into the following in memory:

instances:
  - server: db_prod
    user: decrypted_db_prod_user
    password: decrypted_db_prod_password

You can configure the Agent to use the binary to resolve secrets by adding the following:

secret_backend_command: /path/to/binary

Agent security requirements

The Agent runs the provided executable as a sub-process. The execution patterns differ on Linux and Windows.

Note: Your executable shares the same environment variables as the Agent.

Refreshing API/APP keys at runtime

Starting in Agent version v7.67, you can configure the Agent to refresh its API and APP keys at regular intervals without requiring a restart. This relies on the API key and APP key being pulled as secrets.

To enable this, set secret_refresh_interval (in seconds) in your datadog.yaml file:

api_key: ENC[<secret_handle>]

secret_refresh_interval: 3600  # refresh every hour

By default the Agent randomly spreads its first refresh within the specified secret_refresh_interval window. This means that it resolves the API key at startup, then refreshes it within the first interval and every interval after that. This avoids having a fleet of Agents refreshing their API/APP key at the same time.

To prevent downtime, only invalidate the previous API key and APP key when your entire fleet of Agents has pulled the updated keys from your secret management solution. You can track usage of your API keys in the Fleet Management page.

You can disable this behavior by setting:

secret_refresh_scatter: false

To refresh manually, use:

datadog-agent secret refresh

Enabling DDOT collector refresh

If you are using DDOT collector and want to enable API/APP refresh you must add the following additional configuration to your datadog.yaml file:

agent_ipc:
  port: 5051
  config_refresh_interval: 3600

This ensures the DDOT collector remains in-sync with the Agent after secrets are refreshed. Similar to how the Agent periodically verifies its configuration state, the DDOT collector uses this setting to regularly check for updated values from the Agent.

Troubleshooting

Listing detected secrets

The secret command in the Agent CLI shows any errors related to your setup. For example, if the rights on the executable are incorrect. It also lists all handles found, and where they are located.

On Linux, the command outputs file mode, owner and group for the executable. On Windows, ACL rights are listed.

datadog-agent secret
=== Checking executable rights ===
Executable path: /path/to/you/executable
Check Rights: OK, the executable has the correct rights

Rights Detail:
file mode: 100700
Owner username: dd-agent
Group name: dd-agent

=== Secrets stats ===
Number of secrets decrypted: 3
Secrets handle decrypted:
- api_key: from datadog.yaml
- db_prod_user: from postgres.yaml
- db_prod_password: from postgres.yaml

PS C:\> & "$env:ProgramFiles\Datadog\Datadog Agent\bin\agent.exe" secret
=== Checking executable rights ===
Executable path: C:\path\to\you\executable.exe
Check Rights: OK, the executable has the correct rights

Rights Detail:
Acl list:
stdout:

Path   : Microsoft.PowerShell.Core\FileSystem::C:\path\to\you\executable.exe
Owner  : BUILTIN\Administrators
Group  : WIN-ITODMBAT8RG\None
Access : NT AUTHORITY\SYSTEM Allow  FullControl
         BUILTIN\Administrators Allow  FullControl
         WIN-ITODMBAT8RG\ddagentuser Allow  ReadAndExecute, Synchronize
Audit  :
Sddl   : O:BAG:S-1-5-21-2685101404-2783901971-939297808-513D:PAI(A;;FA;;;SY)(A;;FA;;;BA)(A;;0x1200
         a9;;;S-1-5-21-2685101404-2783901971-939297808-1001)

=== Secrets stats ===
Number of secrets decrypted: 3
Secrets handle decrypted:
- api_key: from datadog.yaml
- db_prod_user: from sqlserver.yaml
- db_prod_password: from sqlserver.yaml

Seeing configurations after secrets were injected

To quickly see how the check’s configurations are resolved, you can use the configcheck command:

sudo -u dd-agent -- datadog-agent configcheck

=== a check ===
Source: File Configuration Provider
Instance 1:
host: <decrypted_host>
port: <decrypted_port>
password: <obfuscated_password>
~
===

=== another check ===
Source: File Configuration Provider
Instance 1:
host: <decrypted_host2>
port: <decrypted_port2>
password: <obfuscated_password2>
~
===

Note: The Agent needs to be restarted to pick up changes on configuration files.

Debugging your secret_backend_command

To test or debug outside of the Agent, you can mimic how the Agent runs it:

sudo -u dd-agent bash -c "echo '{\"version\": \"1.0\", \"secrets\": [\"secret1\", \"secret2\"]}' | /path/to/the/secret_backend_command"

.\Set-SecretPermissions.ps1 -SecretBinaryPath C:\secrets\decrypt_secrets.exe
ddagentuser SID: S-1-5-21-3139760116-144564943-2741514060-1076
=== Checking executable permissions ===
Executable path: C:\secrets\decrypt_secrets.exe
Executable permissions: OK, the executable has the correct permissions

Permissions Detail:

stdout:
Path   : Microsoft.PowerShell.Core\FileSystem::C:\secrets\decrypt_secrets.exe
Owner  : BUILTIN\Administrators
Group  : BUILTIN\Administrators
Access : NT AUTHORITY\SYSTEM Allow  FullControl
         BUILTIN\Administrators Allow  FullControl
         DESKTOP-V03BB2P\ddagentuser Allow  ReadAndExecute, Synchronize
Audit  :
Sddl   : O:BAG:BAD:PAI(A;;FA;;;SY)(A;;FA;;;BA)(A;;0x1200a9;;;S-1-5-21-3139760116-144564943-2741514
         060-1076)
stderr:


=== Secrets stats ===
Number of secrets resolved: 0
Secrets handle resolved:

.\secrets_tester.ps1 -user ddagentuser -password a_new_password -executable C:\path\to\your\executable.exe -payload '{"version": "1.0", "secrets": ["secret_ID_1", "secret_ID_2"]}'
Creating new Process with C:\path\to\your\executable.exe
Waiting a second for the process to be up and running
Writing the payload to Stdin
Waiting a second so the process can fetch the secrets
stdout:
{"secret_ID_1":{"value":"secret1"},"secret_ID_2":{"value":"secret2"}}
stderr: None
exit code:
0

Agent refusing to start

The first thing the Agent does on startup is to load datadog.yaml and decrypt any secrets in it. This is done before setting up the logging. This means that on platforms like Windows, errors occurring when loading datadog.yaml aren’t written in the logs, but on stderr. This can occur when the executable given to the Agent for secrets returns an error.

If you have secrets in datadog.yaml and the Agent refuses to start:

• Try to start the Agent manually to be able to see stderr.
• Remove the secrets from datadog.yaml and test with secrets in a check configuration file first.

Testing Kubernetes Permissions

When reading Secrets directly from Kubernetes you can double check your permissions with the kubectl auth command. The general form of this is:

kubectl auth can-i get secret/<SECRET_NAME> -n <SECRET_NAMESPACE> --as system:serviceaccount:<AGENT_NAMESPACE>:<AGENT_SERVICE_ACCOUNT>

Consider the previous Kubernetes Secrets example, where the Secret Secret:database-secret exists in the Namespace: database, and the Service Account ServiceAccount:datadog-agent exists in the Namespace: default.

In this case, use the following command:

kubectl auth can-i get secret/database-secret -n database --as system:serviceaccount:default:datadog-agent

This command returns whether the permissions are valid for the Agent to view this Secret.

Remove trailing line breaks

Some secret management tools automatically add a line break when exporting secrets through files. You can remove these line breaks by setting secret_backend_remove_trailing_line_break: true in the datadog.yaml configuration file, or use the environment variable DD_SECRET_BACKEND_REMOVE_TRAILING_LINE_BREAK to do the same, especially in containerized environments.

Autodiscovery variables in secret handles

It is also possible to use Autodiscovery variables in secret handles. The Agent resolves these variables before resolving the secret. For example:

instances:
  - server: %%host%%
    user: ENC[db_prod_user_%%host%%]
    password: ENC[db_prod_password_%%host%%]

Further Reading

Additional helpful documentation, links, and articles:

AutodiscoveryDOCUMENTATION