Overview

This check monitors Vault cluster health and leader changes.

Setup

Installation

The Vault check is included in the Datadog Agent package.

Starting with version 3.4.0, this OpenMetrics-based integration has a latest mode (use_openmetrics: true) and a legacy mode (use_openmetrics: false). To get all the most up-to-date features, Datadog recommends enabling the latest mode. Note that latest mode requires Python 3. For more information, see Latest and Legacy Versioning For OpenMetrics-based Integrations.

1. Ensure you have enabled Prometheus metrics in the Vault configuration.

2. For the Vault check to work properly, you need to either enable unauthenticated access to Vault metrics (using Vault version 1.3.0 or later) or provide a Vault client token:

   • To enable unauthenticated access, set Vault’s unauthenticated_metrics_access configuration to true. This allows unauthenticated access to the /v1/sys/metrics endpoint.

     Note: The /sys/metrics endpoint requires Vault v1.1.0 or later to collect metrics.

   • To use a Vault client token, follow the example below. The example uses the JWT auth method, but you can also use additional auth methods.

The Vault integration requires the following capabilities:

• Content of metrics_policy.hcl:

  path "sys/metrics*" {
    capabilities = ["read", "list"]
  }
  

• Setup policy and role:

  $ vault policy write metrics /path/to/metrics_policy.hcl
  $ vault auth enable jwt
  $ vault write auth/jwt/config jwt_supported_algs=RS256 jwt_validation_pubkeys=@<PATH_TO_PUBLIC_PEM>
  $ vault write auth/jwt/role/datadog role_type=jwt bound_audiences=<AUDIENCE> user_claim=name token_policies=metrics
  $ vault agent -config=/path/to/agent_config.hcl
  

• Content of agent_config.hcl:

  exit_after_auth = true
  pid_file = "/tmp/agent_pid"
  
  auto_auth {
    method "jwt" {
      config = {
        path = "<JWT_CLAIM_PATH>"
        role = "datadog"
      }
    }
  
    sink "file" {
      config = {
        path = "<CLIENT_TOKEN_PATH>"
      }
    }
  }
  
  vault {
    address = "http://0.0.0.0:8200"
  }
  

Configuration

Validation

Run the Agent’s status subcommand and look for vault under the Checks section.

Data Collected

Metrics

[Metrics starting with vault.replication.fetchRemoteKeys, vault.replication.merkleDiff, and vault.replication.merkleSync] are not reported unless the replication is in an unhealthy state.

Events

vault.leader_change: This event fires when the cluster leader changes.

Service Checks

vault.can_connect
Returns Critical if the Agent cannot connect to the Vault endpoint, OK otherwise.
Statuses: ok, critical

vault.unsealed
Returns Critical if the vault is sealed, OK otherwise.
Statuses: ok, critical

vault.initialized
Returns Critical if the vault isn’t initialized, OK otherwise.
Statuses: ok, critical

vault.prometheus.health
Returns CRITICAL if the check cannot access the Metrics endpoint. Returns OK otherwise. This service check is only available with the legacy mode (use_openmetrics: false).
Statuses: ok, critical

vault.openmetrics.health
Returns CRITICAL if the Agent is unable to connect to the OpenMetrics endpoint, otherwise returns OK. This service check is only available with the latest mode (use_openmetrics: true).
Statuses: ok, critical

Troubleshooting

Need help? Contact Datadog support.

Further Reading

Additional helpful documentation, links, and articles:

• Monitor HashiCorp Vault with Datadog
• Monitor HashiCorp Vault metrics and logs
• Tools for HashiCorp Vault monitoring
• How to monitor HashiCorp Vault with Datadog