DBM Setup Architectures

Overview

The steps required for setting up Database Monitoring in Datadog varies based on the type of database you’re using (Postgres, MySQL, SQL Server), and the host provider (self-hosted, AWS, Google Cloud SQL, or Azure). No matter which database or host provider you use, to be able to use Database Monitoring for your databases, you need the following:

  • A Datadog Agent
  • Host for your Datadog Agent
  • Read-only access for your databases

Agent

The Datadog Agent is lightweight software that monitors system metrics such as CPU, memory, and network activity. It also connects to the database as a SQL user to collect data about database performance.

For self-hosted databases, you install the agent directly onto the host that is hosting your database. For cloud-managed databases such as AWS RDS and Azure SQL, you configure the Agent to connect to your databases remotely.

Self-hosted databases

The self-hosted setup goes through the database process on the database host, which also hosts the Agent. Then after connecting to the internet, it goes through to Datadog's backend.

In a self-hosted setup, the Datadog Agent collects system metrics from the operating system host, database metrics directly from the database, and log events from database logs.

For self-hosted setups, you install the Agent directly onto the database host so that you have full visibility into the health of your system running the database process.

You grant the Agent read-only access to your database, and configure the integration. The Agent must log in as a user so it can run read-only queries on your database.

Instructions for setting up Database Monitoring with a self-hosted provider:

Cloud-managed databases

If your setup is cloud-managed (with providers such as AWS RDS or Aurora, Google Cloud SQL, or Azure), you install the Agent on a separate host and configure it to connect to each managed instance.

Database Monitoring collects system metrics such as CPU, memory, disk usage, logs, and related telemetry directly from the cloud provider using the Datadog integration with that provider.

The database instance is separate from the Agent host, which is separate from the Datadog backend. The cloud API connects to the Datadog AWS integration through the internet.

You can install the Agent on any cloud VM (for example, EC2) provided it can connect to your database instances.

If you are not running your own Kubernetes cluster, Datadog recommends using your cloud provider’s orchestration tools. For example, you can use AWS ECS to host the Datadog Agent, as the Agent already exists as a Docker container.

Kubernetes

If you are running your apps on Kubernetes, use the Datadog Cluster Agent with Database Monitoring, which can run cluster checks across your pods.

Database instances in a cloud provider connect to nodes in a Kubernetes cluster, which then connect to the Datadog backend through the internet. The cloud API connects directly to the Datadog AWS integration.

The Cluster Agent automatically distributes the database instances across a pool of Agents. This ensures that only one instance of each check runs, as opposed to each node-based Agent pod running this corresponding check. The Cluster Agent holds the configurations and dynamically dispatches them to node-based Agents. The Agents on each node connect to the Cluster Agent every 10 seconds and retrieve the configurations to run.

If an Agent stops reporting, the Cluster Agent removes it from the active pool and dispatches the configurations to other Agents. This ensures one (and only one) instance always runs even as nodes are added and removed from the cluster. This becomes important when you have a large number of database instances — the Cluster Agent spreads the cluster checks across the different nodes.

Aurora

If you are using Aurora, the Agent must be connected to the individual Aurora instance (not the cluster endpoint) because the Agent must connect directly to the host being monitored.

For monitoring Aurora databases, the Agent should not connect to the database through a proxy, load balancer, connection pooler such as pgbouncer, or the Aurora cluster endpoint. Each Datadog Agent must have knowledge of the underlying hostname and should be a single host for its lifetime, even in cases of failover. Otherwise, the values of metrics become incorrect.

Further Reading