Setting Up Database Monitoring for Amazon RDS managed MySQL

Database Monitoring provides deep visibility into your MySQL databases by exposing query metrics, query samples, explain plans, connection data, system metrics, and telemetry for the InnoDB storage engine.

The Agent collects telemetry directly from the database by logging in as a read-only user. Do the following setup to enable Database Monitoring with your MySQL database:

  1. Configure the AWS integration
  2. Configure database parameters
  3. Grant the Agent access to the database
  4. Install and configure the Agent
  5. Install the RDS integration

Before you begin

Supported MySQL versions
5.6, 5.7, or 8.0+
Supported MariaDB versions
10.5, 10.6, or 10.11

Database Monitoring for MariaDB is supported with known limitations.
Supported Agent versions
7.36.1+
Performance impact
The default Agent configuration for Database Monitoring is conservative, but you can adjust settings such as the collection interval and query sampling rate to better suit your needs. For most workloads, the Agent represents less than one percent of query execution time on the database and less than one percent of CPU.

Database Monitoring runs as an integration on top of the base Agent (see benchmarks).
Proxies, load balancers, and connection poolers
The Datadog Agent must connect directly to the host being monitored, preferably through the instance endpoint. The Agent should not connect to the database through a proxy, load balancer, or connection pooler. If the Agent connects to different hosts while it is running (as in the case of failover, load balancing, and so on), the Agent calculates the difference in statistics between two hosts, producing inaccurate metrics.
Data security considerations
See Sensitive information for information about what data the Agent collects from your databases and how to ensure it is secure.

Configure the AWS integration

Enable Standard Collection in the Resource Collection section of your Amazon Web Services integration tile.

Configure MySQL settings

Configure the following in the DB Parameter Group and then restart the server for the settings to take effect:

ParameterValueDescription
performance_schema1Required. Enables the Performance Schema.
max_digest_length4096Required for collection of larger queries. Increases the size of SQL digest text in events_statements_* tables. If left at the default value then queries longer than 1024 characters will not be collected.
performance_schema_max_digest_length4096Must match max_digest_length.
performance_schema_max_sql_text_length4096Must match max_digest_length.
ParameterValueDescription
performance_schema1Required. Enables the Performance Schema.
max_digest_length4096Required for collection of larger queries. Increases the size of SQL digest text in events_statements_* tables. If left at the default value then queries longer than 1024 characters will not be collected.
performance_schema_max_digest_length4096Must match max_digest_length.

Grant the Agent access

The Datadog Agent requires read-only access to the database in order to collect statistics and queries.

The following instructions grant the Agent permission to login from any host using datadog@'%'. You can restrict the datadog user to be allowed to login only from localhost by using datadog@'localhost'. See the MySQL documentation for more info.

Create the datadog user and grant basic permissions:

CREATE USER datadog@'%' IDENTIFIED by '<UNIQUEPASSWORD>';
ALTER USER datadog@'%' WITH MAX_USER_CONNECTIONS 5;
GRANT REPLICATION CLIENT ON *.* TO datadog@'%';
GRANT PROCESS ON *.* TO datadog@'%';
GRANT SELECT ON performance_schema.* TO datadog@'%';

Create the datadog user and grant basic permissions:

CREATE USER datadog@'%' IDENTIFIED BY '<UNIQUEPASSWORD>';
GRANT REPLICATION CLIENT ON *.* TO datadog@'%' WITH MAX_USER_CONNECTIONS 5;
GRANT PROCESS ON *.* TO datadog@'%';
GRANT SELECT ON performance_schema.* TO datadog@'%';

Create the following schema:

CREATE SCHEMA IF NOT EXISTS datadog;
GRANT EXECUTE ON datadog.* to datadog@'%';
GRANT CREATE TEMPORARY TABLES ON datadog.* TO datadog@'%';

Create the explain_statement procedure to enable the Agent to collect explain plans:

DELIMITER $$
CREATE PROCEDURE datadog.explain_statement(IN query TEXT)
    SQL SECURITY DEFINER
BEGIN
    SET @explain := CONCAT('EXPLAIN FORMAT=json ', query);
    PREPARE stmt FROM @explain;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
END $$
DELIMITER ;

Additionally, create this procedure in every schema from which you want to collect explain plans. Replace <YOUR_SCHEMA> with your database schema:

DELIMITER $$
CREATE PROCEDURE <YOUR_SCHEMA>.explain_statement(IN query TEXT)
    SQL SECURITY DEFINER
BEGIN
    SET @explain := CONCAT('EXPLAIN FORMAT=json ', query);
    PREPARE stmt FROM @explain;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
END $$
DELIMITER ;
GRANT EXECUTE ON PROCEDURE <YOUR_SCHEMA>.explain_statement TO datadog@'%';

Runtime setup consumers

With RDS, performance schema consumers can’t be enabled permanently in a configuration. Create the following procedure to give the Agent the ability to enable performance_schema.events_* consumers at runtime.

DELIMITER $$
CREATE PROCEDURE datadog.enable_events_statements_consumers()
    SQL SECURITY DEFINER
BEGIN
    UPDATE performance_schema.setup_consumers SET enabled='YES' WHERE name LIKE 'events_statements_%';
    UPDATE performance_schema.setup_consumers SET enabled='YES' WHERE name = 'events_waits_current';
END $$
DELIMITER ;
GRANT EXECUTE ON PROCEDURE datadog.enable_events_statements_consumers TO datadog@'%';

Securely store your password

Store your password using secret management software such as Vault. You can then reference this password as ENC[<SECRET_NAME>] in your Agent configuration files: for example, ENC[datadog_user_database_password]. See Secrets Management for more information.

The examples on this page use datadog_user_database_password to refer to the name of the secret where your password is stored. It is possible to reference your password in plain text, but this is not recommended.

Install and configure the Agent

To monitor RDS hosts, install the Datadog Agent in your infrastructure and configure it to connect to each instance endpoint remotely. The Agent does not need to run on the database, it only needs to connect to it. For additional Agent installation methods not mentioned here, see the Agent installation instructions.

To configure this check for an Agent running on a host, for example when you provision a small EC2 instance for the Agent to collect from an RDS database:

Edit the mysql.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory to start collecting your MySQL metrics. See the sample mysql.d/conf.yaml for all available configuration options, including those for custom metrics.

Add this configuration block to your mysql.d/conf.yaml to collect MySQL metrics:

init_config:

instances:
  - dbm: true
    host: '<AWS_INSTANCE_ENDPOINT>'
    port: 3306
    username: datadog
    password: 'ENC[datadog_user_database_password]' # from the CREATE USER step earlier, stored as a secret

    # After adding your project and instance, configure the Datadog AWS integration to pull additional cloud data such as CPU and Memory.
    aws:
      instance_endpoint: '<AWS_INSTANCE_ENDPOINT>'

Restart the Agent to start sending MySQL metrics to Datadog.

To configure the Database Monitoring Agent running in a Docker container such as in ECS or Fargate, you can set the Autodiscovery Integration Templates as Docker labels on your agent container.

Note: The Agent must have read permission on the Docker socket for Autodiscovery of labels to work.

Command line

Get up and running quickly by executing the following command to run the agent from your command line. Replace the values to match your account and environment:

export DD_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export DD_AGENT_VERSION=7.36.1

docker run -e "DD_API_KEY=${DD_API_KEY}" \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -l com.datadoghq.ad.check_names='["mysql"]' \
  -l com.datadoghq.ad.init_configs='[{}]' \
  -l com.datadoghq.ad.instances='[{
    "dbm": true,
    "host": "<AWS_INSTANCE_ENDPOINT>",
    "port": 3306,
    "username": "datadog",
    "password": "<UNIQUEPASSWORD>"
  }]' \
  gcr.io/datadoghq/agent:${DD_AGENT_VERSION}

Dockerfile

Labels can also be specified in a Dockerfile, so you can build and deploy a custom agent without changing any infrastructure configuration:

FROM gcr.io/datadoghq/agent:7.36.1

LABEL "com.datadoghq.ad.check_names"='["mysql"]'
LABEL "com.datadoghq.ad.init_configs"='[{}]'
LABEL "com.datadoghq.ad.instances"='[{"dbm": true, "host": "<AWS_INSTANCE_ENDPOINT>", "port": 3306,"username": "datadog","password": "ENC[datadog_user_database_password]"}]'

If you have a Kubernetes cluster, use the Datadog Cluster Agent for Database Monitoring.

Follow the instructions to enable the cluster checks if not already enabled in your Kubernetes cluster. You can declare the MySQL configuration either with static files mounted in the Cluster Agent container or using service annotations:

Helm

Complete the following steps to install the Datadog Cluster Agent on your Kubernetes cluster. Replace the values to match your account and environment.

  1. Complete the Datadog Agent installation instructions for Helm.

  2. Update your YAML configuration file (datadog-values.yaml in the Cluster Agent installation instructions) to include the following:

    clusterAgent:
      confd:
        postgres.yaml: |-
          cluster_check: true
          init_config:
          instances:
            - dbm: true
              host: <INSTANCE_ADDRESS>
              port: 3306
              username: datadog
              password: 'ENC[datadog_user_database_password]'      
    
    clusterChecksRunner:
      enabled: true
    
  3. Deploy the Agent with the above configuration file from the command line:

    helm install datadog-agent -f datadog-values.yaml datadog/datadog
    
For Windows, append --set targetSystem=windows to the helm install command.

Configure with mounted files

To configure a cluster check with a mounted configuration file, mount the configuration file in the Cluster Agent container on the path /conf.d/mysql.yaml:

cluster_check: true  # Make sure to include this flag
init_config:
instances:
  - dbm: true
    host: '<AWS_INSTANCE_ENDPOINT>'
    port: 3306
    username: datadog
    password: 'ENC[datadog_user_database_password]'

Configure with Kubernetes service annotations

Rather than mounting a file, you can declare the instance configuration as a Kubernetes Service. To configure this check for an Agent running on Kubernetes, create a Service in the same namespace as the Datadog Cluster Agent:

apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    tags.datadoghq.com/env: '<ENV>'
    tags.datadoghq.com/service: '<SERVICE>'
  annotations:
    ad.datadoghq.com/service.check_names: '["mysql"]'
    ad.datadoghq.com/service.init_configs: '[{}]'
    ad.datadoghq.com/service.instances: |
      [
        {
          "dbm": true,
          "host": "<AWS_INSTANCE_ENDPOINT>",
          "port": 3306,
          "username": "datadog",
          "password": "ENC[datadog_user_database_password]"
        }
      ]      
spec:
  ports:
  - port: 3306
    protocol: TCP
    targetPort: 3306
    name: mysql

The Cluster Agent automatically registers this configuration and begins running the MySQL check.

Validate

Run the Agent’s status subcommand and look for mysql under the Checks section, or see the Databases page to get started!

Example Agent Configurations

One agent connecting to multiple hosts

It is common to configure a single Agent host to connect to multiple remote database instances (see Agent installation architectures for DBM). To connect to multiple hosts, create an entry for each host in the MySQL integration config. In these cases, Datadog recommends limiting the number of instances per Agent to a maximum of 10 database instances to guarantee reliable performance.

init_config:
instances:
  - dbm: true
    host: example-service-primary.example-host.com
    port: 3306
    username: datadog
    password: 'ENC[datadog_user_database_password]'
    tags:
      - 'env:prod'
      - 'team:team-discovery'
      - 'service:example-service'
  - dbm: true
    host: example-service-replica-1.example-host.com
    port: 3306
    username: datadog
    password: 'ENC[datadog_user_database_password]'
    options:
      replication: true
    tags:
      - 'env:prod'
      - 'team:team-discovery'
      - 'service:example-service'
  - dbm: true
    host: example-service-replica-2.example-host.com
    port: 3306
    username: datadog
    password: 'ENC[datadog_user_database_password]'
    options:
      replication: true
    tags:
      - 'env:prod'
      - 'team:team-discovery'
      - 'service:example-service'
    [...]

Running custom queries

To collect custom metrics, use the custom_queries option. See the sample mysql.d/conf.yaml for more details.

init_config:
instances:
  - dbm: true
    host: localhost
    port: 3306
    username: datadog
    password: 'ENC[datadog_user_database_password]'
    custom_queries:
    - query: SELECT age, salary, hours_worked, name FROM hr.employees;
      columns:
        - name: custom.employee_age
          type: gauge
        - name: custom.employee_salary
           type: gauge
        - name: custom.employee_hours
           type: count
        - name: name
           type: tag
      tags:
        - 'table:employees'

Working with hosts through a proxy

If the Agent must connect through a proxy such as the Cloud SQL Auth proxy, all telemetry is tagged with the hostname of the proxy rather than the database instance. Use the reported_hostname option to set a custom override of the hostname detected by the Agent.

init_config:
instances:
  - dbm: true
    host: localhost
    port: 5000
    username: datadog
    password: 'ENC[datadog_user_database_password]'
    reported_hostname: example-service-primary
  - dbm: true
    host: localhost
    port: 5001
    username: datadog
    password: 'ENC[datadog_user_database_password]'
    reported_hostname: example-service-replica-1

Install the RDS Integration

To see infrastructure metrics from AWS, such as CPU, alongside the database telemetry in DBM, install the RDS integration (optional).

Troubleshooting

If you have installed and configured the integrations and Agent as described and it is not working as expected, see Troubleshooting.

Further reading

Additional helpful documentation, links, and articles: