Amazon RDS

Crawler

Overview

Amazon Relational Database Service (RDS) is a web service used to setup, operate, and scale a relational database in the cloud. Enable this integration to see all your RDS metrics in Datadog.

Note: If you are setting up this integration for the EU, set the environment variable DD_SITE outside of the code to datadoghq.eu or set the variable in the code as follows:

DD_SITE = os.getenv("DD_SITE", default="datadoghq.eu")

There are 3 options for monitoring RDS instances. Choose to use standard or enhanced, and then optionally turn on the native database integration.

• Standard RDS Integration - The standard integration requires selecting RDS on the left side of the AWS integration tile. This allows you to receive metrics about your instance as often as your CloudWatch integration allows. All RDS Engine types are supported.

• Enhanced RDS Integration - The enhanced integration requires additional configuration and is only available for MySQL, Aurora, PostgreSQL, and MariaDB engines. Additional metrics are available but an AWS Lambda is required to submit the metrics to Datadog. The higher granularity and additional required services may result in additional AWS charges.

• RDS + Native Database Integration - The Native Database Integration is optional and available for MySQL, Aurora, MariaDB, SQL Server, and PostgreSQL engine types. To get the metrics from RDS and the ones from the native integration to match up, use the dbinstanceidentifier tag on the native integration based on the identifier you assign to the RDS instance. The RDS instances are automatically assigned the tag.

Setup

Installation

Standard RDS Integration

If you haven’t already, set up the Amazon Web Services integration first.

Enhanced RDS Integration

Enable Enhanced Monitoring for your RDS instance. This can either be done during instance creation or afterwards by choosing Modify under Instance Actions. It is recommended to choose 15 for Monitoring Granularity.

Create your KMS Key

1. Open the KMS home at https://console.aws.amazon.com/kms/home.
2. Go into Customer managed keys.
3. Choose Create Key.
4. Enter an Alias for the key, such as lambda-datadog-key. Note: An alias cannot begin with aws. Aliases that begin with aws are reserved by Amazon Web Services to represent AWS-managed CMKs in your account.
5. Add the appropriate administrators to determine who can administer this key.
6. No roles need to be added.
7. Save your KMS key.

Create your Lambda function

1. From the Lambda Management Console, create a new Lambda Function. Your Lambda function must be in the same region as the KMS key you created.
2. Choose Serverless Application Repository, search for and select Datadog-RDS-Enhanced
3. Give the application a unique name.
4. Paste the Id of the key created in the previous section in the KMSKeyId parameter and deploy.
5. Once the application is deployed, open up the newly created Lambda Function (click on the function under “Resource”).

6. Scroll down to the Environment variables section. Replace <YOUR_API_KEY> with your Datadog API key in the format: {"api_key":"<YOUR_API_KEY>"}:

7. Open the Encryption configuration section and select Enable helpers for encryption in transit.
8. In KMS key to encrypt in transit select the same key as the one below in KMS key to encrypt at rest.
9. Press the Encrypt button next to the JSON blob you just entered.
10. Go up and press Save.
11. Create a new trigger using the RDSOSMetrics Cloudwatch log group as the source.
12. Give the filter a name and optional filter pattern and press Save.

When clicking on test button for your lambda function you might get this error:

{
  "stackTrace": [ [ "/var/task/lambda_function.py",
    109,
    "lambda_handler",
    "event = json.loads(gzip.GzipFile(fileobj=StringIO(event['awslogs']['data'].decode('base64'))).read())"
    ]
  ],
  "errorType": "KeyError",
  "errorMessage": "'awslogs'"
}

This can be ignored. The Test button doesn’t work with this setup.

Native Database Integration

1. Navigate to the AWS Console and open the RDS section to find the instance you want to monitor.

2. Make note of the endpoint URL (e.g. mysqlrds.blah.us-east1.rds.amazonaws.com:3306) which is used to configure the Agent. Also make a note of the DB Instance identifier (e.g. mysqlrds) which is used to create graphs and dashboards.

Configuration

Standard or Enhanced RDS Integration

1. In the AWS integration tile, ensure that RDS is checked under metric collection.

2. Add the following permissions to your Datadog IAM policy in order to collect Amazon RDS metrics. For more information on RDS policies, review the documentation on the AWS website.

1. Install the Datadog - AWS RDS integration.

Native Database Integration

Configure an Agent and connect to your RDS instance by editing the appropriate yaml file in your conf.d directory and then restart your Agent:

For RDS Aurora, edit the YAML file of the database flavor you are using.

If you are using MySQL or MariaDB, then edit mysql.yaml:

init_config:

instances:
  - server: mysqlrds.blah.us-east1-rds.amazonaws.com # The endpoint URL from the AWS console
    user: <USERNAME>
    pass: <PASSWORD>
    port: 3306
    tags:
      - dbinstanceidentifier:<INSTANCE_NAME>

If you are using PostgreSQL, then edit postgres.yaml:

init_config:

instances:
  - host: mysqlrds.blah.us-east1-rds.amazonaws.com
    port: 5432
    username: <USERNAME>
    password: <PASSWORD>
    dbname: <DB_NAME>
    tags:
      - dbinstanceidentifier:<DB_INSTANCE_NAME>

If you are using Microsoft SQL Server, then edit sqlserver.yaml:

init_config:

instances:
  - host: mysqlrds.blah.us-east1-rds.amazonaws.com,1433
    username: <USERNAME>
    password: <PASSWORD>
    tags:
      - dbinstanceidentifier:<DB_INSTANCE_NAME>

Validation

To validate that the native database integration is working, run sudo /etc/init.d/datadog-agent info. You should see something like the following:

Checks
======

[...]

  mysql
  -----
      - instance #0 [OK]
      - Collected 8 metrics & 0 events

Usage

After a few minutes, RDS metrics and metrics from MySQL, Aurora, MariaDB, SQL Server, or PostgreSQL are accessible in Datadog from the metrics explorer, dashboards, and alerts. Here’s an example of an Aurora dashboard displaying a number of metrics from both RDS and the MySQL integration. Metrics from both integrations on the instance quicktestrds are unified using the dbinstanceidentifier tag.

Here is the default dashboard for MySQL on Amazon RDS:

Learn more about how to monitor MySQL on Amazon RDS performance metrics in our series of posts. We detail the key performance metrics, how to collect them, and how to use Datadog to monitor MySQL on Amazon RDS.

Log collection

Enable RDS Logging

It is possible to forward MySQL, MariaDB, and Postgres logs to Amazon CloudWatch. Follow instruction here to start sending your RDS logs to CloudWatch.

Send logs to Datadog

1. If you haven’t already, set up the Datadog log collection AWS Lambda function.
2. Once the lambda function is installed, manually add a trigger on the CloudWatch Log group that contains your RDS logs in the AWS console:
   Select the corresponding CloudWatch Log group, add a filter name (but feel free to leave the filter empty) and add the trigger:

Once done, go in your Datadog Log section to start exploring your logs!

Data Collected

In addition to the metrics collected from the database engines, you also receive the following RDS metrics.

Metrics

Each of the metrics retrieved from AWS is assigned the same tags that appear in the AWS console, including but not limited to host name, security-groups, and more.

Events

The AWS RDS integration includes events related to DB instances, security groups, snapshots, and parameter groups. See example events below:

Service Checks

aws.rds.read_replica_status Monitors the read replication status. This check returns one of the following statuses:

• OK - replicating or connecting
• CRITICAL - error or terminated
• WARNING - stopped
• UNKNOWN - other

Troubleshooting

Need help? Contact Datadog support.

Further Reading

Additional helpful documentation, links, and articles:

Monitoring RDS MySQL performance metricsBLOG Key metrics for AWS RDS PostgreSQL monitoringBLOG Monitoring Amazon Aurora performance metricsBLOG