AWS RDS

Crawler

Overview

Amazon Relational Database Service (RDS) is a web service that makes it easy to setup, operate, and scale a relational database in the cloud. Enable this integration to see all your RDS metrics in Datadog

There are 3 options for monitoring RDS instances. You can choose to use standard or enhanced, and then optionally turn on the native database integration as well if you wish.

• Standard RDS Integration - The standard integration requires selecting RDS on the left side of the AWS integration tile. You will receive metrics about your instance as often as your Cloudwatch integration allows. All RDS Engine types are supported.

• Enhanced RDS Integration - The enhanced integration requires additional configuration and is only available for MySQL, Aurora, PostgreSQL, and MariaDB engines. Additional metrics are available but an AWS Lambda is required to submit the metrics to Datadog. The higher granularity and additional required services may result in additional AWS charges.

• RDS + Native Database Integration - You can also choose to turn on the Native Database Integration. This is available for MySQL, Aurora, MariaDB, SQL Server, and PostgreSQL engine types. To get the metrics from RDS and the ones from the native integration to match up, you will need to use the dbinstanceidentifier tag on the native integration based on the identifier you assign to the RDS instance. The RDS instances will automatically have the tag assigned.

Setup

Installation

Standard RDS Integration

If you haven’t already, set up the Amazon Web Services integration first.

Enhanced RDS Integration

Enable Enhanced Monitoring for your RDS instance. This can either be done during instance creation or afterwards by choosing Modify under Instance Actions. We recommend choosing 15 for Monitoring Granularity.

Create your KMS Key

1. Open the Encryption keys section of the AWS Identity and Access Management (IAM) console at https://console.aws.amazon.com/iam/home#encryptionKeys. For Region, choose the appropriate AWS Region. Do not use the region selector in the navigation bar (top right corner).
2. Choose Create Key.
3. Enter an Alias for the key, such as lambda-datadog-key. Note: An alias cannot begin with aws. Aliases that begin with aws are reserved by Amazon Web Services to represent AWS-managed CMKs in your account.
4. Add the appropriate administrators to determine who can administer this key.
5. No roles need to be added at this time.
6. Save your KMS key.

Create your Lambda function

1. From the Lambda Management Console, create a new Lambda Function. Your Lambda function must be in the same region as the KMS key you created.
2. Choose Serverless Application Repository, search for and select Datadog-RDS-Enhanced
3. Give the application a unique name.
4. Paste the Id of the key created in the previous section in the KMSKeyId parameter and deploy.
5. Once the application is deployed, open up the newly created Lambda Function.
6. Navigate to the Environment variables section. Replace YOUR_KEY with your Datadog API and APP key in the following format: '{"api_key":"<YOUR_API_KEY>", "app_key":"<YOUR_APP_KEY>"}'
7. Open the Encryption configuration section and select Enable helpers for encryption in transit.
8. In KMS key to encrypt in transit select the same key as the one below in KMS key to encrypt at rest.
9. Press the Encrypt button next to the JSON blob you just entered.
10. Go up and press Save.
11. Navigate to the Designer section and add a new trigger on the RDSOSMetrics CloudWatch Logs group.
12. Give the filter a name and optional filter pattern and press Save.

When clicking on test button for your lambda function you might get this error:

{ 
  "stackTrace": [ [ "/var/task/lambda_function.py", 
    109, 
    "lambda_handler", 
    "event = json.loads(gzip.GzipFile(fileobj=StringIO(event['awslogs']['data'].decode('base64'))).read())" 
    ] 
  ], 
  "errorType": "KeyError", 
  "errorMessage": "'awslogs'" 
}

Please ignore it, the Test button doesn’t work with this setup.

Native Database Integration

1. Navigate to the AWS Console and open the RDS section to find the instance you want to monitor.

2. Copy the endpoint URL (e.g. mysqlrds.blah.us-east1.rds.amazonaws.com:3306); You will need it when you configure the agent. Also make a note of the DB Instance identifier (e.g. mysqlrds). You will need it to create graphs and dashboards.

Configuration

Standard RDS Integration

1. In the AWS integration tile, ensure that RDS is checked under metric collection.

2. Add those permissions to your Datadog IAM policy in order to collect Amazon RDS metrics:

   • rds:DescribeDBInstances: Descrive RDS instances to add tags.
   • rds:ListTagsForResource: Add custom tags on RDS instances.
   • rds:DescribeEvents: Add events related to RDS databases.

   For more information on RDS policies, review the documentation on the AWS website.

3. Install the Datadog - AWS RDS integration.

Enhanced RDS Integration

1. In the AWS integration tile, ensure that RDS is checked under metric collection.

2. Add those permissions to your Datadog IAM policy in order to collect Amazon RDS metrics:

   • rds:DescribeDBInstances: Descrive RDS instances to add tags.
   • rds:ListTagsForResource: Add custom tags on RDS instances.
   • rds:DescribeEvents: Add events related to RDS databases.

   For more information on RDS policies, review the documentation on the AWS website.

3. Install the Datadog - AWS RDS integration.

Native Database Integration

Configure an agent and connect to your RDS instance by editing the appropriate yaml file in your conf.d directory and then restart your agent:

If you are using MySQL, MariaDB, or Aurora, then edit mysql.yaml:

init_config:

instances:
  - server: mysqlrds.blah.us-east1-rds.amazonaws.com # The endpoint URL from the AWS console
    user: my_username
    pass: my_password
    port: 3306
    tags:
      - dbinstanceidentifier:my_own_instance

If you are using PostgreSQL, then edit postgres.yaml:

init_config:

instances:
  - host: mysqlrds.blah.us-east1-rds.amazonaws.com
    port: 5432
    username: my_username
    password: my_password
    dbname: db_name
    tags:
      - dbinstanceidentifier:my_own_instance

If you are using Microsoft SQL Server, then edit sqlserver.yaml

init_config:

instances:
  - host: mysqlrds.blah.us-east1-rds.amazonaws.com,1433
    username: my_username
    password: my_password
    tags:
      - dbinstanceidentifier:my_own_instance

Validation

To validate that the native database integration is working, run sudo /etc/init.d/datadog-agent info. You should see something like the following:

Checks
======

[...]

  mysql
  -----
      - instance #0 [OK]
      - Collected 8 metrics & 0 events

Usage

After a few minutes, RDS metrics and metrics from MySQL, Aurora, MariaDB, SQL Server, or PostgreSQL are accessible in Datadog in the Metrics Explorer, in Graphs and in Alerts.
Here’s an example of an Aurora dashboard displaying a number of metrics from both RDS and the MySQL integration. Metrics from both integrations on the instance quicktestrds are unified using the dbinstanceidentifier tag.

Here is the default dashboard for MySQL on Amazon RDS:

Learn more about how to monitor MySQL on Amazon RDS performance metrics thanks to our series of posts. We detail the key performance metrics, how to collect them, and how to use Datadog to monitor MySQL on Amazon RDS.

Log collection

Enable RDS Logging

It is possible to forward MySQL and MariaDB logs to amazon cloudwatch. Follow instruction here to start sending your RDS logs to Cloudwatch.

Send logs to Datadog

1. If you haven’t already, set up the Datadog log collection AWS Lambda function.
2. Once the lambda function is installed, manually add a trigger on the Cloudwatch Log group that contains your API Gateway logs in the AWS console:
   Select the corresponding CloudWatch Log group, add a filter name (but feel free to leave the filter empty) and add the trigger:

Once done, go in your Datadog Log section to start exploring your logs!

Data Collected

Metrics

Each of the metrics retrieved from AWS will be assigned the same tags that appear in the AWS console, including but not limited to host name, security-groups, and more.

Events

The AWS RDS integration does not include any event at this time.

Service Checks

The AWS RDS integration does not include any service check at this time.

Troubleshooting

Need help? Contact Datadog Support.

Further Reading

• Monitoring RDS MySQL performance metrics
  
• Key metrics for AWS RDS PostgreSQL monitoring
  
• Monitoring Amazon Aurora performance metrics