Datadog's Amazon Web Services integration is built to collect ALL metrics from CloudWatch. Datadog strives to continually update the docs to show every sub-integration, but cloud services rapidly release new metrics and services so the list of integrations are sometimes lagging.
For Account ID, enter 464622532012 (Datadog’s account ID). This means that you are granting Datadog read only access to your AWS data.
Select Require external ID and enter the one generated in the AWS integration tile. Make sure you leave Require MFA disabled. For more information about the External ID, see the IAM User Guide.
Click Next: Permissions.
If you’ve already created the policy, search for it on this page and select it, then skip to step 12. Otherwise, click Create Policy, which opens in a new window.
Select the JSON tab. To take advantage of every AWS integration offered by Datadog, use policy snippet below in the textbox. As other components are added to an integration, these permissions may change.
Click Next: Tags and Review policy.
Name the policy DatadogAWSIntegrationPolicy or one of your own choosing, and provide an apt description.
Click Create policy, then close this window.
Back in the “Create role” window, refresh the list of policies and select the policy you just created.
Select the Role Delegation tab and select Manually.
Enter your AWS Account ID without dashes, for example: 123456789012. Your Account ID can be found in the ARN of the role created during the installation of the AWS integration.
Enter the name of the created role. Note: The role name you enter in the integration tile is case sensitive and must exactly match the role name created on the AWS side.
Choose the AWS services to collect metrics from on the left side of the dialog.
Optionally, check the box Enable resource configuration collection if you would like to enable Resource Collection (required for some products and features).
Optionally, add tags to all hosts and metrics.
Optionally, monitor a subset of EC2 instances by entering the AWS tags in the textbox to hosts with tag. Note: This also applies to an instance’s attached EBS volumes.
Optionally, monitor a subset of Lambdas by entering the AWS tags in the textbox to Lambdas with tag.
Click Install Integration.
AWS
In your AWS console, set up an IAM user to be used by the Datadog integration.
Generate an access key and secret key for the Datadog integration IAM user.
Select the Access Keys (GovCloud or China Only) tab.
Enter your AWS Access Key and AWS Secret Key. Only access and secret keys for GovCloud and China are accepted.
Choose the services to collect metrics from on the left side of the dialog.
Optionally, add tags to all hosts and metrics.
Optionally, monitor a subset of EC2 instances by entering the AWS tags in the textbox to hosts with tag. Note: This also applies to an instance’s attached EBS volumes.
Optionally, monitor a subset of Lambdas by entering the AWS tags in the textbox to Lambdas with tag.
Click Install Integration.
AWS IAM Permissions
AWS IAM permissions enable Datadog to collect metrics, tags, CloudWatch events, and other data necessary to monitor your AWS environment.
To correctly set up the AWS Integration, you must attach the relevant IAM policies to the Datadog AWS Integration IAM Role in your AWS account.
AWS Integration IAM Policy
The set of permissions necessary to use all the integrations for individual AWS services.
The following permissions included in the policy document use wild cards such as List* and Get*. If you require strict policies, use the complete action names as listed and reference the Amazon API documentation for your respective services.
There are two ways of sending AWS service logs to Datadog:
Kinesis Firehose destination: Use the Datadog destination in your Kinesis Firehose delivery stream to forward logs to Datadog. It is recommended to use this approach when sending logs from CloudWatch in a very high volume.
Forwarder Lambda function: Deploy the Datadog Forwarder Lambda function, which subscribes to S3 buckets or your CloudWatch log groups and forwards logs to Datadog. You must use this approach to send traces, enhanced metrics, or custom metrics from Lambda functions asynchronously through logs. Datadog also recommends you use this approach to sending logs from S3 or other resources that cannot directly stream data to Kinesis.
Metric collection
There are two ways to send AWS metrics to Datadog:
Metric polling: API polling comes out of the box with the AWS integration. A metric-by-metric crawl of the CloudWatch API pulls data and sends it to Datadog. New metrics are pulled every ten minutes, on average.
Metric streams with Kinesis Firehose: You can use Amazon CloudWatch Metric Streams and Amazon Kinesis Data Firehose to see your metrics. Note: This method has a two to three minute latency, and requires a separate setup.
Resource collection
Some Datadog products leverage information about how your AWS resources (such as S3 Buckets, RDS snapshots, and CloudFront distributions) are configured. Datadog collects this information by making read only API calls into your AWS account.
Cloud Security Posture Management
Setup
If you do not have the AWS integration setup yet for your AWS account, complete the set up process above and make sure to enable resource collection when mentioned.
If you already have the AWS Integration setup for other Datadog products, but do not yet have resource collection enabled, do one of the following:
Automatic - Update your CloudFormation Template
In the CloudFormation console, find the main stack you used to install the datadog integration and select Update
Select Replace current template
Select Amazon S3 URL, enter https://datadog-cloudformation-template.s3.amazonaws.com/aws/main.yaml and click next
Set CloudSecurityPostureManagementPermissions to true and click next without modifying other existing parameters until you reach the Review page. Here you can verify the change set preview.
Check the two acknowledgment boxes at the bottom and click Update stack.
Manual
Attach the AWS managed SecurityAudit policy to your Datadog AWS IAM role. You can find this policy in the AWS console.
Click on the AWS account where you wish to enable resource collection.
Go to the Resource collection section for that account and check the box Route resource data to the Cloud Security Posture Management product
At the bottom left of the tile, click Update Configuration
Alarm collection
There are two ways to send AWS CloudWatch alarms to the Datadog Event Stream:
Alarm polling: Alarm polling comes out of the box with the AWS integration and fetches metric alarms through the DescribeAlarmHistory API. If you follow this method, your alarms are categorized under the event source Amazon Web Services. Note: The crawler does not collect composite alarms.
SNS topic: You can see all AWS CloudWatch alarms in your event stream by subscribing the alarms to an SNS topic, then forwarding the SNS messages to Datadog. To learn how to receive SNS messages as events in Datadog, see Receive SNS messages. If you follow this method, your alarms are categorized under the event source Amazon SNS.
Data Collected
Metrics
aws.logs.incoming_bytes (gauge)
The volume of log events in uncompressed bytes uploaded to Cloudwatch Logs. Shown as byte
aws.logs.incoming_log_events (count)
The number of log events uploaded to Cloudwatch Logs. Shown as event
aws.logs.forwarded_bytes (gauge)
The volume of log events in compressed bytes forwarded to the subscription destination. Shown as byte
aws.logs.forwarded_log_events (count)
The number of log events forwarded to the subscription destination. Shown as event
aws.logs.delivery_errors (count)
The number of log events for which CloudWatch Logs received an error when forwarding data to the subscription destination. Shown as event
aws.logs.delivery_throttling (count)
The number of log events for which CloudWatch Logs was throttled when forwarding data to the subscription destination. Shown as event
aws.events.invocations (count)
Measures the number of times a target is invoked for a rule in response to an event. This includes successful and failed invocations but does not include throttled or retried attempts until they fail permanently.
aws.events.failed_invocations (count)
Measures the number of invocations that failed permanently. This does not include invocations that are retried or that succeeded after a retry attempt
aws.events.triggered_rules (count)
Measures the number of triggered rules that matched with any event.
aws.events.matched_events (count)
Measures the number of events that matched with any rule.
aws.events.throttled_rules (count)
Measures the number of triggered rules that are being throttled.
aws.usage.call_count (count)
The number of specified operations performed in your account Shown as operation
aws.usage.resource_count (count)
The number of specified resources in your account Shown as resource
Events
Events from AWS are collected on a per AWS-service basis. See the your AWS service’s documentation to learn more about collected events.
Tags
The following tags are collected with the AWS integration. Note: Some tags only display on specific metrics.