AWS

Crawler

Overview

Connect to Amazon Web Services (AWS) to:

• See automatic AWS status updates in your event stream
• Get CloudWatch metrics for EC2 hosts without installing the Agent
• Tag your EC2 hosts with EC2-specific information
• See EC2 scheduled maintenance events in your stream
• Collect CloudWatch metrics and events from many other AWS products
• See CloudWatch alarms in your event stream

To quickly get started using the AWS integration, check out the AWS getting started guide.

Setup

Use one of the following methods to integrate your AWS accounts into Datadog for metric, trace, and log collection:

• Role delegation (Automatic): Use a CloudFormation template to automatically set up the necessary AWS role (recommended).
• Role delegation (Manual): Manually create the necessary role and copy the required credentials in the respective form.
• Access keys: Used for GovCloud or China only

AWS IAM Permissions

AWS IAM permissions enable Datadog to collect metrics, tags, CloudWatch events, and other data necessary to monitor your AWS environment.

To correctly set up the AWS Integration, you must attach the relevant IAM policies to the Datadog AWS Integration IAM Role in your AWS account.

AWS Integration IAM Policy

The set of permissions necessary to use all the integrations for individual AWS services.

The following permissions included in the policy document use wild cards such as List* and Get*. If you require strict policies, use the complete action names as listed and reference the Amazon API documentation for your respective services.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "apigateway:GET",
                "autoscaling:Describe*",
                "backup:List*",
                "budgets:ViewBudget",
                "cloudfront:GetDistributionConfig",
                "cloudfront:ListDistributions",
                "cloudtrail:DescribeTrails",
                "cloudtrail:GetTrailStatus",
                "cloudtrail:LookupEvents",
                "cloudwatch:Describe*",
                "cloudwatch:Get*",
                "cloudwatch:List*",
                "codedeploy:List*",
                "codedeploy:BatchGet*",
                "directconnect:Describe*",
                "dynamodb:List*",
                "dynamodb:Describe*",
                "ec2:Describe*",
                "ecs:Describe*",
                "ecs:List*",
                "elasticache:Describe*",
                "elasticache:List*",
                "elasticfilesystem:DescribeFileSystems",
                "elasticfilesystem:DescribeTags",
                "elasticfilesystem:DescribeAccessPoints",
                "elasticloadbalancing:Describe*",
                "elasticmapreduce:List*",
                "elasticmapreduce:Describe*",
                "es:ListTags",
                "es:ListDomainNames",
                "es:DescribeElasticsearchDomains",
                "fsx:DescribeFileSystems",
                "fsx:ListTagsForResource",
                "health:DescribeEvents",
                "health:DescribeEventDetails",
                "health:DescribeAffectedEntities",
                "kinesis:List*",
                "kinesis:Describe*",
                "lambda:GetPolicy",
                "lambda:List*",
                "logs:DeleteSubscriptionFilter",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:DescribeSubscriptionFilters",
                "logs:FilterLogEvents",
                "logs:PutSubscriptionFilter",
                "logs:TestMetricFilter",
                "organizations:DescribeOrganization",
                "rds:Describe*",
                "rds:List*",
                "redshift:DescribeClusters",
                "redshift:DescribeLoggingStatus",
                "route53:List*",
                "s3:GetBucketLogging",
                "s3:GetBucketLocation",
                "s3:GetBucketNotification",
                "s3:GetBucketTagging",
                "s3:ListAllMyBuckets",
                "s3:PutBucketNotification",
                "ses:Get*",
                "sns:List*",
                "sns:Publish",
                "sqs:ListQueues",
                "states:ListStateMachines",
                "states:DescribeStateMachine",
                "support:DescribeTrustedAdvisor*",
                "support:RefreshTrustedAdvisorCheck",
                "tag:GetResources",
                "tag:GetTagKeys",
                "tag:GetTagValues",
                "xray:BatchGetTraces",
                "xray:GetTraceSummaries"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

AWS Security Audit Policy

To use Cloud Security Posture Management, attach AWS’s managed SecurityAudit Policy to your Datadog IAM role.

Log collection

There are two ways of sending AWS service logs to Datadog:

• Kinesis Firehose destination: Use the Datadog destination in your Kinesis Firehose delivery stream to forward logs to Datadog. It is recommended to use this approach when sending logs from CloudWatch in a very high volume.
• Forwarder Lambda function: Deploy the Datadog Forwarder Lambda function, which subscribes to S3 buckets or your CloudWatch log groups and forwards logs to Datadog. You must use this approach to send traces, enhanced metrics, or custom metrics from Lambda functions asynchronously through logs. Datadog also recommends you use this approach to sending logs from S3 or other resources that cannot directly stream data to Kinesis.

Metric collection

There are two ways to send AWS metrics to Datadog:

• Metric polling: API polling comes out of the box with the AWS integration. A metric-by-metric crawl of the CloudWatch API pulls data and sends it to Datadog. New metrics are pulled every ten minutes, on average.
• Metric streams with Kinesis Firehose: You can use Amazon CloudWatch Metric Streams and Amazon Kinesis Data Firehose to see your metrics. Note: This method has a two to three minute latency, and requires a separate setup.

Resource collection

Some Datadog products leverage information about how your AWS resources (such as S3 Buckets, RDS snapshots, and CloudFront distributions) are configured. Datadog collects this information by making read only API calls into your AWS account.

Cloud Security Posture Management

Setup

If you do not have the AWS integration setup yet for your AWS account, complete the set up process above and make sure to enable resource collection when mentioned.

If you already have the AWS Integration setup for other Datadog products, but do not yet have resource collection enabled, do one of the following:

1. Automatic - Update your CloudFormation Template
   1. In the CloudFormation console, find the main stack you used to install the datadog integration and select Update
   2. Select Replace current template
   3. Select Amazon S3 URL, enter https://datadog-cloudformation-template.s3.amazonaws.com/aws/main.yaml and click next
   4. Set CloudSecurityPostureManagementPermissions to true and click next without modifying other existing parameters until you reach the Review page. Here you can verify the change set preview.
   5. Check the two acknowledgment boxes at the bottom and click Update stack.
2. Manual
   1. Attach the AWS managed SecurityAudit policy to your Datadog AWS IAM role. You can find this policy in the AWS console.
3. Go to the Datadog AWS integration tile and do the following:
   1. Click on the AWS account where you wish to enable resource collection.
   2. Go to the Resource collection section for that account and check the box Route resource data to the Cloud Security Posture Management product
   3. At the bottom left of the tile, click Update Configuration

Alarm collection

There are two ways to send AWS CloudWatch alarms to the Datadog Event Stream:

• Alarm polling: Alarm polling comes out of the box with the AWS integration and fetches metric alarms through the DescribeAlarmHistory API. If you follow this method, your alarms are categorized under the event source Amazon Web Services. Note: The crawler does not collect composite alarms.
• SNS topic: You can see all AWS CloudWatch alarms in your event stream by subscribing the alarms to an SNS topic, then forwarding the SNS messages to Datadog. To learn how to receive SNS messages as events in Datadog, see Receive SNS messages. If you follow this method, your alarms are categorized under the event source Amazon SNS.

Data Collected

Metrics

Events

Events from AWS are collected on a per AWS-service basis. See the your AWS service’s documentation to learn more about collected events.

Tags

The following tags are collected with the AWS integration. Note: Some tags only display on specific metrics.

Service Checks

aws.status
Returns CRITICAL if one or more AWS regions are experiencing issues. Returns OK otherwise.
Statuses: ok, critical

Troubleshooting

See the AWS Integration Troubleshooting guide to resolve issues related to the AWS integration.

Further Reading

Additional helpful documentation, links, and articles:

Monitor AWS control plane API usage metrics in DatadogBLOG