AWS

Crawler

Overview

Connect to Amazon Web Services (AWS) to:

• See automatic AWS status updates in your stream
• Get CloudWatch metrics for EC2 hosts without installing the Agent
• Tag your EC2 hosts with EC2-specific information (e.g. availability zone)
• See EC2 scheduled maintenance events in your stream
• Collect CloudWatch metrics and events from many other AWS products

Setup

Use one of the following methods to integrate your AWS accounts into Datadog for metric, trace, and log collection:

• Role delegation (Automatic): Use a CloudFormation template to automatically setup the necessarily AWS role (recommended).
• Role delegation (Manual): Manually create the necessary role and copy the required credentials in the respective form.
• Access keys: Used for GovCloud or China only

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "apigateway:GET",
                "autoscaling:Describe*",
                "budgets:ViewBudget",
                "cloudfront:GetDistributionConfig",
                "cloudfront:ListDistributions",
                "cloudtrail:DescribeTrails",
                "cloudtrail:GetTrailStatus",
                "cloudtrail:LookupEvents",
                "cloudwatch:Describe*",
                "cloudwatch:Get*",
                "cloudwatch:List*",
                "codedeploy:List*",
                "codedeploy:BatchGet*",
                "directconnect:Describe*",
                "dynamodb:List*",
                "dynamodb:Describe*",
                "ec2:Describe*",
                "ecs:Describe*",
                "ecs:List*",
                "elasticache:Describe*",
                "elasticache:List*",
                "elasticfilesystem:DescribeFileSystems",
                "elasticfilesystem:DescribeTags",
                "elasticfilesystem:DescribeAccessPoints",
                "elasticloadbalancing:Describe*",
                "elasticmapreduce:List*",
                "elasticmapreduce:Describe*",
                "es:ListTags",
                "es:ListDomainNames",
                "es:DescribeElasticsearchDomains",
                "fsx:DescribeFileSystems",
                "fsx:ListTagsForResource",
                "health:DescribeEvents",
                "health:DescribeEventDetails",
                "health:DescribeAffectedEntities",
                "kinesis:List*",
                "kinesis:Describe*",
                "lambda:GetPolicy",
                "lambda:List*",
                "logs:DeleteSubscriptionFilter",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:DescribeSubscriptionFilters",
                "logs:FilterLogEvents",
                "logs:PutSubscriptionFilter",
                "logs:TestMetricFilter",
                "organizations:DescribeOrganization",
                "rds:Describe*",
                "rds:List*",
                "redshift:DescribeClusters",
                "redshift:DescribeLoggingStatus",
                "route53:List*",
                "s3:GetBucketLogging",
                "s3:GetBucketLocation",
                "s3:GetBucketNotification",
                "s3:GetBucketTagging",
                "s3:ListAllMyBuckets",
                "s3:PutBucketNotification",
                "ses:Get*",
                "sns:List*",
                "sns:Publish",
                "sqs:ListQueues",
                "states:ListStateMachines",
                "states:DescribeStateMachine",
                "support:*",
                "tag:GetResources",
                "tag:GetTagKeys",
                "tag:GetTagValues",
                "xray:BatchGetTraces",
                "xray:GetTraceSummaries"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "cloudwatch:Get*",
                "cloudwatch:List*",
                "ec2:Describe*",
                "support:*",
                "tag:GetResources",
                "tag:GetTagKeys",
                "tag:GetTagValues"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "acm:DescribeCertificate",
                "acm:ListCertificates",
                "cloudfront:GetDistribution",
                "cloudfront:ListDistributions",
                "cloudtrail:DescribeTrails",
                "cloudtrail:GetEventSelectors",
                "cloudtrail:GetTrailStatus",
                "config:DescribeConfigurationRecorderStatus",
                "config:DescribeConfigurationRecorders",
                "iam:GenerateCredentialReport",
                "iam:GetAccountPasswordPolicy",
                "iam:GetAccountSummary",
                "iam:GetCredentialReport",
                "iam:GetLoginProfile",
                "iam:GetPolicyVersion",
                "iam:ListAttachedUserPolicies",
                "iam:ListEntitiesForPolicy",
                "iam:ListMFADevices",
                "iam:ListPolicies",
                "iam:ListRoles",
                "iam:ListServerCertificates",
                "iam:ListUserPolicies",
                "iam:ListUsers",
                "iam:ListVirtualMFADevices",
                "kms:GetKeyPolicy",
                "kms:GetKeyRotationStatus",
                "kms:ListAliases",
                "kms:ListKeys",
                "lambda:GetPolicy",
                "lambda:ListFunctions",
                "redshift:DescribeClusterParameterGroups",
                "redshift:DescribeClusterParameters",
                "redshift:DescribeLoggingStatus",
                "rds:DescribeDBSecurityGroups",
                "rds:DescribeDBSnapshotAttributes",
                "rds:DescribeDBSnapshots",
                "s3:GetBucketAcl",
                "s3:GetBucketLogging",
                "s3:GetBucketPolicy",
                "s3:GetBucketPolicyStatus",
                "s3:GetBucketPublicAccessBlock",
                "s3:GetBucketVersioning",
                "s3:GetEncryptionConfiguration",
                "sns:GetSubscriptionAttributes",
                "sns:GetTopicAttributes",
                "sns:ListSubscriptions",
                "sns:ListTopics",
                "sqs:GetQueueAttributes",
                "sqs:ListQueues"
            ],
            "Resource": "*"
        }
    ]
}

Log collection

There are two ways of sending AWS service logs to Datadog:

• Kinesis Firehose destination: Use the Datadog destination in your Kinesis Firehose delivery stream to forward logs to Datadog. We recommend using this approach for sending logs from CloudWatch in very high volume.
• Forwarder Lambda function: Deploy the Datadog Forwarder Lambda function, which subscribes to S3 buckets or your CloudWatch log groups and forward logs to Datadog. You must use this approach to send traces, enhanced metrics, or custom metrics from Lambda functions asynchronously through logs. Datadog also recommends you use this approach for sending logs from S3 or other resources that cannot directly stream data to Kinesis.

Metric collection

There are two ways to send AWS metrics to Datadog:

• Metric polling: API polling comes out of the box with the AWS integration and does a metric-by-metric crawl of the CloudWatch API to pull data to send to Datadog. New metrics are pulled every ten minutes on average.
• Metric streams with Kinesis Firehose: You can use Amazon CloudWatch Metric Streams and Amazon Kinesis Data Firehose to see your metrics at a two to three minute latency. This requires a separate setup.

Resource Collection

The products listed below require metadata and configuration information on AWS resources. This information is collected by Datadog making API calls into your AWS account for information on resources such as S3 Buckets, RDS snapshots, and CloudFront distributions (full list of resources and permissions).

Products requiring this:

• Cloud Security Posture Management

Setup:

1. If you do not have the AWS Integration setup yet for your AWS account, complete the set up process above and make sure to enable resource collection when mentioned.
2. If you already have the AWS Integration setup for other Datadog products, but do not yet have resource collection enabled, do one of the following:
   1. Automatic (Recommended) - Update your CloudFormation Template
      1. In the CloudFormation console, find the main stack you used to install the datadog integration and select Update
      2. Select Replace current template
      3. Select Amazon S3 URL, enter https://datadog-cloudformation-template.s3.amazonaws.com/aws/main.yaml and click next
      4. Set ResourceCollectionPermissions to true and click next without modifying other existing parameters until you reach the Review page. Here you can verify the change set preview.
      5. Check the two acknowledgement boxes at the bottom and click Update stack.
   2. Manual
      1. Create a new DatadogAWSResourceCollectionPolicy in your AWS account with these permissions.
      2. Add this new policy to the Datadog role in your account.
3. Go to the Datadog AWS integration tile and do the following:
   1. Click on the AWS account where you wish to enable resource collection.
   2. Go to the Resource configuration collection section for that account and check the box Enable resource configuration collection
   3. At the bottom left of the tile, click Update Configuration

Data Collected

Metrics

Events

Events from AWS are collected on a per AWS-service basis. Please refer to the documentation of specific AWS services to learn more about the events collected.

Tags

The following tags are collected from AWS integrations. Note: Some tags only display on specific metrics.

Troubleshooting

Discrepancy between your data in CloudWatch and Datadog

There are two important distinctions to be aware of:

1. In AWS for counters, a graph that is set to ‘sum’ ‘1minute’ shows the total number of occurrences in one minute leading up to that point, i.e. the rate per 1 minute. Datadog is displaying the raw data from AWS normalized to per second values, regardless of the time frame selected in AWS. This is why you might see Datadog’s value as lower.
2. Overall, min/max/avg have a different meaning within AWS than in Datadog. In AWS, average latency, minimum latency, and maximum latency are three distinct metrics that AWS collects. When Datadog pulls metrics from AWS CloudWatch, the average latency is received as a single time series per ELB. Within Datadog, when you are selecting ‘min’, ‘max’, or ‘avg’, you are controlling how multiple time series are combined. For example, requesting system.cpu.idle without any filter would return one series for each host that reports that metric and those series need to be combined to be graphed. On the other hand, if you requested system.cpu.idle from a single host, no aggregation would be necessary and switching between average and max would yield the same result.

Metrics delayed

When using the AWS integration, Datadog pulls in your metrics via the CloudWatch API. You may see a slight delay in metrics from AWS due to some constraints that exist for their API.

To begin, the CloudWatch API only offers a metric-by-metric crawl to pull data. The CloudWatch APIs have a rate limit that varies based on the combination of authentication credentials, region, and service. Metrics are made available by AWS dependent on the account level. For example, if you are paying for “detailed metrics” within AWS, they are available more quickly. This level of service for detailed metrics also applies to granularity, with some metrics being available per minute and others per five minutes.

Datadog has the ability to prioritize certain metrics within an account to pull them in faster, depending on the circumstances. Please contact Datadog support for more info.

To obtain metrics with virtually zero delay, install the Datadog Agent on the host. For more information, see Datadog’s blog post Don’t fear the Agent: Agent-based monitoring.

Missing metrics

CloudWatch’s API returns only metrics with data points, so if for instance an ELB has no attached instances, it is expected not to see metrics related to this ELB in Datadog.

Wrong count of aws.elb.healthy_host_count

When the cross-zone load balancing option is enabled on an ELB, all the instances attached to this ELB are considered part of all availability zones (on CloudWatch’s side), so if you have 2 instances in 1a and 3 in ab, the metric displays 5 instances per availability zone. As this can be counter intuitive, we’ve added new metrics, aws.elb.healthy_host_count_deduped and aws.elb.un_healthy_host_count_deduped, that display the count of healthy and unhealthy instances per availability zone, regardless of if this cross-zone load balancing option is enabled or not.

Duplicated hosts when installing the Agent

When installing the Agent on an AWS host, you might see duplicated hosts on the infra page for a few hours if you manually set the hostname in the Agent’s configuration. This second host disappears a few hours later, and won’t affect your billing.