AWS

Crawler

Overview

Connect to Amazon Web Services (AWS) to:

• See automatic AWS status updates in your stream
• Get CloudWatch metrics for EC2 hosts without installing the Agent
• Tag your EC2 hosts with EC2-specific information (e.g. availability zone)
• See EC2 scheduled maintenance events in your stream
• Collect CloudWatch metrics and events from many other AWS products

Setup

Setting up the Datadog integration with Amazon Web Services requires configuring role delegation using AWS IAM. To get a better understanding of role delegation, refer to the AWS IAM Best Practices guide.

Role delegation

Choose a method for setting up the necessary AWS role. CloudFormation is recommended.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "apigateway:GET",
                "autoscaling:Describe*",
                "budgets:ViewBudget",
                "cloudfront:GetDistributionConfig",
                "cloudfront:ListDistributions",
                "cloudtrail:DescribeTrails",
                "cloudtrail:GetTrailStatus",
                "cloudwatch:Describe*",
                "cloudwatch:Get*",
                "cloudwatch:List*",
                "codedeploy:List*",
                "codedeploy:BatchGet*",
                "directconnect:Describe*",
                "dynamodb:List*",
                "dynamodb:Describe*",
                "ec2:Describe*",
                "ecs:Describe*",
                "ecs:List*",
                "elasticache:Describe*",
                "elasticache:List*",
                "elasticfilesystem:DescribeFileSystems",
                "elasticfilesystem:DescribeTags",
                "elasticfilesystem:DescribeAccessPoints",
                "elasticloadbalancing:Describe*",
                "elasticmapreduce:List*",
                "elasticmapreduce:Describe*",
                "es:ListTags",
                "es:ListDomainNames",
                "es:DescribeElasticsearchDomains",
                "health:DescribeEvents",
                "health:DescribeEventDetails",
                "health:DescribeAffectedEntities",
                "kinesis:List*",
                "kinesis:Describe*",
                "lambda:AddPermission",
                "lambda:GetPolicy",
                "lambda:List*",
                "lambda:RemovePermission",
                "logs:TestMetricFilter",
                "logs:PutSubscriptionFilter",
                "logs:DeleteSubscriptionFilter",
                "logs:DescribeSubscriptionFilters",
                "rds:Describe*",
                "rds:List*",
                "redshift:DescribeClusters",
                "redshift:DescribeLoggingStatus",
                "route53:List*",
                "s3:GetBucketLogging",
                "s3:GetBucketLocation",
                "s3:GetBucketNotification",
                "s3:GetBucketTagging",
                "s3:ListAllMyBuckets",
                "s3:PutBucketNotification",
                "ses:Get*",
                "sns:List*",
                "sns:Publish",
                "sqs:ListQueues",
                "states:ListStateMachines",
                "states:DescribeStateMachine",
                "support:*",
                "tag:GetResources",
                "tag:GetTagKeys",
                "tag:GetTagValues",
                "xray:BatchGetTraces",
                "xray:GetTraceSummaries"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "cloudwatch:Get*",
                "cloudwatch:List*",
                "ec2:Describe*",
                "support:*",
                "tag:GetResources",
                "tag:GetTagKeys",
                "tag:GetTagValues"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

GovCloud and China

1. Open the AWS integration tile.
2. Select the Access Keys (GovCloud or China Only) tab.
3. Enter your AWS Access Key and AWS Secret Key. Only access and secret keys for GovCloud and China are accepted.
4. Choose the services to collect metrics from on the left side of the dialog.
5. Optionally, add tags to all hosts and metrics.
6. Optionally, monitor a subset of EC2 instances by entering the AWS tags in the textbox to hosts with tag.
7. Optionally, monitor a subset of Lambdas by entering the AWS tags in the textbox to Lambdas with tag.
8. Click Install Integration.

Log collection

AWS service logs are collected with the Datadog Forwarder Lambda function. This Lambda—which triggers on S3 Buckets, Cloudwatch log groups, and Cloudwatch events—forwards logs to Datadog.

To start collecting logs from your AWS services:

1. Set up the Datadog Forwarder Lambda function in your AWS account by following the instructions in the DataDog/datadog-serverless-functions Github repository.
2. Enable logging for your AWS service (most AWS services can log to a S3 bucket or CloudWatch Log Group).

3. Configure the triggers that cause the Lambda to execute. There are two ways to configure the triggers:

   • Automatically: Datadog automatically retrieves the logs for the selected AWS services and adds them as triggers on the Datadog Lambda function. Datadog also keeps the list up to date.
   • Manually: Set up each trigger yourself via the AWS console.

Note: If you are in AWS us-east-1 region, leverage Datadog-AWS Private Link to forward your logs to Datadog. If you do so, your forwarder function must have the VPCLambdaExecutionRole permission.

Advanced: Ingesting logs with Kinesis

Instead of triggering the Datadog Forwarder on the CloudWatch Log groups of your Lambda functions, you can configure a Kinesis stream that is subscribed to the Lambda log groups which you then subscribe the Datadog Forwarder to.

Kinesis log forwarding setup steps

1. Follow these instructions to launch the Datadog Forwarder CloudFormation Stack in your AWS account.
2. Identify the Lambda CloudWatch log groups that you want to forward data to Datadog from. These log groups are named /aws/lambda/{Lambda function name} and can be found by filtering the log groups to the /aws/lambda/ prefix.
3. Check if anything is currently subscribed to your relevant log groups in the Subscriptions column on the log groups index page. Because CloudWatch Log groups can only have one subscription, any existing subscription to the log groups will need to be removed before adding the new Kinesis stream as a subscriber.
   • Note that if you have something else you want to subscribe to these logs, you can subscribe that to the new Kinesis stream after completing this setup.
4. Create a new Kinesis stream (see the Kinesis documentation). Name the stream something descriptive, like DatadogLambdaLogStream, and give it a shard count of 1.
5. Subscribe your new Kinesis stream to the CloudWatch log groups you want to ingest into Datadog. Refer to this CloudWatch Logs doc section to:
   1. Use the aws iam create-role command to create the IAM role that gives CloudWatch Logs permission to put logs data into the Kinesis stream.
   2. Create a permissions policy allowing the kinesis:PutRecord action.
   3. Attach the permissions policy to your newly created IAM role using the aws iam put-role-policy command.
   4. Use the aws logs put-subscription-filter command to subscribe your Kinesis stream to each CloudWatch log group you want to ingest into Datadog.
6. Check the Subscriptions column in the log groups index page to confirm that the new Kinesis stream is now subscribed to your Lambdas’ log groups.
7. Add the Kinesis stream as a trigger to your Datadog Forwarder Lambda. To ensure a low latency of metrics and logs, it is recommended that you set the batch window to 60 seconds.
8. You will start to see your Datadog Forwarder Lambda triggered by the logs from your other Lambda functions, and your logs will appear in Datadog.

Advantages of using Kinesis for streaming logs

• CloudWatch Log groups can only have one subscriber, but Kinesis streams can have multiple subscribers. By subscribing the Kinesis stream to the log groups, you can have multiple consumers of your log data by subscribing them all to the Kinesis stream.
• Because you can set the batch window and batch size of trigger events from Kinesis, you’ll have more control over how often the Datadog Forwarder Lambda is triggered.

Disadvantages of using Kinesis for streaming logs

• Using a Kinesis stream for your logs will not work in conjunction with Datadog’s automatic Forwarder trigger management. You’ll need to add your stream as a subscriber to each log group that you want to ingest data into Datadog from.
• AWS charges for usage of the Kinesis stream. Refer to the Kinesis pricing docs for details.

Enable logging for your AWS service

Any AWS service that generates logs into a S3 bucket or a CloudWatch Log Group is supported. Find specific setup instructions for the most used services in the table below:

Send AWS service logs to Datadog

There are two options when configuring triggers on the Datadog Lambda function:

• Manually set up triggers on S3 buckets, Cloudwatch Log Groups, or Cloudwatch Events.
• Let Datadog automatically set and manage the list of triggers.

Automatically setup triggers

If you are storing logs in many S3 buckets or Cloudwatch Log groups, Datadog can automatically manage triggers for you.

1. If you haven’t already, set up the Datadog log collection AWS Lambda function.

2. Ensure the policy of the IAM role used for Datadog-AWS integration has the following permissions. Information on how these permissions are used can be found in the descriptions below:

   "cloudfront:GetDistributionConfig",
   "cloudfront:ListDistributions",
   "elasticloadbalancing:DescribeLoadBalancers",
   "elasticloadbalancing:DescribeLoadBalancerAttributes",
   "lambda:List*",
   "lambda:AddPermission",
   "lambda:GetPolicy",
   "lambda:RemovePermission",
   "redshift:DescribeClusters",
   "redshift:DescribeLoggingStatus",
   "s3:GetBucketLogging",
   "s3:GetBucketLocation",
   "s3:GetBucketNotification",
   "s3:ListAllMyBuckets",
   "s3:PutBucketNotification",
   "logs:PutSubscriptionFilter",
   "logs:DeleteSubscriptionFilter",
   "logs:DescribeSubscriptionFilters"

   AWS Permission	Description
   cloudfront:GetDistributionConfig	Get the name of the S3 bucket containing CloudFront access logs.
   cloudfront:ListDistributions	List all CloudFront distributions.
   elasticloadbalancing: DescribeLoadBalancers	List all load balancers.
   elasticloadbalancing: DescribeLoadBalancerAttributes	Get the name of the S3 bucket containing ELB access logs.
   lambda:List*	List all Lambda functions.
   lambda:AddPermission	Add permission allowing a particular S3 bucket to trigger a Lambda function.
   lambda:GetPolicy	Gets the Lambda policy when triggers are to be removed.
   lambda:RemovePermission	Remove permissions from a Lambda policy.
   redshift:DescribeClusters	List all Redshift clusters.
   redshift:DescribeLoggingStatus	Get the name of the S3 bucket containing Redshift Logs.
   s3:GetBucketLogging	Get the name of the S3 bucket containing S3 access logs.
   s3:GetBucketLocation	Get the region of the S3 bucket containing S3 access logs.
   s3:GetBucketNotification	Get existing Lambda trigger configurations.
   s3:ListAllMyBuckets	List all S3 buckets.
   s3:PutBucketNotification	Add or remove a Lambda trigger based on S3 bucket events.
   logs:PutSubscriptionFilter	Add a Lambda trigger based on CloudWatch Log events
   logs:DeleteSubscriptionFilter	Remove a Lambda trigger based on CloudWatch Log events
   logs:DescribeSubscriptionFilters	Lists the subscription filters for the specified log group.

3. Navigate to the Collect Logs tab in the AWS Integration tile.

4. Select the AWS Account from where you want to collect logs, and enter the ARN of the Lambda created in the previous section.

5. Select the services from which you’d like to collect logs and hit save. To stop collecting logs from a particular service, uncheck it.

6. If you have logs across multiple regions, you must create additional Lambda functions in those regions and enter them in this tile.

7. To stop collecting all AWS logs, press the x next to each Lambda ARN. All triggers for that function are removed.

8. Within a few minutes of this initial setup, your AWS Logs appear in your Datadog log explorer page in near real time.

Manually setup triggers

Collecting logs from Cloudwatch Log Group

If you are storing logs in a CloudWatch Log Group, send them to Datadog as follows:

1. If you haven’t already, set up the Datadog log collection AWS Lambda function.
2. Once the lambda function is installed, manually add a trigger on the CloudWatch Log Group that contains your logs in the AWS console:

Select the corresponding CloudWatch Log Group, add a filter name (but feel free to leave the filter empty) and add the trigger:

Once done, go into your Datadog Log section to start exploring your logs!

For Terraform users, you can provision and manage your triggers using the aws_cloudwatch_log_subscription_filter resource. See sample code below.

resource "aws_cloudwatch_log_subscription_filter" "datadog_log_subscription_filter" {
  name            = "datadog_log_subscription_filter"
  log_group_name  = <CLOUDWATCH_LOG_GROUP_NAME> # e.g., /aws/lambda/my_lambda_name
  destination_arn = <DATADOG_FORWARDER_ARN> # e.g., arn:aws:lambda:us-east-1:123:function:datadog-forwarder
  filter_pattern  = ""
}

For Serverless Framework, SAM, and CloudFormation users, you can provision and manage your triggers using the CloudFormation AWS::Logs::SubscriptionFilter resource. See sample code below. For Serverless Framework users, you need to put the code snippet under the resources section within your serverless.yml.

Resources:
  MyLogSubscriptionFilter:
    Type: "AWS::Logs::SubscriptionFilter"
    Properties:
      DestinationArn: "<DATADOG_FORWARDER_ARN>"
      LogGroupName: "<CLOUDWATCH_LOG_GROUP_NAME>"
      FilterPattern: ""

Collecting logs from S3 buckets

If you are storing logs in a S3 bucket, send them to Datadog as follows:

1. If you haven’t already, set up the Datadog log collection AWS Lambda function.

2. Once the lambda function is installed, manually add a trigger on the S3 bucket that contains your logs in the AWS console:

3. Select the bucket and then follow the AWS instructions:

4. Set the correct event type on S3 buckets:

Once done, go into your Datadog Log section to start exploring your logs!

For Terraform users, you can provision and manage your triggers using the aws_s3_bucket_notification resource. See sample code below.

resource "aws_s3_bucket_notification" "my_bucket_notification" {
  bucket = my_bucket
  lambda_function {
    lambda_function_arn = "<DATADOG_FORWARDER_ARN>"
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "AWSLogs/"
    filter_suffix       = ".log"
  }
}

resource "aws_lambda_permission" "allow_bucket" {
  statement_id  = "AllowExecutionFromS3Bucket"
  action        = "lambda:InvokeFunction"
  function_name = "<DATADOG_FORWARDER_ARN>"
  principal     = "s3.amazonaws.com"
  source_arn    = "<MY_BUCKET_ARN>"
}

For Serverless Framework, SAM, and CloudFormation users, you can configure triggers using the CloudFormation NotificationConfiguration for your S3 bucket. See sample code below. For Serverless Framework users, you need to put the code snippet under the resources section within your serverless.yml.

Resources:
  Bucket:
    Type: AWS::S3::Bucket
    DependsOn: BucketPermission
    Properties:
      BucketName: "<MY_BUCKET>"
      NotificationConfiguration:
        LambdaConfigurations:
        - Event: 's3:ObjectCreated:*'
          Function: "<DATADOG_FORWARDER_ARN>"
  BucketPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: "<DATADOG_FORWARDER_ARN>"
      Principal: s3.amazonaws.com
      SourceArn: <MY_BUCKET_ARN>
      SourceAccount: !Ref "AWS::AccountId"

Data Collected

Metrics

Events

Events from AWS are collected on a per AWS-service basis. Please refer to the documentation of specific AWS services to learn more about the events collected.

Tag

The following tags are collected from AWS integrations. Note: Some tags only display on specific metrics.

Troubleshooting

Discrepancy between your data in CloudWatch and Datadog

There are two important distinctions to be aware of:

1. In AWS for counters, a graph that is set to ‘sum’ ‘1minute’ shows the total number of occurrences in one minute leading up to that point, i.e. the rate per 1 minute. Datadog is displaying the raw data from AWS normalized to per second values, regardless of the time frame selected in AWS. This is why you might see Datadog’s value as lower.
2. Overall, min/max/avg have a different meaning within AWS than in Datadog. In AWS, average latency, minimum latency, and maximum latency are three distinct metrics that AWS collects. When Datadog pulls metrics from AWS CloudWatch, the average latency is received as a single time series per ELB. Within Datadog, when you are selecting ‘min’, ‘max’, or ‘avg’, you are controlling how multiple time series are combined. For example, requesting system.cpu.idle without any filter would return one series for each host that reports that metric and those series need to be combined to be graphed. On the other hand, if you requested system.cpu.idle from a single host, no aggregation would be necessary and switching between average and max would yield the same result.

Metrics delayed

When using the AWS integration, Datadog pulls in your metrics via the CloudWatch API. You may see a slight delay in metrics from AWS due to some constraints that exist for their API.

To begin, the CloudWatch API only offers a metric-by-metric crawl to pull data. The CloudWatch APIs have a rate limit that varies based on the combination of authentication credentials, region, and service. Metrics are made available by AWS dependent on the account level. For example, if you are paying for “detailed metrics” within AWS, they are available more quickly. This level of service for detailed metrics also applies to granularity, with some metrics being available per minute and others per five minutes.

Datadog has the ability to prioritize certain metrics within an account to pull them in faster, depending on the circumstances. Please contact Datadog support for more info.

To obtain metrics with virtually zero delay, install the Datadog Agent on the host. For more information, see Datadog’s blog post Don’t fear the Agent: Agent-based monitoring.

Missing metrics

CloudWatch’s API returns only metrics with data points, so if for instance an ELB has no attached instances, it is expected not to see metrics related to this ELB in Datadog.

Wrong count of aws.elb.healthy_host_count

When the cross-zone load balancing option is enabled on an ELB, all the instances attached to this ELB are considered part of all availability zones (on CloudWatch’s side), so if you have 2 instances in 1a and 3 in ab, the metric displays 5 instances per availability zone. As this can be counter intuitive, we’ve added new metrics, aws.elb.healthy_host_count_deduped and aws.elb.un_healthy_host_count_deduped, that display the count of healthy and unhealthy instances per availability zone, regardless of if this cross-zone load balancing option is enabled or not.

Duplicated hosts when installing the Agent

When installing the Agent on an AWS host, you might see duplicated hosts on the infra page for a few hours if you manually set the hostname in the Agent’s configuration. This second host disappears a few hours later, and won’t affect your billing.