Send AWS services logs with the Datadog Lambda function
New announcements from Dash: Incident Management, Continuous Profiler, and more! New announcements from Dash!

Send AWS services logs with the Datadog Lambda function

AWS service logs can be collected with the Datadog Forwarder Lambda function. This Lambda—which triggers on S3 Buckets, CloudWatch log groups, and CloudWatch events—forwards logs to Datadog.

To start collecting logs from your AWS services:

  1. Set up the Datadog Forwarder Lambda function in your AWS account.
  2. Enable logging for your AWS service (most AWS services can log to a S3 bucket or CloudWatch Log Group).
  3. Set up the triggers that cause the Forwarder Lambda to execute when there are new logs to be forwarded. There are two ways to configure the triggers.

Note: If you are in AWS us-east-1 region, leverage Datadog-AWS Private Link.

Set up triggers

There are two options when configuring triggers on the Datadog Forwarder Lambda function:

  • Automatically: Datadog automatically retrieves the log locations for the selected AWS services and adds them as triggers on the Datadog Forwarder Lambda function. Datadog also keeps the list up to date.
  • Manually: Set up each trigger yourself.

Automatically set up triggers

If you are storing logs in many S3 buckets or CloudWatch Log groups, Datadog can automatically manage triggers for you.

  1. If you haven’t already, set up the Datadog log collection AWS Lambda function.
  2. Ensure the policy of the IAM role used for Datadog-AWS integration has the following permissions. Information on how these permissions are used can be found in the descriptions below:

    "cloudfront:GetDistributionConfig",
    "cloudfront:ListDistributions",
    "elasticloadbalancing:DescribeLoadBalancers",
    "elasticloadbalancing:DescribeLoadBalancerAttributes",
    "lambda:List*",
    "lambda:AddPermission",
    "lambda:GetPolicy",
    "lambda:RemovePermission",
    "redshift:DescribeClusters",
    "redshift:DescribeLoggingStatus",
    "s3:GetBucketLogging",
    "s3:GetBucketLocation",
    "s3:GetBucketNotification",
    "s3:ListAllMyBuckets",
    "s3:PutBucketNotification",
    "logs:PutSubscriptionFilter",
    "logs:DeleteSubscriptionFilter",
    "logs:DescribeSubscriptionFilters"
    AWS PermissionDescription
    cloudfront:GetDistributionConfigGet the name of the S3 bucket containing CloudFront access logs.
    cloudfront:ListDistributionsList all CloudFront distributions.
    elasticloadbalancing:
    DescribeLoadBalancers
    List all load balancers.
    elasticloadbalancing:
    DescribeLoadBalancerAttributes
    Get the name of the S3 bucket containing ELB access logs.
    lambda:List*List all Lambda functions.
    lambda:AddPermissionAdd permission allowing a particular S3 bucket to trigger a Lambda function.
    lambda:GetPolicyGets the Lambda policy when triggers are to be removed.
    lambda:RemovePermissionRemove permissions from a Lambda policy.
    redshift:DescribeClustersList all Redshift clusters.
    redshift:DescribeLoggingStatusGet the name of the S3 bucket containing Redshift Logs.
    s3:GetBucketLoggingGet the name of the S3 bucket containing S3 access logs.
    s3:GetBucketLocationGet the region of the S3 bucket containing S3 access logs.
    s3:GetBucketNotificationGet existing Lambda trigger configurations.
    s3:ListAllMyBucketsList all S3 buckets.
    s3:PutBucketNotificationAdd or remove a Lambda trigger based on S3 bucket events.
    logs:PutSubscriptionFilterAdd a Lambda trigger based on CloudWatch Log events
    logs:DeleteSubscriptionFilterRemove a Lambda trigger based on CloudWatch Log events
    logs:DescribeSubscriptionFiltersLists the subscription filters for the specified log group.
  3. Navigate to the Collect Logs tab in the AWS Integration tile.

  4. Select the AWS Account from where you want to collect logs, and enter the ARN of the Lambda created in the previous section.

  5. Select the services from which you’d like to collect logs and hit save. To stop collecting logs from a particular service, uncheck it.

  6. If you have logs across multiple regions, you must create additional Lambda functions in those regions and enter them in this tile.

  7. To stop collecting all AWS logs, press the x next to each Lambda ARN. All triggers for that function are removed.

  8. Within a few minutes of this initial setup, your AWS Logs appear in your Datadog log explorer page in near real time.

Manually set up triggers

Collecting logs from CloudWatch Log Group

If you are collecting logs from a CloudWatch Log Group, configure the trigger to the Datadog Forwarder Lambda function using one of the following methods:

Select the corresponding CloudWatch Log Group, add a filter name (but feel free to leave the filter empty) and add the trigger:

Once done, go into your Datadog Log section to start exploring your logs.

For Terraform users, you can provision and manage your triggers using the aws_cloudwatch_log_subscription_filter resource. See sample code below.

resource "aws_cloudwatch_log_subscription_filter" "datadog_log_subscription_filter" {
  name            = "datadog_log_subscription_filter"
  log_group_name  = <CLOUDWATCH_LOG_GROUP_NAME> # e.g., /aws/lambda/my_lambda_name
  destination_arn = <DATADOG_FORWARDER_ARN> # e.g., arn:aws:lambda:us-east-1:123:function:datadog-forwarder
  filter_pattern  = ""
}

For AWS CloudFormation users, you can provision and manage your triggers using the CloudFormation AWS::Logs::SubscriptionFilter resource. See sample code below.

The sample code also work for AWS SAM and Serverless Framework. For Serverless Framework, put the code under the resources section within your serverless.yml.

Resources:
  MyLogSubscriptionFilter:
    Type: "AWS::Logs::SubscriptionFilter"
    Properties:
      DestinationArn: "<DATADOG_FORWARDER_ARN>"
      LogGroupName: "<CLOUDWATCH_LOG_GROUP_NAME>"
      FilterPattern: ""

Collecting logs from S3 buckets

If you are collecting logs from an S3 bucket, configure the trigger to the Datadog Forwarder Lambda function using one of the following methods:

  1. Once the Lambda function is installed, manually add a trigger on the S3 bucket that contains your logs in the AWS console:

  2. Select the bucket and then follow the AWS instructions:

  3. Set the correct event type on S3 buckets:

Once done, go into your Datadog Log section to start exploring your logs!

For Terraform users, you can provision and manage your triggers using the aws_s3_bucket_notification resource. See the sample code below.

resource "aws_s3_bucket_notification" "my_bucket_notification" {
  bucket = my_bucket
  lambda_function {
    lambda_function_arn = "<DATADOG_FORWARDER_ARN>"
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "AWSLogs/"
    filter_suffix       = ".log"
  }
}

For CloudFormation users, you can configure triggers using the CloudFormation NotificationConfiguration for your S3 bucket. See the sample code below.

Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: "<MY_BUCKET>"
      NotificationConfiguration:
        LambdaConfigurations:
        - Event: 's3:ObjectCreated:*'
          Function: "<DATADOG_FORWARDER_ARN>"

Enable logging for your AWS service

Any AWS service that generates logs into a S3 bucket or a CloudWatch Log Group is supported. Find specific setup instructions for the most used services in the table below:

AWS serviceActivate AWS service loggingSend AWS logs to Datadog
API GatewayEnable AWS API Gateway logsManual log collection
CloudfrontEnable AWS Cloudfront logsManual and automatic log collection
CloudtrailEnable AWS Cloudtrail logsManual log collection
DynamoDBEnable AWS DynamoDB logsManual log collection
EC2-Use the Datadog Agent to send your logs to Datadog
ECS-Use the docker agent to gather your logs
Elastic Load Balancing (ELB)Enable AWS ELB logsManual and automatic log collection
Lambda-Manual and automatic log collection
RDSEnable AWS RDS logsManual log collection
Route 53Enable AWS Route 53 logsManual log collection
S3Enable AWS S3 logsManual and automatic log collection
SNSThere is no “SNS Logs”. Process logs and events that are transiting through to the SNS Service.Manual log collection
RedShiftEnable AWS Redshift logsManual and automatic log collection
VPCEnable AWS VPC logsManual log collection