AWS service logs can be collected with the Datadog Forwarder Lambda function. This Lambda—which triggers on S3 Buckets, CloudWatch log groups, and EventBridge events—forwards logs to Datadog.

To start collecting logs from your AWS services:

1. Set up the Datadog Forwarder Lambda function in your AWS account.
2. Enable logging for your AWS service. Find your service in Supported AWS services to see its setup instructions. Most AWS services can log to a S3 bucket or a CloudWatch log group.
3. Set up the triggers that cause the Forwarder Lambda to execute when there are new logs to be forwarded. There are two ways to configure the triggers.

Notes:

• You can use AWS PrivateLink to send your logs over a private connection.
• CloudFormation creates an IAM policy which includes KMS:Decrypt for all resources, and does not align with AWS Security Hub’s best practice. This permission is used to decrypt objects from KMS-encrypted S3 buckets to set up the Lambda function, and the KMS key used to encrypt the S3 buckets cannot be predicted. You can safely delete this permission after the installation finishes successfully.

Supported AWS services

The Datadog Forwarder Lambda function supports any AWS service that generates logs into a S3 bucket or a CloudWatch log group. The following table lists the services it can collect:

• AWS service: The AWS service that generates the logs. Each service name links to its log collection setup instructions. Services without a link require no service-side setup.
• Log source: The source tag Datadog applies to the logs. Use it to find your logs in the Log Explorer.
• Storage: Where the AWS service can write logs that the Forwarder collects.
• Automatic collection: Whether Datadog can automatically set up the triggers for that log source. If not, set up the triggers manually.

¹ EKS control plane logs also use the kubernetes.audit, kube-scheduler, kube-apiserver, kube-controller-manager, and aws-iam-authenticator sources.
² Datadog does not apply a service-specific source tag to these logs.
³ Automatic collection for IoT is available at the account level only.
⁴ Automatic collection for PCS is available for CloudWatch log groups only.
⁵ RDS engine logs also use the postgresql, mariadb, and mysql sources.
⁶ Covers both DNS query logs and Resolver query logs.
⁷ Automatic collection is available for CloudWatch log groups. For S3 buckets, set up the trigger manually.

Note: The Datadog Forwarder automatically creates subscription filters on CloudWatch log groups. Each filter is named in the format DD_LOG_SUBSCRIPTION_FILTER_<LOG_GROUP_NAME>.

Services collected through another method

The following AWS services are supported for log collection, but do not use the Datadog Forwarder Lambda function in the same way:

Set up triggers

There are two options when configuring triggers on the Datadog Forwarder Lambda function:

• Automatically: Datadog automatically retrieves the log locations for the selected AWS services and adds them as triggers on the Datadog Forwarder Lambda function. Datadog also keeps the list up to date.
• Manually: Set up each trigger yourself.

Automatically set up triggers

Datadog can automatically configure triggers on the Datadog Forwarder Lambda function to collect AWS logs. However, automatic subscription does not support creating triggers across different AWS accounts or regions. For scenarios where logs are published to S3 buckets in a separate account, we recommend manually creating a trigger in the same account as the bucket to work around this limitation.

To see which services support automatic collection, and the storage locations they support, see Supported AWS services.

1. If you haven’t already, set up the Datadog log collection AWS Lambda function.

2. Ensure the policy of the IAM role used for Datadog-AWS integration has the following permissions. Information on how these permissions are used can be found in the descriptions below:

   "airflow:GetEnvironment",
   "airflow:ListEnvironments",
   "appsync:ListGraphqlApis",
   "batch:DescribeJobDefinitions",
   "cloudfront:GetDistributionConfig",
   "cloudfront:ListDistributions",
   "cloudtrail:GetTrail",
   "cloudtrail:ListTrails",
   "codebuild:BatchGetProjects",
   "codebuild:ListProjects",
   "dms:DescribeReplicationInstances",
   "ec2:DescribeFlowLogs",
   "ec2:DescribeVerifiedAccessInstanceLoggingConfigurations",
   "ec2:DescribeVpnConnections",
   "ecs:DescribeTaskDefinition",
   "ecs:ListTaskDefinitionFamilies",
   "eks:DescribeCluster",
   "eks:ListClusters",
   "elasticbeanstalk:DescribeEnvironments",
   "elasticloadbalancing:DescribeLoadBalancerAttributes",
   "elasticloadbalancing:DescribeLoadBalancers",
   "glue:BatchGetJobs",
   "glue:GetJobs",
   "glue:GetJob",
   "glue:ListJobs",
   "iot:GetV2LoggingOptions",
   "lambda:GetMicrovmImageVersion",
   "lambda:GetPolicy",
   "lambda:InvokeFunction",
   "lambda:List*",
   "logs:DeleteSubscriptionFilter",
   "logs:DescribeDeliveries",
   "logs:DescribeDeliverySources",
   "logs:DescribeLogGroups",
   "logs:DescribeSubscriptionFilters",
   "logs:GetDeliveryDestination",
   "logs:PutSubscriptionFilter",
   "network-firewall:DescribeLoggingConfiguration",
   "network-firewall:ListFirewalls",
   "rds:DescribeDBClusters",
   "rds:DescribeDBInstances",
   "redshift-serverless:ListNamespaces",
   "redshift:DescribeClusters",
   "redshift:DescribeLoggingStatus",
   "route53:ListQueryLoggingConfigs",
   "route53resolver:ListResolverQueryLogConfigs",
   "s3:GetBucketLocation",
   "s3:GetBucketLogging",
   "s3:GetBucketNotification",
   "s3:ListAllMyBuckets",
   "s3:PutBucketNotification",
   "ssm:GetServiceSetting",
   "ssm:ListCommands",
   "states:DescribeStateMachine",
   "states:ListStateMachines",
   "wafv2:ListLoggingConfigurations"
   

   AWS Permission	Description
   airflow:ListEnvironments	List all MWAA environment names.
   airflow:GetEnvironment	Get information about a MWAA environment.
   appsync:ListGraphqlApis	List all GraphQL Apis.
   batch:DescribeJobDefinitions	List all Batch job definitions.
   cloudfront:GetDistributionConfig	Get the name of the S3 bucket containing CloudFront access logs.
   cloudfront:ListDistributions	List all CloudFront distributions.
   cloudtrail:GetTrail	Get Trail logging information.
   cloudtrail:ListTrails	List all Cloudtrail trails.
   codebuild:BatchGetProjects	List all CodeBuild projects.
   codebuild:ListProjects	Get information on CodeBuild projects.
   dms:DescribeReplicationInstances	List all replication instances for DMS.
   ec2:DescribeFlowLogs	List all Flow log configurations.
   ec2:DescribeVerifiedAccessInstanceLoggingConfigurations	List all Verified Access instance logging configurations.
   ec2:DescribeVpnConnections	List all VPN connections.
   ecs:DescribeTaskDefinition	Describe ECS task definition.
   ecs:ListTaskDefinitionFamilies	List all task definition families.
   elasticloadbalancing: DescribeLoadBalancers	List all load balancers.
   elasticloadbalancing: DescribeLoadBalancerAttributes	Get the name of the S3 bucket containing ELB access logs.
   glue:BatchGetJobs	Get information about multiple Glue jobs.
   glue:GetJob	Get information about a Glue job.
   glue:GetJobs	List all Glue jobs.
   glue:ListJobs	List all Glue job names.
   eks:DescribeCluster	Describe an EKS cluster.
   eks:ListClusters	List all EKS clusters.
   elasticbeanstalk:DescribeEnvironments	List all Elastic Beanstalk environments.
   iot:GetV2LoggingOptions	Get IoT V2 logging options.
   lambda:InvokeFunction	Invoke a Lambda function.
   lambda:List*	List all Lambda functions.
   lambda:GetPolicy	Get the Lambda policy when triggers are to be removed.
   lambda:GetMicrovmImageVersion	Get information about a Lambda MicroVM image version.
   logs:PutSubscriptionFilter	Add a Lambda trigger based on CloudWatch Log events.
   logs:DeleteSubscriptionFilter	Remove a Lambda trigger based on CloudWatch Log events.
   logs:DescribeLogGroups	Describe CloudWatch log groups.
   logs:DescribeDeliveries	Describe CloudWatch log deliveries.
   logs:DescribeDeliverySources	Describe CloudWatch log delivery sources.
   logs:DescribeSubscriptionFilters	List the subscription filters for the specified log group.
   logs:GetDeliveryDestination	Get a CloudWatch log delivery destination.
   network-firewall:DescribeLoggingConfiguration	Get the logging configuration of a firewall.
   network-firewall:ListFirewalls	List all Network Firewall firewalls.
   rds:DescribeDBClusters	List all RDS clusters.
   rds:DescribeDBInstances	List all RDS instances.
   redshift:DescribeClusters	List all Redshift clusters.
   redshift:DescribeLoggingStatus	Get the name of the S3 bucket containing Redshift Logs.
   redshift-serverless:ListNamespaces	List all Redshift Serverless namespaces.
   route53:ListQueryLoggingConfigs	List all DNS query logging configurations for Route 53.
   route53resolver:ListResolverQueryLogConfigs	List all Resolver query logging configurations for Route 53.
   s3:GetBucketLogging	Get the name of the S3 bucket containing S3 access logs.
   s3:GetBucketLocation	Get the region of the S3 bucket containing S3 access logs.
   s3:GetBucketNotification	Get existing Lambda trigger configurations.
   s3:ListAllMyBuckets	List all S3 buckets.
   s3:PutBucketNotification	Add or remove a Lambda trigger based on S3 bucket events.
   ssm:GetServiceSetting	Get the SSM service setting for customer script log group name.
   ssm:ListCommands	List all SSM commands.
   states:ListStateMachines	List all Step Functions.
   states:DescribeStateMachine	Get logging details about a Step Function.
   wafv2:ListLoggingConfigurations	List all logging configurations of the Web Application Firewall.

3. In the AWS Integration page, select the AWS Account to collect logs from and click on the Log Collection tab.

4. In the Datadog Forwarder Lambda section, enter the ARN of the Lambda created in the previous section and click Add. The Lambda function appears in the table below with its name, version, and region.

5. In the Log Autosubscription section, under Log Sources, enable the services from which you’d like to collect logs by toggling them on. To stop collecting logs from a particular service, toggle the log source off.

6. (Optional) In the Log Source Tag Filters section, you can filter log collection by resource tags for each log source. Select a log source from the dropdown menu and add tags in key:value format to limit which resources’ logs are collected. Note: Resource tags are automatically lowercased to match Datadog platform conventions. Define your tag filters in lowercase to avoid mismatches.

7. If you have logs across multiple regions, you must create additional Lambda functions in those regions and add them in the Datadog Forwarder Lambda section.

8. To stop collecting all AWS logs from a specific Lambda function, hover over the Lambda in the table and click the delete icon. All triggers for that function are removed.

9. Within a few minutes of this initial setup, your AWS Logs appear in the Datadog Log Explorer.

Manually set up triggers

Collecting logs from CloudWatch log group

If you are collecting logs from a CloudWatch log group, configure the trigger to the Datadog Forwarder Lambda function using one of the following methods:

data "aws_cloudwatch_log_group" "some_log_group" {
  name = "/some/log/group"
}

resource "aws_lambda_permission" "lambda_permission" {
  action        = "lambda:InvokeFunction"
  function_name = "datadog-forwarder" # this is the default but may be different in your case
  principal     = "logs.amazonaws.com" # or logs.amazonaws.com.cn for China*
  source_arn    = data.aws_cloudwatch_log_group.some_log_group.arn
}

resource "aws_cloudwatch_log_subscription_filter" "datadog_log_subscription_filter" {
  name            = "datadog_log_subscription_filter"
  log_group_name  = <CLOUDWATCH_LOG_GROUP_NAME> # for example, /some/log/group
  destination_arn = <DATADOG_FORWARDER_ARN> # for example,  arn:aws:lambda:us-east-1:123:function:datadog-forwarder
  filter_pattern  = ""
}

Resources:
  MyLogSubscriptionFilter:
    Type: "AWS::Logs::SubscriptionFilter"
    Properties:
      DestinationArn: "<DATADOG_FORWARDER_ARN>"
      LogGroupName: "<CLOUDWATCH_LOG_GROUP_NAME>"
      FilterPattern: ""

Collecting logs from S3 buckets

If you are collecting logs from an S3 bucket, configure the trigger to the Datadog Forwarder Lambda function using one of the following methods:

resource "aws_s3_bucket_notification" "my_bucket_notification" {
  bucket = my_bucket
  lambda_function {
    lambda_function_arn = "<DATADOG_FORWARDER_ARN>"
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "AWSLogs/"
    filter_suffix       = ".log"
  }
}

Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: "<MY_BUCKET>"
      NotificationConfiguration:
        LambdaConfigurations:
        - Event: 's3:ObjectCreated:*'
          Function: "<DATADOG_FORWARDER_ARN>"

Scrubbing and filtering

You can scrub emails or IP address from logs sent by the Lambda function, or define a custom scrubbing rule in the Lambda parameters. You can also exclude or send only those logs that match a specific pattern by using the filtering option.

Further reading

Additional helpful documentation, links, and articles:

Send AWS LogsLEARNING CENTER Configure AWS Lambda for Serverless Monitoring with DatadogLEARNING CENTER Learn how to explore your logsDOCUMENTATION Perform Log AnalyticsDOCUMENTATION Learn how to process your logsDOCUMENTATION How to send logs to Datadog while reducing data transfer feesGUIDE