--- title: Send AWS Services Logs With The Datadog Lambda Function description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: >- Docs > Log Management > Logs Guides > Send AWS Services Logs With The Datadog Lambda Function --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Send AWS Services Logs With The Datadog Lambda Function AWS service logs can be collected with the Datadog Forwarder Lambda function. This Lambda—which triggers on S3 Buckets, CloudWatch log groups, and EventBridge events—forwards logs to Datadog. To start collecting logs from your AWS services: 1. Set up the [Datadog Forwarder Lambda function](https://docs.datadoghq.com/serverless/forwarder.md) in your AWS account. 1. Enable logging for your AWS service. Find your service in Supported AWS services to see its setup instructions. Most AWS services can log to a S3 bucket or a CloudWatch log group. 1. Set up the triggers that cause the Forwarder Lambda to execute when there are new logs to be forwarded. There are two ways to configure the triggers. **Notes**: - You can use [AWS PrivateLink](https://docs.datadoghq.com/serverless/forwarder.md#aws-privatelink-support) to send your logs over a private connection. - CloudFormation creates an IAM policy which includes `KMS:Decrypt` for all resources, and does not align with AWS Security Hub's best practice. This permission is used to decrypt objects from KMS-encrypted S3 buckets to set up the Lambda function, and the KMS key used to encrypt the S3 buckets cannot be predicted. You can safely delete this permission after the installation finishes successfully. ## Supported AWS services{% #supported-aws-services %} The Datadog Forwarder Lambda function supports any AWS service that generates logs into a S3 bucket or a CloudWatch log group. The following table lists the services it can collect: - **AWS service**: The AWS service that generates the logs. Each service name links to its log collection setup instructions. Services without a link require no service-side setup. - **Log source**: The `source` tag Datadog applies to the logs. Use it to find your logs in the [Log Explorer](https://app.datadoghq.com/logs). - **Storage**: Where the AWS service can write logs that the Forwarder collects. - **Automatic collection**: Whether Datadog can automatically set up the triggers for that log source. If not, set up the triggers manually. | AWS service | Log source | Storage | Automatic collection | | ----------------------------------------------------------------------------------------------------- | ---------------------------- | -------------- | -------------------- | | [API Gateway](https://docs.datadoghq.com/integrations/amazon_api_gateway.md#log-collection) | `source:apigateway` | CloudWatch, S3 | Yes | | [AppSync](https://docs.datadoghq.com/integrations/amazon-appsync.md#send-logs-to-datadog) | `source:appsync` | CloudWatch | Yes | | Batch | `source:batch` | CloudWatch | Yes | | [Bedrock](https://docs.datadoghq.com/integrations/amazon-bedrock.md) | `source:bedrock` | CloudWatch, S3 | No | | Bedrock Agentcore | `source:bedrock-agentcore` | CloudWatch, S3 | Yes | | [CloudFront](https://docs.datadoghq.com/integrations/amazon_cloudfront.md#log-collection) | `source:cloudfront` | CloudWatch, S3 | Yes | | [CloudTrail](https://docs.datadoghq.com/integrations/amazon_cloudtrail.md#send-logs-to-datadog) | `source:cloudtrail` | CloudWatch, S3 | Yes | | [CodeBuild](https://docs.datadoghq.com/integrations/amazon-codebuild.md#send-logs-to-datadog) | `source:codebuild` | CloudWatch, S3 | Yes | | [DMS](https://docs.datadoghq.com/integrations/amazon-dms.md#send-logs-to-datadog) | `source:dms` | CloudWatch, S3 | Yes | | [DocumentDB](https://docs.datadoghq.com/integrations/amazon-documentdb.md#send-logs-to-datadog) | `source:docdb` | CloudWatch, S3 | Yes | | [ECS](https://docs.datadoghq.com/containers/amazon_ecs/logs.md) | `source:ecs` | CloudWatch | Yes | | [EKS](https://docs.datadoghq.com/integrations/amazon-eks.md#log-collection) | `source:eks` 1 | CloudWatch | Yes | | [Elastic Beanstalk](https://docs.datadoghq.com/integrations/amazon-elastic-beanstalk.md) | - 2 | CloudWatch | Yes | | [Elastic Load Balancing (ELB)](https://docs.datadoghq.com/integrations/amazon_elb.md#log-collection) | `source:elb` | CloudWatch, S3 | Yes | | [FSx](https://docs.datadoghq.com/integrations/amazon_fsx.md#log-collection) | `source:aws.fsx` | CloudWatch, S3 | No | | [Glue](https://docs.datadoghq.com/integrations/amazon_glue.md#log-collection) | `source:glue` | CloudWatch, S3 | Yes | | [IoT](https://docs.datadoghq.com/integrations/amazon-iot.md#enable-logging) | `source:iot` | CloudWatch | Partial 3 | | [Lambda](https://docs.datadoghq.com/integrations/amazon_lambda.md#log-collection) | `source:lambda` | CloudWatch | Yes | | Lambda@Edge | `source:lambda` | CloudWatch | Yes | | [MWAA](https://docs.datadoghq.com/integrations/amazon_mwaa.md#log-collection) | `source:mwaa` | CloudWatch | Yes | | [Network Firewall](https://docs.datadoghq.com/integrations/amazon_network_firewall.md#log-collection) | `source:network-firewall` | CloudWatch, S3 | Yes | | [OpenSearch](https://docs.datadoghq.com/integrations/amazon_es.md#log-collection) | `source:opensearch` | CloudWatch | No | | [PCS](https://docs.datadoghq.com/integrations/amazon-pcs.md) | - 2 | CloudWatch | Partial 4 | | [RDS](https://docs.datadoghq.com/integrations/amazon_rds.md#log-collection) | `source:rds` 5 | CloudWatch | Yes | | [Redshift](https://docs.datadoghq.com/integrations/amazon-redshift.md#log-collection) | `source:redshift` | CloudWatch, S3 | Yes | | Redshift Serverless | `source:redshift-serverless` | CloudWatch | Yes | | [Route 53](https://docs.datadoghq.com/integrations/amazon_route53.md#send-logs-to-datadog) | `source:route53` 6 | CloudWatch | Yes | | [S3](https://docs.datadoghq.com/integrations/amazon_s3.md#enable-s3-access-logs) | `source:s3` | S3 | Yes | | SSM | `source:ssm` | CloudWatch | Yes | | [Step Functions](https://docs.datadoghq.com/integrations/amazon_step_functions.md#log-collection) | `source:stepfunction` | CloudWatch | Yes | | [Transit Gateway](https://docs.datadoghq.com/integrations/amazon_transit_gateway.md#log-collection) | `source:transitgateway` | CloudWatch, S3 | No | | [Verified Access](https://docs.datadoghq.com/integrations/amazon-verified-access.md#log-collection) | `source:verified-access` | CloudWatch, S3 | Yes | | [VPC](https://docs.datadoghq.com/integrations/amazon_vpc.md#log-collection) | `source:vpc` | CloudWatch, S3 | Yes | | [VPN](https://docs.datadoghq.com/integrations/amazon-vpn.md#send-logs-to-datadog) | - 2 | CloudWatch, S3 | Yes 7 | | [Web Application Firewall](https://docs.datadoghq.com/integrations/amazon_waf.md#log-collection) | `source:waf` | S3 | Yes | 1 EKS control plane logs also use the `kubernetes.audit`, `kube-scheduler`, `kube-apiserver`, `kube-controller-manager`, and `aws-iam-authenticator` sources. 2 Datadog does not apply a service-specific source tag to these logs. 3 Automatic collection for IoT is available at the account level only. 4 Automatic collection for PCS is available for CloudWatch log groups only. 5 RDS engine logs also use the `postgresql`, `mariadb`, and `mysql` sources. 6 Covers both DNS query logs and Resolver query logs. 7 Automatic collection is available for CloudWatch log groups. For S3 buckets, set up the trigger manually. **Note**: The Datadog Forwarder automatically creates [subscription filters](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters) on CloudWatch log groups. Each filter is named in the format `DD_LOG_SUBSCRIPTION_FILTER_`. ### Services collected through another method{% #services-collected-through-another-method %} The following AWS services are supported for log collection, but do not use the Datadog Forwarder Lambda function in the same way: | AWS service | How logs are collected | | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [DynamoDB](https://docs.datadoghq.com/integrations/amazon_dynamodb.md) | DynamoDB does not generate its own logs. API activity is captured through CloudTrail. See [Send logs to Datadog](https://docs.datadoghq.com/integrations/amazon_dynamodb.md#send-logs-to-datadog). | | [EC2](https://docs.datadoghq.com/integrations/amazon_ec2.md) | Use the [Datadog Agent](https://docs.datadoghq.com/integrations/amazon_ec2.md) to send your logs to Datadog. | | [SNS](https://docs.datadoghq.com/integrations/amazon_sns.md) | SNS does not provide logs, but you can process logs and events that are transiting through to the SNS service. See [Send logs to Datadog](https://docs.datadoghq.com/integrations/amazon_sns.md#send-logs-to-datadog). | ## Set up triggers{% #set-up-triggers %} There are two options when configuring triggers on the Datadog Forwarder Lambda function: - Automatically: Datadog automatically retrieves the log locations for the selected AWS services and adds them as triggers on the Datadog Forwarder Lambda function. Datadog also keeps the list up to date. - Manually: Set up each trigger yourself. ### Automatically set up triggers{% #automatically-set-up-triggers %} Datadog can automatically configure triggers on the Datadog Forwarder Lambda function to collect AWS logs. However, automatic subscription does not support creating triggers across different AWS accounts or regions. For scenarios where logs are published to S3 buckets in a separate account, we recommend manually creating a trigger in the same account as the bucket to work around this limitation. To see which services support automatic collection, and the storage locations they support, see Supported AWS services. 1. If you haven't already, set up the [Datadog log collection AWS Lambda function](https://docs.datadoghq.com/serverless/forwarder.md). 1. Ensure the policy of the IAM role used for [Datadog-AWS integration](https://docs.datadoghq.com/integrations/amazon_web_services.md) has the following permissions. Information on how these permissions are used can be found in the descriptions below: ```text "airflow:GetEnvironment", "airflow:ListEnvironments", "appsync:ListGraphqlApis", "batch:DescribeJobDefinitions", "cloudfront:GetDistributionConfig", "cloudfront:ListDistributions", "cloudtrail:GetTrail", "cloudtrail:ListTrails", "codebuild:BatchGetProjects", "codebuild:ListProjects", "dms:DescribeReplicationInstances", "ec2:DescribeFlowLogs", "ec2:DescribeVerifiedAccessInstanceLoggingConfigurations", "ec2:DescribeVpnConnections", "ecs:DescribeTaskDefinition", "ecs:ListTaskDefinitionFamilies", "eks:DescribeCluster", "eks:ListClusters", "elasticbeanstalk:DescribeEnvironments", "elasticloadbalancing:DescribeLoadBalancerAttributes", "elasticloadbalancing:DescribeLoadBalancers", "glue:BatchGetJobs", "glue:GetJobs", "glue:GetJob", "glue:ListJobs", "iot:GetV2LoggingOptions", "lambda:GetPolicy", "lambda:InvokeFunction", "lambda:List*", "logs:DeleteSubscriptionFilter", "logs:DescribeDeliveries", "logs:DescribeDeliverySources", "logs:DescribeLogGroups", "logs:DescribeSubscriptionFilters", "logs:GetDeliveryDestination", "logs:PutSubscriptionFilter", "network-firewall:DescribeLoggingConfiguration", "network-firewall:ListFirewalls", "rds:DescribeDBClusters", "rds:DescribeDBInstances", "redshift-serverless:ListNamespaces", "redshift:DescribeClusters", "redshift:DescribeLoggingStatus", "route53:ListQueryLoggingConfigs", "route53resolver:ListResolverQueryLogConfigs", "s3:GetBucketLocation", "s3:GetBucketLogging", "s3:GetBucketNotification", "s3:ListAllMyBuckets", "s3:PutBucketNotification", "ssm:GetServiceSetting", "ssm:ListCommands", "states:DescribeStateMachine", "states:ListStateMachines", "wafv2:ListLoggingConfigurations" ``` | AWS Permission | Description | | --------------------------------------------------------- | ---------------------------------------------------------------- | | `airflow:ListEnvironments` | List all MWAA environment names. | | `airflow:GetEnvironment` | Get information about a MWAA environment. | | `appsync:ListGraphqlApis` | List all GraphQL Apis. | | `batch:DescribeJobDefinitions` | List all Batch job definitions. | | `cloudfront:GetDistributionConfig` | Get the name of the S3 bucket containing CloudFront access logs. | | `cloudfront:ListDistributions` | List all CloudFront distributions. | | `cloudtrail:GetTrail` | Get Trail logging information. | | `cloudtrail:ListTrails` | List all Cloudtrail trails. | | `codebuild:BatchGetProjects` | List all CodeBuild projects. | | `codebuild:ListProjects` | Get information on CodeBuild projects. | | `dms:DescribeReplicationInstances` | List all replication instances for DMS. | | `ec2:DescribeFlowLogs` | List all Flow log configurations. | | `ec2:DescribeVerifiedAccessInstanceLoggingConfigurations` | List all Verified Access instance logging configurations. | | `ec2:DescribeVpnConnections` | List all VPN connections. | | `ecs:DescribeTaskDefinition` | Describe ECS task definition. | | `ecs:ListTaskDefinitionFamilies` | List all task definition families. | | `elasticloadbalancing:``DescribeLoadBalancers` | List all load balancers. | | `elasticloadbalancing:``DescribeLoadBalancerAttributes` | Get the name of the S3 bucket containing ELB access logs. | | `glue:BatchGetJobs` | Get information about multiple Glue jobs. | | `glue:GetJob` | Get information about a Glue job. | | `glue:GetJobs` | List all Glue jobs. | | `glue:ListJobs` | List all Glue job names. | | `eks:DescribeCluster` | Describe an EKS cluster. | | `eks:ListClusters` | List all EKS clusters. | | `elasticbeanstalk:DescribeEnvironments` | List all Elastic Beanstalk environments. | | `iot:GetV2LoggingOptions` | Get IoT V2 logging options. | | `lambda:InvokeFunction` | Invoke a Lambda function. | | `lambda:List*` | List all Lambda functions. | | `lambda:GetPolicy` | Get the Lambda policy when triggers are to be removed. | | `logs:PutSubscriptionFilter` | Add a Lambda trigger based on CloudWatch Log events. | | `logs:DeleteSubscriptionFilter` | Remove a Lambda trigger based on CloudWatch Log events. | | `logs:DescribeLogGroups` | Describe CloudWatch log groups. | | `logs:DescribeDeliveries` | Describe CloudWatch log deliveries. | | `logs:DescribeDeliverySources` | Describe CloudWatch log delivery sources. | | `logs:DescribeSubscriptionFilters` | List the subscription filters for the specified log group. | | `logs:GetDeliveryDestination` | Get a CloudWatch log delivery destination. | | `network-firewall:DescribeLoggingConfiguration` | Get the logging configuration of a firewall. | | `network-firewall:ListFirewalls` | List all Network Firewall firewalls. | | `rds:DescribeDBClusters` | List all RDS clusters. | | `rds:DescribeDBInstances` | List all RDS instances. | | `redshift:DescribeClusters` | List all Redshift clusters. | | `redshift:DescribeLoggingStatus` | Get the name of the S3 bucket containing Redshift Logs. | | `redshift-serverless:ListNamespaces` | List all Redshift Serverless namespaces. | | `route53:ListQueryLoggingConfigs` | List all DNS query logging configurations for Route 53. | | `route53resolver:ListResolverQueryLogConfigs` | List all Resolver query logging configurations for Route 53. | | `s3:GetBucketLogging` | Get the name of the S3 bucket containing S3 access logs. | | `s3:GetBucketLocation` | Get the region of the S3 bucket containing S3 access logs. | | `s3:GetBucketNotification` | Get existing Lambda trigger configurations. | | `s3:ListAllMyBuckets` | List all S3 buckets. | | `s3:PutBucketNotification` | Add or remove a Lambda trigger based on S3 bucket events. | | `ssm:GetServiceSetting` | Get the SSM service setting for customer script log group name. | | `ssm:ListCommands` | List all SSM commands. | | `states:ListStateMachines` | List all Step Functions. | | `states:DescribeStateMachine` | Get logging details about a Step Function. | | `wafv2:ListLoggingConfigurations` | List all logging configurations of the Web Application Firewall. | 1. In the [AWS Integration page](https://app.datadoghq.com/integrations/amazon-web-services), select the AWS Account to collect logs from and click on the Log Collection tab. 1. In the Datadog Forwarder Lambda section, enter the ARN of the Lambda created in the previous section and click Add. The Lambda function appears in the table below with its name, version, and region. 1. In the Log Autosubscription section, under Log Sources, enable the services from which you'd like to collect logs by toggling them on. To stop collecting logs from a particular service, toggle the log source off. 1. (Optional) In the Log Source Tag Filters section, you can filter log collection by resource tags for each log source. Select a log source from the dropdown menu and add tags in `key:value` format to limit which resources' logs are collected. **Note**: Resource tags are automatically lowercased to match Datadog platform conventions. Define your tag filters in lowercase to avoid mismatches. 1. If you have logs across multiple regions, you must create additional Lambda functions in those regions and add them in the **Datadog Forwarder Lambda** section. 1. To stop collecting all AWS logs from a specific Lambda function, hover over the Lambda in the table and click the delete icon. All triggers for that function are removed. 1. Within a few minutes of this initial setup, your AWS Logs appear in the Datadog [Log Explorer](https://app.datadoghq.com/logs). ### Manually set up triggers{% #manually-set-up-triggers %} #### Collecting logs from CloudWatch log group{% #collecting-logs-from-cloudwatch-log-group %} If you are collecting logs from a CloudWatch log group, configure the trigger to the [Datadog Forwarder Lambda function](https://docs.datadoghq.com/serverless/forwarder.md) using one of the following methods: {% tab title="AWS console" %} 1. In the AWS console, go to Lambda. 1. Click Functions and select the Datadog Forwarder. 1. Click Add trigger and select CloudWatch Logs. 1. Select the log group from the dropdown menu. 1. Enter a name for your filter, and optionally specify a filter pattern. 1. Click Add. 1. Go to the [Datadog Log section](https://app.datadoghq.com/logs) to explore any new log events sent to your log group. {% /tab %} {% tab title="Terraform" %} For Terraform users, you can provision and manage your triggers using the [aws_cloudwatch_log_subscription_filter](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_subscription_filter) resource. See sample code below. ``` data "aws_cloudwatch_log_group" "some_log_group" { name = "/some/log/group" } resource "aws_lambda_permission" "lambda_permission" { action = "lambda:InvokeFunction" function_name = "datadog-forwarder" # this is the default but may be different in your case principal = "logs.amazonaws.com" # or logs.amazonaws.com.cn for China* source_arn = data.aws_cloudwatch_log_group.some_log_group.arn } resource "aws_cloudwatch_log_subscription_filter" "datadog_log_subscription_filter" { name = "datadog_log_subscription_filter" log_group_name = # for example, /some/log/group destination_arn = # for example, arn:aws:lambda:us-east-1:123:function:datadog-forwarder filter_pattern = "" } ``` \* *All use of Datadog Services in (or in connection with environments within) mainland China is subject to the disclaimer published in the [Restricted Service Locations](https://www.datadoghq.com/legal/restricted-service-locations/) section on our website.* {% /tab %} {% tab title="CloudFormation" %} For AWS CloudFormation users, you can provision and manage your triggers using the CloudFormation [AWS::Logs::SubscriptionFilter](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-subscriptionfilter.html) resource. See sample code below. The sample code also work for AWS [SAM](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html) and [Serverless Framework](https://www.serverless.com/). For Serverless Framework, put the code under the [resources](https://www.serverless.com/framework/docs/providers/aws/guide/resources/) section within your `serverless.yml`. ```yaml Resources: MyLogSubscriptionFilter: Type: "AWS::Logs::SubscriptionFilter" Properties: DestinationArn: "" LogGroupName: "" FilterPattern: "" ``` {% /tab %} #### Collecting logs from S3 buckets{% #collecting-logs-from-s3-buckets %} If you are collecting logs from an S3 bucket, configure the trigger to the [Datadog Forwarder Lambda function](https://docs.datadoghq.com/serverless/forwarder.md) using one of the following methods: {% tab title="AWS Console" %} 1. Once the Lambda function is installed, manually add a trigger on the S3 bucket that contains your logs in the AWS console: {% image source="https://docs.dd-static.net/images/logs/aws/adding_trigger.bf800f54049690ef4d358361e887e221.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/logs/aws/adding_trigger.bf800f54049690ef4d358361e887e221.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Adding trigger" /%} 1. Select the bucket and then follow the AWS instructions: {% image source="https://docs.dd-static.net/images/logs/aws/integration_lambda.facd5ed4e39ceb4ac342317606e8fa7c.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/logs/aws/integration_lambda.facd5ed4e39ceb4ac342317606e8fa7c.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Integration Lambda" /%} 1. Set the correct event type on S3 buckets: {% image source="https://docs.dd-static.net/images/logs/aws/object_created.050320f60595af799e53dfbfbe5ecf1e.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/logs/aws/object_created.050320f60595af799e53dfbfbe5ecf1e.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Object Created" /%} Once done, go into your [Datadog Log section](https://app.datadoghq.com/logs) to start exploring your logs! {% /tab %} {% tab title="Terraform" %} For Terraform users, you can provision and manage your triggers using the [aws_s3_bucket_notification](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_notification) resource. See the sample code below. ``` resource "aws_s3_bucket_notification" "my_bucket_notification" { bucket = my_bucket lambda_function { lambda_function_arn = "" events = ["s3:ObjectCreated:*"] filter_prefix = "AWSLogs/" filter_suffix = ".log" } } ``` {% /tab %} {% tab title="CloudFormation" %} For CloudFormation users, you can configure triggers using the CloudFormation [NotificationConfiguration](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-notificationconfig.html) for your S3 bucket. See the sample code below. ```yaml Resources: Bucket: Type: AWS::S3::Bucket Properties: BucketName: "" NotificationConfiguration: LambdaConfigurations: - Event: 's3:ObjectCreated:*' Function: "" ``` {% /tab %} ## Scrubbing and filtering{% #scrubbing-and-filtering %} You can scrub emails or IP address from logs sent by the Lambda function, or define a custom scrubbing rule [in the Lambda parameters](https://github.com/DataDog/datadog-serverless-functions/tree/master/aws/logs_monitoring#log-scrubbing-optional). You can also exclude or send only those logs that match a specific pattern by using the [filtering option](https://github.com/DataDog/datadog-serverless-functions/tree/master/aws/logs_monitoring#log-filtering-optional). ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Send AWS Logs](https://learn.datadoghq.com/courses/send-aws-logs) - [Configure AWS Lambda for Serverless Monitoring with Datadog](https://learn.datadoghq.com/courses/visibility-aws-lambda) - [Learn how to explore your logs](https://docs.datadoghq.com/logs/explorer.md) - [Perform Log Analytics](https://docs.datadoghq.com/logs/explorer.md#visualize) - [Learn how to process your logs](https://docs.datadoghq.com/logs/log_configuration/processors.md) - [How to send logs to Datadog while reducing data transfer fees](https://docs.datadoghq.com/logs/guide/reduce_data_transfer_fees.md)