--- title: AWS Account-Level Log Subscriptions description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: Docs > Log Management > Logs Guides > AWS Account-Level Log Subscriptions --- # AWS Account-Level Log Subscriptions ## Overview{% #overview %} You can use account-level log subscriptions in your AWS environment to automatically forward all of your CloudWatch logs to Datadog. With an account-level log subscription, you don't need to manually configure log forwarding when you have a new log source, or when AWS releases a new service. You can also define your own selection criteria or filter pattern, for more control over which logs are forwarded. ## Create an account-level log subscription{% #create-an-account-level-log-subscription %} There are two ways to create an account-level log subscription, through CloudFormation and through manual setup. For the simplest setup, use CloudFormation to create an Amazon Data Firehose and associated resources in each of your selected regions. ### CloudFormation (recommended){% #cloudformation-recommended %} 1. Copy the URL of the CloudFormation template: ```bash https://datadog-cloudformation-template.s3.amazonaws.com/aws_account_level_logs/main.yaml ``` Go to [CloudFormation](https://console.aws.amazon.com/cloudformation/home) in the AWS console.Click Create stack. - Select `With new resources (standard)`. Leave the options to Choose an existing template and Amazon S3 URL checked.In the Amazon S3 URL field, paste the URL of the CloudFormation template.Click Next.In the Stack name field, provide a descriptive name such as `datadog-account-level-logs-stack`.In the ApiKey field, paste a valid [Datadog API key](https://app.datadoghq.com/organization-settings/api-keys) value.In the Regions field, enter a comma-separated list of AWS region codes (for example, `us-east-1`) corresponding to the regions to include for the account-level log subscription.In the DatadogHttpEndpointUrl field, select the URL corresponding to your [Datadog site](https://docs.datadoghq.com/getting_started/site.md).Click Next.Configure additional stack options as desired.Click Next.Review the stack options, and click the checkbox stating `I acknowledge that AWS CloudFormation might create IAM resources with custom names`.Click Submit. ### Manual{% #manual %} {% tab title="Lambda Forwarder" %} 1. If you haven't already, set up the [Datadog Forwarder](https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function.md) Lambda function. 1. Use the [AWS CLI](https://aws.amazon.com/cli/) to grant CloudWatch Logs the permission to execute your function. - Replace `` with the region containing your Datadog Forwarder Lambda function. - Replace `` with your 12-digit AWS account ID (excluding dashes). ```bash aws lambda add-permission \ --region "" \ --function-name "forwarder-function" \ --statement-id "forwarder-function" \ --principal "logs.amazonaws.com" \ --action "lambda:InvokeFunction" \ --source-arn "arn:aws:logs:::log-group:*" \ --source-account "" ``` Create an account-level subscription filter policy. In the example provided below, all log events that contain the string `ERROR` are streamed, except those in the log groups named `LogGroupToExclude1` and `LogGroupToExclude2`. - Replace `FORWARDER_ARN` with the ARN of the Datadog Forwarder Lambda function. ```bash aws logs put-account-policy \ --policy-name "ExamplePolicyLambda" \ --policy-type "SUBSCRIPTION_FILTER_POLICY" \ --policy-document '{"DestinationArn":"", "FilterPattern": "", "Distribution": "Random"}' \ --scope "ALL" ``` **Note**: To exclude certain log groups from log forwarding, use the `--selection-criteria` option as outlined in the [command reference](https://docs.aws.amazon.com/cli/latest/reference/logs/put-account-policy.html). {% /tab %} {% tab title="Amazon Data Firehose" %} #### Create an S3 bucket and a role for Amazon Data Firehose{% #create-an-s3-bucket-and-a-role-for-amazon-data-firehose %} The following steps guide you through creating a bucket and IAM role. This role grants Amazon Data Firehose permission to put data into your Amazon S3 bucket in case of delivery failures. 1. Use the [AWS CLI](https://aws.amazon.com/cli/) to create an S3 Bucket. Optionally, you can use an existing bucket. - Replace `` with the name for your S3 bucket. - Replace `` with the region for your S3 bucket. ``` aws s3api create-bucket \ --bucket MY-BUCKET \ --create-bucket-configuration LocationConstraint= ``` Create a `TrustPolicyForFirehose.json` file with the following statement: ```bash { "Statement": { "Effect": "Allow", "Principal": { "Service": "firehose.amazonaws.com" }, "Action": "sts:AssumeRole" } } ``` Create an IAM role, specifying the trust policy file: **Note**: The returned `Role.Arn` value is used in a later step. ```bash aws iam create-role \ --role-name FirehosetoS3Role \ --assume-role-policy-document file://./TrustPolicyForFirehose.json ``` Create a `PermissionsForFirehose.json` file with the following statement: - Replace `` with the name of your S3 bucket. ```bash { "Statement": [ { "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::", "arn:aws:s3:::/*" ] } ] } ``` Associate the permissions policy with the role: ```bash aws iam put-role-policy \ --role-name FirehosetoS3Role \ --policy-name Permissions-Policy-For-Firehose \ --policy-document file://./PermissionsForFirehose.json ``` #### Create Amazon Data Firehose delivery stream{% #create-amazon-data-firehose-delivery-stream %} The following steps guide you through creating and configuring an Amazon Data Firehose delivery stream. 1. Go to [Amazon Data Firehose](https://console.aws.amazon.com/firehose/home) in the AWS console. 1. Click Create Firehose stream. 1. In the Source field, select the source of your logs: - Select `Amazon Kinesis Data Streams` if your logs are coming from a Kinesis Data Stream - Select `Direct PUT` if your logs are coming directly from a CloudWatch log group 1. In the Destination field, select `Datadog`. 1. If your Source is `Amazon Kinesis Data Streams`, select your Kinesis data stream under Source settings. 1. Optionally, give the Firehose stream a descriptive name. 1. In the Destination settings section, choose the Datadog logs HTTP endpoint URL that corresponds to your [Datadog site](https://docs.datadoghq.com/getting_started/site.md). 1. For Authentication, a valid [Datadog API key](https://app.datadoghq.com/organization-settings/api-keys) is needed. You can either: - Select Use API key and paste the key's value in the API key field. - Select Use AWS Secrets Manager and choose a secret containing your valid Datadog API key value in the Secret name dropdown. 1. For Content encoding, select `GZIP`. 1. Optionally, configure the Retry duration, the buffer settings, or add Parameters (which are attached as tags to your logs).**Note**: Datadog recommends setting the Buffer size to `2` MiB if the logs are single-line messages. 1. In the Backup settings section, select the S3 bucket for receiving any failed events that exceed the retry duration.**Note**: To ensure any logs that fail to be delivered by the delivery stream are still sent to Datadog, set the Datadog Forwarder Lambda function to [forward logs from this S3 bucket](https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function.md?tab=automaticcloudformation#collecting-logs-from-s3-buckets). 1. Click Create Firehose stream. #### Create role for CloudWatch Logs{% #create-role-for-cloudwatch-logs %} The following steps guide you through creating an IAM role for CloudWatch logs. This role grants CloudWatch Logs permission to put data into your Firehose delivery stream. 1. Create a `./TrustPolicyForCWL.json` file with the following statement: - Replace `` with your 12-digit AWS account ID (excluding dashes). - Replace `` with the region of your CloudWatch logs. ```bash { "Statement": { "Effect": "Allow", "Principal": { "Service": "logs.amazonaws.com" }, "Action": "sts:AssumeRole", "Condition": { "StringLike": { "aws:SourceArn": "arn:aws:logs:::*" } } } } ``` Create an IAM role, specifying the trust policy file: ```bash aws iam create-role \ --role-name CWLtoKinesisFirehoseRole \ --assume-role-policy-document file://./TrustPolicyForCWL.json ``` **Note**: The returned `Role.Arn` value is used in a later step. Create a `./PermissionsForCWL.json` file with the following statement: - Replace `` with the region containing your Datadog Forwarder Lambda function. - Replace `` with your 12-digit AWS account ID (excluding dashes). - Replace `` with the name of your delivery stream. ```bash { "Statement":[ { "Effect":"Allow", "Action":["firehose:PutRecord"], "Resource":[ "arn:aws:firehose:::deliverystream/"] } ] } ``` Associate the permissions policy with the role: ```bash aws iam put-role-policy \ --role-name CWLtoKinesisFirehoseRole \ --policy-name Permissions-Policy-For-CWL \ --policy-document file://./PermissionsForCWL.json ``` #### Create the CloudWatch Logs account-level subscription filter policy{% #create-the-cloudwatch-logs-account-level-subscription-filter-policy %} Before completing this step, the Amazon Data Firehose delivery stream must be in the `Active` state. 1. Create the CloudWatch Logs account-level subscription filter policy. The policy immediately starts the flow of real-time log data from the chosen log group to your Amazon Data Firehose delivery stream: - Replace `` with a name for the subscription filter policy. - Replace `` with the ARN of the CloudWatch logs role. - Replace `` with the ARN of the Amazon Data Firehose delivery stream. ```bash aws logs put-account-policy \ --policy-name "" \ --policy-type "SUBSCRIPTION_FILTER_POLICY" \ --policy-document '{"RoleArn":"", "DestinationArn":"", "FilterPattern": "", "Distribution": "Random"}' \ --scope "ALL" ``` **Note**: To exclude certain log groups from log forwarding, use the `--selection-criteria` option as outlined in the [command reference](https://docs.aws.amazon.com/cli/latest/reference/logs/put-account-policy.html). {% /tab %} ### Validation{% #validation %} Go to the [Log Explorer](https://app.datadoghq.com/logs) and enter the search query `@aws.firehose.arn:""` to view logs forwarded by the Amazon Data Firehose. - Replace `` with the ARN of the log-streaming [Firehose](https://console.aws.amazon.com/firehose/home). ## Further reading{% #further-reading %} - [Learn how to explore your logs](https://docs.datadoghq.com/logs/explorer.md)