AWS service logs are usually stored in S3 buckets or CloudWatch Log groups. It is possible to subscribe to these logs and forward them to an Amazon Kinesis stream to then forward them to one or multiple destinations. Datadog is one of the default destinations for Amazon Kinesis Delivery streams.
AWS fully manages Amazon Kinesis Data Firehose, so you don’t need to maintain any additional infrastructure or forwarding configurations for streaming logs. You can set up a Kinesis Firehose Delivery Stream in the AWS Firehose console, or automatically set up the destination using a CloudFormation template.
Datadog recommends using a Kinesis data stream as input when using the Datadog Kinesis destination. It gives you the ability to forward your logs to multiple destinations in case Datadog is not the only consumer for those logs. If Datadog is the only destination for your logs, or if you already have a Kinesis data stream with your logs, you can ignore step one.
Optionally, use the Create a Data Stream section of the Amazon Kinesis Data Streams developer guide in AWS to create a new Kinesis data stream. Name the stream something descriptive, like DatadogLogStream.
Create a new delivery stream.
a. Set the source:
Amazon Kinesis Data Streams if your logs are coming from a Kinesis data streamDirect PUT if your logs are coming directly from a CloudWatch log groupb. Set the destination as Datadog.
c. Provide a name for the delivery stream.
d. In the Destination settings, choose the Datadog logs HTTP endpoint URL that corresponds to your Datadog site.
e. Paste your API key into the API key field. You can get or create an API key from the Datadog API Keys page.
f. Optionally, configure the Retry duration, the buffer settings, or add Parameters, which are attached as tags to your logs.
g. In the Backup settings, select an S3 backup bucket to receive any failed events that exceed the retry duration.
Note: To ensure that logs that fail through the delivery stream are still sent to Datadog, set the Datadog Forwarder Lambda function to forward logs from this S3 bucket.
h. Click Create delivery stream.
Alternatively, customize this CloudFormation template and install it from the AWS Console. See the full Kinesis CloudFormation template.
Subscribe your new Kinesis stream to the CloudWatch log groups you want to ingest into Datadog. You can check the Subscriptions column on the log groups index page to see current subscriptions to your log groups. Subscriptions can be created through the AWS console or API with the specifications below.
Note: Each CloudWatch Log group can only have two subscriptions.
Create an IAM role and permissions policy to enable CloudWatch Logs to put data into your Kinesis stream.
logs.amazonaws.com or logs.<region>.amazonaws.com is configured as the service principal in the role’s Trust relationships.firehose:PutRecord firehose:PutRecordBatch, kinesis:PutRecord, and kinesis:PutRecords actions.Use the Subscription filters with Kinesis example (steps 3 to 6) for an example of setting this up with the AWS CLI.
The following example creates a subscription filter through the AWS CLI:
```
aws logs put-subscription-filter \
--log-group-name "<MYLOGGROUPNAME>" \
--filter-name "<MyFilterName>" \
--filter-pattern "" \
--destination-arn "<DESTINATIONARN> (data stream or delivery stream)" \
--role-arn "<MYROLEARN>"
```
You can also create a subscription filter through the AWS console.
Go to your log group in CloudWatch and click on the Subscription filters tab, then Create.
Create Kinesis subscription filter.Create Kinesis Firehose subscription filter.Select the data stream or Firehose delivery stream as applicable, as well as the IAM role previously created.
Provide a name for the subscription filter, and click Start streaming.
Important note: The destination of the subscription filter must be in the same account as the log group, as described in the Amazon CloudWatch Logs API Reference.
Check the Subscriptions column in the log groups index page in CloudWatch to confirm that the new Kinesis stream is now subscribed to your log groups.
After you have set up an Amazon Kinesis delivery stream, you can analyze the logs subscribed to your delivery stream in Datadog.
To populate all logs by ARN:
@aws.firehose.arn:"<ARN>", replace <ARN> with your Amazon Kinesis Data Firehose ARN, and press Enter.Additional helpful documentation, links, and articles:
