If you don’t see logs forwarded from a Datadog forwarder Lambda function in the Log Explorer, follow the troubleshooting steps below. If you continue to have trouble after following these steps, contact Datadog support for further assistance.
source:cloudwatchbut you can find other possible sources in the
parse_event_sourcefunction in the Lambda function.
forwardernametag to all the logs it forwards. You can filter on this tag by searching for
Open your forwarder Lambda function.
Click the Monitoring tab.
The monitoring tab displays a series of graphs indicating the following information about your Lambda function:
If you don’t see any data points on the Invocations graph, there may be a problem with the triggers you set for your function. See Manage Your Function Triggers. To get insight into your Lambda invocations without using the monitoring tab, see Viewing Lambda metrics in Datadog.
If you see data points on the “Error count and success rate” graph, check the Lambda function logs to see what error messages are being reported.
If you have enabled AWS Lambda metrics, you can view metrics related to Lambda invocations and errors within Datadog. The following metrics are all tagged with the
|Count of times the Lambda function has been triggered/invoked|
|Count of errors that occurred when the function was invoked|
|Average amount of time (in milliseconds) that it took for the Lambda function to finish executing|
|Maximum amount of time (in milliseconds) that it took for the Lambda function to finish executing|
|Count of invocation attempts that were throttled due to invocation rates exceeding customer limits|
For more information on these and other AWS Lambda metrics, see Amazon Lambda Metrics.
For logs to be forwarded, the forwarder Lambda function needs to have triggers (CloudWatch Logs or S3) set up. Follow the steps below to ensure the triggers are set up correctly.
Does the source of your log (CloudWatch log group or S3 bucket) show up in the “Triggers” list in the forwarder Lambda console? If yes, ensure it’s enabled. Otherwise, follow the steps below to check in the S3 or CloudWatch log group console, because the “Triggers” list displayed in the Lambda console is known to be incomprehensive.
For S3 bucket, navigate to the bucket’s “Properties” tab and scroll down to the “Advanced settings” and “Events” tile, or make a query using the AWS CLI command below. Do you see any event notification configured to trigger the forwarder Lambda function? If not, you need to configure a trigger.
aws s3api get-bucket-notification-configuration --bucket <BUCKET_NAME>
For CloudWatch log group, navigate to the log group’s console’s “Subscriptions” field under the “Log group details” section. Alternatively, you can make a query using AWS CLI command below. If the log group is not subscribed by the forwarder Lambda function, you need to configure a trigger.
aws logs describe-subscription-filters --log-group-name <LOG_GROUP_NAME>
Note, AWS doesn’t allow for more than one resource to be subscribed to a log source. If your log source is already subscribed by a different resource, you need to remove that subscription first.
For CloudWatch log group, you can use the following metrics within the Datadog platform to confirm whether logs are delivered from the log group to the forwarder Lambda function. Use the
log_group tag to filter the data when viewing the metrics.
|The number of log events uploaded to CloudWatch Logs|
|The number of log events forwarded to the subscription destination|
|The number of log events failed to be delivered to the subscription destination|
|The number of log events throttled for delivering to the subscription destination|
Find the most recent log stream.
Do you see any errors? Try searching “?ERROR ?Error ?error”.
Set environment variable “DD_LOG_LEVEL” to “debug” on the forwarder Lambda function to enable the debugging logs for further debugging. The debugging logs are quite verbose; remember to disable it after debugging.