Amazon S3

Docs > Integrations > Amazon S3

Monitor S3 metrics and optimize storage costs at the prefix level with Storage Monitoring (Preview).

Overview

Amazon S3 is a highly available and scalable cloud storage service.

Enable this integration to see in Datadog all your S3 metrics.

Note: This integration requires the permission s3:GetBucketTagging to be fully enabled.

Setup

Installation

If you haven’t already, set up the Amazon Web Services integration first.

Metric collection

In the AWS integration page, ensure that S3 is enabled under the Metric Collection tab.
Install the Datadog - Amazon S3 integration.

Log collection

Enable S3 access logs

Go to the S3 bucket.
Click Properties.
Go to the Services Access Logging section and click Edit.
Select Enable.
Select the S3 bucket to send the logs to.

For more information, see Enabling Amazon S3 server access logging.

Send logs to Datadog

Note: Datadog’s automatic trigger setup is available for S3 buckets only. For CloudWatch log groups, use the manual trigger setup.

If you haven’t already, set up the Datadog Forwarder Lambda function in your AWS account.
Once the Lambda function is installed, there are two ways to collect your S3 access logs:
- Automatically: S3 logs are managed automatically if you grant Datadog access with a set of permissions. See Automatically Set Up Triggers for more information on configuring automatic log collection on the Datadog Forwarder Lambda function.
- Manually: In the AWS console, add a trigger on the S3 bucket that contains your S3 access logs. See the manual installation steps.

Manual installation steps

If you haven’t already, set up the Datadog Forwarder Lambda function in your AWS account.
Once set up, go to the Datadog Forwarder Lambda function. In the Function Overview section, click on Add Trigger.
Select the S3 trigger for the Trigger Configuration.
Select the S3 bucket that contains your S3 logs.
Leave the event type as All object create events.
Click Add to add the trigger to your Lambda.

Go to the Log Explorer to start exploring your logs.

For more information on collecting AWS Services logs, see Send AWS Services Logs with the Datadog Lambda Function.

Data Collected

Metrics


aws.s3.4xx_errors (count)	The total number of HTTP 4xx client error status code requests made to a bucket
aws.s3.5xx_errors (count)	The total number of HTTP 5xx server error status code requests made to a bucket
aws.s3.all_requests (count)	The total number of HTTP requests made to a bucket, regardless of type.
aws.s3.bucket_size_bytes (gauge)	The amount of data in bytes stored in a bucket in the Standard storage class, Standard - Infrequent Access (Standard_IA) storage class, or the Reduced Redundancy Storage (RRS) class. Shown as byte
aws.s3.bytes_downloaded (count)	The total number bytes downloaded from the bucket. Shown as byte
aws.s3.bytes_pending_replication (gauge)	The total number of bytes of objects pending replication Shown as byte
aws.s3.bytes_uploaded (count)	The total number bytes uploaded to the bucket. Shown as byte
aws.s3.delete_requests (count)	The number of HTTP DELETE requests made for objects in a bucket. This also includes Delete Multiple Objects requests.
aws.s3.first_byte_latency (gauge)	The average per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond
aws.s3.first_byte_latency.maximum (gauge)	The maximum per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond
aws.s3.first_byte_latency.minimum (gauge)	The minimum per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond
aws.s3.first_byte_latency.p50 (gauge)	The 50 percentile per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond
aws.s3.first_byte_latency.p90 (gauge)	The 90 percentile per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond
aws.s3.first_byte_latency.p95 (gauge)	The 95 percentile per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond
aws.s3.first_byte_latency.p99 (gauge)	The 99 percentile per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond
aws.s3.first_byte_latency.p99.99 (gauge)	The 99.99 percentile per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond
aws.s3.get_requests (count)	The number of HTTP GET requests made for objects in a bucket. This doesn’t include list operations.
aws.s3.head_requests (count)	The number of HTTP HEAD requests made to a bucket.
aws.s3.list_requests (count)	The number of HTTP requests that list the contents of a bucket.
aws.s3.number_of_objects (gauge)	The total number of objects stored in a bucket for all storage classes except for the GLACIER storage class.
aws.s3.operations_failed_replication (gauge)	The average number of operations that failed to replicate for a given replication rule Shown as operation
aws.s3.operations_failed_replication.sum (count)	The total number of operations that failed to replicate for a given replication rule Shown as operation
aws.s3.operations_failed_replication.samplecount (count)	The total number of replication operations Shown as operation
aws.s3.operations_pending_replication (gauge)	The number of operations pending replication Shown as operation
aws.s3.post_requests (count)	The number of HTTP POST requests made to a bucket.
aws.s3.put_requests (count)	The number of HTTP PUT requests made for objects in a bucket.
aws.s3.replication_latency (gauge)	The maximum number of seconds by which the replication destination Region is behind the source Region Shown as second
aws.s3.total_request_latency (gauge)	The average elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond
aws.s3.total_request_latency.maximum (gauge)	The maximum elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond
aws.s3.total_request_latency.minimum (gauge)	The minimum elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond
aws.s3.total_request_latency.p50 (gauge)	The 50 percentile elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond
aws.s3.total_request_latency.p90 (gauge)	The 90 percentile elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond
aws.s3.total_request_latency.p95 (gauge)	The 95 percentile elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond
aws.s3.total_request_latency.p99 (gauge)	The 90 percentile elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond
aws.s3.total_request_latency.p99.99 (gauge)	The 99.99 percentile elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond

Daily storage metrics

AWS reports aws.s3.bucket_size_bytes and aws.s3.number_of_objects one time per day. These metrics might not appear for up to 24 hours after integration setup, and they reflect the previous day’s values. After deleting objects or emptying a bucket, aws.s3.number_of_objects might continue reporting the previous count until the next daily update from AWS.

Request metrics

S3 request metrics (such as aws.s3.get_requests) require configuration for each S3 bucket in the AWS console. They are not enabled by default. See Monitoring metrics with Amazon CloudWatch for setup instructions.

Events

The Amazon S3 integration does not include any events.

Service Checks

The Amazon S3 integration does not include any service checks.

Troubleshooting

Need help? Contact Datadog support.