Features
Integrations
Dashboards
Log Management
APM
Continuous Profiler
Security Monitoring
Network Monitoring
Synthetic Monitoring
Real User Monitoring
Serverless
Alerts
API
Amazon S3
Overview
Amazon Simple Storage Service (S3) is a highly available and scalable cloud storage service.
Enable this integration to see in Datadog all your S3 metrics.
Setup
Installation
If you haven’t already, set up the Amazon Web Services integration first.
Metric collection
- In the AWS integration tile, ensure that
S3is checked under metric collection. Add those permissions to your Datadog IAM policy in order to collect Amazon S3 metrics:
s3:ListAllMyBuckets: Used to list available bucketss3:GetBucketTagging: Used to get custom bucket tags
For more information on S3 policies, review the documentation on the AWS website.
Install the Datadog - AWS S3 integration.
(optional) To gather Request Metrics, Enable Requests metrics on your Amazon S3 buckets from the AWS console.
Log collection
Enable S3 access logs
Select the S3 bucket and click on the properties tab:
Then click on Server access logging and choose enable:
Then select the s3 bucket the logs should be written into. For more information see the dedicated S3 AWS documentation
Send logs to Datadog
- If you haven’t already, set up the Datadog log collection AWS Lambda function.
Once the lambda function is installed, there are two ways to collect your S3 access logs:
- Automatic: we manage the S3 access logs for you if you grant us a set of permissions. Refer to the main Amazon web service to configure the automatic log collection
- Manual: Add manually a trigger on the s3 bucket that contains your S3 access logs in the AWS console
Manual installation steps
- If you haven’t already, set up the Datadog log collection AWS Lambda function.
- Once the lambda function is installed, manually add a trigger on the S3 bucket that contains your S3 logs in the AWS console, in your Lambda, click on S3 in the trigger list:Configure your trigger by choosing the S3 bucket that contains your S3 logs and change the event type to
Object Created (All)then click on the add button.
Once done, go in your Datadog Log section to start exploring your logs!
Data Collected
Metrics
| aws.s3.bucket_size_bytes (gauge) | The amount of data in bytes stored in a bucket in the Standard storage class, Standard - Infrequent Access (Standard_IA) storage class, or the Reduced Redundancy Storage (RRS) class. Shown as byte |
| aws.s3.number_of_objects (gauge) | The total number of objects stored in a bucket for all storage classes except for the GLACIER storage class. |
| aws.s3.all_requests (count) | The total number of HTTP requests made to a bucket, regardless of type. |
| aws.s3.get_requests (count) | The number of HTTP GET requests made for objects in a bucket. This doesn't include list operations. |
| aws.s3.put_requests (count) | The number of HTTP PUT requests made for objects in a bucket. |
| aws.s3.delete_requests (count) | The number of HTTP DELETE requests made for objects in a bucket. This also includes Delete Multiple Objects requests. |
| aws.s3.head_requests (count) | The number of HTTP HEAD requests made to a bucket. |
| aws.s3.post_requests (count) | The number of HTTP POST requests made to a bucket. |
| aws.s3.list_requests (count) | The number of HTTP requests that list the contents of a bucket. |
| aws.s3.bytes_downloaded (count) | The total number bytes downloaded from the bucket. Shown as byte |
| aws.s3.bytes_uploaded (count) | The total number bytes uploaded to the bucket. Shown as byte |
| aws.s3.4xx_errors (count) | The total number of HTTP 4xx server error status code requests made to a bucket |
| aws.s3.5xx_errors (count) | The total number of HTTP 5xx server error status code requests made to a bucket |
| aws.s3.first_byte_latency (gauge) | The average per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond |
| aws.s3.first_byte_latency.minimum (gauge) | The minimum per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond |
| aws.s3.first_byte_latency.maximum (gauge) | The maximum per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond |
| aws.s3.first_byte_latency.p50 (gauge) | The 50 percentile per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond |
| aws.s3.first_byte_latency.p90 (gauge) | The 90 percentile per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond |
| aws.s3.first_byte_latency.p95 (gauge) | The 95 percentile per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond |
| aws.s3.first_byte_latency.p99 (gauge) | The 99 percentile per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond |
| aws.s3.first_byte_latency.p99.99 (gauge) | The 99.99 percentile per-request time from the complete request being received by a bucket to when the response starts to be returned. Shown as millisecond |
| aws.s3.total_request_latency (gauge) | The average elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond |
| aws.s3.total_request_latency.minimum (gauge) | The minimum elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond |
| aws.s3.total_request_latency.maximum (gauge) | The maximum elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond |
| aws.s3.total_request_latency.p50 (gauge) | The 50 percentile elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond |
| aws.s3.total_request_latency.p90 (gauge) | The 90 percentile elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond |
| aws.s3.total_request_latency.p95 (gauge) | The 95 percentile elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond |
| aws.s3.total_request_latency.p99 (gauge) | The 90 percentile elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond |
| aws.s3.total_request_latency.p99.99 (gauge) | The 99.99 percentile elapsed per-request time from the first byte received to the last byte sent to a bucket Shown as millisecond |
| aws.s3.replication_latency.maximum (gauge) | The maximum number of seconds by which the replication destination Region is behind the source Region Shown as second |
| aws.s3.bytes_pending_replication.maximum (gauge) | The total number of bytes of objects pending replication Shown as byte |
| aws.s3.operations_pending_replication.maximum (gauge) | The number of operations pending replication |
Each of the metrics retrieved from AWS will be assigned the same tags that appear in the AWS console, including but not limited to host name, security-groups, and more.
Events
The AWS S3 integration does not include any events.
Service Checks
The AWS S3 integration does not include any service checks.
Troubleshooting
CloudTrail encrypted log
If your AWS CloudTrail log data is encrypted by KMS in your AWS S3, allow the Datadog role to decrypt the Cloudtrail log data with the following policy: kms:Decrypt. Learn more about your KMS encrypt/decrypt policy.
On this Page