Ensure all S3-related permissions are granted for Resource Collection.

1. Add all S3 permissions required for Storage Management to your Datadog IAM policy. Storage Management relies on permissions in AWS Integration IAM policy and Resource Collection IAM policy to bring together bucket details. In addition, the following 3 permissions allows Datadog to enable S3 inventory on your source buckets and read the generated reports from the destination buckets.

   • s3:PutInventoryConfiguration
   • s3:GetObject (scoped to the destination bucket(s))
   • s3:ListBucket (scoped to the destination bucket(s))

   Example IAM policy with all required permissions for Storage Management:

         {
           "Version": "2012-10-17",
           "Statement": [
   
             {
               "Sid": "DatadogS3BucketInfo",
               "Effect": "Allow",
               "Action": [
                 "s3:ListAllMyBuckets",
                 "s3:GetAccelerateConfiguration",
                 "s3:GetAnalyticsConfiguration",
                 "s3:GetBucket*",
                 "s3:GetEncryptionConfiguration",
                 "s3:GetInventoryConfiguration",
                 "s3:GetLifecycleConfiguration",
                 "s3:GetMetricsConfiguration",
                 "s3:GetReplicationConfiguration",
                 "s3:ListBucket",
                 "s3:GetBucketLocation",
                 "s3:GetBucketLogging",
                 "s3:GetBucketTagging",
                 "s3:PutInventoryConfiguration"
               ],
               "Resource": "*"
             },
             {
               "Sid": "DatadogReadInventoryFromDestinationBucket",
               "Effect": "Allow",
               "Action": [
                 "s3:ListBucket",
                 "s3:GetObject"
               ],
               "Resource": [
                 "arn:aws:s3:::storage-management-inventory-destination",
                 "arn:aws:s3:::storage-management-inventory-destination/*"
               ]
             }
           ]
         }
   

2. On the Enable it for me tab, select the regions or accounts you want to enable and assign a destination bucket per region or per account to store S3 Inventory reports. You can either use an existing bucket or create one in AWS.

   • If you select a destination bucket containing inventories from multiple source buckets, all of those source buckets become enabled for monitoring.
   • The destination buckets must allow the source buckets to write inventory data. See Creating a destination bucket policy in the AWS documentation for details.

3. Complete the inventory configuration. The first inventory report may take up to 24 hours to generate.

To get prefix-level access metrics including request counts, server-side latency, and cold data identification for cost optimization, follow these additional steps:

1. Set up the Datadog Lambda Forwarder (if not already configured):

   • Follow the Datadog Forwarder installation instructions to deploy the Datadog Lambda function in your AWS account
   • This Lambda function collects and forwards your S3 access logs to Datadog

2. Configure S3 Access Logs for each source bucket:

   • Go to your S3 bucket properties in the AWS Console
   • Navigate to Server access logging
   • Enable logging and specify your destination bucket (for simplicity, you can use the destination bucket for your inventory files)
   • Set the target prefix to access-logs/ to organize log files separately from inventory data

3. Set up the Lambda trigger:

   Option A: Automatic (Recommended)

   • In the Datadog AWS integration page, navigate to the Log Collection tab
   • Enable automatic log collection for S3 by checking the S3 Access Logs checkbox
   • Datadog automatically configures triggers on your Forwarder Lambda function for S3 access logs

   Option B: Manual

   • In the AWS console, go to your Datadog Forwarder Lambda function
   • Click Add trigger and select S3
   • Select the bucket containing your access logs
   • Set the event type to All object create events
   • Set the prefix to access-logs/ (matching your access log prefix)
   After sending, S3 Access Logs are also available in Datadog Log Management.

The inventory generation process starts in AWS within 24 hours of the first report. Data from your buckets is visible after this period.

This template configures your existing S3 bucket to generate inventory reports, which Datadog uses to generate detailed metrics about your bucket prefixes.

1. Download the source-bucket-inventory-cfn.yaml template.

2. In AWS CloudFormation, click Create stack in the top right corner and select With existing resources (import resources).

3. In the Specify template step, select Upload a template file.

4. Click Choose file and select the source-bucket-inventory-cfn.yaml file, then click Next.

5. Enter the bucket name you want AWS to start generating inventories for, and click Next.

6. Fill in the required parameters:

   • DestinationBucketName: The bucket for storing inventory files. Note: You must only use one destination bucket for all inventory files generated in an AWS account.
   • SourceBucketName: The bucket you want to monitor and start generating inventory files for

   Optional parameters:

   • SourceBucketPrefix: (Optional) Limit monitoring to a specific path in the source bucket
   • DestinationBucketPrefix: Specific path within the destination bucket. Ensure this path doesn’t include trailing slashes (/)

7. Click Next.
8. Wait for AWS to locate your source bucket, and click Import resources in the bottom right corner.

Notes:

• This CloudFormation template can be rolled back, but rolling back doesn’t delete the created resources. This is to ensure the existing bucket doesn’t get deleted. You can manually delete the inventory configurations by going on the Management tab in the bucket view.
• Review Amazon S3 pricing for costs related to inventory generation.

This template creates two IAM policies:

• A policy to allow Datadog to read inventory files from the destination bucket
• A policy to allow your source bucket to write inventory files to the destination bucket

1. Download the cloud-inventory-policies-cfn.yaml template.

2. In AWS CloudFormation, click Create stack in the top right corner and select With new resources (standard).

3. In the Specify template step, select Upload a template file.

4. Click Choose file and select the cloud-inventory-policies-cfn.yaml file, then click Next.

5. Fill in the required parameters:

   • DatadogIntegrationRole: Your Datadog AWS integration role name
   • DestinationBucketName: The name of the bucket that receives your inventory files. Note: You must only use one destination bucket for all inventory files generated in an AWS account.
   • SourceBucketName: The name of the bucket you want to start generating inventory files for

   Optional parameters:

   • SourceBucketPrefix: This parameter limits the inventory generation to a specific prefix in the source bucket

   • DestinationBucketPrefix: If you want to reuse an existing bucket as the destination, this parameter allows the inventory files to be shipped to a specific prefix in that bucket. Ensure that any prefixes do not include trailing slashes (/)

6. On the Review and create step, verify the parameters have been entered correctly, and click Submit.

Finish setting up S3 buckets for Storage Management

After completing the CloudFormation setup, enable buckets for Storage Management from the Datadog UI:

• Navigate to Storage Management → Enable Buckets.
• In step 2, under Enable S3 Inventory to get prefix level monitoring, select I enabled it myself.
• Choose the destination buckets that contain the inventory files for the source buckets you want to monitor and click Confirm.

You can use the Terraform aws_s3_bucket_inventory resource to enable S3 Inventory.

The following example shows how to enable daily inventory on an S3 bucket for Datadog monitoring. To use this example:

• Replace <MY_MONITORED_BUCKET> with the name of the bucket to be monitored.
• Replace <MY_INVENTORY_DESTINATION> with the name of the bucket that receives your inventory files.
• Replace <DESTINATION_ACCOUNT_ID> with the AWS account ID that owns the destination bucket.

resource "aws_s3_bucket" "monitored" {
  bucket = "<MY_MONITORED_BUCKET>"
}

resource "aws_s3_bucket" "inventory_destination" {
  bucket = "<MY_INVENTORY_DESTINATION>"
}

resource "aws_s3_bucket_inventory" "daily_inventory" {
  bucket = aws_s3_bucket.monitored.id
  name   = "datadog-daily-inventory"


  included_object_versions = "All"
  schedule {
    frequency = "Daily"
  }
  destination {
    bucket {
      account_id = "<DESTINATION_ACCOUNT_ID>"
      bucket_arn = aws_s3_bucket.inventory_destination.arn
      format     = "CSV"
      prefix     = "datadog-inventory/"
    }
  }
  optional_fields = [
    "Size",
    "StorageClass",
    "LastModifiedDate",
    "ETag",
    "IsMultipartUploaded",
    "ReplicationStatus",
    "EncryptionStatus",
    "ObjectLockRetainUntilDate",
    "ObjectLockMode",
    "ObjectLockLegalHoldStatus",
    "IntelligentTieringAccessTier",
    "BucketKeyStatus",
    "ChecksumAlgorithm",
    "ObjectAccessControlList",
    "ObjectOwner"
  ]
}

Notes:

• The destination bucket can be your source bucket, but for security and logical separation, many organizations use a separate bucket.
• The optional_fields section is required for Datadog prefix metrics and cost optimization insights like duplicate objects.

After the inventory configuration is set up and your inventory files begin appearing in the destination bucket, enable buckets for Storage Management from the Datadog UI:

• Navigate to Storage Management → Enable Buckets.
• In Step 2, under Enable S3 Inventory to get prefix level monitoring, select I enabled it myself.
• Choose the destination buckets that contain the inventory files for the source buckets you want to monitor and click Confirm.

Note: If you don’t see a list of your existing destination buckets under I enabled it myself, you need to provide required S3 permissions as part of AWS Resource Collection.

1. Create an S3 bucket to store your inventory files. This bucket acts as the central location for inventory reports. Note: You must only use one destination bucket for all inventory files generated in an AWS account.
2. Create a prefix within the destination bucket (optional).

1. Ensure the Datadog AWS integration role has s3:GetObject and s3:ListBucket permissions on the destination bucket. These permissions allow Datadog to read the generated inventory files.

2. Follow the steps in the Amazon S3 user guide to add a bucket policy to your destination bucket allowing write access (s3:PutObject) from your source buckets.

For each bucket you want to monitor:

1. Go to the Amazon S3 buckets page in the AWS console, and select the bucket.
2. Navigate to the bucket’s Management tab.
3. Click Create inventory configuration.
4. Configure the following settings:

   • Set a configuration name

   • (Optional) Specify a source bucket prefix

   • Object versions: Datadog recommends selecting Include all versions (required to see non-current version metrics)

   • Destination: Select the common destination bucket for inventory files in your AWS account. For example, if the bucket is named destination-bucket, enter s3://your-destination-bucket

     Note: If you want to use a prefix on the destination bucket, add this as well

   • Frequency: Datadog recommends choosing Daily. This setting determines how often your prefix-level metrics are updated in Datadog

   • Output format: CSV

   • Status: Enabled

   • Server-side encryption: Don’t specify an encryption key

   • Select all the available Additional metadata fields. Minimally, the following fields are required:

Note: Review Amazon S3 pricing for costs related to inventory generation.