Storage Management for Amazon S3

이 페이지는 아직 한국어로 제공되지 않습니다. 번역 작업 중입니다.
현재 번역 프로젝트에 대한 질문이나 피드백이 있으신 경우 언제든지 연락주시기 바랍니다.

Setup

    The fastest way to configure Storage Management is through the Enable Buckets page, where you can enable S3 inventory and configure monitoring for multiple buckets at once.

    As an alternative, you can set up S3 inventory manually or with Terraform and enable Storage Management using your existing setup. For details, see Existing S3 Inventory.

    Select buckets for enabling Storage Monitoring

    Ensure all S3-related permissions are granted for Resource Collection.

    - Source bucket: The S3 bucket you want to monitor with Storage Management
    - Destination bucket: Used to store inventory reports (one per AWS region, can be reused cross-account)

    1. Add all S3 permissions required for Storage Management to your Datadog IAM policy. Storage Management relies on permissions in AWS Integration IAM policy and Resource Collection IAM policy to bring together bucket details. In addition, the following 3 permissions allows Datadog to enable S3 inventory on your source buckets and read the generated reports from the destination buckets.

      • s3:PutInventoryConfiguration
      • s3:GetObject (scoped to the destination bucket(s))
      • s3:ListBucket (scoped to the destination bucket(s))

      Example IAM policy with all required permissions for Storage Management:

            {
        "Version": "2012-10-17",
        "Statement": [

          {
            "Sid": "DatadogS3BucketInfo",
            "Effect": "Allow",
            "Action": [
              "s3:ListAllMyBuckets",
              "s3:GetAccelerateConfiguration",
              "s3:GetAnalyticsConfiguration",
              "s3:GetBucket*",
              "s3:GetEncryptionConfiguration",
              "s3:GetInventoryConfiguration",
              "s3:GetLifecycleConfiguration",
              "s3:GetMetricsConfiguration",
              "s3:GetReplicationConfiguration",
              "s3:ListBucket",
              "s3:GetBucketLocation",
              "s3:GetBucketLogging",
              "s3:GetBucketTagging",
              "s3:PutInventoryConfiguration"
            ],
            "Resource": "*"
          },
          {
            "Sid": "DatadogReadInventoryFromDestinationBucket",
            "Effect": "Allow",
            "Action": [
              "s3:ListBucket",
              "s3:GetObject"
            ],
            "Resource": [
              "arn:aws:s3:::storage-management-inventory-destination",
              "arn:aws:s3:::storage-management-inventory-destination/*"
            ]
          }
        ]
      }

    2. On the Enable it for me tab, select the regions or accounts you want to enable and assign a destination bucket per region or per account to store S3 Inventory reports. You can either use an existing bucket or create one in AWS.

      • If you select a destination bucket containing inventories from multiple source buckets, all of those source buckets become enabled for monitoring.
      • The destination buckets must allow the source buckets to write inventory data. See Creating a destination bucket policy in the AWS documentation for details.
      Select buckets for enabling Storage Monitoring

    3. Complete the inventory configuration in Datadog. The first inventory report may take up to 24 hours to generate.

    4. Navigate to S3 > Destination bucket > Permissions > Bucket policy. Add or update the bucket policy on the destination bucket to allow the S3 service (s3.amazonaws.com) to write inventory objects from the source bucket(s).

      Use the following example bucket policy to allow S3 to write inventory files to your destination bucket. Replace <DESTINATION_BUCKET>, <DESTINATION_PREFIX> (optional), and <ACCOUNT_ID> with your actual bucket name, bucket prefix, and AWS account ID.

      {
  "Sid": "AllowS3InventoryWriteFromAccountBuckets",
  "Effect": "Allow",
  "Principal": { "Service": "s3.amazonaws.com" },
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::<DESTINATION_BUCKET>/<DESTINATION_PREFIX>/*",
  "Condition": {
    "ArnLike": {
      "aws:SourceArn": "arn:aws:s3:::*"
    },
    "StringEquals": {
      "aws:SourceAccount": "<ACCOUNT_ID>",
      "s3:x-amz-acl": "bucket-owner-full-control"
    }
  }
}

    To get prefix-level access metrics including request counts, server-side latency, and cold data identification for cost optimization, follow these additional steps:

    1. Set up the Datadog Lambda Forwarder (if not already configured):

    2. Configure S3 Access Logs for each source bucket:

      • Go to your S3 bucket properties in the AWS Console
      • Navigate to Server access logging
      • Enable logging and specify your destination bucket (for simplicity, you can use the destination bucket for your inventory files)
      • Set the target prefix to access-logs/ to organize log files separately from inventory data

    3. Set up the Lambda trigger:

      Option A: Automatic (Recommended)

      • In the Datadog AWS integration page, navigate to the Log Collection tab
      • Enable automatic log collection for S3 by checking the S3 Access Logs checkbox
      • Datadog automatically configures triggers on your Forwarder Lambda function for S3 access logs

      Option B: Manual

      • In the AWS console, go to your Datadog Forwarder Lambda function
      • Click Add trigger and select S3
      • Select the bucket containing your access logs
      • Set the event type to All object create events
      • Set the prefix to access-logs/ (matching your access log prefix)
      After sending, S3 Access Logs are also available in Datadog Log Management.

    The inventory generation process starts in AWS within 24 hours of the first report. Data from your buckets is visible after this period.

    You can also set up Storage Management using the provided CloudFormation templates. This process involves two steps:

    This template configures your existing S3 bucket to generate inventory reports, which Datadog uses to generate detailed metrics about your bucket prefixes.

    1. Download the source-bucket-inventory-cfn.yaml template.

    2. In AWS CloudFormation, click Create stack in the top right corner and select With existing resources (import resources).

    3. In the Specify template step, select Upload a template file.

    4. Click Choose file and select the source-bucket-inventory-cfn.yaml file, then click Next.

    5. Enter the bucket name you want AWS to start generating inventories for, and click Next.

      Identify S3 resources to start generating inventory

    6. Fill in the required parameters:

      • DestinationBucketName: The bucket for storing inventory files. Note: You must only use one destination bucket for all inventory files generated in an AWS account.
      • SourceBucketName: The bucket you want to monitor and start generating inventory files for

      Optional parameters:

      • SourceBucketPrefix: (Optional) Limit monitoring to a specific path in the source bucket
      • DestinationBucketPrefix: Specific path within the destination bucket. Ensure this path doesn’t include trailing slashes (/)
    Specify stack details
    1. Click Next.
    2. Wait for AWS to locate your source bucket, and click Import resources in the bottom right corner.

    Notes:

    • This CloudFormation template can be rolled back, but rolling back doesn’t delete the created resources. This is to ensure the existing bucket doesn’t get deleted. You can manually delete the inventory configurations by going on the Management tab in the bucket view.
    • Review Amazon S3 pricing for costs related to inventory generation.

    This template creates two IAM policies:

    • A policy to allow Datadog to read inventory files from the destination bucket
    • A policy to allow your source bucket to write inventory files to the destination bucket

    1. Download the cloud-inventory-policies-cfn.yaml template.

    2. In AWS CloudFormation, click Create stack in the top right corner and select With new resources (standard).

    3. In the Specify template step, select Upload a template file.

    4. Click Choose file and select the cloud-inventory-policies-cfn.yaml file, then click Next.

    5. Fill in the required parameters:

      • DatadogIntegrationRole: Your Datadog AWS integration role name
      • DestinationBucketName: The name of the bucket that receives your inventory files. Note: You must only use one destination bucket for all inventory files generated in an AWS account.
      • SourceBucketName: The name of the bucket you want to start generating inventory files for

      Optional parameters:

      • SourceBucketPrefix: This parameter limits the inventory generation to a specific prefix in the source bucket

      • DestinationBucketPrefix: If you want to reuse an existing bucket as the destination, this parameter allows the inventory files to be shipped to a specific prefix in that bucket. Ensure that any prefixes do not include trailing slashes (/)

        Stack parameters for bucket policy

    6. On the Review and create step, verify the parameters have been entered correctly, and click Submit.

    Finish setting up S3 buckets for Storage Management

    After completing the CloudFormation setup, enable buckets for Storage Management from the Datadog UI:

    • Navigate to Storage Management → Enable Buckets.
    • In step 2, under Enable S3 Inventory to get prefix level monitoring, select Use existing inventories.
    • Choose the destination buckets that contain the inventory files for the source buckets you want to monitor and click Confirm.
    Select destination buckets for enabling Storage Monitoring

    Use the official Datadog Storage Management Terraform module to configure S3 Inventory and forward S3 Access logs for Storage Management. This module configures all required permissions on the AWS Integration IAM role, adds a bucket policy to allow Datadog to read inventory files from the destination bucket path, and enables S3 Access Log collection if you already have a Forwarder set up.

    To use this example:

    • Replace <AWS_REGION> with your AWS region.
    • Replace <MODULE_NAME> with a unique name for this module instance.
    • Replace <DATADOG_AWS_INTEGRATION_ROLE_NAME> with the name of your Datadog AWS Integration IAM role.
    • Replace <SOURCE_BUCKET_1>, <SOURCE_BUCKET_2>, etc. with the names of the buckets to be monitored.
    • Replace <DESTINATION_BUCKET_NAME> with the name of the bucket that receives your inventory files.
    • Replace <DATADOG_FORWARDER_FUNCTION_NAME> with the name of your Datadog Forwarder Lambda function (only required if enabling access logs).

    For more options, see the module documentation.

    provider "aws" {
  region = "<AWS_REGION>"
}

provider "datadog" {
  # Configure via environment variables:
  #   DD_API_KEY, DD_APP_KEY, DD_SITE
}

module "datadog_storage_management" {
  source = "DataDog/storage-management/aws"

  name                              = "<MODULE_NAME>"
  datadog_aws_integration_role_name = "<DATADOG_AWS_INTEGRATION_ROLE_NAME>"
  source_bucket_names               = ["<SOURCE_BUCKET_1>", "<SOURCE_BUCKET_2>"]
  destination_bucket_name           = "<DESTINATION_BUCKET_NAME>"

  # Bucket policy: "none", "create", or "merge" (default)
  destination_bucket_policy_management = "merge"

  # Optional: Enable S3 Access Logs for prefix-level request and latency metrics
  enable_access_logging           = true
  datadog_forwarder_function_name = "<DATADOG_FORWARDER_FUNCTION_NAME>"
}

    After enabling S3 inventory, it may take up to 24 hours for the first inventory reports to be generated. To check that inventories are being created, go to the AWS Console, navigate to your destination bucket, and check that inventory files appear in the destination prefix you specified during setup.

    Once you’ve confirmed inventory files are present, verify Storage Management is enabled on your buckets by navigating to Storage Management > Enable Buckets > Use existing inventories and confirming your destination bucket is listed and enabled.

    To manually set up the required Amazon S3 Inventory and related configuration, follow these steps:

    1. Create an S3 bucket to store your inventory files. This bucket acts as the central location for inventory reports. Note: You must only use one destination bucket for all inventory files generated in an AWS account.
    2. Create a prefix within the destination bucket (optional).

    1. Ensure the Datadog AWS integration role has s3:GetObject and s3:ListBucket permissions on the destination bucket. These permissions allow Datadog to read the generated inventory files.

    2. Ensure the destination bucket policy allows S3 to write inventory files to your destination bucket.

      Example bucket policy:

      {
  "Sid": "AllowS3InventoryWriteFromAccountBuckets",
  "Effect": "Allow",
  "Principal": { "Service": "s3.amazonaws.com" },
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::<DESTINATION_BUCKET>/<DESTINATION_PREFIX>/*",
  "Condition": {
    "ArnLike": {
      "aws:SourceArn": "arn:aws:s3:::*"
    },
    "StringEquals": {
      "aws:SourceAccount": "<ACCOUNT_ID>",
      "s3:x-amz-acl": "bucket-owner-full-control"
    }
  }
}

    3. Follow the steps in the Amazon S3 User Guide to add a bucket policy to your destination bucket that allows Amazon S3 to write inventory objects (s3:PutObject) from your source bucket or buckets.

    For each bucket you want to monitor:

    1. Go to the Amazon S3 buckets page in the AWS console, and select the bucket.
    2. Navigate to the bucket’s Management tab.
    3. Click Create inventory configuration.
    4. Configure the following settings:

      • Set a configuration name

      • (Optional) Specify a source bucket prefix

      • Object versions: Datadog recommends selecting Include all versions (required to see non-current version metrics)

        Select destination buckets for enabling Storage Monitoring

      • Destination: Select the common destination bucket for inventory files in your AWS account. For example, if the bucket is named destination-bucket, enter s3://your-destination-bucket

        Note: If you want to use a prefix on the destination bucket, add this as well

      • Frequency: Datadog recommends choosing Daily. This setting determines how often your prefix-level metrics are updated in Datadog

      • Output format: CSV

      • Status: Enabled

      • Server-side encryption: Don’t specify an encryption key

      • Select all the available Additional metadata fields. Minimally, the following fields are required:

        Additional metadata fields. Size, Last modified, Multipart upload, Replication status, Encryption, Object ACL, Storage class, Intelligent-Tiering: Access tier, ETag, and Checksum function are all selected. Bucket key status, Object owner, and All Object Lock configurations are unselected.

    Note: Review Amazon S3 pricing for costs related to inventory generation.

    Post-setup steps

    After the inventory configuration is set up and your inventory files begin appearing in the destination bucket, enable buckets for Storage Management from the Datadog UI:

    • Navigate to Storage Management → Enable Buckets.
    • In Step 2, under Enable S3 Inventory to get prefix level monitoring, select Use existing inventories.
    • Choose the destination buckets that contain the inventory files for the source buckets you want to monitor and click Confirm.

    Note: If you don’t see a list of your existing destination buckets under Use existing inventories, you need to provide required S3 permissions as part of AWS Resource Collection.

    Select destination buckets for enabling Storage Monitoring

    If you have already configured S3 Inventory for the buckets you want to monitor, enable buckets for Storage Monitoring from the Datadog UI.

    Note: Storage Management only supports CSV format for inventories.

    1. Navigate to Storage Management > Enable Buckets.
    2. In Step 2, under Enable S3 Inventory to get prefix level monitoring, select Use existing inventories.
    3. Choose the destination buckets that contain the inventory files for the source buckets you want to monitor and click Confirm.

    Note: If you don’t see a list of your existing destination buckets under Use existing inventories, you need to provide required S3 permissions as part of AWS Resource Collection.

    Select destination buckets for enabling Storage Monitoring

    Validation

    To verify your setup:

    1. Wait for the first inventory report to generate (up to 24 hours for daily inventories).
    2. Navigate to Infrastructure > Storage Management to see if the bucket(s) you configured are showing in the explorer list when “Monitored buckets” is selected.
    Validate bucket is enabled for monitoring

    Best practices

    Follow these best practices to optimize Storage Management setup:

    • Configure lifecycle policies for inventory destination buckets: S3 Inventory reports are generated daily and stored in your destination bucket. To prevent old inventory files from accumulating and incurring storage costs, add a lifecycle policy to automatically delete inventory reports older than three days.

    • Configure lifecycle policies for S3 access logs: If you have enabled S3 access logs for prefix-level request metrics, the raw log files accumulate in your destination bucket. After these logs are forwarded to Datadog, the raw files are no longer needed for Storage Management purposes. To automatically delete access log files after forwarding to Datadog, add a lifecycle rule.

      Note: Before enabling automatic deletion, verify that there are no compliance or audit requirements in your organization that mandate retaining raw S3 access logs for a specific period.

    Troubleshooting

    If you don’t see data for buckets you set up for Storage Management:

    • Check Storage Management > Amazon S3 > Enable Buckets for any errors:

      • IAM Role(s) Lacking Permissions: Ensure s3:GetObject and s3:ListBucket permissions for the destination buckets are set on the Datadog AWS Integration Role. Verify all S3-related permissions are granted as part of Resource Collection.

      • Problem reading inventory: Ensure the destination bucket policy allows S3 to write inventory files. See Example Bucket Policy. Note: This error may also appear if the first inventory file has not yet been generated (takes up to 24 hours) or if the buckets you are enabling are empty.

    • If you’re still encountering issues, contact Datadog.

    Visualize granular S3 usage with inventory metrics

    Metric NameNotable TagsDescription
    aws.s3.inventory.total_prefix_sizebucketname, prefix, region, storagetype, extension, delete_marker, is_latestTotal amount of data, in bytes, stored in a prefix.
    aws.s3.inventory.average_prefix_sizebucketname, prefix, regionAverage object size, in bytes, for objects in a prefix.
    aws.s3.inventory.prefix_object_countbucketname, prefix, region, storagetype, extension, delete_marker, is_latestThe total number of objects stored in a prefix.
    aws.s3.inventory.prefix_object_count.levelsbucketname, prefixN*, region, storagetype, extension, delete_markerObject counts aggregated to hierarchical prefix levels, used for treemap visualizations.
    aws.s3.inventory.total_prefix_size.levelsbucketname, prefixN*, region, storagetype, extension, delete_markerPrefix size aggregated to hierarchical prefix levels, used for treemap visualizations.
    aws.s3.inventory.prefix_age_daysbucketname, prefix, regionAge, in days, of the oldest object in the bucket or prefix.
    aws.s3.inventory.access_logs.total_requests_by_methodbucketname, prefix, region, methodTotal number of requests for objects in a prefix, optionally split by request method (for example, GET or PUT). Requires S3 Access Logs in Datadog.
    aws.s3.inventory.access_logs.request_latency_by_methodbucketname, prefix, region, methodServer response time for requests in a prefix, optionally split by request method. Requires S3 Access Logs in Datadog.

    *prefixN refers to prefix levels such as prefix0, prefix1, prefix2, and so on.

    Note: For the most accurate monitoring and visualization, ensure that S3 inventory reports use the CSV format and include all object versions if you wish to view non-current object recommendations or metrics.

    Act on optimizations with Storage Management Recommendations

    Storage Management generated recommendations are available in Cloud Cost Recommendations. Storage Management generates recommendations to reduce your S3 costs with granular details, like which prefixes you can optimize, by combining observability data with access logs. In order to see these recommendations, you must enable Storage Management for S3 buckets and enable Cloud Cost Management to see the Recommendations.

    Recommendations are run on a daily basis and are automatically refreshed in your account as soon as the recommendations are released.

    Prerequisites

    Seeing recommendations requires the following prerequisites:

    1. Ensure you’ve configured S3 buckets for Storage Management by following the steps above on this page.
    2. If you would like to see recommendations for moving infrequently accessed data to cheaper tiers by prefix, enable and forward S3 Access Logs to Datadog (Datadog Log Management fees apply).
    3. If you would like to see recommendations for identifying non-current versions in prefixes, ensure you’re including “All versions” as part of the S3 inventory configuration.

    Available recommendations

    • Transition S3 bucket prefix to Infrequent Access

    • Expire old non-current version objects in S3 bucket prefix

    • Consolidate small files in prefix to minimize per-object storage costs

      Storage Management Recommendations in CCM