The fastest way to configure Storage Monitoring is through the Add Buckets page in Datadog, where you can set up multiple S3 buckets at the same time.

1. Go to Datadog > Infrastructure > Storage Monitoring.
2. Click Add Buckets.

3. Enable Amazon S3 Integration and Resource collection for all the AWS accounts you want to monitor.

   1. Allow Datadog to read from your destination buckets. Add the following permissions to the Datadog IAM integration role for the account that owns the destination buckets:

      • s3:GetObject
      • s3:ListBucket

      Scope these read-only permissions to only the destination buckets containing your S3 inventory files.

   2. Allow source buckets to write to destination buckets. The destination buckets must include a policy that allows the source buckets to write inventory data. See Creating a destination bucket policy in the AWS documentation for details.

   Example source-bucket policy:

     {
       "Version": "2012-10-17",
       "Statement": [
         {
           "Sid": "AllowListInventoryBucket",
           "Effect": "Allow",
           "Action": "s3:ListBucket",
           "Resource": "arn:aws:s3:::storage-monitoring-s3-inventory-destination"
         },
         {
           "Sid": "AllowGetInventoryObjects",
           "Effect": "Allow",
           "Action": "s3:GetObject",
           "Resource": "arn:aws:s3:::storage-monitoring-s3-inventory-destination/*"
         }
       ]
     }
   

4. Select the S3 buckets you want to monitor with Storage Monitoring. You can select buckets from multiple AWS accounts at once.

5. Assign a destination bucket per region to store S3 inventory reports from the source buckets. This can be an existing AWS bucket or a new one.

   • Source bucket: The S3 bucket you want to monitor with Storage Monitoring
   • Destination bucket: Used to store inventory reports (one per AWS region, can be reused)

6. Complete the configuration. The inventory generation process will start within AWS within 24 hours of the first report.

7. Return to Infrastructure > Storage Monitoring to see your bucket(s) appear.

You can also set up Storage Monitoring using the provided CloudFormation templates. This process involves two steps:

Step 1: Configure inventory generation

This template configures your existing S3 bucket to generate inventory reports, which Datadog uses to generate detailed metrics about your bucket prefixes.

1. Download the source-bucket-inventory-cfn.yaml template.

2. In AWS CloudFormation, click Create stack in the top right corner and select With existing resources (import resources).

3. In the Specify template step, select Upload a template file.

4. Click Choose file and select the source-bucket-inventory-cfn.yaml file, then click Next.

5. Enter the bucket name you want AWS to start generating inventories for, and click Next.

6. Fill in the required parameters:

   • DestinationBucketName: The bucket for storing inventory files. Note: You must only use one destination bucket for all inventory files generated in an AWS account.
   • SourceBucketName: The bucket you want to monitor and start generating inventory files for

   Optional parameters:

   • SourceBucketPrefix: (Optional) Limit monitoring to a specific path in the source bucket
   • DestinationBucketPrefix: Specific path within the destination bucket. Ensure this path doesn’t include trailing slashes (/)

7. Click Next.
8. Wait for AWS to locate your source bucket, and click Import resources in the bottom right corner.

Note: This CloudFormation template can be rolled back, but rolling back doesn’t delete the created resources. This is to ensure the existing bucket doesn’t get deleted. You can manually delete the inventory configurations by going on the Management tab in the bucket view.

Note: Review Amazon S3 pricing for costs related to inventory generation.

Step 2: Configure required permissions

This template creates two IAM policies:

• A policy to allow Datadog to read inventory files from the destination bucket
• A policy to allow your source bucket to write inventory files to the destination bucket

1. Download the cloud-inventory-policies-cfn.yaml template.

2. In AWS CloudFormation, click Create stack in the top right corner and select With new resources (standard).

3. In the Specify template step, select Upload a template file.

4. Click Choose file and select the cloud-inventory-policies-cfn.yaml file, then click Next.

5. Fill in the required parameters:

   • DatadogIntegrationRole: Your Datadog AWS integration role name
   • DestinationBucketName: The name of the bucket that receives your inventory files. Note: You must only use one destination bucket for all inventory files generated in an AWS account.
   • SourceBucketName: The name of the bucket you want to start generating inventory files for

   Optional parameters:

   • SourceBucketPrefix: This parameter limits the inventory generation to a specific prefix in the source bucket
   • DestinationBucketPrefix: If you want to reuse an existing bucket as the destination, this parameter allows the inventory files to be shipped to a specific prefix in that bucket. Ensure that any prefixes do not include trailing slashes (/)

6. On the Review and create step, verify the parameters have been entered correctly, and click Submit.

Post-setup steps

After completing the CloudFormation setup, fill out the post-setup form with the following required information:

1. Name of the destination bucket holding the inventory files.
2. Prefix where the files are stored in the destination bucket (if any).
3. Name of the source bucket you want to monitor (the bucket producing inventory files).
4. AWS region of the destination bucket holding the inventory files.
5. AWS account ID containing the buckets.
6. Datadog org ID.

You can use the Terraform aws_s3_bucket_inventory resource to set up Storage Monitoring.

The following example shows how to enable daily inventory on an S3 bucket for Datadog monitoring. To use this example:

• Replace <MY_MONITORED_BUCKET> with the name of the bucket to be monitored.
• Replace <MY_INVENTORY_DESTINATION> with the name of the bucket that receives your inventory files.
• Replace <DESTINATION_ACCOUNT_ID> with the AWS account ID that owns the destination bucket.

resource "aws_s3_bucket" "monitored" {
  bucket = "<MY_MONITORED_BUCKET>"
}

resource "aws_s3_bucket" "inventory_destination" {
  bucket = "<MY_INVENTORY_DESTINATION>"
}

resource "aws_s3_bucket_inventory" "daily_inventory" {
  bucket = aws_s3_bucket.monitored.id
  name   = "datadog-daily-inventory"


  included_object_versions = "All"
  schedule {
    frequency = "Daily"
  }
  destination {
    bucket {
      account_id = "<DESTINATION_ACCOUNT_ID>"
      bucket_arn = aws_s3_bucket.inventory_destination.arn
      format     = "CSV"
      prefix     = "datadog-inventory/"
    }
  }
  optional_fields = [
    "Size",
    "StorageClass",
    "LastModifiedDate"
  ]
}

Notes:

• The destination bucket can be your source bucket, but for security and logical separation, many organizations use a separate bucket.
• The optional_fields section is recommended for Datadog prefix metrics.

Post-setup steps

Once the inventory configuration is set up and your inventory files begin appearing in the destination bucket, fill out this form to provide your S3 configuration details. This allows Datadog to begin generating prefix metrics for your storage.

Use modules for complex setups

If you need to manage multiple buckets, complex inventory policies, encryption, or cross-account setups, you can use the terraform-aws-s3-bucket module.

Troubleshooting

• S3 Inventory files are delivered daily, and may take up to 24 hours to appear after setup.
• Ensure IAM permissions allow S3 to write inventory files to your destination bucket.
• If cross-account access is needed, confirm that the inventory destination prefix (datadog-inventory/ in the example) is correct and accessible to Datadog.

To manually set up the required Amazon S3 Inventory and related configuration, follow these steps:

Step 1: Create a destination bucket

1. Create an S3 bucket to store your inventory files. This bucket acts as the central location for inventory reports. Note: You must only use one destination bucket for all inventory files generated in an AWS account.
2. Create a prefix within the destination bucket (optional).

Step 2: Configure the bucket and integration role policies

1. Follow the steps in the Amazon S3 user guide to add a bucket policy to your destination bucket allowing write access (s3:PutObject) from your source buckets.

2. Ensure the Datadog AWS integration role has s3:GetObject and s3:ListObjects permissions on the destination bucket. These permissions allow Datadog to read the generated inventory files.

Step 3: Configure inventory generation

For each bucket you want to monitor:

1. Go to the Amazon S3 buckets page in the AWS console, and select the bucket.
2. Navigate to the bucket’s Management tab.
3. Click Create inventory configuration.
4. Configure the following settings:

• Set a configuration name
• (Optional) Specify a source bucket prefix
• Object versions: Datadog recommends selecting Current Versions Only
• Destination: Select the common destination bucket for inventory files in your AWS account. For example, if the bucket is named destination-bucket, enter s3://your-destination-bucket Note: If you want to use a prefix on the destination bucket, add this as well
• Frequency: Datadog recommends choosing Daily. This setting determines how often your prefix-level metrics are updated in Datadog
• Output format: CSV
• Status: Enabled
• Server-side encryption: Don’t specify an encryption key
• Select the following Additional metadata fields:
  1. Size
  2. Last Modified
  3. Storage Class

Note: Review Amazon S3 pricing for costs related to inventory generation.

Post-setup steps

After completing the above steps, fill out the post-setup form with the following required information:

1. Name of the destination bucket where inventories are stored.
2. Prefix where the files are stored in the destination bucket (optional).
3. Region of the destination bucket.
4. AWS account ID containing the bucket.
5. Datadog role name that has the permissions to read objects in destination bucket.
6. Datadog org ID.

If you have already configured S3 inventory for the buckets you want to monitor, choose one of the following options:

• Fill out this form to share your configurations with Datadog
• Reach out to us to use an API for setting up multiple buckets

Use the Google Cloud CLI to run the following command:

gcloud storage insights inventory-reports create <SOURCE_BUCKET_URL> \
  --no-csv-header \
  --display-name=datadog-storage-monitoring \
  --destination=gs://<DESTINATION_BUCKET>/<SOURCE_BUCKET>/{{date}}> \
  --metadata-fields=project,bucket,name,size,updated,storageClass \
  --schedule-starts=<YYYY-MM-DD> \
  --schedule-repeats=<DAILY|WEEKLY> \
  --schedule-repeats-until=<YYYY-MM-DD>

Copy the following Terraform template, substitute the necessary arguments, and apply it in the Google Cloud project that contains your bucket.

locals {
  source_bucket      = "" # The name of the bucket you want to monitor
  destination_bucket = "" # The bucket where inventory reports are written
  frequency          = "" # Possible values: Daily, Weekly (report generation frequency)
  location           = "" # The location of your source and destination buckets
}

data "google_project" "project" {
}

resource "google_storage_insights_report_config" "config" {
  display_name = "datadog-storage-monitoring"
  location     = local.location
  frequency_options {
    frequency = local.frequency
    start_date {
      day   = "" # Fill in the day
      month = "" # Fill in the month
      year  = "" # Fill in the year
    }
    end_date {
      day   = "" # Fill in the day
      month = "" # Fill in the month
      year  = "" # Fill in the year
    }
  }
  csv_options {
    record_separator = "\n"
    delimiter        = ","
    header_required  = false
  }
  object_metadata_report_options {
    metadata_fields = ["bucket", "name", "project", "size", "updated", "storageClass"]
    storage_filters {
      bucket = local.source_bucket
    }
    storage_destination_options {
      bucket           = google_storage_bucket.report_bucket.name
      destination_path = "${local.source_bucket}/{{date}}"
    }
  }

  depends_on = [
    google_storage_bucket_iam_member.admin
  ]
}

resource "google_storage_bucket" "report_bucket" {
  name                        = local.destination_bucket
  location                    = local.location
  force_destroy               = true
  uniform_bucket_level_access = true
}

resource "google_storage_bucket_iam_member" "admin" {
  bucket = google_storage_bucket.report_bucket.name
  role   = "roles/storage.admin"
  member = "serviceAccount:service-${data.google_project.project.number}@gcp-sa-storageinsights.iam.gserviceaccount.com"
}

You can allow Datadog to handle the inventory report configuration by providing the proper permissions to your service account:

1. Navigate to IAM & Admin -> Service accounts
2. Find your Datadog service account and add the roles/storageinsights.Admin role
3. Navigate to the source bucket you want to monitor and grant these permissions:
   • roles/storage.insightsCollectorService
   • roles/storage.ObjectViewer
4. Navigate to the destination bucket and grant these permissions:
   • roles/storage.objectCreator
   • roles/storage.insightsCollectorService

Alternatively, you can create a custom role specifically for Datadog with these required permissions:

storage.buckets.get
storage.objects.list
storage.buckets.getObjectInsights
storage.buckets.get
storage.objects.create
storageinsights.reportConfigs.get
storageinsights.reportConfigs.create
storageinsights.reportConfigs.list
storageinsights.reportConfigs.update
storage.objects.get
storageinsights.reportDetails.get
storageinsights.reportDetails.list

After granting the necessary permissions, Datadog can create the inventory report configuration with your setup details.

Step 5: Add the Storage Object Viewer role to your Datadog service account

Grant Datadog permission to access and extract the generated inventory reports from Google. This permission should be on the destination bucket where the inventory reports are stored.

1. Select the destination bucket for your inventory reports
2. In the bucket details page, click the Permissions tab
3. Under Permissions, click Grant Access to add a new principal
4. Principal: Enter the Datadog Service Account email
5. Role: Select Storage Object Viewer (roles/storage.objectViewer)

Enable inventories for the selected storage accounts in each subscription by running the following script in your Azure Cloud Shell:

curl https://datadogstoragemonitoring.blob.core.windows.net/scripts/install.sh \
  | bash -s -- <CLIENT_ID> <SUBSCRIPTION_ID> <COMMA_SEPARATED_STORAGE_ACCOUNT_NAMES>

Before running the script, set your shell environment to Bash and replace the various placeholder inputs with the correct values:

• <CLIENT_ID>: The client ID of an App Registration already set up using the Datadog Azure integration
• <SUBSCRIPTION_ID>: The subscription ID of the Azure subscription containing the storage accounts
• <COMMA_SEPARATED_STORAGE_ACCOUNT_NAMES>: A comma-separated list of the storage accounts you want to monitor (for example, storageaccount1,storageaccount2)

For Each Storage Account you wish to monitor, follow all of the steps here:

Create a blob inventory policy

1. In the Azure portal, navigate to your Storage Account.
2. Go to Data management > Blob inventory.
3. Click Add.
4. Configure the policy:
   • Name: datadog-storage-monitoring
   • Destination container:
     • Click Create new, and enter the name datadog-storage-monitoring.
   • Object type to inventory: Blob
   • Schedule: Daily
   • Blob types: Select Block blobs, Append blobs, and Page blobs.
   • Subtypes: Select Include blob versions
   • Schema fields: Select All, or ensure that at least the following are selected:
     • Name
     • Access tier
     • Last modified
     • Content length
     • Server encrypted
     • Current version status
     • Version ID
   • Exclude prefix: datadog-storage-monitoring
5. Click Add.

Add the role assignment

1. In the Azure portal, navigate to your Storage Account.
2. Go to Data storage > Containers.
3. Click on the datadog-storage-monitoring container.
4. Click on Access control (IAM) in the left-hand menu.
5. Click Add > Add role assignment.
6. Fill out the role assignment:
   • Role: Select Storage Blob Data Reader. Click Next.
   • Assign access to: User, group, or service principal.
   • Members: Click + Select members and search for your App Registration by its name and select it.
   • Note: This should be an App Registration set up in the Datadog Azure integration. Keep in mind the Client ID for later.
7. Click Review + assign.