Overview

To set up Cloud Cost Management in Datadog, you should:

1. Have an AWS account with billing access
2. Have the AWS integration installed in Datadog
3. Follow the steps below to create a Cost and Usage report

Setup

Prerequisite: generate a Cost and Usage Report

Create a Cost and Usage Report in AWS under the Legacy Pages section. At this time, there is no support for creating Cost and Usage Report data exports.

Select the following content options:

• Include resource IDs
• Split cost allocation data (Enables ECS Cost Allocation. You must also opt in to AWS Split Cost Allocation in Cost Explorer preferences).
• “Refresh automatically”

Select the following Delivery options:

• Time granularity: Hourly
• Report versioning: Create new report version
• Compression type: GZIP or Parquet
• Format: text/csv or Parquet

Configure the AWS integration

Navigate to Setup & Configuration and select an AWS account from the dropdown menu to pull costs from.

Note: Datadog recommends sending a Cost and Usage Report from an AWS management account for cost visibility into related member accounts. If you send a Cost and Usage report from an AWS member account, ensure that you have selected the following options in your management account’s preferences:

• Linked Account Access
• Linked Account Refunds and Credits
• Linked Account Discounts

This ensures complete cost accuracy by allowing periodic cost calculations against the AWS Cost Explorer.

Locate the Cost and Usage Report

If you have navigated away from the report that you created in the setup prerequisites section, follow AWS documentation to find and view your Cost and Usage Reports details.

To enable Datadog to locate the Cost and Usage Report, complete the fields with their corresponding details:

• Region: This is the region your bucket is located. For example, us-east-1.
• Bucket Name: This is the name of the s3 bucket that the CUR is saved to.
• Report Path Prefix: This is the folder name. If viewing Report path prefix from the AWS details page, this is the first section of the path. For example, if Report path prefix is displayed as cur-report-dir/cost-report, you would enter cur-report-dir.
• Report Name: This is the name you entered when you generated the report in the prerequisite section. If viewing the Report path prefix from the AWS details page, this is the second half of the path. For example, if Report path prefix is displayed as cur-report-dir/cost-report, you would enter cost-report.

Note: Datadog only supports CURs generated by AWS. Do not modify or move the files generated by AWS, or attempt to provide access to files generated by a 3rd party.

Configure access to the Cost and Usage Report

Configure AWS to ensure Datadog has permissions to access the CUR and the s3 bucket it is stored in by creating a policy using the following JSON:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "DDCloudCostReadBucket",
          "Effect": "Allow",
          "Action": [
              "s3:ListBucket"
          ],
          "Resource": "arn:aws:s3:::BUCKETNAME"
      },
      {
          "Sid": "DDCloudCostGetBill",
          "Effect": "Allow",
          "Action": [
              "s3:GetObject"
          ],
          "Resource": "arn:aws:s3:::BUCKETNAME/REPORT_PREFIX/REPORT_NAME/*"
      },
      {
          "Sid": "DDCloudCostCheckAccuracy",
          "Effect": "Allow",
          "Action": [
              "ce:Get*"
          ],
          "Resource": "*"
      },
      {
          "Sid": "DDCloudCostListCURs",
          "Action": [
              "cur:DescribeReportDefinitions"
          ],
          "Effect": "Allow",
          "Resource": "*"
      },
      {
          "Sid": "DDCloudCostListOrganizations",
          "Action": [
              "organizations:Describe*",
              "organizations:List*"
          ],
          "Effect": "Allow",
          "Resource": "*"
      }
  ]
}

Tip: Make note of the name you created for this policy for next steps.

Attach the policy to the Datadog integration role

Attach the new S3 policy to the Datadog integration role.

1. Navigate to Roles in the AWS IAM console.
2. Locate the role used by the Datadog integration. By default it is named DatadogIntegrationRole, but the name may vary if your organization has renamed it. Click the role name to open the role summary page.
3. Click Attach policies.
4. Enter the name of the S3 bucket policy created above.
5. Click Attach policy.

Note: Data can take up to 48 to 72 hours after setup to stabilize in Datadog.

Cost types

Visualize your ingested data using out-of-the-box cost types. The cost types differ mainly in how they report on discount rates, savings plans and reservations.

On-demand

On-demand costs are the costs of usage at the public, on-demand rate published by AWS. This excludes all savings plans, reservations, discounts, taxes, and fees.

In most cases, on-demand costs are not a reliable source to estimate actual costs.

Amortized and unblended costs

Amortized cost metrics distribute commitment savings throughout the discount term. This is also called accrual basis. Reservations and savings plans are drawn down from a monthly commitment and applied directly to covered usage, at the time of usage. Any unused remainder appears as a fee.

In contrast, unblended cost metrics show all charges on the date that they are incurred. This is also called cost basis. Reservation and savings plan fees show up on the date they were charged, and are not applied directly to covered usage. After billing data for a month is finalized, unblended metrics match the AWS invoice exactly.

Net costs

Net costs apply private discounts directly to usage. The cost of usage for a specific resource represents the effective cost after all savings are realized.

In contrast, other metrics show private discounts as separate, negative-valued line items with no resource attribution tags. Rather than attributing the discounts directly to usage, those metrics subtract discounts from the total cost.

Net amortized costs provide the most accurate representation for cost allocation, with all savings applied directly to usage. Net cost metrics are available if your AWS account has privately negotiated enterprise discounts. If your account doesn’t have net metrics available, use amortized cost instead.

Container allocation

Container allocation metrics contain all of the same costs as the AWS metrics, but with additional breakdowns and insights for container workloads. See container cost allocation for more details.

Example

The following scenario demonstrates how different cost types behave. Imagine you have:

• An EC2 instance running for one hour with the cost of $3 per compute-hour.
• A savings plan which prices this instance type at $2 per compute-hour.
• A negotiated EDP discount of 10% on top of all other discounts.

Here’s how the instance cost, savings plan hourly commitment, and discount appear in each cost type:

Cost metrics summary

In general:

• aws.cost.net.amortized.shared.resources.allocated provides the most complete cost allocation for specific workloads and teams.
• If you do not have container cost allocation, use aws.cost.net.amortized.
• If you do not have net amortized costs, use aws.cost.amortized.shared.resources.allocated or aws.cost.amortized.

Tag enrichment

Datadog adds tags to the ingested cost data using many sources, described in detail below.

• Cost and Usage Report columns
• AWS Resource tags
• AWS Account tags
• AWS Integration tags
• Out-of-the-box tags
• Container workload tags
• Tag pipelines

Cost and Usage Report columns

All string-valued columns from the AWS Cost and Usage Report (CUR) are added as tags on cost metrics.

To ensure consistency, Datadog normalizes tag keys using underscores and lower case. For example, the CUR column lineItem/ResourceId maps to the tag key line_item/resource_id. Tag values are generally unmodified - maintaining exact casing and most special characters.

Examples:

AWS resource tags

AWS resource tags are user-defined tags that appear in the AWS console when viewing a particular resource, like an EC2 instance or S3 bucket.

When you enable the Datadog AWS integration, Datadog automatically collects resource tags for most AWS resources. These tags are applied to all costs found in the CUR for a given resource. Resource tags are retrieved regularly and are applied to cost data starting from the day they are created or modified. Historical tag values are not overwritten when tags change.

If the AWS integration is not enabled, you can enable resource tag enrichment by activating cost allocation tags in AWS billing. This allows you to select a subset of resource tag keys to include as columns in the AWS CUR. Datadog automatically includes those columns as tags when processing the CUR.

AWS organization and account tags

AWS Organizations support user-defined tags on organizational units and accounts. Datadog automatically fetches and applies these tags to cost data. Account tags are applied to all usage associated with those accounts. Organization tags are applied to all billing data for the matching payer account.

Requires the Datadog AWS Integration on the organization account.

AWS integration tags

AWS integration tags are tags set on the AWS Integration tile in the Datadog integrations page. They are applied to all costs found in the CUR for the associated AWS account.

Out-of-the-box tags

Datadog adds out-of-the-box tags to ingested cost data to help you further break down and allocate your costs. These tags are derived from your Cost and Usage Report (CUR) and make it easier to discover and understand cost data.

The following out-of-the-box tags are available for filtering and grouping data:

Cost and observability correlation

Viewing costs in context of observability data is important to understand how infrastructure changes impact costs, identify why costs change, and optimize infrastructure for both costs and performance. Datadog updates resource identifying tags on cost data for top AWS products to simplify correlating observability and cost metrics.

For example, to view cost and utilization for each RDS database, you can make a table with aws.cost.amortized, aws.rds.cpuutilization, and aws.rds.freeable_memory (or any other RDS metric) and group by dbinstanceidentifier. Or, to see Lambda usage and costs side by side, you can graph aws.lambda.concurrent_executions and aws.cost.amortized grouped by functionname.

The following out-of-the-box tags are available:

Container orchestrators

Container cost allocation adds tags from the workloads incurring cost. Examples include tags from Kubernetes pods and nodes and ECS tasks and containers.

Requires container cost allocation, and applies only to shared.resources.allocated metrics.

Tag pipelines

Finally, all of your tag pipeline rulesets are applied, providing complete cost allocation when infrastructure tagging is not possible.

Billing conductor

AWS Billing Conductor is a custom billing service for AWS Marketplace Channel Partners (Partners) and organizations that have chargeback requirements. Billing Conductor enables customers to create a second, pro forma version of their costs to share with their customers or account owners. Billing rates, credits and fees, and overhead costs can be customized at your discretion. You can also select which accounts to include in the CUR.

Pro Forma Cost and Usage Reports do not include discounts and taxes, which makes it difficult to compare costs in Datadog to AWS Cost Explorer. Additionally, adding accounts to a billing group impacts how Reservations and Savings Plans are shared across AWS accounts.

To create a billing conductor CUR, follow the AWS Cost and Usage Reports user guide. Ensure the CUR meets Datadog’s requirements. After the billing conductor CUR is created, follow the Cloud Cost Management instructions above to set it up in Datadog.

Further reading

Additional helpful documentation, links, and articles:

Cloud Cost ManagementDOCUMENTATION Gain insights into your Azure billDOCUMENTATION Gain insights into your Google Cloud billDOCUMENTATION