Overview

To set up Cloud Cost Management in Datadog, you should:

  1. Have an AWS account with billing access
  2. Have the AWS integration installed in Datadog
  3. Follow the steps below to create a Cost and Usage Report

Setup

Prerequisite: generate a Cost and Usage Report

Create a Legacy Cost and Usage Report in AWS under the Data Exports section.

Select the Export type Legacy CUR export.

Select the following content options:

  • Export type: Legacy CUR export
  • Include resource IDs
  • Split cost allocation data (Enables ECS Cost Allocation. You must also opt in to AWS Split Cost Allocation in Cost Explorer preferences).
  • “Refresh automatically”

Select the following delivery options:

  • Time granularity: Hourly
  • Report versioning: Create new report version
  • Compression type: GZIP or Parquet

Configure the AWS integration

Navigate to Setup & Configuration and select an AWS account from the dropdown menu to pull costs from.

Note: Datadog recommends sending a Cost and Usage Report from an AWS management account for cost visibility into related member accounts. If you send a Cost and Usage report from an AWS member account, ensure that you have selected the following options in your management account’s preferences:

  • Linked Account Access
  • Linked Account Refunds and Credits
  • Linked Account Discounts

This ensures complete cost accuracy by allowing periodic cost calculations against the AWS Cost Explorer.

Locate the Cost and Usage Report

If you have navigated away from the report that you created in the prerequisites section, follow AWS documentation to view your Data Exports. Select the legacy CUR export that you created, then select Edit to see the details of the export.

To enable Datadog to locate the Cost and Usage Report, complete the fields with their corresponding details:

  • Bucket Name: This is the name of the S3 bucket in the Data export storage settings section.
  • Bucket Region: This is the region your bucket is located. For example, us-east-1.
  • Export Path Prefix: This is the S3 path prefix in the Data export storage settings section.
  • Export Name: This is the Export name in the Export name section.

Note: Datadog only supports legacy CURs generated by AWS. Do not modify or move the files generated by AWS, or attempt to provide access to files generated by a 3rd party.

Configure access to the Cost and Usage Report

Create a policy in AWS to ensure Datadog has permissions to access the CUR and the S3 bucket it is stored in. Use the following JSON:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "DDCloudCostReadBucket",
          "Effect": "Allow",
          "Action": [
              "s3:ListBucket"
          ],
          "Resource": "arn:aws:s3:::BUCKETNAME"
      },
      {
          "Sid": "DDCloudCostGetBill",
          "Effect": "Allow",
          "Action": [
              "s3:GetObject"
          ],
          "Resource": "arn:aws:s3:::BUCKETNAME/REPORT_PREFIX/REPORT_NAME/*"
      },
      {
          "Sid": "DDCloudCostCheckAccuracy",
          "Effect": "Allow",
          "Action": [
              "ce:Get*"
          ],
          "Resource": "*"
      },
      {
          "Sid": "DDCloudCostListCURs",
          "Action": [
              "cur:DescribeReportDefinitions"
          ],
          "Effect": "Allow",
          "Resource": "*"
      },
      {
          "Sid": "DDCloudCostListOrganizations",
          "Action": [
              "organizations:Describe*",
              "organizations:List*"
          ],
          "Effect": "Allow",
          "Resource": "*"
      }
  ]
}

Tip: Make note of the name you created for this policy for next steps.

Attach the policy to the Datadog integration role

Attach the new S3 policy to the Datadog integration role.

  1. Navigate to Roles in the AWS IAM console.
  2. Locate the role used by the Datadog integration. By default it is named DatadogIntegrationRole, but the name may vary if your organization has renamed it. Click the role name to open the role summary page.
  3. Click Attach policies.
  4. Enter the name of the S3 bucket policy created above.
  5. Click Attach policy.

Note: Data can take up to 48 to 72 hours after setup to stabilize in Datadog.

Cost types

Visualize your ingested data using out-of-the-box cost types. The cost types differ mainly in how they report on discount rates, savings plans and reservations.

On-demand

On-demand costs are the costs of usage at the public, on-demand rate published by AWS. This excludes all savings plans, reservations, discounts, taxes, and fees.

In most cases, on-demand costs are not a reliable source to estimate actual costs.

Amortized and unblended costs

Amortized cost metrics distribute commitment savings throughout the discount term. This is also called accrual basis. Reservations and savings plans are drawn down from a monthly commitment and applied directly to covered usage, at the time of usage. Any unused remainder appears as a fee.

In contrast, unblended cost metrics show all charges on the date that they are incurred. This is also called cost basis. Reservation and savings plan fees show up on the date they were charged, and are not applied directly to covered usage. After billing data for a month is finalized, unblended metrics match the AWS invoice exactly.

Net costs

Net costs apply private discounts directly to usage. The cost of usage for a specific resource represents the effective cost after all savings are realized.

In contrast, other metrics show private discounts as separate, negative-valued line items with no resource attribution tags. Rather than attributing the discounts directly to usage, those metrics subtract discounts from the total cost.

Net amortized costs provide the most accurate representation for cost allocation, with all savings applied directly to usage. Net cost metrics are available if your AWS account has privately negotiated enterprise discounts. If your account does not have enterprise discounts, then net amortized cost and amortized cost are equivalent.

Container allocation

Container allocation metrics contain all of the same costs as the AWS metrics, but with additional breakdowns and insights for container workloads. See container cost allocation for more details.

Example

The following scenario demonstrates how different cost types behave. Imagine you have:

  • An EC2 instance running for one hour with the cost of $3 per compute-hour.
  • A savings plan which prices this instance type at $2 per compute-hour.
  • A negotiated EDP discount of 10% on top of all other discounts.

Here’s how the instance cost, savings plan hourly commitment, and discount appear in each cost type:

Cost typeUsageSavings PlanDiscountExplanation
On Demand$3.00This is the public on-demand rate.
Unblended$3.00$2.00-$0.20Savings plan recurring fee and EDP discount are separate line items, not associated with a specific resource. (Note: the $3 resource cost is offset with SavingsPlanNegation.)
Net Unblended$1.80Savings plan recurring fee appears as a line item with the discount applied; the cost is not associated with a specific resource.
Amortized$2.00-$0.20Savings plan discount is applied directly to the resource cost. EDP discount is a separate line item.
Net Amortized$1.80Savings plan and EDP discounts are applied directly to resource cost.
Net Amortized - Shared Resources Allocated$1.80The same cost as Net Amortized, but this cost can be further broken down by Kubernetes dimensions and pod tags.

Cost metrics summary

In general:

  • aws.cost.net.amortized.shared.resources.allocated provides the most complete cost allocation for specific workloads and teams.
  • If you do not have container cost allocation, use aws.cost.net.amortized.
  • If you do not have net amortized costs, use aws.cost.amortized.shared.resources.allocated or aws.cost.amortized.
MetricDescription
aws.cost.net.amortized.shared.resources.allocatedAll of your AWS net amortized costs, with additional breakdowns and insights for container workloads. Requires container cost allocation.
aws.cost.net.amortizedNet amortized costs, without container cost breakdowns.
aws.cost.net.unblendedNet unblended costs, without container cost breakdowns. Matches the AWS invoice, with specialized discounts pre-calculated within usage costs.
aws.cost.amortized.shared.resources.allocatedAll of your AWS amortized costs, with additional breakdowns and insights for container workloads. Requires container cost allocation.
aws.cost.amortizedAmortized costs, without container cost breakdowns.
aws.cost.unblendedUnblended costs, without container cost breakdowns. Matches the AWS invoice.
aws.cost.ondemandCosts based on the list rate provided by AWS, excluding all savings plans, reservations, discounts, taxes, and fees.

Tag enrichment

Datadog adds tags to the ingested cost data using many sources, described in detail below.

  • Cost and Usage Report columns
  • AWS Resource tags
  • AWS Account tags
  • AWS Integration tags
  • Out-of-the-box tags
  • Container workload tags
  • Tag pipelines

Cost and Usage Report columns

All string-valued columns from the AWS Cost and Usage Report (CUR) are added as tags on cost metrics.

To ensure consistency, Datadog normalizes tag keys using underscores and lower case. For example, the CUR column lineItem/ResourceId maps to the tag key line_item/resource_id. Tag values are generally unmodified - maintaining exact casing and most special characters.

Examples:

CUR ColumnCUR ValueCloud Cost Tag
lineItem/ResourceIdi-12345678a9b12cd3eline_item/resource_id:i-12345678a9b12cd3e
product/regionus-east-1product/region:us-east-1
product/usagetypeDataTransfer-Regional-Bytesproduct/usagetype:DataTransfer-Regional-Bytes

AWS resource tags

AWS resource tags are user-defined tags that appear in the AWS console when viewing a particular resource, like an EC2 instance or S3 bucket.

When you enable the Datadog AWS integration, Datadog automatically collects resource tags for most AWS resources. These tags are applied to all costs found in the CUR for a given resource. Resource tags are retrieved regularly and are applied to cost data starting from the day they are created or modified. Historical tag values are not overwritten when tags change.

If the AWS integration is not enabled, you can enable resource tag enrichment by activating cost allocation tags in AWS billing. This allows you to select a subset of resource tag keys to include as columns in the AWS CUR. Datadog automatically includes those columns as tags when processing the CUR.

AWS organization and account tags

AWS Organizations support user-defined tags on organizational units and accounts. Datadog automatically fetches and applies these tags to cost data. Account tags are applied to all usage associated with those accounts. Organization tags are applied to all billing data for the matching payer account.

Requires the Datadog AWS Integration on the organization account.

AWS integration tags

AWS integration tags are tags set on the AWS Integration tile in the Datadog integrations page. They are applied to all costs found in the CUR for the associated AWS account.

Out-of-the-box tags

Datadog adds out-of-the-box tags to ingested cost data to help you further break down and allocate your costs. These tags are derived from your Cost and Usage Report (CUR) and make it easier to discover and understand cost data.

The following out-of-the-box tags are available for filtering and grouping data:

TagDescription
aws_productThe AWS service being billed.
aws_product_familyThe category for the AWS service being billed (for example, Compute or Storage).
aws_management_account_nameThe AWS management account name associated with the item.
aws_management_account_idThe AWS management account ID associated with the item.
aws_member_account_nameThe AWS member account name associated with the item.
aws_member_account_idThe AWS member account ID associated with the item.
aws_cost_typeThe type of charge covered by this item (for example, Usage, or Tax).
aws_pricing_termWhether the usage is Reserved, Spot, or On-Demand.
aws_reservation_arnThe ARN of the Reserved Instance that the item benefited from.
aws_savings_plan_arnThe ARN of the Savings Plan the item benefited from.
aws_usage_typeThe usage details of the item (for example, BoxUsage:i3.8xlarge).
aws_operationThe operation associated with the item (for example, RunInstances).
aws_regionThe region associated with the item (such as us-east-1).
aws_availability_zoneThe availability zone associated with the item.
aws_resource_idThe resource ID associated with the item.
aws_instance_typeThe instance type of the item.
aws_instance_familyThe instance family associated with your item (for example, Storage optimized).
is_aws_ec2_computeWhether the usage is related to EC2 compute.
is_aws_ec2_compute_on_demandWhether the usage is on-demand.
is_aws_ec2_compute_reservationWhether the usage is associated with a Reserved Instance.
is_aws_ec2_capacity_reservationWhether the usage is associated with a Capacity Reservation.
is_aws_ec2_spot_instanceWhether the usage is associated with a Spot Instance.
is_aws_ec2_savings_planWhether the usage is associated with a Savings Plan.
aws_bill_entityThe AWS seller that your account is with. Transactions can either be an AWS Marketplace purchase (AWS Marketplace) or a purchase of other AWS services (AWS).
aws_bill_typeThe type of bill that this report covers (such as Purchase).
aws_cost_typeThe type of charge covering the line item (such as SavingsPlanCoveredUsage).
aws_discount_lease_termThe length of time that a Reserved instance is reserved for.
aws_discount_purchase_optionHow you chose to pay for a reservation (such as All Upfront).
aws_ec2_compute_product_familyThe type of usage for an EC2 Compute line item (such as BoxUsage or SpotUsage).
aws_pricing_usage_unitThe pricing unit that AWS used for calculating the usage cost (such as Hours).
aws_reservation_modification_statusIndicates whether the RI lease was modified or unaltered (such as Manual).
bill/billing_entityThe AWS seller that your account is with. Transactions can either be an AWS Marketplace purchase (AWS Marketplace) or a purchase of other AWS services (AWS).
bill/bill_typeThe type of bill that this report covers (such as Purchase).
bill/invoicing_entityThe AWS entity that issues the invoice.
bill/payer_account_idThe account ID of the paying account. For an organization in AWS Organizations, this is the account ID of the management account.
is_aws_ec2_compute_savings_plantrue for line items that represent EC2 Compute usage, paid for using a Savings Plan.
line_item/currency_codeThe currency that this line item is shown in (USD by default).
line_item/legal_entityThe provider of your AWS services.
line_item/line_item_typeThe type of charge covered by the line item (such as Credit).
line_item/operationThe specific AWS operation covered by the line item (such as RunInstances).
line_item/product_codeThe code of the product measured (such as Amazon EC2 for Amazon Elastic Cloud Compute).
line_item/resource_idThe individual resource ID associated with the line item (Optional).
line_item/tax_typeThe type of tax that AWS applied to the line item.
line_item/usage_account_idThe ID of the account that used the line item.
line_item/usage_typeThe usage details of the line item (such as USW2-BoxUsage:m2.2xlarge).
pricing/lease_contract_lengthThe length of time that the RI is reserved for.
pricing/purchase_optionHow you chose to pay for the line item (such as All Upfront).
pricing/termWhether your AWS usage is Reserved or On-Demand.
pricing/unitThe pricing unit that AWS used for calculating the usage cost (such as Hours).
reservation/availability_zoneThe Availability Zone of the resource associated with the line item (such as us-east-1).
reservation/modification_statusShows whether the RI lease was modified or unaltered (such as Manual).
reservation/reservation_arnThe ARN of the RI that the line item benefited from.
reservation/subscription_idThe unique ID that maps the line item with the associated offer.
savings_plan/instance_type_familyThe instance family that is associated with the specified usage (such as m4).
savings_plan/offering_typeThe type of Savings Plan purchased (such as ComputeSavingsPlans).
savings_plan/payment_optionThe payment options available for the Savings Plan (such as All Upfront).
savings_plan/purchase_termDescribes the duration or term of the Savings Plan (such as 1yr).
savings_plan/regionThe AWS Region that hosts the AWS services (such as US East (N. Virginia)).
savings_plan/savings_plan_arnThe unique Savings Plan identifier.

Cost and observability correlation

Viewing costs in context of observability data is important to understand how infrastructure changes impact costs, identify why costs change, and optimize infrastructure for both costs and performance. Datadog updates resource identifying tags on cost data for top AWS products to simplify correlating observability and cost metrics.

For example, to view cost and utilization for each RDS database, you can make a table with aws.cost.amortized, aws.rds.cpuutilization, and aws.rds.freeable_memory (or any other RDS metric) and group by dbinstanceidentifier. Or, to see Lambda usage and costs side by side, you can graph aws.lambda.concurrent_executions and aws.cost.amortized grouped by functionname.

The following out-of-the-box tags are available:

AWS ProductTag
ec2instance_id
s3bucketname
rdsdbinstanceidentifier
lambdafunctionname
dynamodbtablename
elasticachecacheclusterid
cloudfront (distribution)distributionid
cloudfront (function)functionname
ec2 natgatewaynatgatewayid
redshiftclusteridentifier
kinesisstreamname
queuequeuename
snstopicname
elb (application, gateway, network)loadbalancer
elb (all other costs)loadbalancername

Container orchestrators

Container cost allocation adds tags from the workloads incurring cost. Examples include tags from Kubernetes pods and nodes and ECS tasks and containers.

Requires container cost allocation, and applies only to shared.resources.allocated metrics.

Tag pipelines

Finally, all of your tag pipeline rulesets are applied, providing complete cost allocation when infrastructure tagging is not possible.

Billing conductor

AWS Billing Conductor is a custom billing service for AWS Marketplace Channel Partners (Partners) and organizations that have chargeback requirements. Billing Conductor enables customers to create a second, pro forma version of their costs to share with their customers or account owners. Billing rates, credits and fees, and overhead costs can be customized at your discretion. You can also select which accounts to include in the CUR.

Pro Forma Cost and Usage Reports do not include discounts and taxes, which makes it difficult to compare costs in Datadog to AWS Cost Explorer. Additionally, adding accounts to a billing group impacts how Reservations and Savings Plans are shared across AWS accounts.

To create a billing conductor CUR, follow the AWS Cost and Usage Reports user guide. Ensure the CUR meets Datadog’s requirements. After the billing conductor CUR is created, follow the Cloud Cost Management instructions above to set it up in Datadog.

Further reading