AWS

Overview

Connect to Amazon Web Services (AWS) to:

  • See automatic AWS status updates in your Events Explorer
  • Get CloudWatch metrics for EC2 hosts without installing the Agent
  • Tag your EC2 hosts with EC2-specific information
  • See EC2 scheduled maintenance events in your stream
  • Collect CloudWatch metrics and events from many other AWS products
  • See CloudWatch alarms in your Events Explorer

To quickly get started using the AWS integration, check out the AWS getting started guide.

Datadog’s Amazon Web Services integration collects logs, events, and most metrics from CloudWatch for over 90 AWS services.

Setup

Use one of the following methods to integrate your AWS accounts into Datadog for metric, event, tag, and log collection.

Automatic

Manual

  • Role delegation
    To set up the AWS integration manually with role delegation, see the manual setup guide.

  • Access keys (GovCloud or China* Only)
    To set up the AWS integration with access keys, see the manual setup guide.

    * All use of Datadog Services in (or in connection with environments within) mainland China is subject to the disclaimer published in the Restricted Service Locations section on our website.

AWS IAM permissions

AWS IAM permissions enable Datadog to collect metrics, tags, EventBridge events, and other data necessary to monitor your AWS environment.

To correctly set up the AWS Integration, you must attach the relevant IAM policies to the Datadog AWS Integration IAM Role in your AWS account.

AWS integration IAM policy

The set of permissions necessary to use all the integrations for individual AWS services.

The following permissions included in the policy document use wild cards such as List* and Get*. If you require strict policies, use the complete action names as listed and reference the Amazon API documentation for your respective services.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "apigateway:GET",
                "aoss:BatchGetCollection",
                "aoss:ListCollections",
                "autoscaling:Describe*",
                "backup:List*",
                "bcm-data-exports:GetExport",
                "bcm-data-exports:ListExports",
                "bedrock:GetAgent",
                "bedrock:GetAgentAlias",
                "bedrock:GetFlow",
                "bedrock:GetFlowAlias",
                "bedrock:GetGuardrail",
                "bedrock:GetImportedModel",
                "bedrock:GetInferenceProfile",
                "bedrock:GetMarketplaceModelEndpoint",
                "bedrock:ListAgentAliases",
                "bedrock:ListAgents",
                "bedrock:ListFlowAliases",
                "bedrock:ListFlows",
                "bedrock:ListGuardrails",
                "bedrock:ListImportedModels",
                "bedrock:ListInferenceProfiles",
                "bedrock:ListMarketplaceModelEndpoints",
                "bedrock:ListPromptRouters",
                "bedrock:ListProvisionedModelThroughputs",
                "budgets:ViewBudget",
                "cassandra:Select",
                "cloudfront:GetDistributionConfig",
                "cloudfront:ListDistributions",
                "cloudtrail:DescribeTrails",
                "cloudtrail:GetTrailStatus",
                "cloudtrail:LookupEvents",
                "cloudwatch:Describe*",
                "cloudwatch:Get*",
                "cloudwatch:List*",
                "codeartifact:DescribeDomain",
                "codeartifact:DescribePackageGroup",
                "codeartifact:DescribeRepository",
                "codeartifact:ListDomains",
                "codeartifact:ListPackageGroups",
                "codeartifact:ListPackages",
                "codedeploy:BatchGet*",
                "codedeploy:List*",
                "codepipeline:ListWebhooks",
                "cur:DescribeReportDefinitions",
                "directconnect:Describe*",
                "dynamodb:Describe*",
                "dynamodb:List*",
                "ec2:Describe*",
                "ec2:GetAllowedImagesSettings",
                "ec2:GetEbsDefaultKmsKeyId",
                "ec2:GetInstanceMetadataDefaults",
                "ec2:GetSerialConsoleAccessStatus",
                "ec2:GetSnapshotBlockPublicAccessState",
                "ec2:GetTransitGatewayPrefixListReferences",
                "ec2:SearchTransitGatewayRoutes",
                "ecs:Describe*",
                "ecs:List*",
                "elasticache:Describe*",
                "elasticache:List*",
                "elasticfilesystem:DescribeAccessPoints",
                "elasticfilesystem:DescribeFileSystems",
                "elasticfilesystem:DescribeTags",
                "elasticloadbalancing:Describe*",
                "elasticmapreduce:Describe*",
                "elasticmapreduce:List*",
                "emr-containers:ListManagedEndpoints",
                "emr-containers:ListSecurityConfigurations",
                "emr-containers:ListVirtualClusters",
                "es:DescribeElasticsearchDomains",
                "es:ListDomainNames",
                "es:ListTags",
                "events:CreateEventBus",
                "fsx:DescribeFileSystems",
                "fsx:ListTagsForResource",
                "glacier:GetVaultNotifications",
                "glue:ListRegistries",
                "grafana:DescribeWorkspace",
                "greengrass:GetComponent",
                "greengrass:GetConnectivityInfo",
                "greengrass:GetCoreDevice",
                "greengrass:GetDeployment",
                "health:DescribeAffectedEntities",
                "health:DescribeEventDetails",
                "health:DescribeEvents",
                "kinesis:Describe*",
                "kinesis:List*",
                "lambda:GetPolicy",
                "lambda:List*",
                "lightsail:GetInstancePortStates",
                "logs:DeleteSubscriptionFilter",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:DescribeSubscriptionFilters",
                "logs:FilterLogEvents",
                "logs:PutSubscriptionFilter",
                "logs:TestMetricFilter",
                "macie2:GetAllowList",
                "macie2:GetCustomDataIdentifier",
                "macie2:ListAllowLists",
                "macie2:ListCustomDataIdentifiers",
                "macie2:ListMembers",
                "macie2:GetMacieSession",
                "managedblockchain:GetAccessor",
                "managedblockchain:GetMember",
                "managedblockchain:GetNetwork",
                "managedblockchain:GetNode",
                "managedblockchain:GetProposal",
                "managedblockchain:ListAccessors",
                "managedblockchain:ListInvitations",
                "managedblockchain:ListMembers",
                "managedblockchain:ListNodes",
                "managedblockchain:ListProposals",
                "memorydb:DescribeAcls",
                "memorydb:DescribeMultiRegionClusters",
                "memorydb:DescribeParameterGroups",
                "memorydb:DescribeReservedNodes",
                "memorydb:DescribeSnapshots",
                "memorydb:DescribeSubnetGroups",
                "memorydb:DescribeUsers",
                "oam:ListAttachedLinks",
                "oam:ListSinks",
                "organizations:Describe*",
                "organizations:List*",
                "osis:GetPipeline",
                "osis:GetPipelineBlueprint",
                "osis:ListPipelineBlueprints",
                "osis:ListPipelines",
                "proton:GetComponent",
                "proton:GetDeployment",
                "proton:GetEnvironment",
                "proton:GetEnvironmentAccountConnection",
                "proton:GetEnvironmentTemplate",
                "proton:GetEnvironmentTemplateVersion",
                "proton:GetRepository",
                "proton:GetService",
                "proton:GetServiceInstance",
                "proton:GetServiceTemplate",
                "proton:GetServiceTemplateVersion",
                "proton:ListComponents",
                "proton:ListDeployments",
                "proton:ListEnvironmentAccountConnections",
                "proton:ListEnvironmentTemplateVersions",
                "proton:ListEnvironmentTemplates",
                "proton:ListEnvironments",
                "proton:ListRepositories",
                "proton:ListServiceInstances",
                "proton:ListServiceTemplateVersions",
                "proton:ListServiceTemplates",
                "proton:ListServices",
                "qldb:ListJournalKinesisStreamsForLedger",
                "rds:Describe*",
                "rds:List*",
                "redshift:DescribeClusters",
                "redshift:DescribeLoggingStatus",
                "redshift-serverless:ListEndpointAccess",
                "redshift-serverless:ListManagedWorkgroups",
                "redshift-serverless:ListNamespaces",
                "redshift-serverless:ListRecoveryPoints",
                "redshift-serverless:ListSnapshots",
                "route53:List*",
                "s3:GetBucketLocation",
                "s3:GetBucketLogging",
                "s3:GetBucketNotification",
                "s3:GetBucketTagging",
                "s3:ListAccessGrants",
                "s3:ListAllMyBuckets",
                "s3:PutBucketNotification",
                "s3express:GetBucketPolicy",
                "s3express:GetEncryptionConfiguration",
                "s3express:ListAllMyDirectoryBuckets",
                "savingsplans:DescribeSavingsPlanRates",
                "savingsplans:DescribeSavingsPlans",
                "secretsmanager:GetResourcePolicy",
                "ses:Get*",
                "ses:ListAddonInstances",
                "ses:ListAddonSubscriptions",
                "ses:ListAddressLists",
                "ses:ListArchives",
                "ses:ListContactLists",
                "ses:ListCustomVerificationEmailTemplates",
                "ses:ListMultiRegionEndpoints",
                "ses:ListIngressPoints",
                "ses:ListRelays",
                "ses:ListRuleSets",
                "ses:ListTemplates",
                "ses:ListTrafficPolicies",
                "sns:GetSubscriptionAttributes",
                "sns:List*",
                "sns:Publish",
                "sqs:ListQueues",
                "states:DescribeStateMachine",
                "states:ListStateMachines",
                "support:DescribeTrustedAdvisor*",
                "support:RefreshTrustedAdvisorCheck",
                "tag:GetResources",
                "tag:GetTagKeys",
                "tag:GetTagValues",
                "timestream:DescribeEndpoints",
                "timestream:ListTables",
                "waf-regional:GetRule",
                "waf-regional:GetRuleGroup",
                "waf-regional:ListRuleGroups",
                "waf-regional:ListRules",
                "waf:GetRule",
                "waf:GetRuleGroup",
                "waf:ListRuleGroups",
                "waf:ListRules",
                "wafv2:GetIPSet",
                "wafv2:GetLoggingConfiguration",
                "wafv2:GetRegexPatternSet",
                "wafv2:GetRuleGroup",
                "wafv2:ListLoggingConfigurations",
                "workmail:DescribeOrganization",
                "workmail:ListOrganizations",
                "xray:BatchGetTraces",
                "xray:GetTraceSummaries"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

AWS resource collection IAM policy

To use resource collection, you must attach AWS’s managed SecurityAudit Policy to your Datadog IAM role.

Notes:

  • Warning messages appear on the AWS integration tile in Datadog if you enable resource collection, but do not have the AWS Security Audit Policy attached to your Datadog IAM role.
  • To enable Datadog to collect account management resources from account.GetAlternateContact and account.GetContactInformation, you need to enable trusted access for AWS account management.
  • AWS Govcloud and AWS China accounts are not currently supported.

Log collection

There are two ways of sending AWS service logs to Datadog:

  • Amazon Data Firehose destination: Use the Datadog destination in your Amazon Data Firehose delivery stream to forward logs to Datadog. It is recommended to use this approach when sending logs from CloudWatch in a very high volume.
  • Forwarder Lambda function: Deploy the Datadog Forwarder Lambda function, which subscribes to S3 buckets or your CloudWatch log groups and forwards logs to Datadog. Datadog also recommends you use this approach for sending logs from S3 or other resources that cannot directly stream data to Amazon Data Firehose.

Metric collection

There are two ways to send AWS metrics to Datadog:

  • Metric polling: API polling comes out of the box with the AWS integration. A metric-by-metric crawl of the CloudWatch API pulls data and sends it to Datadog. New metrics are pulled every ten minutes, on average.
  • Metric streams with Amazon Data Firehose: You can use Amazon CloudWatch Metric Streams and Amazon Data Firehose to see your metrics. Note: This method has a two to three minute latency, and requires a separate setup.

You can find a full list of the available sub-integrations on the Integrations page. Many of these integrations are installed by default when Datadog recognizes data coming in from your AWS account. See the AWS Integration Billing page for options to exclude specific resources for cost control.

Resource collection

Some Datadog products leverage information about how your AWS resources (such as S3 buckets, RDS snapshots, and CloudFront distributions) are configured. Datadog collects this information by making read-only API calls to your AWS account.

AWS resource collection IAM policy

To use resource collection, you must attach AWS’s managed SecurityAudit Policy to your Datadog IAM role.

Notes:

  • Warning messages appear on the AWS integration tile in Datadog if you enable resource collection, but do not have the AWS Security Audit Policy attached to your Datadog IAM role.
  • To enable Datadog to collect account management resources from account.GetAlternateContact and account.GetContactInformation, you need to enable trusted access for AWS account management.
  • AWS Govcloud and AWS China accounts are not currently supported.

Cloud Security Management

Setup

If you do not have the AWS integration set up for your AWS account, complete the set up process above. Ensure that you enable Cloud Security Management when mentioned.

Note: The AWS integration must be set up with Role delegation to use this feature.

To add Cloud Security Management to an existing AWS integration, follow the steps below to enable resource collection.

  1. Provide the necessary permissions to the Datadog IAM role by attaching the AWS managed SecurityAudit policy to your Datadog AWS IAM role. You can find this policy in the AWS console.

  2. Complete the setup in the Datadog AWS integration page with the steps below. Alternatively, you can use the Update an AWS Integration API endpoint.

    1. Select the AWS account where you wish to enable resource collection.
    2. On the Resource collection tab, click Enable next to Cloud Security Management. You are redirected to the Cloud Security Management Setup page, and a setup dialog automatically opens for the selected account.
    3. On the setup dialog, switch the Enable Resource Scanning toggle to the on position.
    4. Click Done to complete the setup.

Alarm collection

There are two ways to send AWS CloudWatch alarms to the Datadog Events Explorer:

  • Alarm polling: Alarm polling comes out of the box with the AWS integration and fetches metric alarms through the DescribeAlarmHistory API. If you follow this method, your alarms are categorized under the event source Amazon Web Services. Note: The crawler does not collect composite alarms.
  • SNS topic: You can see all AWS CloudWatch alarms in your Events Explorer by subscribing the alarms to an SNS topic, then forwarding the SNS messages to Datadog. To learn how to receive SNS messages as events in Datadog, see Receive SNS messages. If you follow this method, your alarms are categorized under the event source Amazon SNS.

Data Collected

Metrics

aws.events.failed_invocations
(count)		Measures the number of invocations that failed permanently. This does not include invocations that are retried or that succeeded after a retry attempt
aws.events.invocations
(count)		Measures the number of times a target is invoked for a rule in response to an event. This includes successful and failed invocations but does not include throttled or retried attempts until they fail permanently.
aws.events.matched_events
(count)		Measures the number of events that matched with any rule.
aws.events.throttled_rules
(count)		Measures the number of triggered rules that are being throttled.
aws.events.triggered_rules
(count)		Measures the number of triggered rules that matched with any event.
aws.logs.delivery_errors
(count)		The number of log events for which CloudWatch Logs received an error when forwarding data to the subscription destination.
Shown as event
aws.logs.delivery_throttling
(count)		The number of log events for which CloudWatch Logs was throttled when forwarding data to the subscription destination.
Shown as event
aws.logs.forwarded_bytes
(gauge)		The volume of log events in compressed bytes forwarded to the subscription destination.
Shown as byte
aws.logs.forwarded_log_events
(count)		The number of log events forwarded to the subscription destination.
Shown as event
aws.logs.incoming_bytes
(gauge)		The volume of log events in uncompressed bytes uploaded to Cloudwatch Logs.
Shown as byte
aws.logs.incoming_log_events
(count)		The number of log events uploaded to Cloudwatch Logs.
Shown as event
aws.usage.call_count
(count)		The number of specified operations performed in your account
Shown as operation
aws.usage.resource_count
(count)		The number of specified resources in your account
Shown as resource

Note: You can enable the collection of AWS custom metrics, as well as metrics from services that Datadog doesn’t have an integration for. See the AWS Integration and CloudWatch FAQ for more information.

Events

Events from AWS are collected on a per AWS-service basis. See your AWS service’s documentation to learn more about collected events.

Tags

The following tags are collected with the AWS integration. Note: Some tags only display on specific metrics.

IntegrationDatadog Tag Keys
Allregion
API Gatewayapiid, apiname, method, resource, stage
App Runnerinstance, serviceid, servicename
Auto Scalingautoscalinggroupname, autoscaling_group
Billingaccount_id, budget_name, budget_type, currency, servicename, time_unit
CloudFrontdistributionid
CodeBuildproject_name
CodeDeployapplication, creator, deployment_config, deployment_group, deployment_option, deployment_type, status
DirectConnectconnectionid
DynamoDBglobalsecondaryindexname, operation, streamlabel, tablename
EBSvolumeid, volume-name, volume-type
EC2autoscaling_group, availability-zone, image, instance-id, instance-type, kernel, name, security_group_name
ECSclustername, servicename, instance_id
EFSfilesystemid
ElastiCachecachenodeid, cache_node_type, cacheclusterid, cluster_name, engine, engine_version, preferred_availability-zone, replication_group
ElasticBeanstalkenvironmentname, enviromentid
ELBavailability-zone, hostname, loadbalancername, name, targetgroup
EMRcluster_name, jobflowid
ESdedicated_master_enabled, ebs_enabled, elasticsearch_version, instance_type, zone_awareness_enabled
Firehosedeliverystreamname
FSxfilesystemid, filesystemtype
Healthevent_category, status, service
IoTactiontype, protocol, rulename
Kinesisstreamname, name, state
KMSkeyid
Lambdafunctionname, resource, executedversion, memorysize, runtime
Machine Learningmlmodelid, requestmode
MQbroker, queue, topic
OpsWorksstackid, layerid, instanceid
Pollyoperation
RDSauto_minor_version_upgrade, dbinstanceclass, dbclusteridentifier, dbinstanceidentifier, dbname, engine, engineversion, hostname, name, publicly_accessible, secondary_availability-zone
RDS Proxyproxyname, target, targetgroup, targetrole
Redshiftclusteridentifier, latency, nodeid, service_class, stage, wlmid
Route 53healthcheckid
S3bucketname, filterid, storagetype
SESTag keys are custom set in AWS.
SNStopicname
SQSqueuename
VPCnategatewayid, vpnid, tunnelipaddress
WorkSpacesdirectoryid, workspaceid

Service Checks

aws.status
Returns CRITICAL if one or more AWS regions are experiencing issues. Returns OK otherwise.
Statuses: ok, critical

Troubleshooting

See the AWS Integration Troubleshooting guide to resolve issues related to the AWS integration.

Further Reading

Additional helpful documentation, links, and articles:

Monitor AWS control plane API usage metrics in DatadogBLOG more more Highlights from AWS re:Invent 2022BLOG more more Best practices for creating least-privilege AWS IAM policiesBLOG more more