--- isPrivate: true title: Install Datadog Instrumentation through the AWS Integration description: >- Instrument your Amazon EC2 instances and AWS Lambda functions directly from the AWS integration, without connecting to each host or redeploying each function. breadcrumbs: >- Docs > Integrations > Integration Guides > Install Datadog Instrumentation through the AWS Integration --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Install Datadog Instrumentation through the AWS Integration {% callout %} # Important note for users on the following Datadog sites: us2.ddog-gov.com {% alert level="info" %} To find out if this integration is available in your organization, see your [Datadog Integrations](https://app.datadoghq.com/integrations) page or ask your organization administrator. To initiate an exception request to enable this integration for your organization, email [support@ddog-gov.com](mailto:support@ddog-gov.com). {% /alert %} {% /callout %} ## Overview{% #overview %} The [AWS integration](https://docs.datadoghq.com/integrations/amazon_web_services.md) collects metrics, events, and logs from Amazon CloudWatch without installing anything on your resources. Datadog instrumentation collects telemetry data from inside your AWS workloads that CloudWatch alone can't provide, including host-level metrics, distributed traces (APM), live processes, and detailed logs. You can instrument your AWS workloads directly from Datadog, without connecting to each host or redeploying each function. Enable instrumentation while you set up the AWS integration, or at any time afterward. ## Supported workloads{% #supported-workloads %} | Workload | What Datadog installs | | -------------------- | ----------------------------------------------------------------------------------------------------------------------- | | Amazon EC2 instances | The Datadog Agent | | AWS Lambda functions | The Datadog Lambda extension and, for supported runtimes, the Datadog tracing layer that matches the function's runtime | Amazon EKS is not supported. For Lambda functions, Datadog also offers remote instrumentation, a separate product. To decide which to use, see the following section. {% alert level="warning" %} A Lambda function can be managed by only one Datadog instrumentation product. Datadog skips any function that remote instrumentation already manages, and skips any function you instrumented yourself. {% /alert %} ## Choose between the AWS integration and remote instrumentation{% #choose-between-the-aws-integration-and-remote-instrumentation %} Datadog offers two ways to add instrumentation to Lambda functions without redeploying them yourself: - **Instrumentation through the AWS integration**, covered by this guide, is managed entirely from Datadog. Datadog updates your functions with the AWS integration IAM role created by the CloudFormation stack, and deploys no compute into your account. - **[Remote instrumentation](https://docs.datadoghq.com/serverless/aws_lambda/remote_instrumentation.md)** deploys a Datadog instrumenter function, `datadog-remote-instrumenter`, into your own account. That function applies the instrumentation and keeps it in place. Both add the same Datadog Lambda extension and tracing layers, and both restore instrumentation that is changed outside of Datadog. They differ in where the work runs, how functions are selected, and what you install. | Aspect | Instrumentation through the AWS integration | Remote instrumentation | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | | Workloads | Amazon EC2 instances and AWS Lambda functions | AWS Lambda functions | | What runs in your account | No Datadog compute. Datadog calls the AWS APIs with the AWS integration IAM role created by the CloudFormation stack | The instrumenter Lambda function | | Scope of setup | One CloudFormation stack per AWS account | One CloudFormation stack per account and region | | Selecting functions | You write a query on function attributes, select specific functions, or add all eligible functions. Datadog shows the matched set before you save | You write targeting rules on function names and tags, with logical operators | | Functions that match later | Instrumented automatically, whether they were created after you saved the rule or started matching it after a tag change | Instrumented automatically when they match your targeting rules | | Layer versions | Datadog selects and updates them | You set them, and they stay fixed until you change them | | How instrumented functions authenticate | [Workload Identity Federation](https://docs.datadoghq.com/account_management/workload_identity_federation.md), with no Datadog API key on the function | A Datadog API key with Remote Configuration enabled | | Datadog permissions | Hosts Read and Agent Install | Serverless AWS Instrumentation Read and Write | | Removing instrumentation | Remove from Datadog | Delete the CloudFormation stack in that region | Both products deploy a CloudFormation stack in your account. The stack for remote instrumentation also creates a CloudTrail trail and supporting resources. For what the stack in this guide creates, including the EventBridge resources that send change events to Datadog, see [How Datadog instrumentation through the AWS integration works](https://docs.datadoghq.com/integrations/guide/aws-agent-installation-technical-reference.md). Use instrumentation through the AWS integration when you want to instrument both EC2 instances and Lambda functions from one place, or when you want to narrow the function list by region, runtime, and memory size. Use remote instrumentation when you want to match on the tags in `DD_TAGS`, or when you want to set the layer versions applied to your functions and keep them fixed. Both products can match on AWS resource tags. ## Prerequisites{% #prerequisites %} For all workloads, confirm the following: - **CloudFormation access**: You can approve a CloudFormation stack in the target AWS account. Instrumentation deploys a stack in your account, so you (or a teammate) need permission to review and create it. For the required permissions and why they're needed, see the Required AWS permissions section. - **Datadog permissions**: Viewing instrumentation rules requires the **Hosts Read** permission. Creating, editing, or deleting rules requires the **Agent Install** permission. ### Amazon EC2 instances{% #amazon-ec2-instances %} - **SSM Agent**: The [AWS Systems Manager (SSM) Agent](https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html) must already be present on the target instances. Datadog installs the Agent through SSM and can't install the SSM Agent for you, so instances built from custom AMIs without the SSM Agent are not eligible. Datadog flags these instances so you can address them. - **Supported platforms**: Linux (x86_64 and arm64) and Windows (x86_64). macOS and Windows on arm64 are not supported. ### AWS Lambda functions{% #aws-lambda-functions %} - **Resource collection**: [Resource collection](https://docs.datadoghq.com/integrations/amazon_web_services.md#resource-collection) must be enabled on the AWS integration. Datadog uses it to list your functions and preview which ones a rule matches. - **AWS partition**: The function must be in the commercial `aws` partition. Functions in AWS GovCloud or the AWS China partitions are not supported, because Lambda instrumentation authenticates through [Workload Identity Federation](https://docs.datadoghq.com/account_management/workload_identity_federation.md), which does not support those partitions. - **Package type**: The function must use the Zip package type. Container image functions are not supported, because Datadog instrumentation is distributed as Lambda layers, which container image functions can't use. - **Architecture**: The function must use a single architecture, either `x86_64` or `arm64`. - **Lambda@Edge**: The function must not be a Lambda@Edge function. Datadog excludes both the replicas and the functions they replicate. - **Layer count**: AWS limits a function to five layers. Datadog adds two layers, or one for OS-only runtimes, so the function must have room for them after its existing layers. - **Supported runtimes**: | Runtime | Versions | | ------- | ----------------------------------------------------------------------------- | | Node.js | 16.x, 18.x, 20.x, 22.x, 24.x, 26.x | | Python | 3.8, 3.9, 3.10, 3.11, 3.12, 3.13, 3.14 | | Ruby | 3.2, 3.3, 3.4, 4.0 | | Java | 8 (`java8` and `java8.al2`), 11, 17, 21, 25 | | .NET | 6, 8, 10 | | OS-only | `provided.al2` and `provided.al2023` (extension layer only, no tracing layer) | Datadog marks any function that doesn't meet these conditions as ineligible in the rule preview, so you can see what is excluded before you apply a rule. ## Required AWS permissions{% #required-aws-permissions %} Datadog instrumentation requires permissions beyond the base [AWS integration IAM policy](https://docs.datadoghq.com/integrations/amazon_web_services.md#aws-iam-permissions). For **Amazon EC2 instances**, Datadog needs permissions to: - Store your Datadog API key (AWS Secrets Manager). - Create and attach the IAM role used to install the Agent (IAM). - Install the Agent on your EC2 instances (AWS Systems Manager). For **AWS Lambda functions**, Datadog needs permissions to: - Find your functions and read their configuration and tags (AWS Lambda). - Add and remove the Datadog Lambda layers on a function (AWS Lambda). - Identify Lambda@Edge functions so Datadog can skip them (Amazon CloudFront). For both workloads, Datadog also uses Amazon EventBridge to maintain instrumentation as your AWS resources change. {% alert level="info" %} All write actions run inside your own AWS account through the role created by the CloudFormation stack. Datadog does not hold persistent write credentials to your account. {% /alert %} The CloudFormation stack you approve during setup adds these permissions to your AWS integration IAM role, scoped to the workloads you selected. There is no policy to apply manually. The following sections list the permissions that all workloads share, followed by the permissions specific to each workload. ### Change notification permissions{% #change-notification-permissions %} These permissions let Datadog react to changes to your AWS resources. They apply to every workload: | Permission | Why Datadog needs it | | ------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `events:PutRule`, `events:PutTargets`, `events:DescribeRule`, `events:ListTargetsByRule`, `events:RemoveTargets`, `events:DeleteRule` | Set up the change notifications that let Datadog react to resource changes | | `iam:GetRole`, `iam:PassRole` | Read and pass the EventBridge cross-region role. Both are restricted to the `datadog-eventbridge-cross-region-role` role, and `iam:PassRole` is further restricted to the EventBridge service | ### Amazon EC2 permissions{% #amazon-ec2-permissions %} | Permission | Why Datadog needs it | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ec2:DescribeInstances` | Find your instances and check which ones match your rule (state, tags, OS, architecture) | | `ssm:DescribeInstanceInformation` | Confirm the SSM Agent is running before Datadog attempts anything | | `ssm:GetDocument`, `ssm:CreateDocument`, `ssm:UpdateDocument`, `ssm:UpdateDocumentDefaultVersion` | Publish the install script in your account and keep it up to date | | `ssm:SendCommand`, `ssm:ListCommandInvocations` | Run the install and confirm when it finishes | | `secretsmanager:DescribeSecret`, `secretsmanager:CreateSecret` | Store the API key so it is never passed in a command | | `iam:CreateRole`, `iam:CreateInstanceProfile`, `iam:AddRoleToInstanceProfile`, `iam:AttachRolePolicy`, `iam:PutRolePolicy`, `iam:PassRole`, `ec2:AssociateIamInstanceProfile`, and the matching `Get` and `List` reads | Give an instance the minimum access it needs in case it does not have an IAM role: reachable by Systems Manager, and able to read its own API key secret | | `iam:Detach*`, `iam:Delete*`, `iam:RemoveRoleFromInstanceProfile`, `ec2:Disassociate*`, `ec2:DescribeIamInstanceProfileAssociations` | Cleanly undo the resources above when you uninstall | | `ecs:ListClusters`, `ecs:ListContainerInstances` | Recognize Amazon Elastic Container Service (ECS) container instances so Datadog skips them (they are handled at the cluster level) | `iam:CreateRole` and `iam:PassRole` are the most sensitive grants. `iam:CreateRole` is restricted to role names matching `datadog-ec2-instrumenter/datadog-ssm-*` in your account, and `iam:PassRole` is further restricted to the Amazon EC2 service. ### AWS Lambda permissions{% #aws-lambda-permissions %} | Permission | Why Datadog needs it | | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `lambda:ListFunctions` | Find the functions in your account and region | | `cloudfront:ListDistributions` | Identify Lambda@Edge functions so Datadog skips them | | `lambda:GetFunctionConfiguration`, `lambda:ListTags` | Read a function's configuration and tags to check which functions match your rule | | `lambda:UpdateFunctionConfiguration` | Add the Datadog layers and environment variables, and remove them on uninstall | | `lambda:GetLayerVersion` | Satisfy the AWS requirement that every layer submitted during a function update be authorized, including your own unchanged layers | Lambda instrumentation needs no Secrets Manager, Systems Manager, or IAM write permissions. Function reads and updates are restricted to Lambda functions in your own account. ## How it works{% #how-it-works %} An **instrumentation rule** is the saved **Scope to** selection for an AWS account. It determines which resources Datadog automatically instruments, including resources that match the selection later. Datadog evaluates the selection, instruments each covered resource inside your own account, and keeps it instrumented: 1. You write a query describing the resources to cover, select specific resources, or add all eligible resources. 1. Datadog evaluates the rule against your account and records which resources it covers. 1. Datadog instruments each covered resource: on EC2, by installing the Agent through AWS Systems Manager; on Lambda, by adding the Datadog layers and environment variables to the function. 1. Datadog keeps the covered resources instrumented, reinstalling instrumentation that goes missing and retrying anything that failed. You approve one CloudFormation stack, one time, during initial setup. After that, instrumentation runs automatically from Datadog, with no new CloudFormation template to launch. For the full technical and security details, including the AWS resources Datadog creates, the instrumentation mechanism, and how Datadog keeps instrumentation in place, see [How Datadog instrumentation through the AWS integration works](https://docs.datadoghq.com/integrations/guide/aws-agent-installation-technical-reference.md). {% image source="https://docs.dd-static.net/images/integrations/amazon_web_services/aws-agent-installation-how-it-works.0ef5f094b54de732dee8bbad85456d18.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/integrations/amazon_web_services/aws-agent-installation-how-it-works.0ef5f094b54de732dee8bbad85456d18.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Flowchart of the AWS Agent installation process, showing which steps happen in Datadog and which run inside your AWS account." /%} ### Choose how your rule matches resources{% #choose-how-your-rule-matches-resources %} Because Datadog re-evaluates the rule over time, the query you write determines how coverage behaves as your infrastructure changes. **To cover resources as they appear**, match tags and attributes already present in your infrastructure, such as `env:prod`. Any resource that matches is instrumented, including resources created or retagged after you save the rule. Use this when you want new matching resources monitored automatically without updating the rule. **To cover a fixed set**, select the resources individually from the resource list. The rule matches only the resources you selected, so resources that appear later are not added. **When a fixed set is too large to select individually**, match a tag you control, such as `datadog:true`. Apply that tag only to the resources you want instrumented. Coverage then changes only when you change the tags, so your infrastructure-as-code determines which resources are covered. {% alert level="warning" %} Coverage works in both directions. When a resource stops matching the rule, Datadog removes instrumentation from it. A tag change made in AWS can therefore remove monitoring from a resource without anyone editing the rule in Datadog. {% /alert %} ### Best practices for rules and tags{% #best-practices-for-rules-and-tags %} **Match tags your team owns.** When a rule matches a tag that another team controls, that team can add or remove monitoring by retagging, without opening Datadog. Keeping the tag and the rule under the same ownership keeps that decision with the people who made it. **Avoid tags that change during normal operations.** Tags that change with an environment promotion, a deployment, or an autoscaling template can move resources in and out of coverage. Match on attributes that stay stable for the life of the resource. **Treat the rule as the complete configuration for the account.** Each AWS account has one rule per resource type. Every edit re-scopes all coverage for that resource type rather than adding to the existing coverage. Review the matching resources before you save. **Carve out exceptions with exclusions.** When a broad rule covers resources you want to skip, exclude them from the same rule instead of switching to an individually selected list. Exclusions keep the rule readable and preserve automatic coverage for everything else. ## What Datadog changes on a Lambda function{% #what-datadog-changes-on-a-lambda-function %} Datadog preserves your existing layers and environment variables. For Node.js and Python functions, Datadog redirects the handler to the Datadog handler and keeps your original handler in an environment variable. Datadog records exactly what it changed, so uninstalling restores your original configuration. For the specific layers, environment variables, and handler changes Datadog makes, see [What Datadog changes on a function](https://docs.datadoghq.com/integrations/guide/aws-agent-installation-technical-reference.md#what-datadog-changes-on-a-function) in the technical reference. **No Datadog API key is written into your function.** The extension authenticates with the function's own execution role through [Workload Identity Federation](https://docs.datadoghq.com/account_management/workload_identity_federation.md), so no Datadog credential is stored in your account for Lambda instrumentation. Datadog sets up this authentication for you, so there is nothing to configure. To tune what the extension collects, set the standard Datadog environment variables on the function. For the full list, see [Configure Serverless Monitoring for AWS Lambda](https://docs.datadoghq.com/serverless/aws_lambda/configuration.md). For what instrumentation collects and the Lambda monitoring features it enables, see [Serverless Monitoring for AWS Lambda](https://docs.datadoghq.com/serverless/aws_lambda.md). ## Install Datadog instrumentation{% #install-datadog-instrumentation %} {% alert level="info" %} This feature is available only to trial customers and customers who enabled it during their trial. This applies to both AWS integration setup and Fleet Automation. {% /alert %} You can start instrumentation from two entry points, depending on how much control you want over which resources are instrumented: - **AWS integration setup (instrument all eligible resources)**: When you [set up the AWS integration](https://docs.datadoghq.com/getting_started/integrations/aws.md), enable the instrumentation toggle on the [AWS integration page](https://app.datadoghq.com/integrations/amazon-web-services), next to log and resource collection. Then select the workloads you want. Datadog instruments all eligible resources for those workloads and keeps instrumenting eligible resources as they appear. - **Fleet Automation (instrument specific resources)**: Open the [AWS Install Agents page](https://app.datadoghq.com/fleet/install-agent/latest?platform=aws) at any time to select the specific resources you want. The instrumentation toggle appears during setup, with a workload selector listing **EC2 Instances**, **Lambda Functions**, and **EKS Clusters**. Only **EC2 Instances** and **Lambda Functions** can be selected: {% image source="https://docs.dd-static.net/images/integrations/amazon_web_services/aws-agent-installation-setup-toggle.10db8977458c0c434c4284453f98859f.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/integrations/amazon_web_services/aws-agent-installation-setup-toggle.10db8977458c0c434c4284453f98859f.png?auto=format&fit=max&w=850&dpr=2 2x" alt="The Install the Datadog Agent step in AWS setup, with the install toggle enabled and the Hosts (EC2) workload toggle turned on." /%} To install from the AWS Install Agents page: 1. Select the workload you want to instrument: **EC2 Instances** or **Lambda Functions**. 1. Write a query describing the resources to cover, select specific resources from the list, or add all eligible resources. For Lambda, you can narrow the list by region, runtime, and memory size. 1. Review the preview of matching resources. Resources Datadog can't instrument appear as ineligible, with the reason. 1. Review the generated CloudFormation stack, then continue to AWS and create it. Datadog prompts you for this only once. 1. Return to Datadog. Instrumentation proceeds automatically, and Datadog reports progress as resources are instrumented. ## Verify instrumentation{% #verify-instrumentation %} After instrumentation completes: - **EC2**: The newly installed Agents appear in the [Infrastructure List](https://app.datadoghq.com/infrastructure) and on the host map. Fleet Automation lists the same Agents in the Fleet View. - **Lambda**: The instrumented functions appear on the [Serverless](https://app.datadoghq.com/functions) page, and their traces appear in [APM](https://app.datadoghq.com/apm/traces). If a function is instrumented but its telemetry doesn't arrive, see [Troubleshoot AWS Lambda Monitoring](https://docs.datadoghq.com/serverless/aws_lambda/troubleshooting.md). ## Manage instrumented resources{% #manage-instrumented-resources %} Use the [AWS Install Agents page](https://app.datadoghq.com/fleet/install-agent/latest?platform=aws) in Fleet Automation to manage the resources you've instrumented through the AWS integration. From this page, you can: - View the instrumented resources and their status. - Instrument new resources in your AWS environment. - Remove instrumentation from resources you no longer want to monitor. The rule is the source of truth. To stop coverage, update the rule. If you remove instrumentation from a covered resource yourself, Datadog restores it. For EC2, manage Agent configuration and version upgrades through [Fleet Automation](https://docs.datadoghq.com/agent/fleet_automation.md). For Lambda, Datadog updates layer versions automatically. ## Remove Datadog instrumentation{% #remove-datadog-instrumentation %} To remove instrumentation, remove resources from a rule, edit the rule's query, or delete the rule. Deleting a rule removes instrumentation from everything the rule covered. - **EC2**: Datadog removes the Datadog Agent and any IAM role or instance profile it created for each instance. - **Lambda**: Datadog removes the layers it added and restores the environment variables and handler the function had beforehand. Layers and environment variables you added yourself are left in place. ## Troubleshooting{% #troubleshooting %} ### The SSM Agent is not present on an EC2 instance{% #the-ssm-agent-is-not-present-on-an-ec2-instance %} Agent installation on EC2 relies on the AWS Systems Manager (SSM) Agent, which Datadog can't install for you. Datadog flags any instance that lacks it as ineligible, including those built from custom AMIs. Install the SSM Agent on the instance, then retry. See [Working with SSM Agent](https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html) in the AWS documentation. ### A permission or IAM error occurs{% #a-permission-or-iam-error-occurs %} If instrumentation can't complete because of missing permissions, Datadog shows a notification linking to the CloudFormation resource that needs the new permission. Update your existing stack to grant the required permissions. You don't need to create a new stack. ### A Lambda function is skipped as already instrumented{% #a-lambda-function-is-skipped-as-already-instrumented %} Datadog skips any function that carries Datadog layers, a Datadog handler, or Datadog environment variables that Datadog did not apply. Skipping these functions prevents layer and configuration conflicts. To manage the function from the AWS integration instead, remove your existing Datadog instrumentation from it. Datadog then instruments the function automatically. Functions managed by [remote instrumentation](https://docs.datadoghq.com/serverless/aws_lambda/remote_instrumentation.md) are also skipped, and Datadog tells you which of the two applies. A function can be managed by only one Datadog instrumentation product. ### A Lambda function exceeds the layer limit{% #a-lambda-function-exceeds-the-layer-limit %} AWS limits a function to five layers, and Datadog adds two layers, or one for OS-only runtimes. When a function already carries enough layers that instrumentation would exceed the limit, Datadog reports it and stops rather than retrying. Remove a layer from the function to make room. Datadog then instruments the function automatically. ### A Lambda function uses a non-Datadog execution wrapper{% #a-lambda-function-uses-a-non-datadog-execution-wrapper %} Java and .NET instrumentation sets `AWS_LAMBDA_EXEC_WRAPPER`. When a function already sets that variable to something other than the Datadog wrapper, Datadog skips the function rather than overwrite your wrapper. To instrument the function through the AWS integration, remove the custom wrapper from the function. If the function needs its own wrapper, instrument it yourself instead; see [Instrumenting AWS Lambda](https://docs.datadoghq.com/serverless/aws_lambda/instrumentation.md). ### A Lambda function appears as ineligible{% #a-lambda-function-appears-as-ineligible %} Datadog marks a function ineligible when it doesn't meet the Lambda prerequisites. The most common reasons are a container image package type, an unsupported runtime or architecture, a function outside the commercial `aws` partition, and Lambda@Edge functions. Lambda@Edge replicas and the functions they replicate are both excluded. ## Further reading{% #further-reading %} Additional helpful documentation, links, and articles: - [How Datadog instrumentation through the AWS integration works](https://docs.datadoghq.com/integrations/guide/aws-agent-installation-technical-reference.md) - [AWS Integration](https://docs.datadoghq.com/integrations/amazon_web_services.md) - [AWS Manual Setup Guide](https://docs.datadoghq.com/integrations/guide/aws-manual-setup.md) - [Why install the Datadog Agent on your cloud instances?](https://docs.datadoghq.com/agent/guide/why-should-i-install-the-agent-on-my-cloud-instances.md) - [Fleet Automation](https://docs.datadoghq.com/agent/fleet_automation.md) - [Agent Configuration](https://docs.datadoghq.com/agent/configuration.md) - [Serverless Monitoring for AWS Lambda](https://docs.datadoghq.com/serverless/aws_lambda.md) - [Configure Serverless Monitoring for AWS Lambda](https://docs.datadoghq.com/serverless/aws_lambda/configuration.md) - [Instrumenting AWS Lambda](https://docs.datadoghq.com/serverless/aws_lambda/instrumentation.md) - [Troubleshoot AWS Lambda Monitoring](https://docs.datadoghq.com/serverless/aws_lambda/troubleshooting.md) - [Workload Identity Federation](https://docs.datadoghq.com/account_management/workload_identity_federation.md)