--- title: Remote instrumentation for AWS Lambda description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: >- Docs > Serverless > Serverless Monitoring for AWS Lambda > Remote instrumentation for AWS Lambda --- # Remote instrumentation for AWS Lambda {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com {% alert level="danger" %} This feature is not supported for your selected Datadog site. (). {% /alert %} {% /callout %} {% image source="https://datadog-docs.imgix.net/images/serverless/lambda/svl_lambda_remote.b31a8635300975e0be9a5cf4512b6fe5.png?auto=format" alt="AWS Remote Instrumentation page in Datadog, showing a 'Scope functions to instrument using tags' box and functions available for remote instrumentation." /%} You can use remote instrumentation to quickly add instrumentation to your AWS Lambda functions and keep them instrumented securely. The *instrumenter* is a Lambda function that ensures your target functions have the Datadog Lambda extension and Datadog Lambda library added. The instrumenter also ensures that your functions remain instrumented and can send telemetry to Datadog. The instrumenter must be deployed to every region and account where you want to instrument functions. ### Prerequisites{% #prerequisites %} - The [Datadog-AWS integration](https://docs.datadoghq.com/integrations/amazon_web_services) is set up, and [resource collection](https://app.datadoghq.com/integrations/amazon-web-services?panel=resource-collection) is enabled. - Your user account has the following permissions: - Serverless AWS Instrumentation Read - Serverless AWS Instrumentation Write - Your API key must have [Remote Configuration access](https://docs.datadoghq.com/agent/remote_config/?tab=configurationyamlfile#setup) {% image source="https://datadog-docs.imgix.net/images/agent/remote_config/RC_Key_updated.b0e2299082792781d28ea9a3c1e88950.png?auto=format" alt="API Key properties with Remote Configuration capability Enable button." /%} ## Setup{% #setup %} 1. On the [Serverless > AWS Lambda](https://app.datadoghq.com/functions?cloud=aws) page, select **Instrument Functions**. 1. On the **Select AWS Region and Launch CloudFormation** modal: - Click **Select API Key** to select the Datadog API key to use to send data to your AWS account. - Use the **Select a region** drop-down to choose the region where you want to enable instrumentation. 1. Click **Launch CloudFormation Template**. You are prompted to deploy the template into your environment. Launching the template can take a few minutes. Important alert (level: info): Datadog recommends that you always test remote instrumentation on development functions before moving to staging and production. The CloudFormation stack deploys the instrumenter function, **datadog-remote-instrumenter**, into your account and region. The stack also creates a CloudTrail and some adjacent resources. 1. After the instrumenter function is deployed, select functions to instrument. You can select functions by function name, tags, or combinations of tags. See the Selecting functions section for more details. After you finish your selections, click **Next**. 1. Confirm your function selections. You can also set layer versions and toggle logging and tracing. These settings are used for all future instrumentation and remain fixed until you manually update them. Updates can take a few minutes to be applied. After you set your configuration, the instrumenter automatically instruments any functions that newly satisfy your configured targeting rules. The instrumenter also keeps your functions instrumented. If Datadog layers or environment variables on a matching function are modified outside of remote instrumentation, the instrumenter automatically re-instruments your function. ### Selecting functions{% #selecting-functions %} You can select functions by: - Function name - AWS resource tags - Tags configured in `DD_TAGS` Tag matching is case-insensitive. To enable instrumentation for multiple functions at once, you could apply a custom tag (for example, `dd_serverless_instrument:true`) to all the functions you want to instrument, and use this tag to select all of your desired functions. #### Logical operators{% #logical-operators %} You can use logical operators on your tags: - `OR`: Use a union operator to select multiple tags with the same key - **Example**: `name:(app-staging OR app-prod)` - `AND`: Use an intersection operator to select multiple tags with different keys - **Example**: `name:app-staging AND team:engineering` - `!`: Use a negation operator to exclude tags or function names - **Example**: `team:engineering AND !function_name:lambda-name` ## Skipped functions{% #skipped-functions %} Functions that have pre-existing Datadog layers or environment variables are considered manually instrumented. Manually instrumented functions are marked `manual` and skipped by the remote instrumenter to ensure there are no layer conflicts. Datadog recommends that you only instrument Lambda functions with a memory size greater than 256 MB. To skip instrumenting smaller Lambdas, make sure they are not selected. ## Verification{% #verification %} After the remote instrumenter Lambda has applied instrumentation to your functions, the status column displays an **Instrumented** status. You can also confirm your functions are instrumented by opening your AWS Console and ensuring that two layers (Datadog Lambda extension and a Datadog tracing layer) have been added to each selected function. ## Upgrading to a new version{% #upgrading-to-a-new-version %} 1. Find [datadog-remote-instrument](https://console.aws.amazon.com/cloudformation/home#/stacks?filteringText=datadog-remote-instrument) (if you didn't rename it) CloudFormation stack. 1. Find the current version of the stack template in the **Template** tab. ```yaml Mappings: Constants: DdRemoteInstrumentApp: Version: ``` Note down the value of the template version, such as `1.10.0`, in case you run into issues with the new version and need to roll back. 1. Update the stack using template URL `https://datadog-cloudformation-template.s3.amazonaws.com/aws/remote-instrument/latest.yaml`. You can also replace `latest` with a specific version, such as `1.10.0`, if needed. Check the [releases page](https://github.com/DataDog/serverless-remote-instrumentation/releases) for new features and fixes. Make sure to review the changesets before applying the update. ## Removing instrumentation{% #removing-instrumentation %} Deleting the CloudFormation stack in a region automatically removes instrumentation from all functions in that region. ## Troubleshooting{% #troubleshooting %} If you see issues related to IAM roles, ensure that you have permission to create resources for the following services: - EventBridge - S3 - CloudTrail - Lambda ## Further reading{% #further-reading %} - [Bring faster visibility into AWS Lambda functions with remote instrumentation](https://www.datadoghq.com/blog/faster-visibility-into-aws-lambda-functions)