Instrumenting Python Serverless Applications

The Datadog CLI modifies existing Lambda functions’ configurations to enable instrumentation without requiring a new deployment. It is the quickest way to get started with Datadog’s serverless monitoring.

1. Install the Datadog CLI client

   npm install -g @datadog/datadog-ci
   

2. If you are new to Datadog serverless monitoring, launch the Datadog CLI in the interactive mode to guide your first installation for a quick start, and you can ignore the remaining steps. To permanently install Datadog for your production applications, skip this step and follow the remaining ones to run the Datadog CLI command in your CI/CD pipelines after your normal deployment.

   datadog-ci lambda instrument -i
   

3. Configure the AWS credentials

   Datadog CLI requires access to the AWS Lambda service, and depends on the AWS JavaScript SDK to resolve the credentials. Ensure your AWS credentials are configured using the same method you would use when invoking the AWS CLI.

4. Configure the Datadog site

   export DATADOG_SITE="<DATADOG_SITE>"
   

   Replace <DATADOG_SITE> with (ensure the correct SITE is selected on the right).

5. Configure the Datadog API key

   Datadog recommends saving the Datadog API key in AWS Secrets Manager for security and easy rotation. The key needs to be stored as a plaintext string (not a JSON blob). Ensure your Lambda functions have the required secretsmanager:GetSecretValue IAM permission.

   export DATADOG_API_KEY_SECRET_ARN="<DATADOG_API_KEY_SECRET_ARN>"
   

   For quick testing purposes, you can also set the Datadog API key in plaintext:

   export DATADOG_API_KEY="<DATADOG_API_KEY>"
   

6. Instrument your Lambda functions

   Note: Instrument your Lambda functions in a dev or staging environment first. If the instrumentation result is unsatisfactory, run uninstrument with the same arguments to revert the changes.

   To instrument your Lambda functions, run the following command.

   datadog-ci lambda instrument -f <functionname> -f <another_functionname> -r <aws_region> -v 92 -e 56
   

   To fill in the placeholders:

   • Replace <functionname> and <another_functionname> with your Lambda function names. Alternatively, you can use --functions-regex to automatically instrument multiple functions whose names match the given regular expression.
   • Replace <aws_region> with the AWS region name.

   Additional parameters can be found in the CLI documentation.

The Datadog Serverless Plugin automatically configures your functions to send metrics, traces, and logs to Datadog through the Datadog Lambda Extension.

To install and configure the Datadog Serverless Plugin, follow these steps:

1. Install the Datadog Serverless Plugin:

   serverless plugin install --name serverless-plugin-datadog
   

2. Update your serverless.yml:

   custom:
     datadog:
       site: <DATADOG_SITE>
       apiKeySecretArn: <DATADOG_API_KEY_SECRET_ARN>
   

   To fill in the placeholders:

   • Replace <DATADOG_SITE> with (ensure the correct SITE is selected on the right).
   • Replace <DATADOG_API_KEY_SECRET_ARN> with the ARN of the AWS secret where your Datadog API key is securely stored. The key needs to be stored as a plaintext string (not a JSON blob). The secretsmanager:GetSecretValue permission is required. For quick testing, you can instead use apiKey and set the Datadog API key in plaintext.

   For more information and additional settings, see the plugin documentation.

The Datadog CloudFormation macro automatically transforms your SAM application template to install Datadog on your functions using Lambda layers, and configures your functions to send metrics, traces, and logs to Datadog through the Datadog Lambda Extension.

1. Install the Datadog CloudFormation macro

   Run the following command with your AWS credentials to deploy a CloudFormation stack that installs the macro AWS resource. You only need to install the macro once for a given region in your account. Replace create-stack with update-stack to update the macro to the latest version.

   aws cloudformation create-stack \
     --stack-name datadog-serverless-macro \
     --template-url https://datadog-cloudformation-template.s3.amazonaws.com/aws/serverless-macro/latest.yml \
     --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM
   

   The macro is now deployed and ready to use.

2. Instrument your Lambda functions

   Add the DatadogServerless transform after the AWS::Serverless transform under the Transform section in your template.yml file for SAM.

   Transform:
     - AWS::Serverless-2016-10-31
     - Name: DatadogServerless
       Parameters:
         stackName: !Ref "AWS::StackName"
         pythonLayerVersion: 92
         extensionLayerVersion: 56
         site: "<DATADOG_SITE>"
         apiKeySecretArn: "<DATADOG_API_KEY_SECRET_ARN>"
   

   To fill in the placeholders:

   • Replace <DATADOG_SITE> with (ensure the correct SITE is selected on the right).
   • Replace <DATADOG_API_KEY_SECRET_ARN> with the ARN of the AWS secret where your Datadog API key is securely stored. The key needs to be stored as a plaintext string (not a JSON blob). The secretsmanager:GetSecretValue permission is required. For quick testing, you can use apiKey instead and set the Datadog API key in plaintext.

   More information and additional parameters can be found in the macro documentation.

The Datadog CDK Construct automatically installs Datadog on your functions using Lambda Layers, and configures your functions to send metrics, traces, and logs to Datadog through the Datadog Lambda Extension.

1. Install the Datadog CDK constructs library

   # For AWS CDK v1
   pip install datadog-cdk-constructs
   
   # For AWS CDK v2
   pip install datadog-cdk-constructs-v2
   

2. Instrument your Lambda functions

   # For AWS CDK v1
   from datadog_cdk_constructs import Datadog
   
   # For AWS CDK v2
   from datadog_cdk_constructs_v2 import Datadog
   
   datadog = Datadog(self, "Datadog",
       python_layer_version=92,
       extension_layer_version=56,
       site="<DATADOG_SITE>",
       api_key_secret_arn="<DATADOG_API_KEY_SECRET_ARN>",
     )
   datadog.add_lambda_functions([<LAMBDA_FUNCTIONS>])
   

   To fill in the placeholders:

   • Replace <DATADOG_SITE> with (ensure the correct SITE is selected on the right).
   • Replace <DATADOG_API_KEY_SECRET_ARN> with the ARN of the AWS secret where your Datadog API key is securely stored. The key needs to be stored as a plaintext string (not a JSON blob). The secretsmanager:GetSecretValue permission is required. For quick testing, you can use apiKey instead and set the Datadog API key in plaintext.

   More information and additional parameters can be found on the Datadog CDK documentation.

1. Install the Datadog Lambda Library

   If you are deploying your Lambda function as a container image, you cannot use the Datadog Lambda library as a Lambda Layer. Instead, you must install the Datadog Lambda library as a dependency of your function within the image.

   pip install datadog-lambda
   

   Note that the minor version of the datadog-lambda package always matches the layer version. For example, datadog-lambda v0.5.0 matches the content of layer version 5.

2. Install the Datadog Lambda Extension

   Add the Datadog Lambda Extension to your container image by adding the following to your Dockerfile:

   COPY --from=public.ecr.aws/datadog/lambda-extension:<TAG> /opt/. /opt/
   

   Replace <TAG> with either a specific version number (for example, 56) or with latest. Alpine is also supported with specific version numbers (such as 56-alpine) or with latest-alpine. You can see a complete list of possible tags in the Amazon ECR repository.

3. Redirect the handler function

   • Set your image’s CMD value to datadog_lambda.handler.handler. You can set this in AWS or directly in your Dockerfile. Note that the value set in AWS overrides the value in the Dockerfile if you set both.
   • Set the environment variable DD_LAMBDA_HANDLER to your original handler, for example, myfunc.handler.

   Note: If you are using a third-party security or monitoring tool that is incompatible with the Datadog handler redirection, you can apply the Datadog wrapper in your function code instead.

4. Configure the Datadog site, API key, and tracing in your Dockerfile

   • Set the environment variable DD_SITE to (ensure the correct SITE is selected on the right).
   • Set the environment variable DD_API_KEY_SECRET_ARN with the ARN of the AWS secret where your Datadog API key is securely stored. The key needs to be stored as a plaintext string (not a JSON blob). The secretsmanager:GetSecretValue permission is required. For quick testing, you can use DD_API_KEY instead and set the Datadog API key in plaintext.
   • Set the environment variable DD_TRACE_ENABLED to true.

To use Terraform, wrap your Lambda function handler with the Datadog Lambda library. Set your function’s handler to the Datadog handler function, datadog_lambda.handler.handler.

Use this format for your Terraform resource:

resource "aws_lambda_function" "lambda" {
  "function_name" = ...
  ...

  # Remember sure to choose the right layers based on your Lambda architecture and AWS regions

  layers = [
    <DATADOG_TRACER_ARN>,
    <DATADOG_EXTENSION_ARN>
  ]

  handler = "datadog_lambda.handler.handler"

  environment {
    variables = {
      DD_SITE                     = <DATADOG_SITE>
      DD_API_KEY_SECRET_ARN       = <API_KEY>
      DD_LAMBDA_HANDLER           = <LAMBDA_HANDLER>
    }
  }
}

Fill in variables accordingly:

1. Replace <DATADOG_TRACER_ARN> with the ARN of the appropriate Datadog tracer for your region and architecture:

   AWS REGIONS	ARCHITECTURE	LAYER
   Commercial	x86_64	arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-<RUNTIME>:92
   	arm64	arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-<RUNTIME>-ARM:
   GovCloud	x86_64	arn:aws:lambda:<AWS_REGION>:002406178527:layer:Datadog-<RUNTIME>:92
   	arm64	arn:aws:lambda:<AWS_REGION>:002406178527:layer:Datadog-<RUNTIME>-ARM:92

   In the ARN, replace <AWS_REGION> with a valid AWS region, such as us-east-1. Replace <RUNTIME> with one of the following: Python38, Python39, Python310, Python311, Python312.

2. Replace <DATADOG_EXTENSION_ARN> with the ARN of the appropriate Datadog Lambda Extension for your region and architecture:

   AWS REGIONS	ARCHITECTURE	LAYER
   Commercial	x86_64	arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:56
   	arm64	arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:56
   GovCloud	x86_64	arn:aws-us-gov:lambda:<AWS_REGION>:002406178527:layer:Datadog-Extension:56
   	arm64	arn:aws-us-gov:lambda:<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:56

   In the ARN, replace <AWS_REGION> with a valid AWS region, such as us-east-1.

3. Replace <DATADOG_SITE> with (ensure the correct SITE is selected on the right).

4. Replace <API_KEY> with the ARN of the AWS secret where your Datadog API key is securely stored. The key needs to be stored as a plaintext string (not a JSON blob). The secretsmanager:GetSecretValue permission is required. For quick testing, use DD_API_KEY instead of DD_API_KEY_SECRET_ARN and set the value to your Datadog API key in plaintext.

5. Replace <LAMBDA_HANDLER> with your original handler. For example, myfunc.handler.

Full example

resource "aws_lambda_function" "lambda" {
  "function_name" = ...
  ...

  # Remember sure to choose the right layers based on your Lambda architecture and AWS regions

  layers = [
    "arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Python312:92",
    "arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Extension:56"
  ]

  handler = "datadog_lambda.handler.handler"

  environment {
    variables = {
      DD_SITE                     = "datadoghq.com"
      DD_API_KEY_SECRET_ARN       = "arn:aws..."
      DD_LAMBDA_HANDLER           = "myfunc.handler"

    }
  }
}

1. Install the Datadog Lambda library

   The Datadog Lambda Library can be imported either as a layer (recommended) OR as a Python package.

   The minor version of the datadog-lambda package always matches the layer version. For example, datadog-lambda v0.5.0 matches the content of layer version 5.

   • Option A: Configure the layers for your Lambda function using the ARN in the following format:

     # Use this format for x86-based Lambda deployed in AWS commercial regions
     arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-<RUNTIME>:92
     
     # Use this format for arm64-based Lambda deployed in AWS commercial regions
     arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-<RUNTIME>-ARM:92
     
     # Use this format for x86-based Lambda deployed in AWS GovCloud regions
     arn:aws-us-gov:lambda:<AWS_REGION>:002406178527:layer:Datadog-<RUNTIME>:92
     
     # Use this format for arm64-based Lambda deployed in AWS GovCloud regions
     arn:aws-us-gov:lambda:<AWS_REGION>:002406178527:layer:Datadog-<RUNTIME>-ARM:92
     

     Replace <AWS_REGION> with a valid AWS region, such as us-east-1. The available <RUNTIME> options are: Python38, Python39, Python310, Python311, Python312.

   • Option B: If you cannot use the prebuilt Datadog Lambda layer, alternatively install the datadog-lambda package and its dependencies locally to your function project folder using your favorite Python package manager, such as pip.

     pip install datadog-lambda -t ./
     

     Note: datadog-lambda depends on ddtrace, which uses native extensions; therefore it must be installed and compiled in a Linux environment on the right architecture (x86_64 or arm64). For example, you can use dockerizePip for the Serverless Framework and –use-container for AWS SAM. For more details, see how to add dependencies to your function deployment package.

     See the latest release.

2. Install the Datadog Lambda Extension

   Configure the layers for your Lambda function using the ARN in the following format:

   # Use this format for x86-based Lambda deployed in AWS commercial regions
   arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension:56
   
   # Use this format for arm64-based Lambda deployed in AWS commercial regions
   arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Extension-ARM:56
   
   # Use this format for x86-based Lambda deployed in AWS GovCloud regions
   arn:aws-us-gov:lambda:<AWS_REGION>:002406178527:layer:Datadog-Extension:56
   
   # Use this format for arm64-based Lambda deployed in AWS GovCloud regions
   arn:aws-us-gov:lambda:<AWS_REGION>:002406178527:layer:Datadog-Extension-ARM:56
   

   Replace <AWS_REGION> with a valid AWS region, such as us-east-1.

3. Redirect the handler function

   • Set your function’s handler to datadog_lambda.handler.handler.
   • Set the environment variable DD_LAMBDA_HANDLER to your original handler, for example, myfunc.handler.

   Note: If you are using a third-party security or monitoring tool that is incompatible with the Datadog handler redirection, you can apply the Datadog wrapper in your function code instead.

4. Configure the Datadog site, API key, and tracing

   • Set the environment variable DD_SITE to (ensure the correct SITE is selected on the right).
   • Set the environment variable DD_API_KEY_SECRET_ARN with the ARN of the AWS secret where your Datadog API key is securely stored. The key needs to be stored as a plaintext string, instead of being inside a json blob. The secretsmanager:GetSecretValue permission is required. For quick testings, you can use DD_API_KEY instead and set the Datadog API key in plaintext.
   • Set the environment variable DD_TRACE_ENABLED to true.

5. (AWS Chalice only) Register the middleware

   If you are using AWS Chalice, you must install datadog-lambda using pip, and register datadog_lambda_wrapper as a middleware in your app.py:

   from chalice import Chalice, ConvertToMiddleware
   from datadog_lambda.wrapper import datadog_lambda_wrapper
   
   app = Chalice(app_name='hello-chalice')
   
   app.register_middleware(ConvertToMiddleware(datadog_lambda_wrapper))
   
   @app.route('/')
   def index():
       return {'hello': 'world'}