--- title: Instrumenting Python Serverless Applications Using the Datadog Forwarder description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: >- Docs > Serverless > Serverless Monitoring Guides > Instrumenting Python Serverless Applications Using the Datadog Forwarder --- # Instrumenting Python Serverless Applications Using the Datadog Forwarder ## Overview{% #overview %} {% alert level="danger" %} If you are a new user of Datadog Serverless, follow the [instructions to instrument your Lambda functions using the Datadog Lambda Extension](https://docs.datadoghq.com/serverless/installation/python.md) instead. If you have setup Datadog Serverless with the Datadog Forwarder before Lambda offered out-of-the-box functionality, use this guide to maintain your instance. {% /alert %} ## Prerequisites{% #prerequisites %} The [Datadog Forwarder Lambda function](https://docs.datadoghq.com/serverless/forwarder.md) is required to ingest AWS Lambda traces, enhanced metrics, custom metrics, and logs. ## Configuration{% #configuration %} {% tab title="Datadog CLI" %} The Datadog CLI modifies existing Lambda functions' configurations to enable instrumentation without requiring a new deployment. It is the quickest way to get started with Datadog's serverless monitoring. You can also add the command to your CI/CD pipelines to enable instrumentation for all your serverless applications. Run the command *after* your normal serverless application deployment, so that changes made by the Datadog CLI command are not overridden. ### Install{% #install %} Install the Datadog CLI with NPM or Yarn: ```sh # NPM npm install -g @datadog/datadog-ci @datadog/datadog-ci-plugin-lambda # Yarn yarn global add @datadog/datadog-ci @datadog/datadog-ci-plugin-lambda ``` ### Instrument{% #instrument %} To instrument the function, run the following command with your [AWS credentials](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html). ```sh datadog-ci lambda instrument -f -f -r -v --forwarder ``` To fill in the placeholders: - Replace `` and `` with your Lambda function names. - Replace `` with the AWS region name. - Replace `` with the desired version of the Datadog Lambda Library. The latest version is `123`. - Replace `` with the Forwarder ARN (see the [Forwarder documentation](https://docs.datadoghq.com/serverless/forwarder.md)). For example: ```sh datadog-ci lambda instrument -f my-function -f another-function -r us-east-1 -v 123 --forwarder "arn:aws:lambda:us-east-1:000000000000:function:datadog-forwarder" ``` If your Lambda function is configured to use code signing, you must add Datadog's Signing Profile ARN (`arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc`) to your function's [Code Signing Configuration](https://docs.aws.amazon.com/lambda/latest/dg/configuration-codesigning.html#config-codesigning-config-update) before you can instrument it with the Datadog CLI. More information and additional parameters can be found in the [CLI documentation](https://docs.datadoghq.com/serverless/serverless_integrations/cli.md). {% /tab %} {% tab title="Serverless Framework" %} The [Datadog Serverless Plugin](https://docs.datadoghq.com/serverless/serverless_integrations/plugin.md) automatically adds the Datadog Lambda library to your functions using a layer, and configures your functions to send metrics, traces, and logs to Datadog through the [Datadog Forwarder](https://docs.datadoghq.com/serverless/forwarder.md). If your Lambda function is configured to use code signing, you must add Datadog's Signing Profile ARN (`arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc`) to your function's [Code Signing Configuration](https://docs.aws.amazon.com/lambda/latest/dg/configuration-codesigning.html#config-codesigning-config-update) before you install the Datadog Serverless Plugin. To install and configure the Datadog Serverless Plugin, follow these steps: 1. Install the Datadog Serverless Plugin: ``` yarn add --dev serverless-plugin-datadog ``` 1. In your `serverless.yml`, add the following: ``` plugins: - serverless-plugin-datadog ``` 1. In your `serverless.yml`, also add the following section: ``` custom: datadog: forwarderArn: # The Datadog Forwarder ARN goes here. ``` More information on the Datadog Forwarder ARN or installation can be found [here](https://docs.datadoghq.com/serverless/forwarder.md). For additional settings, see the [plugin documentation](https://docs.datadoghq.com/serverless/serverless_integrations/plugin.md). {% /tab %} {% tab title="AWS SAM" %} The [Datadog CloudFormation macro](https://docs.datadoghq.com/serverless/serverless_integrations/macro.md) automatically transforms your SAM application template to add the Datadog Lambda library to your functions using layers, and configure your functions to send metrics, traces, and logs to Datadog through the [Datadog Forwarder](https://docs.datadoghq.com/serverless/forwarder.md). ### Install{% #install %} Run the following command with your [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to deploy a CloudFormation stack that installs the macro AWS resource. You only need to install the macro **once** for a given region in your account. Replace `create-stack` with `update-stack` to update the macro to the latest version. ```sh aws cloudformation create-stack \ --stack-name datadog-serverless-macro \ --template-url https://datadog-cloudformation-template.s3.amazonaws.com/aws/serverless-macro/latest.yml \ --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM ``` The macro is now deployed and ready to use. ### Instrument{% #instrument %} To instrument your function, add the following to `template.yml` under the `Transform` section, **after** the `AWS::Serverless` transform for SAM. ```yaml Transform: - AWS::Serverless-2016-10-31 - Name: DatadogServerless Parameters: pythonLayerVersion: "123" stackName: !Ref "AWS::StackName" forwarderArn: "" service: "" # Optional env: "" # Optional ``` To fill in the placeholders: - Replace `` with Forwarder ARN (see the [Forwarder documentation](https://docs.datadoghq.com/serverless/forwarder.md)). - Replace `` and `` with your service and environment values. If your Lambda function is configured to use code signing, you must add Datadog's Signing Profile ARN (`arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc`) to your function's [Code Signing Configuration](https://docs.aws.amazon.com/lambda/latest/dg/configuration-codesigning.html#config-codesigning-config-update) before you can use the macro. More information and additional parameters can be found in the [macro documentation](https://docs.datadoghq.com/serverless/serverless_integrations/macro.md). {% /tab %} {% tab title="AWS CDK" %} The [Datadog CloudFormation macro](https://docs.datadoghq.com/serverless/serverless_integrations/macro.md) automatically transforms the CloudFormation template generated by the AWS CDK to add the Datadog Lambda library to your functions using layers, and configure your functions to send metrics, traces, and logs to Datadog through the [Datadog Forwarder](https://docs.datadoghq.com/serverless/forwarder.md). ### Install{% #install %} Run the following command with your [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to deploy a CloudFormation stack that installs the macro AWS resource. You only need to install the macro **once** for a given region in your account. Replace `create-stack` with `update-stack` to update the macro to the latest version. ```sh aws cloudformation create-stack \ --stack-name datadog-serverless-macro \ --template-url https://datadog-cloudformation-template.s3.amazonaws.com/aws/serverless-macro/latest.yml \ --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM ``` The macro is now deployed and ready to use. ### Instrument{% #instrument %} To instrument the function, add the `DatadogServerless` transform and the `CfnMapping` to your `Stack` object in your AWS CDK app. See the sample code below in Python (the usage in other language should be similar). ```python from aws_cdk import core class CdkStack(core.Stack): def __init__(self, scope: core.Construct, id: str, **kwargs) -> None: super().__init__(scope, id, **kwargs) self.add_transform("DatadogServerless") mapping = core.CfnMapping(self, "Datadog", mapping={ "Parameters": { "pythonLayerVersion": "123", "forwarderArn": "", "stackName": self.stackName, "service": "", # Optional "env": "", # Optional } }) ``` To fill in the placeholders: - Replace `` with Forwarder ARN (see the [Forwarder documentation](https://docs.datadoghq.com/serverless/forwarder.md)). - Replace `` and `` with your service and environment values. If your Lambda function is configured to use code signing, you must add Datadog's Signing Profile ARN (`arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc`) to your function's [Code Signing Configuration](https://docs.aws.amazon.com/lambda/latest/dg/configuration-codesigning.html#config-codesigning-config-update) before you can use the macro. More information and additional parameters can be found in the [macro documentation](https://docs.datadoghq.com/serverless/serverless_integrations/macro.md). {% /tab %} {% tab title="Zappa" %} ### Update settings{% #update-settings %} 1. Add the following settings to your `zappa_settings.json`: ```json { "dev": { "layers": ["arn:aws:lambda::464622532012:layer:Datadog-:"], "lambda_handler": "datadog_lambda.handler.handler", "aws_environment_variables": { "DD_LAMBDA_HANDLER": "handler.lambda_handler", "DD_TRACE_ENABLED": "true", "DD_FLUSH_TO_LOG": "true", }, } } ``` 1. Replace the placeholder ``, `` and `` in the layer ARN with appropriate values. The available `RUNTIME` options are `Python38`, `Python39`, `Python310`, `Python311`, `Python312`, `Python313`, `Python314`. The latest `VERSION` is `123`. For example: ``` # For regular regions arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Python314:123 # For us-gov regions arn:aws-us-gov:lambda:us-gov-east-1:002406178527:layer:Datadog-Python314:123 ``` 1. If your Lambda function is configured to use code signing, add Datadog's Signing Profile ARN (`arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc`) to your function's [Code Signing Configuration](https://docs.aws.amazon.com/lambda/latest/dg/configuration-codesigning.html#config-codesigning-config-update). ### Subscribe{% #subscribe %} Subscribe the Datadog Forwarder Lambda function to each of your function's log groups, to send metrics, traces, and logs to Datadog. 1. [Install the Datadog Forwarder](https://docs.datadoghq.com/serverless/forwarder.md) if you haven't. 1. [Subscribe the Datadog Forwarder to your function's log groups](https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function.md#collecting-logs-from-cloudwatch-log-group). {% /tab %} {% tab title="Chalice" %} ### Update the project{% #update-the-project %} 1. Set environment variables `DD_TRACE_ENABLED` and `DD_FLUSH_TO_LOG` to `"true"` in your `config.json`: ```json { "version": "2.0", "app_name": "hello-chalice", "stages": { "dev": { "api_gateway_stage": "api", "environment_variables": { "DD_TRACE_ENABLED": "true", "DD_FLUSH_TO_LOG": "true" } } } } ``` 1. Add `datadog_lambda` to your `requirements.txt`. 1. Register `datadog_lambda_wrapper` as a [middleware](https://aws.github.io/chalice/topics/middleware.html?highlight=handler#registering-middleware) in your `app.py`: ```python from chalice import Chalice, ConvertToMiddleware from datadog_lambda.wrapper import datadog_lambda_wrapper app = Chalice(app_name='hello-chalice') app.register_middleware(ConvertToMiddleware(datadog_lambda_wrapper)) @app.route('/') def index(): return {'hello': 'world'} ``` 1. If your Lambda function is configured to use code signing, add Datadog's Signing Profile ARN (`arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc`) to your function's [Code Signing Configuration](https://docs.aws.amazon.com/lambda/latest/dg/configuration-codesigning.html#config-codesigning-config-update). ### Subscribe{% #subscribe %} Subscribe the Datadog Forwarder Lambda function for each of your function's log groups, to send metrics, traces, and logs to Datadog. 1. [Install the Datadog Forwarder](https://docs.datadoghq.com/serverless/forwarder.md) if you haven't. 1. [Subscribe the Datadog Forwarder to your function's log groups](https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function.md#collecting-logs-from-cloudwatch-log-group). {% /tab %} {% tab title="Container Image" %} ### Install{% #install %} If you are deploying your Lambda function as a container image, you cannot use the Datadog Lambda library as a layer. Instead, you must install the Datadog Lambda library as a dependency of your function within the image. ```sh pip install datadog-lambda ``` Note that the minor version of the `datadog-lambda` package always matches the layer version. For example, `datadog-lambda v0.5.0` matches the content of layer version 5. ### Configure{% #configure %} Follow these steps to configure the function: 1. Set your image's `CMD` value to `datadog_lambda.handler.handler`. You can set this in AWS or directly in your Dockerfile. Note that the value set in AWS overrides the value in the Dockerfile if you set both. 1. Set the following environment variables in AWS: - Set `DD_LAMBDA_HANDLER` to your original handler, for example, `myfunc.handler`. - Set `DD_TRACE_ENABLED` to `true`. - Set `DD_FLUSH_TO_LOG` to `true`. Optionally add `service` and `env` tags with appropriate values to your function. ### Subscribe{% #subscribe %} Subscribe the Datadog Forwarder Lambda function to each of your functions' log groups in order to send metrics, traces, and logs to Datadog. 1. [Install the Datadog Forwarder if you haven't](https://docs.datadoghq.com/serverless/forwarder.md). 1. [Subscribe the Datadog Forwarder to your function's log groups](https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function.md#collecting-logs-from-cloudwatch-log-group). {% /tab %} {% tab title="Custom" %} ### Install{% #install %} You can either install the Datadog Lambda library as a layer (recommended) or Python package. The minor version of the `datadog-lambda` package always matches the layer version. For example, datadog-lambda v0.5.0 matches the content of layer version 5. #### Using the layer{% #using-the-layer %} [Configure the layers](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) for your Lambda function using the ARN in the following format: ``` # For us,us3,us5,ap1, ap2, and eu regions arn:aws:lambda::464622532012:layer:Datadog-: # For us-gov regions arn:aws-us-gov:lambda::002406178527:layer:Datadog-: ``` The available `RUNTIME` options are `Python38`, `Python39`, `Python310`, `Python311`, `Python312`, `Python313`, `Python314`. The latest `VERSION` is `123`. For example: ``` arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Python314:123 ``` If your Lambda function is configured to use code signing, you must add Datadog's Signing Profile ARN (`arn:aws:signer:us-east-1:464622532012:/signing-profiles/DatadogLambdaSigningProfile/9vMI9ZAGLc`) to your function's [Code Signing Configuration](https://docs.aws.amazon.com/lambda/latest/dg/configuration-codesigning.html#config-codesigning-config-update) before you can add the Datadog Lambda library as a layer. #### Using the package{% #using-the-package %} Install `datadog-lambda` and its dependencies locally to your function project folder. **Note**: `datadog-lambda` depends on `ddtrace`, which uses native extensions; therefore they must be installed and compiled in a Linux environment. For example, you can use [dockerizePip](https://github.com/UnitedIncome/serverless-python-requirements#cross-compiling) for the Serverless Framework and [–use-container](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-build.html) for AWS SAM. For more details, see [how to add dependencies to your function deployment package](https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-dependencies). ``` pip install datadog-lambda -t ./ ``` See the [latest release](https://pypi.org/project/datadog-lambda/). ### Configure{% #configure %} Follow these steps to configure the function: 1. Set your function's handler to `datadog_lambda.handler.handler`. 1. Set the environment variable `DD_LAMBDA_HANDLER` to your original handler, for example, `myfunc.handler`. 1. Set the environment variable `DD_TRACE_ENABLED` to `true`. 1. Set the environment variable `DD_FLUSH_TO_LOG` to `true`. 1. Optionally add a `service` and `env` tag with appropriate values to your function. ### Subscribe{% #subscribe %} Subscribe the Datadog Forwarder Lambda function to each of your function's log groups, to send metrics, traces, and logs to Datadog. 1. [Install the Datadog Forwarder](https://docs.datadoghq.com/serverless/forwarder.md) if you haven't. 1. [Subscribe the Datadog Forwarder to your function's log groups](https://docs.datadoghq.com/logs/guide/send-aws-services-logs-with-the-datadog-lambda-function.md#collecting-logs-from-cloudwatch-log-group). {% /tab %} ### Tag{% #tag %} Although it's optional, Datadog recommends tagging your serverless applications with the `env`, `service`, and `version` tags following the [unified service tagging documentation](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging.md#aws-lambda-functions). ## Explore{% #explore %} After configuring your function following the steps above, view your metrics, logs, and traces on the [Serverless homepage](https://app.datadoghq.com/functions). ## Monitor custom business logic{% #monitor-custom-business-logic %} If you would like to submit a custom metric or span, see the sample code below: ```python import time from ddtrace import tracer from datadog_lambda.metric import lambda_metric def lambda_handler(event, context): # add custom tags to the lambda function span, # does NOT work when X-Ray tracing is enabled current_span = tracer.current_span() if current_span: current_span.set_tag('customer.id', '123456') # submit a custom span with tracer.trace("hello.world"): print('Hello, World!') # submit a custom metric lambda_metric( metric_name='coffee_house.order_value', value=12.45, timestamp=int(time.time()), # optional, must be within last 20 mins tags=['product:latte', 'order:online'] ) return { 'statusCode': 200, 'body': get_message() } # trace a function @tracer.wrap() def get_message(): return 'Hello from serverless!' ``` For more information on custom metric submission, see [here](https://docs.datadoghq.com/serverless/custom_metrics.md?tab=python). For additional details on custom instrumentation, see the Datadog APM documentation for [custom instrumentation](https://docs.datadoghq.com/tracing/custom_instrumentation/python.md). ## Further Reading{% #further-reading %}