Google Cloud Run

Overview

Google Cloud Run is a fully managed serverless platform for deploying and scaling container-based applications. Datadog provides monitoring and log collection for Cloud Run through the Google Cloud integration. Datadog also provides a solution, now in public beta, for instrumenting your Cloud Run applications with a purpose-built Agent to enable tracing, custom metrics, and direct log collection.

This feature is in public beta. You can provide feedback through a feedback form, or through your standard support channels. During the beta period, Cloud Run monitoring and APM tracing are available without a direct cost. Existing APM customers may incur increased span ingestion and volume costs.

Getting started

Prerequisites

Make sure you have a Datadog API Key and are using a programming language supported by a Datadog tracing library.

1. Install Agent

You can install the Agent using Dockerfile or a buildpack. If you use buildpack, you must install your tracing library first.

Install Agent with Dockerfile

You can instrument your application with a Datadog Agent by adding the following lines to your Dockerfile. You may need to adjust these examples depending on your existing Dockerfile setup.

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-go
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in Cloud Run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# execute your binary application wrapped in the entrypoint. Adapt this line to your needs
CMD ["/path/to/your-go-binary"]

Instrument your application with the Datadog Agent by adding the following lines to your Dockerfile. You may need to adjust these examples depending on your existing Dockerfile setup.

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# install the python tracing library here or in requirements.txt
RUN pip install --no-cache-dir ddtrace==1.7.3

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-python
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in cloud run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# execute your binary application wrapped in the entrypoint, launched by the Datadog trace library. Adapt this line to your needs
CMD ["ddtrace-run", "python", "app.py"]

Instrument your application with the Datadog Agent by adding the following lines to your Dockerfile. You may need to adjust these examples depending on your existing Dockerfile setup.

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# install the Datadog js tracing library, either here or in package.json

npm i dd-trace@2.2.0

# enable the Datadog tracing library
ENV NODE_OPTIONS="--require dd-trace/init"

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-nodejs
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in cloud run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# execute your binary application wrapped in the entrypoint. Adapt this line to your needs
CMD ["/nodejs/bin/node", "/path/to/your/app.js"]

Instrument your application with the Datadog Agent by adding the following lines to your Dockerfile. You may need to adjust these examples depending on your existing Dockerfile setup.

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-java
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in cloud run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# execute your binary application wrapped in the entrypoint. Adapt this line to your needs
CMD ["./mvnw", "spring-boot:run"]

Instrument your application with the Datadog Agent by adding the following lines to your Dockerfile. You may need to adjust these examples depending on your existing Dockerfile setup.

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-dotnet
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in cloud run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# execute your binary application wrapped in the entrypoint. Adapt this line to your needs
CMD ["dotnet", "helloworld.dll"]

Instrument your application with the Datadog Agent by adding the following lines to your Dockerfile. You may need to adjust these examples depending on your existing Dockerfile setup.

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-ruby
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in cloud run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# execute your binary application wrapped in the entrypoint. Adapt this line to your needs
CMD ["rails", "server", "-b", "0.0.0.0"] (adapt this line to your needs)

Install Agent with buildpack

Pack Buildpacks provide a convenient way to package your container without using a Dockerfile. This example uses the Google Cloud container registry and Datadog serverless buildpack.

Note: Install the tracing library for your language before running the buildpack.

Build your application by running the following command:

pack build --builder=gcr.io/buildpacks/builder \
--buildpack from=builder \
--buildpack datadog/serverless-buildpack:latest \
gcr.io/YOUR_PROJECT/YOUR_APP_NAME

Note: Not compatible with Alpine.

2. Install tracing library

If you used buildpack, you can skip to configure your application.

Follow these instructions to install and configure the Go tracing library in your application to capture and submit traces.

Sample code for a simple Go application.

Follow these instructions to install and configure the Python tracing library in your application to capture and submit traces.

Sample code for a simple Python application.

Follow these instructions to install and configure the Node tracing library in your application to capture and submit traces.

Sample code for a simple Node.js application.

Follow these instructions to install and configure the Java tracing library in your application to capture and submit traces.

Sample code for a simple Java application.

Follow the instructions to install and configure the .NET Core tracing library and the .NET Framework tracing library.

Follow these instructions to install and configure the Ruby tracing library in your application to capture and submit traces.

Sample code for a simple Ruby application.

3. Configure your application

Once the container is built and pushed to your registry, the last step is to set the required environment variables for the Datadog Agent:

  • DD_API_KEY: Datadog API key, used to send data to your Datadog account. It should be configured as a Google Cloud Secret for privacy and safety issue.
  • DD_SITE: Datadog endpoint and website. Select your site on the right side of this page. Your site is: .
  • DD_TRACE_ENABLED: set to true to enable tracing

For more environment variables and their function, see Additional Configurations.

This command deploys the service and allows any external connection to reach it. Set DD_API_KEY as an environment variable, and set your service listening to port 80.

gcloud run deploy APP_NAME --image=gcr.io/YOUR_PROJECT/APP_NAME \
  --port=80 \
  --update-env-vars=DD_API_KEY=$DD_API_KEY \
  --update-env-vars=DD_TRACE_ENABLED=true \
  --update-env-vars=DD_SITE='datadoghq.com' \
  --allow-unauthenticated

3. Results

Once the deployment is completed, your metrics and traces are sent to Datadog. In Datadog, navigate to Infrastructure->Serverless to see your serverless metrics and traces.

Additional configurations

  • Advanced Tracing: The Datadog Agent already provides some basic tracing for popular frameworks. Follow the advanced tracing guide for more information.

  • Logs: If you use the Google Cloud integration, your logs are already being collected. Alternatively, you can set the DD_LOGS_ENABLED environment variable to true to capture application logs through the serverless instrumentation directly.

  • Custom Metrics: You can submit custom metrics using a DogStatsd client. For monitoring Cloud Run and other serverless applications, use distribution metrics. Distributions provide avg, sum, max, min, and count aggregations by default. On the Metric Summary page, you can enable percentile aggregations (p50, p75, p90, p95, p99) and also manage tags. To monitor a distribution for a gauge metric type, use avg for both the time and space aggregations. To monitor a distribution for a count metric type, use sum for both the time and space aggregations.

  • Trace Propagation: In order to propagate trace context for distributed tracing, set the DD_TRACE_PROPAGATION_STYLE environment variable to 'datadog' for your Cloud Run app and any Datadog-instrumented services downstream of it.

Environment Variables

VariableDescription
DD_API_KEYDatadog API Key - Required
DD_SITEDatadog site - Required
DD_LOGS_ENABLEDWhen true, send logs (stdout and stderr) to Datadog. Defaults to false.
DD_SERVICESee Unified Service Tagging.
DD_VERSIONSee Unified Service Tagging.
DD_ENVSee Unified Service Tagging.
DD_SOURCESee Unified Service Tagging.
DD_TAGSSee Unified Service Tagging.

OpenTelemetry

Follow these steps to send OpenTelemetry (OTel) data to Datadog.

  1. Tell OTel to export spans to Datadog serverless-init.

    // instrument.js
    
    const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
    const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
    const { Resource } = require('@opentelemetry/resources');
    const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
    const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
    
    const provider = new NodeTracerProvider({
       resource: new Resource({
           [ SemanticResourceAttributes.SERVICE_NAME ]: '<your-service-name>',
       })
    });
    
    provider.addSpanProcessor(
       new SimpleSpanProcessor(
           new OTLPTraceExporter(
               { url: 'http://localhost:4318/v1/traces' },
           ),
       ),
    );
    provider.register();
    
  2. Add OTel’s instrumentation for Express. This is akin to adding ddtrace.

    // instrument.js
    
    const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
    const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
    const { registerInstrumentations } = require('@opentelemetry/instrumentation');
    
    registerInstrumentations({
       instrumentations: [
           new HttpInstrumentation(),
           new ExpressInstrumentation(),
       ],
    });
    
  3. Add instrumentation at runtime. For instance, for Node.js, use NODE_OPTIONS.

    # Dockerfile
    
    FROM node
    
    WORKDIR /app
    COPY package.json index.js instrument.js /app/
    RUN npm i
    
    ENV NODE_OPTIONS="--require ./instrument"
    
    CMD npm run start
    
  4. Add the Datadog serverless-init.

    # Dockerfile
    
    COPY --from=datadog/serverless-init /datadog-init /app/datadog-init
    ENTRYPOINT ["/app/datadog-init"]
    
  5. Enable OTel in the Datadog serverless-init using the DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT or DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT environment variable.

    # Dockerfile
    
    ENV DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT="localhost:4318"
    

Troubleshooting

This integration depends on your runtime having a full SSL implementation. If you are using a slim image for Node, you may need to add the following command to your Dockerfile to include certificates.

RUN apt-get update && apt-get install -y ca-certificates

Further reading

Additional helpful documentation, links, and articles: