Google Cloud Run

Overview

Google Cloud Run is a fully managed serverless platform for deploying and scaling container-based applications. Datadog provides monitoring and log collection for Cloud Run through the Google Cloud integration. Datadog also provides a solution, now in public beta, for instrumenting your Cloud Run applications with a purpose-built Agent to enable tracing, custom metrics, and direct log collection.

Getting started

Prerequisites

Make sure you have a Datadog API Key and are using a programming language supported by a Datadog tracing library.

1. Install Agent

You can install the Agent using Dockerfile or a buildpack. If you use buildpack, you must install your tracing library first.

Install Agent with Dockerfile

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-go
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in Cloud Run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# execute your binary application wrapped in the entrypoint. Adapt this line to your needs
CMD ["/path/to/your-go-binary"]

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# install the python tracing library here or in requirements.txt
RUN pip install --no-cache-dir ddtrace==1.7.3

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-python
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in cloud run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# execute your binary application wrapped in the entrypoint, launched by the Datadog trace library. Adapt this line to your needs
CMD ["ddtrace-run", "python", "app.py"]

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# install the Datadog js tracing library, either here or in package.json

npm i dd-trace@2.2.0

# enable the Datadog tracing library
ENV NODE_OPTIONS="--require dd-trace/init"

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-nodejs
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in cloud run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# execute your binary application wrapped in the entrypoint. Adapt this line to your needs
CMD ["/nodejs/bin/node", "/path/to/your/app.js"]

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-java
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in cloud run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# execute your binary application wrapped in the entrypoint. Adapt this line to your needs
CMD ["./mvnw", "spring-boot:run"]

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-dotnet
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in cloud run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# execute your binary application wrapped in the entrypoint. Adapt this line to your needs
CMD ["dotnet", "helloworld.dll"]

# copy the Datadog `serverless-init` into your Docker image
COPY --from=datadog/serverless-init /datadog-init /app/datadog-init

# optionally add Datadog tags
ENV DD_SERVICE=datadog-demo-run-ruby
ENV DD_ENV=datadog-demo
ENV DD_VERSION=1

# this env var is needed for trace propagation to work properly in cloud run.
# ensure to set this variable for all Datadog-instrumented downstream services.
ENV DD_TRACE_PROPAGATION_STYLE=datadog

# change the entrypoint to wrap your application into the Datadog serverless-init process
ENTRYPOINT ["/app/datadog-init"]

# execute your binary application wrapped in the entrypoint. Adapt this line to your needs
CMD ["rails", "server", "-b", "0.0.0.0"] (adapt this line to your needs)

Install Agent with buildpack

Pack Buildpacks provide a convenient way to package your container without using a Dockerfile. This example uses the Google Cloud container registry and Datadog serverless buildpack.

Note: Install the tracing library for your language before running the buildpack.

Build your application by running the following command:

pack build --builder=gcr.io/buildpacks/builder \
--buildpack from=builder \
--buildpack datadog/serverless-buildpack:latest \
gcr.io/YOUR_PROJECT/YOUR_APP_NAME

Note: Not compatible with Alpine.

2. Install tracing library

If you used buildpack, you can skip to configure your application.

3. Configure your application

Once the container is built and pushed to your registry, the last step is to set the required environment variables for the Datadog Agent:

• DD_API_KEY: Datadog API key, used to send data to your Datadog account. It should be configured as a Google Cloud Secret for privacy and safety issue.
• DD_SITE: Datadog endpoint and website. Select your site on the right side of this page. Your site is: .
• DD_TRACE_ENABLED: set to true to enable tracing

For more environment variables and their function, see Additional Configurations.

This command deploys the service and allows any external connection to reach it. Set DD_API_KEY as an environment variable, and set your service listening to port 80.

gcloud run deploy APP_NAME --image=gcr.io/YOUR_PROJECT/APP_NAME \
  --port=80 \
  --update-env-vars=DD_API_KEY=$DD_API_KEY \
  --update-env-vars=DD_TRACE_ENABLED=true \
  --update-env-vars=DD_SITE='datadoghq.com' \
  --allow-unauthenticated

3. Results

Once the deployment is completed, your metrics and traces are sent to Datadog. In Datadog, navigate to Infrastructure->Serverless to see your serverless metrics and traces.

Additional configurations

• Advanced Tracing: The Datadog Agent already provides some basic tracing for popular frameworks. Follow the advanced tracing guide for more information.

• Logs: If you use the Google Cloud integration, your logs are already being collected. Alternatively, you can set the DD_LOGS_ENABLED environment variable to true to capture application logs through the serverless instrumentation directly.

• Custom Metrics: You can submit custom metrics using a DogStatsd client. For monitoring Cloud Run and other serverless applications, use distribution metrics. Distributions provide avg, sum, max, min, and count aggregations by default. On the Metric Summary page, you can enable percentile aggregations (p50, p75, p90, p95, p99) and also manage tags. To monitor a distribution for a gauge metric type, use avg for both the time and space aggregations. To monitor a distribution for a count metric type, use sum for both the time and space aggregations.

• Trace Propagation: In order to propagate trace context for distributed tracing, set the DD_TRACE_PROPAGATION_STYLE environment variable to 'datadog' for your Cloud Run app and any Datadog-instrumented services downstream of it.

Environment Variables

OpenTelemetry

Follow these steps to send OpenTelemetry (OTel) data to Datadog.

1. Tell OTel to export spans to Datadog serverless-init.

   // instrument.js
   
   const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
   const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
   const { Resource } = require('@opentelemetry/resources');
   const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
   const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
   
   const provider = new NodeTracerProvider({
      resource: new Resource({
          [ SemanticResourceAttributes.SERVICE_NAME ]: '<your-service-name>',
      })
   });
   
   provider.addSpanProcessor(
      new SimpleSpanProcessor(
          new OTLPTraceExporter(
              { url: 'http://localhost:4318/v1/traces' },
          ),
      ),
   );
   provider.register();
   

2. Add OTel’s instrumentation for Express. This is akin to adding ddtrace.

   // instrument.js
   
   const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');
   const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
   const { registerInstrumentations } = require('@opentelemetry/instrumentation');
   
   registerInstrumentations({
      instrumentations: [
          new HttpInstrumentation(),
          new ExpressInstrumentation(),
      ],
   });
   

3. Add instrumentation at runtime. For instance, for Node.js, use NODE_OPTIONS.

   # Dockerfile
   
   FROM node
   
   WORKDIR /app
   COPY package.json index.js instrument.js /app/
   RUN npm i
   
   ENV NODE_OPTIONS="--require ./instrument"
   
   CMD npm run start
   

4. Add the Datadog serverless-init.

   # Dockerfile
   
   COPY --from=datadog/serverless-init /datadog-init /app/datadog-init
   ENTRYPOINT ["/app/datadog-init"]
   

5. Enable OTel in the Datadog serverless-init using the DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT or DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT environment variable.

   # Dockerfile
   
   ENV DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT="localhost:4318"
   

Troubleshooting

This integration depends on your runtime having a full SSL implementation. If you are using a slim image for Node, you may need to add the following command to your Dockerfile to include certificates.

RUN apt-get update && apt-get install -y ca-certificates

Further reading

Additional helpful documentation, links, and articles:

Collect traces, logs, and custom metrics from GCR servicesBLOG