--- title: Instrumenting a Go Cloud Run Container with Sidecar description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: >- Docs > Serverless > Google Cloud Run > Choosing an Instrumentation Method for Containers > Sidecar Instrumentation > Instrumenting a Go Cloud Run Container with Sidecar --- # Instrumenting a Go Cloud Run Container with Sidecar {% alert level="info" %} A sample application is [available on GitHub](https://github.com/DataDog/serverless-gcp-sample-apps/tree/main/cloud-run/sidecar/go). {% /alert %} ## Setup{% #setup %} 1. **Install the Datadog Go tracer**. 1. In your main application, add the tracing library from `dd-trace-go`. ```shell go get github.com/DataDog/dd-trace-go/v2/ddtrace/tracer ``` 1. Add the following to your application code to initialize the tracer: ```go tracer.Start() defer tracer.Stop() ``` You can also add additional packages: ```shell # Enable Profiling go get github.com/DataDog/dd-trace-go/v2/profiler # Patch /net/http go get github.com/DataDog/dd-trace-go/contrib/net/http/v2 ``` For more information, see [Tracing Go Applications](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/go/) and the [Tracer README](https://github.com/DataDog/dd-trace-go?tab=readme-ov-file#installing). 1. **Install serverless-init as a sidecar**. Datadog publishes new releases of the `serverless-init` container image to Google's gcr.io, AWS's ECR, and on Docker Hub: | hub.docker.com | gcr.io | public.ecr.aws | | ----------------------- | -------------------------------- | -------------------------------------- | | datadog/serverless-init | gcr.io/datadoghq/serverless-init | public.ecr.aws/datadog/serverless-init | Images are tagged based on semantic versioning, with each new version receiving three relevant tags: - `1`, `1-alpine`: use these to track the latest minor releases, without breaking changes - `1.x.x`, `1.x.x-alpine`: use these to pin to a precise version of the library - `latest`, `latest-alpine`: use these to follow the latest version release, which may include breaking changes {% tab title="Datadog CLI" %} Setup: Install the Datadog CLI ```shell npm install -g @datadog/datadog-ci @datadog/datadog-ci-plugin-cloud-run ``` Install the [gcloud CLI](https://cloud.google.com/sdk/docs/install) and authenticate with `gcloud auth login`. Configuration: Configure the [Datadog site](https://docs.datadoghq.com/getting_started/site/) and Datadog API key, and define the service name to use in Datadog. ```shell export DATADOG_SITE="" export DD_API_KEY="" export DD_SERVICE="" ``` Instrument: If you are new to Datadog serverless monitoring, launch the Datadog CLI in interactive mode to guide your first installation for a quick start. ```shell datadog-ci cloud-run instrument -i ``` To set up the Datadog sidecar for your applications, run the `instrument` command *after* your normal deployment. You can specify multiple services to instrument by passing multiple `--service` flags. ```shell datadog-ci cloud-run instrument --project --service --region ``` You can pin to a specific image with the `--sidecar-image` flag. See the [latest releases on Docker Hub](https://hub.docker.com/r/datadog/serverless-init). Additional parameters can be found in the [CLI documentation](https://github.com/DataDog/datadog-ci/tree/master/packages/plugin-cloud-run#arguments). {% /tab %} {% tab title="Terraform" %} The [Datadog Terraform module for Google Cloud Run](https://github.com/DataDog/terraform-google-cloud-run-datadog) wraps the [`google_cloud_run_v2_service`](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_run_v2_service) resource and automatically configures your Cloud Run app for Datadog Serverless Monitoring by adding required environment variables and the serverless-init sidecar. If you don't already have Terraform set up, [install Terraform](https://developer.hashicorp.com/terraform/install), create a new directory, and make a file called `main.tf`. Then, add the following to your Terraform configuration, updating it as necessary based on your needs: ```tf variable "datadog_api_key" { description = "Your Datadog API key" type = string sensitive = true } module "my-cloud-run-app" { source = "DataDog/cloud-run-datadog/google" version = "~> 1.0" project = "my-gcp-project" name = "my-cloud-run-app" location = "us-central1" datadog_api_key = var.datadog_api_key datadog_service = "test-service" // your application service datadog_version = "0.0.0" // your code version datadog_env = "prod" // your application environment datadog_enable_logging = true deletion_protection = false template = { containers = [ { name = "main" image = "us-docker.pkg.dev/cloudrun/container/hello" resources = { limits = { cpu = "1" memory = "512Mi" } } ports = { container_port = 8080 } env = [ { name = "DD_TRACE_ENABLED", value = "true" }, ] }, ] } } ``` See the Environment Variables for more information on the configuration options available through the `env`. Ensure the container port for the main container is the same as the one exposed in your Dockerfile/service. If you haven't already, initialize your Terraform project: ```shell terraform init ``` To deploy your app, run: ```shell terraform apply ``` {% /tab %} {% tab title="YAML Deploy" %} Create a YAML file that contains your configuration. You can use the following example and adapt it to your needs: ```yaml apiVersion: serving.knative.dev/v1 kind: Service metadata: name: '' labels: cloud.googleapis.com/location: '' service: '' spec: template: metadata: labels: service: '' annotations: # The maximum number of instances that can be created for this service. # https://cloud.google.com/run/docs/reference/rest/v1/RevisionTemplate autoscaling.knative.dev/maxScale: '100' # The startup CPU boost feature for revisions provides additional CPU during # instance startup time and for 10 seconds after the instance has started. # https://cloud.google.com/run/docs/configuring/services/cpu#startup-boost run.googleapis.com/startup-cpu-boost: 'true' spec: containers: - env: - name: DD_SERVICE value: '' image: '' name: run-sidecar-1 ports: - containerPort: 8080 name: http1 resources: limits: cpu: 1000m memory: 512Mi startupProbe: failureThreshold: 1 periodSeconds: 240 tcpSocket: port: 8080 timeoutSeconds: 240 volumeMounts: - mountPath: /shared-volume name: shared-volume - env: - name: DD_SERVERLESS_LOG_PATH value: shared-volume/logs/*.log - name: DD_SITE value: '' - name: DD_ENV value: '' - name: DD_API_KEY value: '' - name: DD_SERVICE value: '' - name: DD_VERSION value: '' - name: DD_LOG_LEVEL value: debug - name: DD_LOGS_INJECTION value: 'true' - name: DD_SOURCE value: 'go' - name: DD_HEALTH_PORT value: '12345' image: gcr.io/datadoghq/serverless-init: name: serverless-init-1 resources: limits: cpu: 1000m memory: 512Mi startupProbe: failureThreshold: 3 periodSeconds: 10 tcpSocket: port: 12345 timeoutSeconds: 1 volumeMounts: - mountPath: /shared-volume name: shared-volume volumes: - emptyDir: medium: Memory sizeLimit: 512Mi name: shared-volume traffic: - latestRevision: true percent: 100 ``` This example uses the latest version of serverless-init. See the [latest releases on Docker Hub](https://hub.docker.com/r/datadog/serverless-init) to pin a specific version. See the Environment Variables for more information. In this example, the environment variables, startup health check, and volume mount are already added. If you don't want to enable logs, remove the shared volume. Ensure the container port for the main container is the same as the one exposed in your Dockerfile/service. To deploy your container, run: ```shell gcloud run services replace .yaml ``` {% /tab %} {% tab title="Other" %} After deploying your Cloud Run app, you can manually modify your app's settings to enable Datadog monitoring. 1. Create a **Volume** with `In-Memory` volume type. 1. Add a **new container** with image URL: `gcr.io/datadoghq/serverless-init:`. See the [latest releases on Docker Hub](https://hub.docker.com/r/datadog/serverless-init) to pin a specific version. 1. Add the volume mount to every container in your application. Choose a path such as `/shared-volume`, and remember it for the next step. 1. Add the following environment variables to your `serverless-init` sidecar container: - `DD_SERVICE`: A name for your service. For example, `gcr-sidecar-test`. - `DD_ENV`: A name for your environment. For example, `dev`. - `DD_SERVERLESS_LOG_PATH`: Your log path. For example, `/shared-volume/logs/*.log`. The path must begin with the mount path you defined in the previous step. - `DD_API_KEY`: Your [Datadog API key](https://app.datadoghq.com/organization-settings/api-keys). For a list of all environment variables, including additional tags, see Environment variables. {% /tab %} 1. **Set up logs**. In the previous step, you created a shared volume. You may have also set the `DD_SERVERLESS_LOG_PATH` environment variable, which defaults to `/shared-volume/logs/app.log`. In this step, configure your logging library to write logs to the file set in `DD_SERVERLESS_LOG_PATH`. In Go, we recommend writing logs in a JSON format. For example, you can use a third-party logging library such as `logrus`: ```go const LOG_FILE = "/shared-volume/logs/app.log" os.MkdirAll(filepath.Dir(LOG_FILE), 0755) logFile, err := os.OpenFile(LOG_FILE, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666) defer logFile.Close() logrus.SetOutput(logFile) logrus.SetFormatter(&logrus.JSONFormatter{}) logrus.AddHook(&dd_logrus.DDContextLogHook{}) logrus.WithContext(ctx).Info("Hello World!") ``` Datadog recommends setting the environment variable `DD_SOURCE=go` in your sidecar container to enable advanced Datadog log parsing. For more information, see [Correlating Go Logs and Traces](https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/go/). 1. **Add a service label in Google Cloud**. In your Cloud Run service's info panel, add a label with the following key and value: | Key | Value | | --------- | ---------------------------------------------------------------------------------------------- | | `service` | The name of your service. Matches the value provided as the `DD_SERVICE` environment variable. | See [Configure labels for services](https://cloud.google.com/run/docs/configuring/services/labels) in the Cloud Run documentation for instructions. **Send custom metrics**. To send custom metrics, [install the DogStatsD client](https://docs.datadoghq.com/extend/dogstatsd/?tab=go#install-the-dogstatsd-client) and [view code examples](https://docs.datadoghq.com/metrics/custom_metrics/dogstatsd_metrics_submission/?tab=go#code-examples-5). In serverless, only the *distribution* metric type is supported. ### Environment variables{% #environment-variables %} | Variable | Description | Container | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | | `DD_API_KEY` | [Datadog API key](https://app.datadoghq.com/organization-settings/api-keys) - **Required** | Sidecar container | | `DD_SITE` | [Datadog site](https://docs.datadoghq.com/getting_started/site/) - **Required** | Sidecar container | | `DD_SERVICE` | Datadog Service name. **Required** | Both containers | | `DD_SERVERLESS_LOG_PATH` | The path where the sidecar should tail logs from. Recommended to set to `/shared-volume/logs/app.log`. | Sidecar container | | `DD_LOGS_INJECTION` | When true, enrich all logs with trace data for supported loggers. See [Correlate Logs and Traces](https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/) for more information. | Application container | | `DD_VERSION` | See [Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging/). | Both containers | | `DD_ENV` | See [Unified Service Tagging](https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging/). | Both containers | | `DD_SOURCE` | Set the log source to enable a [Log Pipeline](https://docs.datadoghq.com/logs/log_configuration/pipelines) for advanced parsing. To automatically apply language-specific parsing rules, set to `go`, or use your custom pipeline. Defaults to `cloudrun`. | Sidecar container | | `DD_TAGS` | Add custom tags to your logs, metrics, and traces. Tags should be comma separated in key/value format (for example: `key1:value1,key2:value2`). | Sidecar container | **Do not set** the following environment variables in your serverless environment. They should only be set in non-serverless environments. - `DD_AGENT_HOST` - `DD_TRACE_AGENT_URL` ## Troubleshooting{% #troubleshooting %} This integration depends on your runtime having a full SSL implementation. If you are using a slim image, you may need to add the following command to your Dockerfile to include certificates: ```dockerfile RUN apt-get update && apt-get install -y ca-certificates ``` To have your Cloud Run services appear in the [software catalog](https://cloud.google.com/run/docs/configuring/services/labels), you must set the `DD_SERVICE`, `DD_VERSION`, and `DD_ENV` environment variables. ## Further reading{% #further-reading %} - [Tracing Go Applications](https://docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/go/) - [Correlating Go Logs and Traces](https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/go/)