---
title: Google Cloud Log Forwarding Setup
description: Datadog, the leading service for cloud-scale monitoring.
breadcrumbs: Docs > Log Management > Logs Guides > Google Cloud Log Forwarding Setup
---

# Google Cloud Log Forwarding Setup

## Overview{% #overview %}

Forwarding logs from your Google Cloud environment enables near real-time monitoring of the resources and activities taking place in your organization or folder. You can set up [log monitors](https://docs.datadoghq.com/monitors/types/log/) to be notified of issues, use [Cloud SIEM](https://docs.datadoghq.com/security/cloud_siem/) to detect threats, or leverage [Watchdog](https://docs.datadoghq.com/watchdog/) to identify unknown issues or anomalous behavior.

Logs are forwarded by [Google Cloud Dataflow](https://cloud.google.com/dataflow) using the [Datadog Dataflow template](https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-to-datadog). This approach offers batching and compression of your log events before forwarding them to Datadog, which is the most network-efficient way to forward your logs. You can specify which logs are forwarded with inclusion and exclusion filters.

## Setup{% #setup %}

{% collapsible-section #quick-start-log-setup %}
#### Quick Start (recommended)

#### Choose the Quick Start setup method if…{% #choose-the-quick-start-setup-method-if %}

- You are setting up log forwarding from Google Cloud for the first time.
- You prefer a UI-based workflow and want to minimize the time it takes to create and configure the necessary resources.
- You want to automate setup steps in scripts or CI/CD pipelines.

##### Prerequisite permissions{% #prerequisite-permissions %}

You must have the following permissions to complete the setup:

##### In Google Cloud:{% #in-google-cloud %}

- [roles/pubsub.admin](https://docs.cloud.google.com/iam/docs/roles-permissions/pubsub#pubsub.admin)
- [roles/storage.admin](https://docs.cloud.google.com/storage/docs/access-control/iam-roles#storage.admin)
- [roles/secretmanager.admin](https://docs.cloud.google.com/iam/docs/roles-permissions/secretmanager#secretmanager.admin)
- [roles/resourcemanager.projectIamAdmin](https://docs.cloud.google.com/resource-manager/docs/access-control-proj#resourcemanager.projectIamAdmin)
- [roles/logging.configWriter](https://docs.cloud.google.com/iam/docs/roles-permissions/logging#logging.configWriter)
- [roles/serviceusage.serviceUsageAdmin](https://docs.cloud.google.com/iam/docs/roles-permissions/serviceusage#serviceusage.serviceUsageAdmin)
- [roles/dataflow.developer](https://docs.cloud.google.com/dataflow/docs/concepts/access-control#dataflow.developer)

##### In Datadog:{% #in-datadog %}

Your Datadog user account must have either the Datadog Admin role, or, if using custom roles, the following permissions:

- `api_keys_read`
- `api_keys_write`

##### Instructions{% #instructions %}

1. In the [Google Cloud integration tile](https://app.datadoghq.com/integrations/gcp), click the **Configure Log Collection** button.
1. Select **Quick Start**. A setup script, configured with your Datadog credentials and site, is automatically generated.
1. Copy the setup script. You can run the script locally or in Google Cloud Shell:
   - Locally: May be faster, but requires your Google Cloud credentials and the [gcloud CLI](https://docs.cloud.google.com/sdk/docs/install) installed on your machine.
   - [Google Cloud Shell](https://docs.cloud.google.com/shell/docs): Click **Open Google Cloud Shell** to run the script.
1. After running the script, return to the Google Cloud integration tile.
1. In the **Select Projects** section, select the folders and projects to forward logs from. If you select a folder, logs are forwarded from all of its child projects.**Note**: Only folders and projects that you have the necessary access and permissions for appear in this section. Likewise, folders and projects without a display name do not appear.
1. In the **Dataflow Job Configuration** section, specify configuration options for the Dataflow job:
   - Select deployment settings (Google Cloud region and project to host the created resources—Pub/Sub topics and subscriptions, a log routing sink, a Secret Manager entry, a service account, a Cloud Storage bucket, and a Dataflow job)
   - Select scaling settings (number of workers and maximum workers)
   - Select performance settings (maximum number of parallel requests and batch size)
   - Select execution options
1. In the **Advanced Configuration** section, optionally specify the machine type for your Dataflow worker VMs. If no machine type is selected, Dataflow automatically chooses an appropriate machine type based on your job requirements.
1. Optionally, choose to specify inclusion and exclusion filters using Google Cloud's [logging query language](https://cloud.google.com/logging/docs/view/logging-query-language).
1. Review the steps to be executed in the **Complete Setup** section. If everything is satisfactory, click **Complete Setup**.

{% /collapsible-section %}

{% collapsible-section #terraform-log-setup %}
#### Terraform

#### Choose the Terraform setup method if…{% #choose-the-terraform-setup-method-if %}

- You manage infrastructure as code and want to keep the Datadog Google Cloud integration under version control.
- You need to configure multiple folders or projects consistently with reusable provider blocks.
- You want a repeatable, auditable deployment process that fits into your Terraform-managed environment.

##### Prerequisite permissions{% #prerequisite-permissions-1 %}

You must have the following permissions to complete the setup:

##### In Google Cloud:{% #in-google-cloud-1 %}

- [roles/pubsub.admin](https://docs.cloud.google.com/iam/docs/roles-permissions/pubsub#pubsub.admin)
- [roles/storage.admin](https://docs.cloud.google.com/storage/docs/access-control/iam-roles#storage.admin)
- [roles/secretmanager.admin](https://docs.cloud.google.com/iam/docs/roles-permissions/secretmanager#secretmanager.admin)
- [roles/resourcemanager.projectIamAdmin](https://docs.cloud.google.com/resource-manager/docs/access-control-proj#resourcemanager.projectIamAdmin)
- [roles/logging.configWriter](https://docs.cloud.google.com/iam/docs/roles-permissions/logging#logging.configWriter)
- [roles/serviceusage.serviceUsageAdmin](https://docs.cloud.google.com/iam/docs/roles-permissions/serviceusage#serviceusage.serviceUsageAdmin)
- [roles/dataflow.developer](https://docs.cloud.google.com/dataflow/docs/concepts/access-control#dataflow.developer)

##### In Datadog:{% #in-datadog-1 %}

Your Datadog user account must have either the Datadog Admin role, or, if using custom roles, the following permissions:

- `api_keys_read`
- `api_keys_write`

##### Instructions{% #instructions-1 %}

{% tab title="Datadog UI-based setup" %}

1. In the [Google Cloud integration tile](https://app.datadoghq.com/integrations/gcp), click the **Configure Log Collection** button.
1. Select **Terraform**.
1. In the **Select Projects** section, select the folders and projects to forward logs from. If you select a folder, logs are forwarded from all of its child projects.**Note**: Only folders and projects that you have the necessary access and permissions for appear in this section. Likewise, folders and projects without a display name do not appear.
1. In the **Dataflow Job Configuration** section, specify configuration options for the Dataflow job:
   - Select deployment settings (Google Cloud region and project to host the created resources—Pub/Sub topics and subscriptions, a log routing sink, a Secret Manager entry, a service account, a Cloud Storage bucket, and a Dataflow job)
   - Select scaling settings (maximum workers)
   - Select performance settings (maximum number of parallel requests and batch size)
   - Select execution options (Streaming Engine is enabled by default; read more about its [benefits](https://docs.cloud.google.com/dataflow/docs/streaming-engine#benefits))
1. In the **Advanced Configuration** section, optionally specify the machine type for your Dataflow worker VMs. If no machine type is selected, Dataflow automatically chooses an appropriate machine type based on your job requirements.
1. Optionally, choose to specify inclusion and exclusion filters using Google Cloud's [logging query language](https://cloud.google.com/logging/docs/view/logging-query-language).

{% /tab %}

{% tab title="Manual setup with Terraform module" %}
See the instructions on the [`terraform-gcp-datadog-integration`](https://github.com/GoogleCloudPlatform/terraform-gcp-datadog-integration?tab=readme-ov-file#log-collection-integration---google-cloud-platform-to-datadog) repo to set up and manage the necessary infrastructure through Terraform.
{% /tab %}

{% /collapsible-section %}

{% collapsible-section #manual-logging-setup %}
#### Manual

The instructions in this section guide you through the process of:

1. Creating a Pub/Sub [topic](https://cloud.google.com/pubsub/docs/create-topic) and [pull subscription](https://cloud.google.com/pubsub/docs/create-subscription) to receive logs from a configured log sink
1. Creating a custom Dataflow worker service account to provide [least privilege](https://cloud.google.com/iam/docs/using-iam-securely#least_privilege) to your Dataflow pipeline workers
1. Creating a [log sink](https://cloud.google.com/logging/docs/export/configure_export_v2#creating_sink) to publish logs to the Pub/Sub topic
1. Creating a Dataflow job using the [Datadog template](https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-to-datadog) to stream logs from the Pub/Sub subscription to Datadog

You have full control over which logs are sent to Datadog through the logging filters you create in the log sink, including GCE and GKE logs. See Google's [Logging query language page](https://cloud.google.com/logging/docs/view/logging-query-language) for information about writing filters. For a detailed examination of the created architecture, see [Stream logs from Google Cloud to Datadog](https://cloud.google.com/architecture/partners/stream-cloud-logs-to-datadog) in the Cloud Architecture Center.

**Note**: You must enable the **Dataflow API** to use Google Cloud Dataflow. See [Enabling APIs](https://cloud.google.com/apis/docs/getting-started#enabling_apis) in the Google Cloud documentation for more information.

To collect logs from applications running in GCE or GKE, you can also use the [Datadog Agent](https://docs.datadoghq.com/agent/).

#### 1. Create a Cloud Pub/Sub topic and subscription

1. Go to the [Cloud Pub/Sub console](https://console.cloud.google.com/cloudpubsub/topicList) and create a new topic. Select the option **Add a default subscription** to simplify the setup.

**Note**: You can also manually configure a [Cloud Pub/Sub subscription](https://console.cloud.google.com/cloudpubsub/subscription/) with the **Pull** delivery type. If you manually create your Pub/Sub subscription, leave the `Enable dead lettering` box **unchecked**. For more details, see [Unsupported Pub/Sub features](https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#unsupported-features).

{% image
   source="https://datadog-docs.imgix.net/images/integrations/google_cloud_platform/create_a_topic.44528bbd99d6bfafeec24361b18433e8.png?auto=format"
   alt="The Create a topic page in the Google Cloud Console with the Add a default subscription checkbox selected" /%}

Give that topic an explicit name such as `export-logs-to-datadog` and click **Create**.

Create an additional topic and default subscription to handle any log messages rejected by the Datadog API. The name of this topic is used within the Datadog Dataflow template as part of the path configuration for the `outputDeadletterTopic` [template parameter](https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-to-datadog#template-parameters). When you have inspected and corrected any issues in the failed messages, send them back to the original `export-logs-to-datadog` topic by running a [Pub/Sub to Pub/Sub template](https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-to-pubsub) job.

Datadog recommends creating a secret in [Secret Manager](https://console.cloud.google.com/security/secret-manager) with your valid Datadog API key value, for later use in the Datadog Dataflow template.

{% alert level="danger" %}
Cloud Pub/Subs are subject to [Google Cloud quotas and limitations](https://cloud.google.com/pubsub/quotas#quotas). If the number of logs you have exceeds those limitations, Datadog recommends you split your logs over several topics. See the Monitor the Pub/Sub Log Forwarding section for information on setting up monitor notifications if you approach those limits.
{% /alert %}

#### 2. Create a custom Dataflow worker service account

The default behavior for Dataflow pipeline workers is to use your project's [Compute Engine default service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account), which grants permissions to all resources in the project. If you are forwarding logs from a **Production** environment, you should instead create a custom worker service account with only the necessary roles and permissions, and assign this service account to your Dataflow pipeline workers.

1. Go to the [Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) page in the Google Cloud console and select your project.
1. Click **CREATE SERVICE ACCOUNT** and give the service account a descriptive name. Click **CREATE AND CONTINUE**.
1. Add the roles in the required permissions table and click **DONE**.

##### Required permissions{% #required-permissions %}

{% dl %}

{% dt %}
[Dataflow Admin](https://cloud.google.com/dataflow/docs/concepts/access-control#dataflow.admin)
{% /dt %}

{% dd %}
`roles/dataflow.admin`Allow this service account to perform Dataflow administrative tasks.
{% /dd %}

{% dt %}
[Dataflow Worker](https://cloud.google.com/dataflow/docs/concepts/access-control#dataflow.worker)
{% /dt %}

{% dd %}
`roles/dataflow.worker`Allow this service account to perform Dataflow job operations.
{% /dd %}

{% dt %}
[Pub/Sub Viewer](https://cloud.google.com/pubsub/docs/access-control#pubsub.viewer)
{% /dt %}

{% dd %}
`roles/pubsub.viewer`Allow this service account to view messages from the Pub/Sub subscription with your Google Cloud logs.
{% /dd %}

{% dt %}
[Pub/Sub Subscriber](https://cloud.google.com/pubsub/docs/access-control#pubsub.subscriber)
{% /dt %}

{% dd %}
`roles/pubsub.subscriber`Allow this service account to consume messages from the Pub/Sub subscription with your Google Cloud logs.
{% /dd %}

{% dt %}
[Pub/Sub Publisher](https://cloud.google.com/pubsub/docs/access-control#pubsub.publisher)
{% /dt %}

{% dd %}
`roles/pubsub.publisher`Allow this service account to publish failed messages to a separate subscription, which allows for analysis or resending the logs.
{% /dd %}

{% dt %}
[Secret Manager Secret Accessor](https://cloud.google.com/secret-manager/docs/access-control#secretmanager.secretAccessor)
{% /dt %}

{% dd %}
`roles/secretmanager.secretAccessor`Allow this service account to access the Datadog API key in Secret Manager.
{% /dd %}

{% dt %}
[Storage Object Admin](https://cloud.google.com/storage/docs/access-control/iam-roles/)
{% /dt %}

{% dd %}
`roles/storage.objectAdmin`Allow this service account to read and write to the Cloud Storage bucket specified for staging files.
{% /dd %}

{% /dl %}

**Note**: If you don't create a custom service account for the Dataflow pipeline workers, ensure that the default Compute Engine service account has the required permissions above.

#### 3. Export logs from Google Cloud Pub/Sub topic

1. Go to [the Logs Explorer page](https://console.cloud.google.com/logs/viewer) in the Google Cloud console.

1. From the **Log Router** tab, select **Create Sink**.

1. Provide a name for the sink.

1. Choose *Cloud Pub/Sub* as the destination and select the Cloud Pub/Sub topic that was created for that purpose. **Note**: The Cloud Pub/Sub topic can be located in a different project.

   {% image
      source="https://datadog-docs.imgix.net/images/integrations/google_cloud_pubsub/creating_sink2.e55ad3ab587722af906178574f88c7f2.png?auto=format"
      alt="Export Google Cloud Pub/Sub Logs to Pub Sub" /%}

1. Choose the logs you want to include in the sink with an optional inclusion or exclusion filter. You can filter the logs with a search query, or use the [sample function](https://cloud.google.com/logging/docs/view/logging-query-language#sample). For example, to include only 10% of the logs with a `severity` level of `ERROR`, create an inclusion filter with `severity="ERROR" AND sample(insertId, 0.1)`.

   {% image
      source="https://datadog-docs.imgix.net/images/integrations/google_cloud_platform/sink_inclusion_filter_2.88c8121e089979bd22de3b8fddf0311d.png?auto=format"
      alt="The inclusion filter for a Google Cloud logging sink with a query of severity=ERROR and sample(insertId, 0.1)" /%}

1. Click **Create Sink**.

**Note**: It is possible to create several exports from Google Cloud Logging to the same Cloud Pub/Sub topic with different sinks.

#### 4. Create and run the Dataflow job

1. Go to the [Create job from template](https://console.cloud.google.com/dataflow/createjob) page in the Google Cloud console.

1. Give the job a name and select a Dataflow regional endpoint.

1. Select `Pub/Sub to Datadog` in the **Dataflow template** dropdown, and the **Required parameters** section appears.

   1. Select the input subscription in the **Pub/Sub input subscription** dropdown.

   1. Enter the following in the **Datadog Logs API URL** field:
https://
**Note**: Ensure that the Datadog site selector on the right of the page is set to your [Datadog site](https://docs.datadoghq.com/getting_started/site/) before copying the URL above.

   1. Select the topic created to receive message failures in the **Output deadletter Pub/Sub topic** dropdown.

   1. Specify a path for temporary files in your storage bucket in the **Temporary location** field.

      {% image
         source="https://datadog-docs.imgix.net/images/integrations/google_cloud_platform/dataflow_parameters.c071c6b343c8ce7492e120cd402a41c6.png?auto=format"
         alt="Required parameters in the Datadog Dataflow template" /%}

1. Under **Optional Parameters**, check `Include full Pub/Sub message in the payload`.

1. If you created a secret in Secret Manager with your Datadog API key value as mentioned in step 1, enter the **resource name** of the secret in the **Google Cloud Secret Manager ID** field.

   {% image
      source="https://datadog-docs.imgix.net/images/integrations/google_cloud_platform/dataflow_template_optional_parameters.ebde6e04c11aec3c531e2ac0773f9544.png?auto=format"
      alt="Optional parameters in the Datadog Dataflow template with Google Cloud Secret Manager ID and Source of the API key passed fields both highlighted" /%}

See [Template parameters](https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-to-datadog#template-parameters) in the Dataflow template for details on using the other available options:

   - `apiKeySource=KMS` with `apiKeyKMSEncryptionKey` set to your [Cloud KMS](https://cloud.google.com/kms/docs) key ID and `apiKey` set to the encrypted API key
   - **Not recommended**: `apiKeySource=PLAINTEXT` with `apiKey` set to the plaintext API key

1. If you created a custom worker service account, select it in the **Service account email** dropdown.

   {% image
      source="https://datadog-docs.imgix.net/images/integrations/google_cloud_platform/dataflow_template_service_account.0f2cb379c840a153b5ab7bd9598ca6a2.png?auto=format"
      alt="Optional parameters in the Datadog Dataflow template with the service account email dropdown highlighted" /%}

1. Click **RUN JOB**.

**Note**: If you have a shared VPC, see the [Specify a network and subnetwork](https://cloud.google.com/dataflow/docs/guides/specifying-networks#shared) page in the Dataflow documentation for guidelines on specifying the `Network` and `Subnetwork` parameters.
{% /collapsible-section %}

{% collapsible-section #pub-sub-push-logging-setup %}
#### Pub/Sub Push subscription (legacy)

Collecting Google Cloud logs with a Pub/Sub Push subscription is in the process of being **deprecated**.

The above documentation for the **Push** subscription is only maintained for troubleshooting or modifying legacy setups.

Datadog recommends instead using a **Pull** subscription with the Datadog Dataflow template, as described in the Quick Start and Terraform setup sections.
{% /collapsible-section %}

See the [Stream logs from Google Cloud to Datadog](https://cloud.google.com/architecture/partners/stream-cloud-logs-to-datadog) guide in the Google Cloud architecture center for a more detailed explanation of the steps and architecture involved in log forwarding. For a deep dive into the benefits of the Pub/Sub to Datadog template, read [Stream your Google Cloud logs to Datadog with Dataflow](https://www.datadoghq.com/blog/stream-logs-datadog-dataflow-template/) in the Datadog blog.

## Validation{% #validation %}

New logging events delivered to the Cloud Pub/Sub topic appear in the [Datadog Log Explorer](https://app.datadoghq.com/logs).

**Note**: You can use the [Google Cloud Pricing Calculator](https://cloud.google.com/products/calculator) to calculate potential costs.

## Monitor the Cloud Pub/Sub log forwarding{% #monitor-the-cloud-pubsub-log-forwarding %}

The [Google Cloud Pub/Sub integration](https://docs.datadoghq.com/integrations/google-cloud-pubsub/) provides helpful metrics to monitor the status of the log forwarding:

- `gcp.pubsub.subscription.num_undelivered_messages` for the number of messages pending delivery
- `gcp.pubsub.subscription.oldest_unacked_message_age` for the age of the oldest unacknowledged message in a subscription

Use the metrics above with a [metric monitor](https://docs.datadoghq.com/monitors/types/metric/) to receive alerts for the messages in your input and deadletter subscriptions.

## Monitor the Dataflow pipeline{% #monitor-the-dataflow-pipeline %}

Use Datadog's [Google Cloud Dataflow integration](https://docs.datadoghq.com/integrations/google-cloud-dataflow/) to monitor all aspects of your Dataflow pipelines. You can see all your key Dataflow metrics on the out-of-the-box dashboard, enriched with contextual data such as information about the GCE instances running your Dataflow workloads, and your Pub/Sub throughput.

You can also use a preconfigured [Recommended Monitor](https://www.datadoghq.com/blog/datadog-recommended-monitors/) to set up notifications for increases in backlog time in your pipeline. For more information, read [Monitor your Dataflow pipelines with Datadog](https://www.datadoghq.com/blog/monitor-dataflow-pipelines-with-datadog/) in the Datadog blog.

## Further reading{% #further-reading %}

- [Stream your Google Cloud logs to Datadog with Dataflow](https://www.datadoghq.com/blog/stream-logs-datadog-dataflow-template/)
- [Getting Started with Observability in Google Cloud with Datadog](https://learn.datadoghq.com/courses/getting-started-gcp)