--- title: Collect Google Cloud Logs with a Pub/Sub Push Subscription description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: >- Docs > Log Management > Logs Guides > Collect Google Cloud Logs with a Pub/Sub Push Subscription --- # Collect Google Cloud Logs with a Pub/Sub Push Subscription {% alert level="warning" %} This page describes deprecated features with configuration information relevant to legacy Pub/Sub Push subscriptions, useful for troubleshooting or modifying legacy setups. Pub/Sub Push subscription is being deprecated for the following reasons: - For Google Cloud VPC, new Push subscriptions cannot be configured with external endpoints (see Google Cloud's [Supported products and limitations](https://cloud.google.com/vpc-service-controls/docs/supported-products#table_pubsub) page for more information) - The Push subscription does not provide compression or batching of events Documentation for the **Push** subscription is only maintained for troubleshooting or modifying legacy setups. Use a **Pull** subscription with the Datadog Dataflow template to forward your Google Cloud logs to Datadog instead. See Log collection on the Google Cloud integration page for instructions. {% /alert %} ## Overview{% #overview %} This guide describes how to forward logs from your Google Cloud services to Datadog through a **Push** subscription to a [Google Cloud Pub/Sub](https://cloud.google.com/pubsub/docs/overview) topic. To collect logs from applications running in GCE or GKE, you can also use the [Datadog Agent](https://docs.datadoghq.com/agent.md). **Note**: If you have a [Google Cloud VPC](https://cloud.google.com/vpc) in your Google Cloud environment, the **Push** subscription cannot access endpoints outside the VPC. ## Setup{% #setup %} ### Prerequisites{% #prerequisites %} The [Google Cloud Platform integration](https://docs.datadoghq.com/integrations/google_cloud_platform.md#installation) is successfully installed. ### Create a Cloud Pub/Sub topic{% #create-a-cloud-pubsub-topic %} 1. Go to the [Cloud Pub/Sub console](https://console.cloud.google.com/cloudpubsub/topicList) and create a new topic. {% image source="https://docs.dd-static.net/images/integrations/google_cloud_platform/create_topic_no_default_sub.20a127f97176635b27d67ffb06856872.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/integrations/google_cloud_platform/create_topic_no_default_sub.20a127f97176635b27d67ffb06856872.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Create a topic without default subscription" /%} 1. Give that topic an explicit name such as `export-logs-to-datadog` and click **Create**. **Warning**: Pub/subs are subject to [Google Cloud quotas and limitations](https://cloud.google.com/pubsub/quotas#quotas). If the number of logs you have is higher than those limitations, Datadog recommends you split your logs over several topics. See the Monitor the Log Forwarding section for information on setting up monitor notifications if you approach those limits. ### Forward logs to Datadog with a Cloud Pub/Sub subscription{% #forward-logs-to-datadog-with-a-cloud-pubsub-subscription %} 1. In the [Cloud Pub/Sub console](https://console.cloud.google.com/cloudpubsub/topicList), select **Subscriptions** in the left hand navigation. Click **Create Subscription**. 1. Create a subscription ID and select the topic you previously created. 1. Select the `Push` method and enter the following command, replacing `` with the value of a valid [Datadog API key](https://docs.datadoghq.com/account_management/api-app-keys.md): ``` https://gcp-intake.logs./api/v2/logs?dd-api-key=&dd-protocol=gcp ``` **Note**: Ensure that the `Datadog site` selector on the right of the page is set to your [Datadog site](https://docs.datadoghq.com/getting_started/site.md) before copying the command above. Configure any additional options, such as **Subscription expiration**, **Acknowledgment deadline**, **Message retention duration**, or **Dead lettering**.Under **Retry policy**, select **Retry after exponential backoff delay**.Click **Create** at the bottom. The Pub/Sub is ready to receive logs from Google Cloud Logging and forward them to Datadog. ### Export logs from Google Cloud{% #export-logs-from-google-cloud %} 1. Go to [the Google Cloud Logs Explorer page](https://console.cloud.google.com/logs/viewer) and filter the logs that need to be exported. 1. From the **Log Router** tab, select **Create Sink**. 1. Provide a name for the sink. 1. Choose **Cloud Pub/Sub** as the destination and select the pub/sub that was created for that purpose. **Note**: The pub/sub can be located in a different project. {% image source="https://docs.dd-static.net/images/integrations/google_cloud_pubsub/creating_sink2.e55ad3ab587722af906178574f88c7f2.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/integrations/google_cloud_pubsub/creating_sink2.e55ad3ab587722af906178574f88c7f2.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Export Google Cloud Pub/Sub Logs to Pub Sub" /%} 1. Click **Create Sink** and wait for the confirmation message to appear. **Note**: It is possible to create several exports from Google Cloud Logging to the same Pub/Sub with different sinks. ## Monitor the log forwarding{% #monitor-the-log-forwarding %} Pub/subs are subject to [Google Cloud quotas and limitations](https://cloud.google.com/pubsub/quotas#quotas). If the number of logs you have is higher than those limitations, Datadog recommends you split your logs over several topics, using different filters. To be automatically notified when you reach this quota, activate the [Pub/Sub metric integration](https://docs.datadoghq.com/integrations/google_cloud_pubsub.md) and set up a monitor on the metric `gcp.pubsub.subscription.num_outstanding_messages`. Filter this monitor on the subscription that exports logs to Datadog to make sure it never goes above 1000, as per the below example: {% image source="https://docs.dd-static.net/images/integrations/google_cloud_platform/log_pubsub_monitoring-v2.9ed4cca650cb30921282a1bc6404cca4.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/integrations/google_cloud_platform/log_pubsub_monitoring-v2.9ed4cca650cb30921282a1bc6404cca4.png?auto=format&fit=max&w=850&dpr=2 2x" alt="Pub Sub monitoring" /%} ### Sampling logs{% #sampling-logs %} You can optionally sample logs while querying by using the [sample function](https://cloud.google.com/logging/docs/view/logging-query-language#sample). For example, to include only 10% of the logs, use `sample(insertId, 0.1)`. ## Further Reading{% #further-reading %} - [Learn how to explore your logs](https://docs.datadoghq.com/logs/explorer.md) - [Perform Log Analytics](https://docs.datadoghq.com/logs/explorer.md#visualize) - [Learn how to process your logs](https://docs.datadoghq.com/logs/log_configuration/processors.md) - [How to send logs to Datadog while reducing data transfer fees](https://docs.datadoghq.com/logs/guide/reduce_data_transfer_fees.md)