Google Cloud Configuration Guide for Cloud SIEM

이 페이지는 아직 한국어로 제공되지 않으며 번역 작업 중입니다. 번역에 관한 질문이나 의견이 있으시면 언제든지 저희에게 연락해 주십시오.

Overview

Datadog Cloud SIEM applies detection rules to all processed logs in Datadog to detect threats, like a targeted attack, a threat intel listed IP communicating with your systems, or an insecure resource modification. The threats are surfaced as Security Signals in the Security Signals Explorer for triaging.

Use Google Cloud Dataflow and the Datadog template to forward logs from your Google Cloud services to Datadog. This guide walks you through the following steps so that you can start detecting threats with your Google Cloud audit logs:

  1. Enable Data Access audit logs
  2. Create a Google Cloud publish/subscription (pub/sub) topic and pull subscription to receive logs from a configured log sink
  3. Create a custom Dataflow worker service account
  4. Create a log sink to publish logs to the pub/sub
  5. Create and run the Dataflow job
  6. Use Cloud SIEM to triage Security Signals

Collecting Google Cloud logs with a Pub/Sub Push subscription is in the process of being deprecated for the following reasons:

  • If you have a Google Cloud VPC, the Push subscription cannot access endpoints outside the VPC.
  • The Push subscription does not provide compression or batching of events, and so is only suitable for a low volume of logs.

Documentation for the Push subscription is only maintained for troubleshooting or modifying legacy setups. Use a Pull subscription with the Datadog Dataflow template to forward your Google Cloud logs to Datadog instead.

Enable Data Access audit logs

  1. Navigate to the IAM & Admin Console > Audit Log.
  2. Select the services for which you want to enable data access logs.
  3. In the Log Types panel, enable Admin Read, Data Read, and Data Write.
  4. Click Save.

Change default configuration for new services

If a new Google Cloud service is added, it inherits your default audit configuration.

To ensure that Data Access audit logs are captured for new Google Cloud services, modify your default audit configuration:

  1. Navigate to the IAM & Admin Console > Audit Log.
  2. Enable Admin Read, Data Read, and Data Write.
  3. Click Save.

Create a Google Cloud publish/subscription (pub/sub) system

  1. Navigate to Pub/Sub > Topics.
  2. Click Create Topic.
  3. Enter a descriptive topic name. For example, export-audit-logs-to-datadog.
  4. Leave Add a default subscription selected, which creates a subscription with default configuration values. The name of the subscription is automatically generated as your topic name with “-sub” appended to it. This subscription name is used when you create your Dataflow job later.
  5. Click Create.

Create an additional topic and subscription for outputDeadletterTopic parameter

Create an additional topic and default subscription to handle any log messages rejected by the Datadog API. This topic is used when you set up the Dataflow job later.

  1. Navigate back to Pub/Sub > Topics
  2. Click Create Topic.
  3. Enter a descriptive topic name.
  4. Leave Add a default subscription selected.
  5. Click Create.

Warning: Pub/subs are subject to Google Cloud quotas and limitations. If the number of logs you have is higher than those limitations, Datadog recommends you split your logs over several topics. See Monitor the Log Forwarding for information on how to set up a monitor to notify when you are close to those limits.

Create a secret in Secret Manager

Datadog recommends creating a secret in Secret Manager with your valid Datadog API key value. This secret is used when you set up the Dataflow job later.

  1. Navigate to Security > Secret Manager.
  2. Click Create Secret.
  3. Enter a name for the secret.
  4. Copy your Datadog API key and paste it into the Secret value section.
  5. Optionally, set the other configurations based on your use case.
  6. Click Create Secret.

Create a custom Dataflow worker service account

The default behavior for Dataflow pipeline workers is to use your project’s Compute Engine default service account, which grants permissions to all resources in the project. If you are forwarding logs from a production environment, create a custom worker service account with only the necessary roles and permissions, and assign this service account to your Dataflow pipeline workers.

Note: If you are not creating a custom service account for the Dataflow pipeline workers, ensure that the default Compute Engine service account has the required permissions below.

  1. Navigate to Google Cloud’s Service Account page.
  2. Select your project.
  3. Click Create Service Account.
  4. Enter a descriptive name for the service account.
  5. Click Create and Continue.
  6. Add the following roles:
    Required permissions
    RolePathDescription
    Dataflow Adminroles/dataflow.adminAllow this service account to perform Dataflow administrative tasks
    Dataflow Workerroles/dataflow.workerAllow this service account to perform Dataflow job operations
    Pub/Sub Viewerroles/pubsub.viewerAllow this service account to view messages from the Pub/Sub subscription with your Google Cloud logs
    Pub/Sub Subscriberroles/pubsub.subscriberAllow this service account to consume messages from the Pub/Sub subscription with your Google Cloud logs
    Pub/Sub Publisherroles/pubsub.publisherAllow this service account to publish failed messages to a separate subscription, which allows for analysis or resending the logs
    Secret Manager Secret Accessorroles/secretmanager.secretAccessorAllow this service account to access the Datadog API key in Secret Manager
    Storage Object Adminroles/storage.objectAdminAllow this service account to read and write to the Cloud Storage bucket specified for staging files
  7. Continue Continue.
  8. Click Done.

Create a log sink to publish logs to the pub/sub

  1. Navigate to Google Cloud’s Logs Explorer.
  2. Select Log Router in the left side menu.
  3. Click Create Sink.
  4. Enter a descriptive name for the sink.
  5. Click Next.
  6. In the Select Sink Service dropdown menu, select Cloud Pub/Sub topic.
    Note: The Cloud Pub/Sub topic can be located in a different project.
  7. In the Select a Cloud Pub/Sub topic, select the Pub/Sub created earlier.
  8. Click Next.
  9. Enter an inclusion filter for the logs you want to send to Datadog.
  10. Click Next.
  11. Optionally, enter an exclusion filter to exclude logs you do not want sent to Datadog.
  12. Click Create Sink.

Note: It is possible to create several exports from Google Cloud Logging to the same pub/sub with different sinks.

Create and run the Dataflow job

  1. Navigate to Google Cloud Dataflow.
  2. Click Create job from template.
  3. Enter a name for the job.
  4. Select a regional endpoint.
  5. In the Dataflow template dropdown menu, select Pub/Sub to Datadog.
  6. In Required Parameters section:
    a. In the Pub/Sub input subscription dropdown menu, select the default subscription that was created earlier when you created a new Pub/Sub system.
    b. Enter the following in the Datadog Logs API URL field:
    https://
    
    Note: Ensure that the Datadog site selector on the right of this documentation page is set to your Datadog site before copying the URL above.
    c. In the Output deadletter Pub/Sub topic field, select the additional topic you created earlier for receiving messages rejected by the Datadog API.
    d. Specify a path for temporary files in your storage bucket in the Temporary location field.
  7. If you created a secret in Secret Manager for your Datadog API key value earlier:
    a. Click Optional Parameters to see the additional fields.
    b. Enter the resource name of the secret in the Google Cloud Secret Manager ID field.
    To get the resource name, go to your secret in Secret Manager. Click on your secret. Click on the three dots under Action and select Copy resource name.
    c. Enter SECRET_MANAGER in the Source of the API key passed field.
  8. If you are not using a secret for your Datadog API key value:
    • Recommended:
      • Set Source of API key passed to KMS.
      • Set Google Cloud KMS key for the API key to your Cloud KMS key ID.
      • Set Logs API Key to the encrypted API key.
    • Not recommended: Source of API key passed set to PLAINTEXT with Logs API Key set to the plaintext API key.
  9. See Template parameters in the Dataflow template for details on other available options.
  10. If you created a custom worker service account, select it in the Service account email dropdown menu.
  11. Click Run Job.

See new logging events delivered to the Cloud Pub/Sub topic in the Datadog Log Explorer.

Use Cloud SIEM to triage Security Signals

Cloud SIEM applies out-of-the-box detection rules to all processed logs, including the Google Cloud audit logs you have just set up. When a threat is detected with a detection rule, a Security Signal is generated and can be viewed in the Security Signals Explorer.

Further reading