Use Observability Pipelines’ Google SecOps destination to send logs to Google SecOps.

The Observability Pipelines Worker uses standard Google authentication methods. See Authentication methods at Google for more information about choosing the authentication method for your use case.

Setup

Set up the Google SecOps destination and its environment variables when you set up a pipeline. The information below is configured in the pipelines UI.

Set up the destination

To set up the Worker’s Google SecOps destination:

1. Enter the identifier for your Google SecOps endpoint URL. If you leave it blank, the default is used.
   • Note: Only enter the identifier for the endpoint URL. Do not enter the actual URL.
2. Enter the customer ID for your Google SecOps instance.
3. If you have a credentials JSON file, enter the path to your credentials JSON file. The credentials file must be placed under DD_OP_DATA_DIR/config. Alternatively, you can use the GOOGLE_APPLICATION_CREDENTIALS environment variable to provide the credential path.
   • If you’re using workload identity on Google Kubernetes Engine (GKE), the GOOGLE_APPLICATION_CREDENTIALS is provided for you.
   • The Worker uses standard Google authentication methods.
4. Select JSON or Raw encoding in the dropdown menu.
5. Enter the log type. See template syntax if you want to route logs to different log types based on specific fields in your logs.
6. Optionally, toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing logs to disk, ensuring buffered logs persist through a Worker restart. See Configurable buffers for destinations for more information.
   • If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
   • To configure a buffer on your destination:
     1. Select the buffer type you want to set (Memory or Disk).
     2. Enter the buffer size and select the unit.
        • Maximum memory buffer size is 128 GB.
        • Maximum disk buffer size is 500 GB.

Note: Logs sent to the Google SecOps destination must have ingestion labels. For example, if the logs are from a A10 load balancer, it must have the ingestion label A10_LOAD_BALANCER. See Google Cloud’s Support log types with a default parser for a list of available log types and their respective ingestion labels.

Set secrets

These are the defaults used for secret identifiers and environment variables.

Note: If you enter identifiers for your secrets and then choose to use environment variables, the environment variable is the identifier entered and prepended with DD_OP. For example, if you entered PASSWORD_1 for a password identifier, the environment variable for that password is DD_OP_PASSWORD_1.

How the destination works

Event batching

A batch of events is flushed when one of these parameters is met. See event batching for more information.