Overview

Use Observability Pipelines’ syslog destinations to send logs to rsyslog or syslog-ng.

Setup

Configure the rsyslog or syslog-ng destination when you set up a pipeline. You can set up a pipeline in the UI, using the API, or with Terraform. The steps in this section are configured in the UI.

After you select the rsyslog or syslog-ng destination in the pipeline UI:

• Enter the identifier for your endpoint URL. If you leave it blank, the default is used.
  • Note: Only enter the identifiers for the syslog endpoint URL and, if applicable, the key pass. Do not enter the actual values.

See Matching log fields to syslog fields for information on how fields are matched.

Optional settings

Enable TLS

Toggle the switch to Enable TLS.

• If you are using Secrets Management, enter the identifier for the key pass. See Set secrets for the default used if the field is left blank.
• The following certificate and key files are required:
  • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
  • Notes:
    • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
    • The file must be readable by the observability-pipelines-worker group and user.

Wait time for TCP keepalive probes

Enter the number of seconds to wait before sending TCP keepalive probes on an idle connection.

Buffering

Toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing data to disk, ensuring buffered data persists through a Worker restart. See Destination buffers for more information.

• If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
• To configure a buffer on your destination:
  1. Select the buffer type you want to set (Memory or Disk).
  2. Enter the buffer size and select the unit.
     1. Maximum memory buffer size is 128 GB.
     2. Maximum disk buffer size is 500 GB.
  3. In the Behavior on full buffer dropdown menu, select whether you want to block events or drop new events when the buffer is full.

Matching log fields to syslog fields

The rsyslog and syslog-ng destinations match these log fields to the following syslog fields:

Secret defaults

These are the defaults used for secret identifiers and environment variables.

Note: If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with DD_OP. For example, if you entered PASSWORD_1 for a password identifier, the environment variable for that password is DD_OP_PASSWORD_1.

How the destination works

Event batching

The rsyslog and syslog-ng destinations do not batch events.