Overview

Use Observability Pipelines’ Socket source to send logs to the Worker over a socket connection (TCP or UDP).

Prerequisites

If your forwarders are globally configured to enable SSL, you need the appropriate TLS certificates and the password for your private key.

Setup

Set up this source when you set up a pipeline. You can set up a pipeline in the UI, using the API, or with Terraform. The instructions in this section are for setting up the source in the UI.

Note: The Worker can only receive logs over TCP or UDP. If your application writes to a UNIX domain socket, see UNIX domain sockets for more information.

After you select the Socket source in the pipeline UI:

1. Enter the identifier for your socket address. If you leave it blank, the default is used.
2. In the Mode dropdown menu, select the socket type to use.
3. In the Framing dropdown menu, select how to delimit the stream of events.

   FRAMING METHOD	DESCRIPTION
   newline_delimited	Byte frames are delimited by a newline character.
   bytes	Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
   character_delimited	Byte frames are delimited by a chosen character.
   chunked_gelf	Byte frames are chunked GELF messages.
   octet_counting	Byte frames are delimited according to the octet counting format.

Optional TLS settings

Toggle the switch to Enable TLS.

• If you are using Secrets Management, enter the identifier for the key pass. See Secret defaults for the default used if the field is left blank.
• Enter the following certificate and key files:
  • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • (Optional) CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) root file in DER, PEM, or CRT (X.509).
  • (Optional) Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER, PEM, or CRT (PKCS #8) format.
  • Notes:
    • The configuration data directory /var/lib/observability-pipelines-worker/config/ is automatically appended to the file paths. See Advanced Worker Configurations for more information.
    • The file must be readable by the observability-pipelines-worker group and user.

• (Optional) Toggle Verify certificate to require connecting clients to present a valid client certificate. This enforces mutual TLS (mTLS), where the Worker verifies the identity of each connecting client.

UNIX domain sockets

The Socket source only supports receiving logs over TCP or UDP. If your application writes to a UNIX domain socket, use socat to bridge it to a TCP or UDP socket to send logs to the Worker.

Standalone bridge

Run socat alongside your application to forward from the UNIX socket to the Worker:

socat UNIX-RECV:/var/run/app.sock TCP:<OPW_HOST>

Replace <OPW_HOST> with the host IP address or the load balancer URL associated with the Observability Pipelines Worker.

Kubernetes sidecar

In Kubernetes, the Worker typically runs as a StatefulSet behind a Service, so it is not reachable over localhost. Run socat as a sidecar container in the same pod as your application, and share a volume for the socket file. For example:

volumes:
  - name: app-socket
    emptyDir: {}

initContainers:
  # Remove any stale socket file before the sidecar starts
  - name: socket-cleanup
    image: busybox:1.36
    command: ["sh", "-c", "rm -f /var/run/app/app.sock"]
    volumeMounts:
      - name: app-socket
        mountPath: /var/run/app

containers:
  # Your application container
  - name: app
    # ...
    volumeMounts:
      - name: app-socket
        mountPath: /var/run/app

  # socat sidecar: bridges the UNIX socket to the Worker's Service
  - name: socat-opw-bridge
    image: alpine/socat:1.8.0.0
    args:
      - UNIX-RECV:/var/run/app/app.sock,fork
      - TCP:<RELEASE_NAME>-observability-pipelines-worker.<NAMESPACE>.svc.cluster.local:5000
    volumeMounts:
      - name: app-socket
        mountPath: /var/run/app

# Monitor and adjust resources as necessary
    resources:
      requests:
        cpu: 10m
        memory: 16Mi
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true

Point the TCP argument at the Worker’s Kubernetes Service endpoint instead of localhost. The Worker’s StatefulSet pods aren’t guaranteed to run on every node, so the Worker pod might not be reachable at localhost. This is especially true if you have dedicated node groups for the Worker and your workloads.

Secret defaults

These are the defaults used for secret identifiers and environment variables.