Use Observability Pipelines’ Kafka source to receive logs from your Kafka topics. Select and set up this source when you set up a pipeline. The Kafka source uses librdkafka.

You can also send Azure Event Hub logs to Observability Pipelines using the Kafka source.

Prerequisites

To use Observability Pipelines’ Kafka source, you need the following information available:

• The hosts and ports of the Kafka bootstrap servers, which clients should use to connect to the Kafka cluster and discover all the other hosts in the cluster.
• The appropriate TLS certificates and the password you used to create your private key, if your forwarders are globally configured to enable SSL.

Set up the source in the pipeline UI

Select and set up this source when you set up a pipeline. The information below is for the source settings in the pipeline UI.

1. Enter the group ID.
2. Enter the topic name. If there is more than one, click Add Field to add additional topics.
3. Optionally, toggle the switch to enable SASL Authentication and select the mechanism (PLAIN, SCHRAM-SHA-256, or SCHRAM-SHA-512) in the dropdown menu.
4. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.
   Note: All file paths are made relative to the configuration data directory, which is /var/lib/observability-pipelines-worker/config/ by default. See Advanced Configurations for more information. The file must be owned by the observability-pipelines-worker group and observability-pipelines-worker user, or at least readable by the group or user.
   • Server Certificate Path: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509).
   • CA Certificate Path: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
   • Private Key Path: The path to the .key private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.
5. Optionally, click Advanced and click Add Option to add additional librdkafka options.
   1. Select an option in the dropdown menu.
   2. Enter a value for that option.
   3. Check your values against the librdkafka documentation to make sure they have the correct type and are within the set range.
   4. Click Add Option to add another librdkafka option.

Set the environment variables

• The host and port of the Kafka bootstrap servers.
  • The bootstrap server that the client uses to connect to the Kafka cluster and discover all the other hosts in the cluster. The host and port must be entered in the format of host:port, such as 10.14.22.123:9092. If there is more than one server, use commas to separate them.
  • Stored as the environment variable: DD_OP_SOURCE_KAFKA_BOOTSTRAP_SERVERS.
• If you enabled SASL:
  • Kafka SASL username
    • Stored as the environment variable: DD_OP_SOURCE_KAFKA_SASL_USERNAME.
  • Kafka SASL password
    • Stored as the environment variable: DD_OP_SOURCE_KAFKA_SASL_PASSWORD.

librdkafka options

These are the available librdkafka options:

• auto.offset.reset
• auto.commit.interval.ms
• client.id
• coordinator.query.interval.ms
• enable.auto.commit
• enable.auto.offset.store
• fetch.max.bytes
• fetch.message.max.bytes
• fetch.min.bytes
• fetch.wait.max.ms
• group.instance.id
• heartbeat.interval.ms
• queued.min.messages
• session.timeout.ms
• socket.timeout.ms

See the librdkafka documentation for more information and to ensure your values have the correct type and are within range.