A source is where data is collected and sent to Observability Pipelines. The source component in a configuration defines how Observability Pipelines collects or receives data from the source.
The namespace to use for logs. This overrides the global setting.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
Whether or not end-to-end acknowledgements are enabled for this source.
default: null
connection_string
required
string
URI for the AMQP server.
The URI has the format of
amqp://<user>:<password>@<host>:<port>/<vhost>?timeout=<seconds>.
The default vhost can be specified by using a value of %2f.
To connect over TLS, a scheme of amqps can be specified instead. For example,
amqps://.... Additional TLS settings, such as client certificate verification, can be
configured under the tls section.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
If empty, the URI is automatically discovered based on the latest version detected.
By default:
The version 4 endpoint base URI is stored in the environment variable ECS_CONTAINER_METADATA_URI_V4.
The version 3 endpoint base URI is stored in the environment variable ECS_CONTAINER_METADATA_URI.
The version 2 endpoint base URI is 169.254.170.2/v2/.
version
optional
<oneOf>
The version of the task metadata endpoint to use.
If empty, the version is automatically discovered based on environment variables.
By default:
Version 4 is used if the environment variable ECS_CONTAINER_METADATA_URI_V4 is defined.
Version 3 is used if the environment variable ECS_CONTAINER_METADATA_URI_V4 is not defined, but the
environment variable ECS_CONTAINER_METADATA_URIis defined.
Version 2 is used if neither of the environment variables ECS_CONTAINER_METADATA_URI_V4 or
ECS_CONTAINER_METADATA_URI are defined.
v2
optional
v2
Version 2.
More information about version 2 of the task metadata endpoint can be found here.
v3
optional
v3
Version 3.
More information about version 3 of the task metadata endpoint can be found here.
v4
optional
v4
Version 4.
More information about version 4 of the task metadata endpoint can be found here.
An optional access key to authenticate requests against.
DEPRECATED: AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If
configured, access_key should be set to the same value. Otherwise, all requests are allowed.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
access_keys
optional
array,null
An optional list of access keys to authenticate requests against.
AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If
configured, access_keys should be set to the same value. Otherwise, all requests are allowed.
store_access_key
required
boolean
Whether or not to store the AWS Firehose Access Key in event secrets.
If set to true, when incoming requests contains an access key sent by AWS Firehose, it is kept in the
event secrets as "aws_kinesis_firehose_access_key".
record_compression
optional
<oneOf>
The compression scheme to use for decompressing records within the Firehose message.
Some services, like AWS CloudWatch Logs, compresses the events with gzip,
before sending them AWS Kinesis Firehose. This option can be used to automatically decompress
them before forwarding them to the next component.
Note that this is different from Content encoding option of the
Firehose HTTP endpoint destination. That option controls the content encoding of the entire HTTP request.
auto
optional
auto
Automatically attempt to determine the compression scheme.
The compression scheme of the object is determined by looking at its file signature, also known
as magic bytes.
If the record fails to decompress with the discovered format, the record is forwarded as is.
Thus, if you know the records are always gzip encoded (for example, if they are coming from AWS CloudWatch Logs),
set gzip in this field so that any records that are not-gzipped are rejected.
none
optional
none
Uncompressed.
gzip
optional
gzip
GZIP.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
The compression scheme used for decompressing objects retrieved from S3.
auto
optional
auto
Automatically attempt to determine the compression scheme.
The compression scheme of the object is determined from its Content-Encoding and
Content-Type metadata, as well as the key suffix (for example, .gz).
It is set to none if the compression scheme cannot be determined.
none
optional
none
Uncompressed.
gzip
optional
gzip
GZIP.
zstd
optional
zstd
ZSTD.
strategy
optional
<oneOf>
The strategy to use to consume objects from S3.
sqs
optional
sqs
Consumes objects by processing bucket notification events sent to an AWS SQS queue.
sqs
optional
<oneOf>
Configuration options for SQS.
Option 1
optional
object
SQS configuration options.
queue_url
required
uri
The URL of the SQS queue to poll for bucket notifications.
poll_secs
optional
integer
How long to wait while polling the queue for new messages, in seconds.
Generally, this should not be changed unless instructed to do so, as if messages are available,
they are always consumed, regardless of the value of poll_secs.
visibility_timeout_secs
optional
integer
The visibility timeout to use for messages, in seconds.
This controls how long a message is left unavailable after it is received. If a message is received, and
takes longer than visibility_timeout_secs to process and delete the message from the queue, it is made available again for another consumer.
This can happen if there is an issue between consuming a message and deleting it.
delete_message
optional
boolean
Whether to delete the message once it is processed.
It can be useful to set this to false for debugging or during the initial setup.
client_concurrency
optional
integer,null
Number of concurrent tasks to create for polling the queue for messages.
Defaults to the number of available CPUs on the system.
Should not typically need to be changed, but it can sometimes be beneficial to raise this
value when there is a high rate of messages being pushed into the queue and the objects
being fetched are small. In these cases, system resources may not be fully utilized without
fetching more messages per second, as the SQS message consumption rate affects the S3 object
retrieval rate.
tls_options
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
assume_role
optional
string,null
The ARN of an [IAM role][iam_role] to assume at startup.
Configuration of the authentication strategy for interacting with AWS services.
multiline
optional
<oneOf>
Multiline aggregation configuration.
If not specified, multiline aggregation is disabled.
Option 1
optional
object
Configuration of multi-line aggregation.
start_pattern
required
string
Regular expression pattern that is used to match the start of a new message.
condition_pattern
required
string
Regular expression pattern that is used to determine whether or not more lines should be read.
This setting must be configured in conjunction with mode.
mode
required
<oneOf>
Aggregation mode.
This setting must be configured in conjunction with condition_pattern.
continue_through
optional
continue_through
All consecutive lines matching this pattern are included in the group.
The first line (the line that matched the start pattern) does not need to match the ContinueThrough pattern.
This is useful in cases such as a Java stack trace, where some indicator in the line (such as a leading
whitespace) indicates that it is an extension of the proceeding line.
continue_past
optional
continue_past
All consecutive lines matching this pattern, plus one additional line, are included in the group.
This is useful in cases where a log message ends with a continuation marker, such as a backslash, indicating
that the following line is part of the same message.
halt_before
optional
halt_before
All consecutive lines not matching this pattern are included in the group.
This is useful where a log line contains a marker indicating that it begins a new message.
halt_with
optional
halt_with
All consecutive lines, up to and including the first line matching this pattern, are included in the group.
This is useful where a log line ends with a termination marker, such as a semicolon.
timeout_ms
required
integer
The maximum amount of time to wait for the next additional line, in milliseconds.
Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.
acknowledgements
optional
object
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
Whether or not end-to-end acknowledgements are enabled for this source.
default: null
tls_options
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
Configuration of the authentication strategy for interacting with AWS services.
queue_url
required
string
The URL of the SQS queue to poll for messages.
poll_secs
optional
integer
How long to wait while polling the queue for new messages, in seconds.
Generally, this should not be changed unless instructed to do so, as if messages are available,
they are always consumed, regardless of the value of poll_secs.
visibility_timeout_secs
optional
integer
The visibility timeout to use for messages, in seconds.
This controls how long a message is left unavailable after it is received. If a message is received, and
takes longer than visibility_timeout_secs to process and delete the message from the queue, it is made available again for another consumer.
This can happen if there is an issue between consuming a message and deleting it.
delete_message
optional
boolean
Whether to delete the message once it is processed.
It can be useful to set this to false for debugging or during the initial setup.
client_concurrency
optional
integer,null
Number of concurrent tasks to create for polling the queue for messages.
Defaults to the number of available CPUs on the system.
Should not typically need to be changed, but it can sometimes be beneficial to raise this
value when there is a high rate of messages being pushed into the queue and the messages
being fetched are small. In these cases, system resources may not be fully utilized without
fetching more messages per second, as it spends more time fetching the messages than
processing them.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
Whether or not end-to-end acknowledgements are enabled for this source.
default: null
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
If this is set to true, when incoming events contain a Datadog API key, it is
stored in the event metadata and used if the event is sent to a Datadog sink.
disable_logs
optional
boolean
If this is set to true, logs are not accepted by the component.
disable_metrics
optional
boolean
If this is set to true, metrics are not accepted by the component.
disable_traces
optional
boolean
If this is set to true, traces are not accepted by the component.
multiple_outputs
optional
boolean
If this is set to true logs, metrics, and traces are sent to different outputs.
For a source component named agent, the received logs, metrics, and traces can then be
configured as input to other components by specifying agent.logs, agent.metrics, and
agent.traces, respectively.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
The amount of time, in seconds, to pause between each batch of output lines.
The default is one batch per second. To remove the delay and output batches as quickly as possible, set
interval to 0.0.
count
optional
integer
The total number of lines to output.
By default, the source continuously prints logs (infinitely).
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
An optional path that deserializes an empty string to None.
socket_path
required
string
Absolute path to the socket file to read DNSTAP data from.
The DNS server must be configured to send its DNSTAP data to this socket file. The socket file is created
if it doesn't already exist when the source first starts.
raw_data_only
optional
boolean,null
Whether or not to skip parsing or decoding of DNSTAP frames.
If set to true, frames are not parsed or decoded. The raw frame data is set as a field on the event
(called rawData) and encoded as a base64 string.
multithreaded
optional
boolean,null
Whether or not to concurrently process DNSTAP frames.
max_frame_handling_tasks
optional
integer,null
Maximum number of frames that can be processed concurrently.
socket_file_mode
optional
integer,null
Unix file mode bits to be applied to the unix socket file as its designated file permissions.
Note: The file mode value can be specified in any numeric format supported by your configuration
language, but it is most intuitive to use an octal number.
socket_receive_buffer_size
optional
integer,null
The size, in bytes, of the receive buffer used for the socket.
This should not typically needed to be changed.
socket_send_buffer_size
optional
integer,null
The size, in bytes, of the send buffer used for the socket.
This should not typically needed to be changed.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global settings.
If absent, the DOCKER_HOST environment variable is used. If DOCKER_HOST is also absent,
the default Docker local socket (/var/run/docker.sock on Unix platforms,
//./pipe/docker_engine on Windows) is used.
exclude_containers
optional
array,null
A list of container IDs or names of containers to exclude from log collection.
Matching is prefix first, so specifying a value of foo would match any container named foo as well as any
container whose name started with foo. This applies equally whether matching container IDs or names.
By default, the source collects logs for all containers. If exclude_containers is configured, any
container that matches a configured exclusion is excluded even if it is also included with
include_containers, so care should be taken when using prefix matches as they cannot be overridden by a
corresponding entry in include_containers, for example, excluding foo by attempting to include foo-specific-id.
This can be used in conjunction with include_containers.
include_containers
optional
array,null
A list of container IDs or names of containers to include in log collection.
Matching is prefix first, so specifying a value of foo would match any container named foo as well as any
container whose name started with foo. This applies equally whether matching container IDs or names.
By default, the source collects logs for all containers. If include_containers is configured, only
containers that match a configured inclusion and are also not excluded get matched.
This can be used in conjunction with exclude_containers.
include_labels
optional
array,null
A list of container object labels to match against when filtering running containers.
Labels should follow the syntax described in the Docker object labels documentation.
include_images
optional
array,null
A list of image names to match against.
If not provided, all images are included.
partial_event_marker_field
optional
string,null
Overrides the name of the log field used to mark an event as partial.
If auto_partial_merge is disabled, partial events are emitted with a log field, set by this
configuration value, indicating that the event is not complete.
auto_partial_merge
optional
boolean
Enables automatic merging of partial events.
retry_backoff_secs
optional
integer
The amount of time to wait before retrying after an error.
multiline
optional
<oneOf>
Multiline aggregation configuration.
If not specified, multiline aggregation is disabled.
Option 1
optional
object
Configuration of multi-line aggregation.
start_pattern
required
string
Regular expression pattern that is used to match the start of a new message.
condition_pattern
required
string
Regular expression pattern that is used to determine whether or not more lines should be read.
This setting must be configured in conjunction with mode.
mode
required
<oneOf>
Aggregation mode.
This setting must be configured in conjunction with condition_pattern.
continue_through
optional
continue_through
All consecutive lines matching this pattern are included in the group.
The first line (the line that matched the start pattern) does not need to match the ContinueThrough pattern.
This is useful in cases such as a Java stack trace, where some indicator in the line (such as a leading
whitespace) indicates that it is an extension of the proceeding line.
continue_past
optional
continue_past
All consecutive lines matching this pattern, plus one additional line, are included in the group.
This is useful in cases where a log message ends with a continuation marker, such as a backslash, indicating
that the following line is part of the same message.
halt_before
optional
halt_before
All consecutive lines not matching this pattern are included in the group.
This is useful where a log line contains a marker indicating that it begins a new message.
halt_with
optional
halt_with
All consecutive lines, up to and including the first line matching this pattern, are included in the group.
This is useful where a log line ends with a termination marker, such as a semicolon.
timeout_ms
required
integer
The maximum amount of time to wait for the next additional line, in milliseconds.
Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.
tls
optional
<oneOf>
Configuration of TLS when connecting to the Docker daemon.
Only relevant when connecting to Docker with an HTTPS URL.
If not configured, the environment variable DOCKER_CERT_PATH is used. If DOCKER_CERT_PATH is absent, then DOCKER_CONFIG is used. If both environment variables are absent, the certificates in ~/.docker/ are read.
Option 1
optional
object
Configuration of TLS when connecting to the Docker daemon.
Only relevant when connecting to Docker with an HTTPS URL.
If not configured, the environment variable DOCKER_CERT_PATH is used. If DOCKER_CERT_PATH is absent, then DOCKER_CONFIG is used. If both environment variables are absent, the certificates in ~/.docker/ are read.
ca_file
required
string
Path to the CA certificate file.
crt_file
required
string
Path to the TLS certificate file.
key_file
required
string
Path to the TLS key file.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
The command is run until it exits, potentially being restarted.
scheduled
optional
<oneOf>
Configuration options for scheduled commands.
Option 1
optional
object
Configuration options for scheduled commands.
exec_interval_secs
optional
integer
The interval, in seconds, between scheduled command runs.
If the command takes longer than exec_interval_secs to run, it is killed.
streaming
optional
<oneOf>
Configuration options for streaming commands.
Option 1
optional
object
Configuration options for streaming commands.
respawn_on_exit
optional
boolean
Whether or not the command should be rerun if the command exits.
respawn_interval_secs
optional
integer
The amount of time, in seconds, before rerunning a streaming command that exited.
command
required
[string]
The command to be run, plus any arguments required.
working_directory
optional
<oneOf>
The directory in which to run the command.
Option 1
optional
string
A file path.
include_stderr
optional
boolean
Whether or not the output from stderr should be included when generating events.
maximum_buffer_size_bytes
optional
integer
The maximum buffer size allowed before a log event is generated.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Option 1
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
Array of file patterns to include. Globbing is supported.
exclude
optional
[string]
Array of file patterns to exclude. Globbing is supported.
Takes precedence over the include option. Note: The exclude patterns are applied after the attempt to glob everything
in include. This means that all files are first matched by include and then filtered by the exclude
patterns. This can be impactful if include contains directories with contents that are not accessible.
file_key
optional
string
Overrides the name of the log field used to add the file path to each event.
The value is the full path to the file where the event was read message.
Set to "" to suppress this key.
start_at_beginning
optional
boolean,null
DEPRECATED: Whether or not to start reading from the beginning of a new file.
ignore_checkpoints
optional
boolean,null
Whether or not to ignore existing checkpoints when determining where to start reading a file.
Checkpoints are still written normally.
read_from
optional
<oneOf>
File position to use when reading a new file.
beginning
optional
beginning
Read from the beginning of the file.
end
optional
end
Start reading from the current end of the file.
ignore_older_secs
optional
integer,null
Ignore files with a data modification date older than the specified number of seconds.
max_line_bytes
optional
integer
The maximum size of a line before it is discarded.
This protects against malformed lines or tailing incorrect files.
host_key
optional
string
Overrides the name of the log field used to add the current hostname to each event.
The directory used to persist file checkpoint positions.
By default, the global data_dir option is used. Make sure the running user has write
permissions to this directory.
Option 1
optional
string
A file path.
offset_key
optional
<oneOf>
Enables adding the file offset to each event and sets the name of the log field used.
The value is the byte offset of the start of the line within the file.
Off by default, the offset is only added to the event if this is set.
Option 1
optional
string
An optional path that deserializes an empty string to None.
glob_minimum_cooldown_ms
optional
integer
Delay between file discovery calls.
This controls the interval at which files are searched. A higher value results in greater
chances of some short-lived files being missed between searches, but a lower value increases
the performance impact of file discovery.
fingerprint
optional
<oneOf>
Configuration for how files should be identified.
This is important for checkpointing when file rotation is used.
Checksum
optional
object
Read lines from the beginning of the file and compute a checksum over them.
bytes
optional
integer,null
Maximum number of bytes to use, from the lines that are read, for generating the checksum.
ignored_header_bytes
required
integer
The number of bytes to skip ahead (or ignore) when reading the data used for generating the checksum.
This can be helpful if all files share a common header that should be skipped.
lines
optional
integer
The number of lines to read for generating the checksum.
If your files share a common header that is not always a fixed size,
If the file has less than this amount of lines, it won’t be read at all.
strategy
required
checksum
Read lines from the beginning of the file and compute a checksum over them.
This may be useful when used with source directories containing dangling symlinks.
message_start_indicator
optional
string,null
DEPRECATED: String value used to identify the start of a multi-line message.
multi_line_timeout
optional
integer
DEPRECATED: How long to wait for more data when aggregating a multi-line message, in milliseconds.
multiline
optional
<oneOf>
Multiline aggregation configuration.
If not specified, multiline aggregation is disabled.
Option 1
optional
object
Configuration of multi-line aggregation.
start_pattern
required
string
Regular expression pattern that is used to match the start of a new message.
condition_pattern
required
string
Regular expression pattern that is used to determine whether or not more lines should be read.
This setting must be configured in conjunction with mode.
mode
required
<oneOf>
Aggregation mode.
This setting must be configured in conjunction with condition_pattern.
continue_through
optional
continue_through
All consecutive lines matching this pattern are included in the group.
The first line (the line that matched the start pattern) does not need to match the ContinueThrough pattern.
This is useful in cases such as a Java stack trace, where some indicator in the line (such as a leading
whitespace) indicates that it is an extension of the proceeding line.
continue_past
optional
continue_past
All consecutive lines matching this pattern, plus one additional line, are included in the group.
This is useful in cases where a log message ends with a continuation marker, such as a backslash, indicating
that the following line is part of the same message.
halt_before
optional
halt_before
All consecutive lines not matching this pattern are included in the group.
This is useful where a log line contains a marker indicating that it begins a new message.
halt_with
optional
halt_with
All consecutive lines, up to and including the first line matching this pattern, are included in the group.
This is useful where a log line ends with a termination marker, such as a semicolon.
timeout_ms
required
integer
The maximum amount of time to wait for the next additional line, in milliseconds.
Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.
max_read_bytes
optional
integer
An approximate limit on the amount of data read from a single file at a given time.
oldest_first
optional
boolean
Instead of balancing read capacity fairly across all watched files, prioritize draining the oldest files before moving on to read data from younger files.
remove_after_secs
optional
integer,null
Timeout from reaching EOF after which the file is removed from the filesystem, unless new data is written in the meantime.
If not specified, files are not removed.
line_delimiter
optional
string
String sequence used to separate one file line from another.
When set, the messages are transcoded from the specified encoding to UTF-8, which is the encoding that is
assumed internally for string-like data. Enable this transcoding operation if you need your data to
be in UTF-8 for further processing. At the time of transcoding, any malformed sequences (that can't be mapped to
UTF-8) is replaced with the Unicode REPLACEMENT
CHARACTER and warnings are
logged.
acknowledgements
optional
object
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
An optional path that deserializes an empty string to None.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Option 1
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
The socket address to listen for connections on, or systemd{#N} to use the Nth socket passed by
systemd socket activation.
If a socket address is used, it must include a port.
connection_limit
optional
integer,null
The maximum number of TCP connections that are allowed at any given time.
keepalive
optional
<oneOf>
TCP keepalive settings for socket-based components.
Option 1
optional
object
TCP keepalive settings for socket-based components.
time_secs
optional
integer,null
The time to wait before starting to send TCP keepalive probes on an idle connection.
receive_buffer_bytes
optional
integer,null
The size of the receive buffer used for each connection.
This generally should not need to be changed.
tls
optional
<oneOf>
TlsEnableableConfig for sources, adding metadata from the client certificate.
Option 1
optional
TlsEnableableConfig for sources, adding metadata from the client certificate.
client_metadata_key
optional
<oneOf>
Event field for client certificate metadata.
Option 1
optional
string
An optional path that deserializes an empty string to None.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
acknowledgements
optional
object
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
The subscription within the project which is configured to receive logs.
endpoint
optional
string
The endpoint from which to pull data.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
max_concurrency
optional
integer
The maximum number of concurrent stream connections to open at once.
full_response_size
optional
integer
The number of messages in a response to mark a stream as
"busy". This is used to determine if more streams should be
started.
The GCP Pub/Sub servers send responses with 100 or more messages when
the subscription is busy.
poll_time_seconds
optional
number
How often to poll the currently active streams to see if they
are all busy and so open a new stream.
ack_deadline_secs
optional
integer
The acknowledgement deadline, in seconds, to use for this stream.
Messages that are not acknowledged when this deadline expires may be retransmitted.
ack_deadline_seconds
optional
integer,null
The acknowledgement deadline, in seconds, to use for this stream.
DEPRECATED: Messages that are not acknowledged when this deadline expires may be retransmitted.
retry_delay_secs
optional
number
The amount of time, in seconds, to wait between retry attempts after an error.
retry_delay_seconds
optional
number,null
DEPRECATED: The amount of time, in seconds, to wait between retry attempts after an error.
keepalive_secs
optional
number
The amount of time, in seconds, with no received activity
before sending a keepalive request. If this is set larger than
60, you may see periodic errors sent from the server.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
skip_authentication
optional
boolean
Skip all authentication handling. For use with integration tests only.
A list of URL query parameters to include in the log event.
These override any values included in the body with conflicting names.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
auth
optional
<oneOf>
HTTP Basic authentication configuration.
Option 1
optional
object
HTTP Basic authentication configuration.
username
required
string
The username for basic authentication.
password
required
string
The password for basic authentication.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
Note: For json and ndjson encodings, the fields of the JSON objects are output as separate fields.
Option 1
optional
<oneOf>
Content encoding.
text
optional
text
Plaintext.
ndjson
optional
ndjson
Newline-delimited JSON.
json
optional
json
JSON.
binary
optional
binary
Binary.
headers
optional
[string]
A list of HTTP headers to include in the log event.
These override any values included in the JSON payload with conflicting names.
query_parameters
optional
[string]
A list of URL query parameters to include in the log event.
These override any values included in the body with conflicting names.
auth
optional
<oneOf>
HTTP Basic authentication configuration.
Option 1
optional
object
HTTP Basic authentication configuration.
username
required
string
The username for basic authentication.
password
required
string
The password for basic authentication.
strict_path
optional
boolean
Whether or not to treat the configured path as an absolute path.
If set to true, only requests using the exact URL path specified in path are accepted. Otherwise,
requests sent to a URL path that starts with the value of path are accepted.
With strict_path set to false and path set to "", the configured HTTP source accepts requests from
any URL path.
path
optional
string
The URL path on which log event POST requests are sent.
path_key
optional
string
The event key in which the requested URL path used to send the request is stored.
method
optional
<oneOf>
Specifies the action of the HTTP request.
HEAD
optional
HEAD
HTTP HEAD method.
GET
optional
GET
HTTP GET method.
POST
optional
POST
HTTP POST method.
PUT
optional
PUT
HTTP Put method.
PATCH
optional
PATCH
HTTP PATCH method.
DELETE
optional
DELETE
HTTP DELETE method.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Option 1
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
One or more values for the same header can be provided.
method
optional
<oneOf>
Specifies the method of the HTTP request.
HEAD
optional
HEAD
HTTP HEAD method.
GET
optional
GET
HTTP GET method.
POST
optional
POST
HTTP POST method.
PUT
optional
PUT
HTTP Put method.
PATCH
optional
PATCH
HTTP PATCH method.
DELETE
optional
DELETE
HTTP DELETE method.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
auth
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP Authentication.
Option 1
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The basic authentication username.
password
required
string
The basic authentication password.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
token
required
string
The bearer authentication token.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
Note: For json and ndjson encodings, the fields of the JSON objects are output as separate fields.
Option 1
optional
<oneOf>
Content encoding.
text
optional
text
Plaintext.
ndjson
optional
ndjson
Newline-delimited JSON.
json
optional
json
JSON.
binary
optional
binary
Binary.
headers
optional
[string]
A list of HTTP headers to include in the log event.
These override any values included in the JSON payload with conflicting names.
query_parameters
optional
[string]
A list of URL query parameters to include in the log event.
These override any values included in the body with conflicting names.
auth
optional
<oneOf>
HTTP Basic authentication configuration.
Option 1
optional
object
HTTP Basic authentication configuration.
username
required
string
The username for basic authentication.
password
required
string
The password for basic authentication.
strict_path
optional
boolean
Whether or not to treat the configured path as an absolute path.
If set to true, only requests using the exact URL path specified in path are accepted. Otherwise,
requests sent to a URL path that starts with the value of path are accepted.
With strict_path set to false and path set to "", the configured HTTP source accepts requests from
any URL path.
path
optional
string
The URL path on which log event POST requests are sent.
path_key
optional
string
The event key in which the requested URL path used to send the request is stored.
method
optional
<oneOf>
Specifies the action of the HTTP request.
HEAD
optional
HEAD
HTTP HEAD method.
GET
optional
GET
HTTP GET method.
POST
optional
POST
HTTP POST method.
PUT
optional
PUT
HTTP Put method.
PATCH
optional
PATCH
HTTP PATCH method.
DELETE
optional
DELETE
HTTP DELETE method.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Option 1
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
A comma-separated list of Kafka bootstrap servers.
These are the servers in a Kafka cluster that a client should use to bootstrap its connection to the cluster,
allowing discovery of all the other hosts in the cluster.
Must be in the form of host:port, and comma-separated.
topics
required
[string]
The Kafka topics names to read events from.
Regular expression syntax is supported if the topic begins with ^.
group_id
required
string
The consumer group name to be used to consume events from Kafka.
auto_offset_reset
optional
string
If offsets for consumer group do not exist, set them using this strategy.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
object
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
object
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
