Cette page n'est pas encore disponible en français, sa traduction est en cours. Si vous avez des questions ou des retours sur notre projet de traduction actuel, n'hésitez pas à nous contacter.
Observability Pipelines is not available on the US1-FED Datadog site.
A source is where data is collected and sent to Observability Pipelines. The source component in a configuration defines how Observability Pipelines collects or receives data from the source.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
The namespace to use for logs. This overrides the global setting.
offset_key
optional
string
The AMQP offset key.
queue
optional
string
The name of the queue to consume.
routing_key_field
optional
string
The AMQP routing key.
connection_string
required
string
URI for the AMQP server.
The URI has the format of
amqp://<user>:<password>@<host>:<port>/<vhost>?timeout=<seconds>.
The default vhost can be specified by using a value of %2f.
To connect over TLS, a scheme of amqps can be specified instead. For example,
amqps://.... Additional TLS settings, such as client certificate verification, can be
configured under the tls section.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
DEPRECATED: AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If
configured, access_key should be set to the same value. Otherwise, all requests are allowed.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
access_keys
optional
array,null
A list of access keys to authenticate requests against.
AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If
configured, access_keys should be set to the same value. Otherwise, all requests are allowed.
acknowledgements
optional
object
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
The namespace to use for logs. This overrides the global setting.
record_compression
optional
<oneOf>
The compression scheme to use for decompressing records within the Firehose message.
Some services, like AWS CloudWatch Logs, compresses the events with gzip,
before sending them AWS Kinesis Firehose. This option can be used to automatically decompress
them before forwarding them to the next component.
Note that this is different from Content encoding option of the
Firehose HTTP endpoint destination. That option controls the content encoding of the entire HTTP request.
auto
optional
auto
Automatically attempt to determine the compression scheme.
The compression scheme of the object is determined by looking at its file signature, also known
as magic bytes.
If the record fails to decompress with the discovered format, the record is forwarded as is.
Thus, if you know the records are always gzip encoded (for example, if they are coming from AWS CloudWatch Logs),
set gzip in this field so that any records that are not-gzipped are rejected.
none
optional
none
Uncompressed.
gzip
optional
gzip
GZIP.
store_access_key
required
boolean
Whether or not to store the AWS Firehose Access Key in event secrets.
If set to true, when incoming requests contains an access key sent by AWS Firehose, it is kept in the
event secrets as "aws_kinesis_firehose_access_key".
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
The namespace to use for logs. This overrides the global setting.
multiline
optional
<oneOf>
Multiline aggregation configuration.
If not specified, multiline aggregation is disabled.
Option 1
optional
object
Configuration of multi-line aggregation.
condition_pattern
required
string
Regular expression pattern that is used to determine whether or not more lines should be read.
This setting must be configured in conjunction with mode.
mode
required
<oneOf>
Aggregation mode.
This setting must be configured in conjunction with condition_pattern.
continue_through
optional
continue_through
All consecutive lines matching this pattern are included in the group.
The first line (the line that matched the start pattern) does not need to match the ContinueThrough pattern.
This is useful in cases such as a Java stack trace, where some indicator in the line (such as a leading
whitespace) indicates that it is an extension of the proceeding line.
continue_past
optional
continue_past
All consecutive lines matching this pattern, plus one additional line, are included in the group.
This is useful in cases where a log message ends with a continuation marker, such as a backslash, indicating
that the following line is part of the same message.
halt_before
optional
halt_before
All consecutive lines not matching this pattern are included in the group.
This is useful where a log line contains a marker indicating that it begins a new message.
halt_with
optional
halt_with
All consecutive lines, up to and including the first line matching this pattern, are included in the group.
This is useful where a log line ends with a termination marker, such as a semicolon.
start_pattern
required
string
Regular expression pattern that is used to match the start of a new message.
timeout_ms
required
integer
The maximum amount of time to wait for the next additional line, in milliseconds.
Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.
sqs
optional
<oneOf>
Configuration options for SQS.
Option 1
optional
object
SQS configuration options.
client_concurrency
optional
integer,null
Number of concurrent tasks to create for polling the queue for messages.
Defaults to the number of available CPUs on the system.
Should not typically need to be changed, but it can sometimes be beneficial to raise this
value when there is a high rate of messages being pushed into the queue and the objects
being fetched are small. In these cases, system resources may not be fully utilized without
fetching more messages per second, as the SQS message consumption rate affects the S3 object
retrieval rate.
delete_message
optional
boolean
Whether to delete the message once it is processed.
It can be useful to set this to false for debugging or during the initial setup.
poll_secs
optional
integer
How long to wait while polling the queue for new messages, in seconds.
Generally, this should not be changed unless instructed to do so, as if messages are available,
they are always consumed, regardless of the value of poll_secs.
queue_url
required
uri
The URL of the SQS queue to poll for bucket notifications.
tls_options
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
visibility_timeout_secs
optional
integer
The visibility timeout to use for messages, in seconds.
This controls how long a message is left unavailable after it is received. If a message is received, and
takes longer than visibility_timeout_secs to process and delete the message from the queue, it is made available again for another consumer.
This can happen if there is an issue between consuming a message and deleting it.
strategy
optional
<oneOf>
The strategy to use to consume objects from S3.
sqs
optional
sqs
Consumes objects by processing bucket notification events sent to an AWS SQS queue.
tls_options
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
endpoint
optional
string,null
Custom endpoint for use with AWS-compatible services.
Whether or not end-to-end acknowledgements are enabled for this source.
default: null
auth
optional
Configuration of the authentication strategy for interacting with AWS services.
client_concurrency
optional
integer,null
Number of concurrent tasks to create for polling the queue for messages.
Defaults to the number of available CPUs on the system.
Should not typically need to be changed, but it can sometimes be beneficial to raise this
value when there is a high rate of messages being pushed into the queue and the messages
being fetched are small. In these cases, system resources may not be fully utilized without
fetching more messages per second, as it spends more time fetching the messages than
processing them.
Whether to delete the message once it is processed.
It can be useful to set this to false for debugging or during the initial setup.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
The namespace to use for logs. This overrides the global setting.
poll_secs
optional
integer
How long to wait while polling the queue for new messages, in seconds.
Generally, this should not be changed unless instructed to do so, as if messages are available,
they are always consumed, regardless of the value of poll_secs.
queue_url
required
string
The URL of the SQS queue to poll for messages.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
visibility_timeout_secs
optional
integer
The visibility timeout to use for messages, in seconds.
This controls how long a message is left unavailable after it is received. If a message is received, and
takes longer than visibility_timeout_secs to process and delete the message from the queue, it is made available again for another consumer.
This can happen if there is an issue between consuming a message and deleting it.
endpoint
optional
string,null
Custom endpoint for use with AWS-compatible services.
If this is set to true, logs are not accepted by the component.
disable_metrics
optional
boolean
If this is set to true, metrics are not accepted by the component.
disable_traces
optional
boolean
If this is set to true, traces are not accepted by the component.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
The namespace to use for logs. This overrides the global setting.
multiple_outputs
optional
boolean
If this is set to true logs, metrics, and traces are sent to different outputs.
For a source component named agent, the received logs, metrics, and traces can then be
configured as input to other components by specifying agent.logs, agent.metrics, and
agent.traces, respectively.
store_api_key
optional
boolean
If this is set to true, when incoming events contain a Datadog API key, it is
stored in the event metadata and used if the event is sent to a Datadog sink.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
Whether or not end-to-end acknowledgements are enabled for this source.
default: null
address
required
string
The socket address to listen for connections on, or systemd{#N} to use the Nth socket passed by
systemd socket activation.
If a socket address is used, it must include a port.
connection_limit
optional
integer,null
The maximum number of TCP connections that are allowed at any given time.
keepalive
optional
<oneOf>
TCP keepalive settings for socket-based components.
Option 1
optional
object
TCP keepalive settings for socket-based components.
time_secs
optional
integer,null
The time to wait before starting to send TCP keepalive probes on an idle connection.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
receive_buffer_bytes
optional
integer,null
The size of the receive buffer used for each connection.
This generally should not need to be changed.
tls
optional
<oneOf>
TlsEnableableConfig for sources, adding metadata from the client certificate.
Option 1
optional
TlsEnableableConfig for sources, adding metadata from the client certificate.
client_metadata_key
optional
<oneOf>
Event field for client certificate metadata.
Option 1
optional
string
An optional path that deserializes an empty string to None.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
The number of messages in a response to mark a stream as
"busy". This is used to determine if more streams should be
started.
The GCP Pub/Sub servers send responses with 100 or more messages when
the subscription is busy.
keepalive_secs
optional
number
The amount of time, in seconds, with no received activity
before sending a keepalive request. If this is set larger than
60, you may see periodic errors sent from the server.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
max_concurrency
optional
integer
The maximum number of concurrent stream connections to open at once.
poll_time_seconds
optional
number
How often to poll the currently active streams to see if they
are all busy and so open a new stream.
project
required
string
The project name from which to pull logs.
retry_delay_seconds
optional
number,null
DEPRECATED: The amount of time, in seconds, to wait between retry attempts after an error.
retry_delay_secs
optional
number
The amount of time, in seconds, to wait between retry attempts after an error.
subscription
required
string
The subscription within the project which is configured to receive logs.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
skip_authentication
optional
boolean
Skip all authentication handling. For use with integration tests only.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
The namespace to use for logs. This overrides the global setting.
query_parameters
optional
[string]
A list of URL query parameters to include in the log event.
These override any values included in the body with conflicting names.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
For json and ndjson encodings, the fields of the JSON objects are output as separate fields.
Option 1
optional
<oneOf>
Content encoding.
text
optional
text
Plaintext.
ndjson
optional
ndjson
Newline-delimited JSON.
json
optional
json
JSON.
binary
optional
binary
Binary.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Option 1
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
A list of HTTP headers to include in the log event.
These override any values included in the JSON payload with conflicting names.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
method
optional
<oneOf>
Specifies the action of the HTTP request.
HEAD
optional
HEAD
HTTP HEAD method.
GET
optional
GET
HTTP GET method.
POST
optional
POST
HTTP POST method.
PUT
optional
PUT
HTTP Put method.
PATCH
optional
PATCH
HTTP PATCH method.
DELETE
optional
DELETE
HTTP DELETE method.
path
optional
string
The URL path on which log event POST requests are sent.
path_key
optional
string
The event key in which the requested URL path used to send the request is stored.
query_parameters
optional
[string]
A list of URL query parameters to include in the log event.
These override any values included in the body with conflicting names.
response_code
optional
integer
Specifies the HTTP response status code that will be returned on successful requests.
strict_path
optional
boolean
Whether or not to treat the configured path as an absolute path.
If set to true, only requests using the exact URL path specified in path are accepted. Otherwise,
requests sent to a URL path that starts with the value of path are accepted.
With strict_path set to false and path set to "", the configured HTTP source accepts requests from
any URL path.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Configuration of the authentication strategy for HTTP requests.
HTTP Authentication.
Option 1
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
password
required
string
The basic authentication password.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The basic authentication username.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
One or more values for the same header can be provided.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
method
optional
<oneOf>
Specifies the method of the HTTP request.
HEAD
optional
HEAD
HTTP HEAD method.
GET
optional
GET
HTTP GET method.
POST
optional
POST
HTTP POST method.
PUT
optional
PUT
HTTP Put method.
PATCH
optional
PATCH
HTTP PATCH method.
DELETE
optional
DELETE
HTTP DELETE method.
query
optional
object
Custom parameters for the HTTP request query string.
One or more values for the same parameter key can be provided.
The parameters provided in this option are appended to any parameters
manually provided in the endpoint option.
scrape_interval_secs
optional
integer
The interval between scrapes. Requests are run concurrently so if a scrape takes longer
than the interval a new scrape will be started. This can take extra resources, set the timeout
to a value lower than the scrape interval to prevent this from happening.
scrape_timeout_secs
optional
number
The timeout for each scrape request.
tls
optional
<oneOf>
TLS configuration.
Option 1
optional
object
TLS configuration.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
For json and ndjson encodings, the fields of the JSON objects are output as separate fields.
Option 1
optional
<oneOf>
Content encoding.
text
optional
text
Plaintext.
ndjson
optional
ndjson
Newline-delimited JSON.
json
optional
json
JSON.
binary
optional
binary
Binary.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Option 1
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
A list of HTTP headers to include in the log event.
These override any values included in the JSON payload with conflicting names.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
method
optional
<oneOf>
Specifies the action of the HTTP request.
HEAD
optional
HEAD
HTTP HEAD method.
GET
optional
GET
HTTP GET method.
POST
optional
POST
HTTP POST method.
PUT
optional
PUT
HTTP Put method.
PATCH
optional
PATCH
HTTP PATCH method.
DELETE
optional
DELETE
HTTP DELETE method.
path
optional
string
The URL path on which log event POST requests are sent.
path_key
optional
string
The event key in which the requested URL path used to send the request is stored.
query_parameters
optional
[string]
A list of URL query parameters to include in the log event.
These override any values included in the body with conflicting names.
response_code
optional
integer
Specifies the HTTP response status code that will be returned on successful requests.
strict_path
optional
boolean
Whether or not to treat the configured path as an absolute path.
If set to true, only requests using the exact URL path specified in path are accepted. Otherwise,
requests sent to a URL path that starts with the value of path are accepted.
With strict_path set to false and path set to "", the configured HTTP source accepts requests from
any URL path.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
A comma-separated list of Kafka bootstrap servers.
These are the servers in a Kafka cluster that a client should use to bootstrap its connection to the cluster,
allowing discovery of all the other hosts in the cluster.
Must be in the form of host:port, and comma-separated.
commit_interval_ms
optional
integer
The frequency that the consumer offsets are committed (written) to offset storage.
Maximum time the broker may wait to fill the response.
framing
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
The namespace to use for logs. This overrides the global setting.
metrics
optional
object
Metrics configuration.
topic_lag_metric
required
boolean
Expose topic lag metrics for all topics and partitions. Metric names are kafka_consumer_lag.
default: false
offset_key
optional
string
Overrides the name of the log field used to add the offset to each event.
The value is the offset of the Kafka message itself.
By default, "offset" is used.
partition_key
optional
string
Overrides the name of the log field used to add the partition to each event.
The value is the partition from which the Kafka message was consumed from.
By default, "partition" is used.
session_timeout_ms
optional
integer
The Kafka session timeout.
socket_timeout_ms
optional
integer
Timeout for network requests.
topic_key
optional
string
Overrides the name of the log field used to add the topic to each event.
The value is the topic from which the Kafka message was consumed from.
By default, "topic" is used.
topics
required
[string]
The Kafka topics names to read events from.
Regular expression syntax is supported if the topic begins with ^.
sasl
optional
<oneOf>
Configuration for SASL authentication when interacting with Kafka.
Option 1
optional
object
Configuration for SASL authentication when interacting with Kafka.
enabled
optional
boolean,null
Enables SASL authentication.
Only PLAIN- and SCRAM-based mechanisms are supported when configuring SASL authentication using sasl.*. For
other mechanisms, librdkafka_options.* must be used directly to configure other librdkafka-specific values.
If using sasl.kerberos.* as an example, where * is service.name, principal, kinit.md, etc., then
librdkafka_options.* as a result becomes librdkafka_options.sasl.kerberos.service.name,
librdkafka_options.sasl.kerberos.principal, etc.
Wrapper for sensitive strings containing credentials
username
optional
string,null
The SASL username.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Whether or not end-to-end acknowledgements are enabled for this source.
default: null
address
required
string
The socket address to listen for connections on, or systemd{#N} to use the Nth socket passed by
systemd socket activation.
If a socket address is used, it must include a port.
connection_limit
optional
integer,null
The maximum number of TCP connections that are allowed at any given time.
keepalive
optional
<oneOf>
TCP keepalive settings for socket-based components.
Option 1
optional
object
TCP keepalive settings for socket-based components.
time_secs
optional
integer,null
The time to wait before starting to send TCP keepalive probes on an idle connection.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
receive_buffer_bytes
optional
integer,null
The size of the receive buffer used for each connection.
tls
optional
<oneOf>
TlsEnableableConfig for sources, adding metadata from the client certificate.
Option 1
optional
TlsEnableableConfig for sources, adding metadata from the client certificate.
client_metadata_key
optional
<oneOf>
Event field for client certificate metadata.
Option 1
optional
string
An optional path that deserializes an empty string to None.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
url
required
string
The NATS URL to connect to.
The URL takes the form of nats://server:port.
If the port is not specified it defaults to 4222.
Whether or not end-to-end acknowledgements are enabled for this source.
default: null
grpc
required
object
Configuration for the opentelemetry gRPC server.
address
required
string
The socket address to listen for connections on.
It must include a port.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
http
required
object
Configuration for the opentelemetry HTTP server.
address
required
string
The socket address to listen for connections on.
It must include a port.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
Whether or not end-to-end acknowledgements are enabled for this source.
default: null
address
required
string
The socket address to accept connections on.
The address must include a port.
auth
optional
<oneOf>
HTTP Basic authentication configuration.
Option 1
optional
object
HTTP Basic authentication configuration.
password
required
string
The password for basic authentication.
username
required
string
The username for basic authentication.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
Acknowledgement configuration for the splunk_hec source.
ack_idle_cleanup
optional
boolean
Whether or not to remove channels after idling for max_idle_time seconds.
A channel is idling if it is not used for sending data or querying acknowledgement statuses.
default: false
enabled
optional
boolean,null
Enables end-to-end acknowledgements.
default: null
max_idle_time
optional
integer
The amount of time, in seconds, a channel is allowed to idle before removal.
Channels can potentially idle for longer than this setting but clients should not rely on such behavior.
Minimum of 1.
default: 300
max_number_of_ack_channels
optional
integer
The maximum number of Splunk HEC channels clients can use with this source.
Minimum of 1.
default: 1000000
max_pending_acks
optional
integer
The maximum number of acknowledgement statuses pending query across all channels.
Equivalent to the max_number_of_acked_requests_pending_query Splunk HEC setting.
Minimum of 1.
default: 10000000
max_pending_acks_per_channel
optional
integer
The maximum number of acknowledgement statuses pending query for a single channel.
Equivalent to the max_number_of_acked_requests_pending_query_per_ack_channel Splunk HEC setting.
Minimum of 1.
default: 1000000
address
optional
string
The socket address to listen for connections on.
The address must include a port.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global settings.
store_hec_token
optional
boolean
Whether or not to forward the Splunk HEC authentication token with events.
If set to true, when incoming requests contain a Splunk HEC token, the token used is kept in the
event metadata and preferentially used if the event is sent to a Splunk HEC sink.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
token
optional
<oneOf>
Optional authorization token.
DEPRECATED: If supplied, incoming requests must supply this token in the Authorization header, just as a client would if
it was communicating with the Splunk HEC endpoint directly.
If not supplied, the Authorization header is ignored and requests are not authenticated.
Option 1
optional
string
Wrapper for sensitive strings containing credentials
valid_tokens
optional
array,null
A list of valid authorization tokens.
If supplied, incoming requests must supply one of these tokens in the Authorization header, just as a client
would if it was communicating with the Splunk HEC endpoint directly.
If not supplied, the Authorization header is ignored and requests are not authenticated.
The socket address to listen for connections on, or systemd{#N} to use the Nth socket passed by
systemd socket activation.
If a socket address is used, it must include a port.
connection_limit
optional
integer,null
The maximum number of TCP connections that are allowed at any given time.
keepalive
optional
<oneOf>
TCP keepalive settings for socket-based components.
Option 1
optional
object
TCP keepalive settings for socket-based components.
time_secs
optional
integer,null
The time to wait before starting to send TCP keepalive probes on an idle connection.
receive_buffer_bytes
optional
integer,null
The size of the receive buffer used for each connection.
shutdown_timeout_secs
optional
integer
The timeout before a connection is forcefully closed during shutdown.
tls
optional
<oneOf>
TlsEnableableConfig for sources, adding metadata from the client certificate.
Option 1
optional
TlsEnableableConfig for sources, adding metadata from the client certificate.
client_metadata_key
optional
<oneOf>
Event field for client certificate metadata.
Option 1
optional
string
An optional path that deserializes an empty string to None.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
mode
required
tcp
Listen on TCP.
Option 2
optional
Listen on UDP.
address
required
string
The socket address to listen for connections on, or systemd{#N} to use the Nth socket passed by
systemd socket activation.
If a socket address is used, it must include a port.
receive_buffer_bytes
optional
integer,null
The size of the receive buffer used for each connection.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Option 1
optional
<oneOf>
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Bytes
optional
object
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
method
required
bytes
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
CharacterDelimited
optional
Byte frames which are delimited by a chosen character.
character_delimited
required
object
Options for the character delimited decoder.
delimiter
required
integer
The character that delimits byte sequences.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
character_delimited
Byte frames which are delimited by a chosen character.
LengthDelimited
optional
object
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
method
required
length_delimited
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
NewlineDelimited
optional
Byte frames which are delimited by a newline character.
newline_delimited
optional
object
Options for the newline delimited decoder.
max_length
optional
integer,null
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
method
required
newline_delimited
Byte frames which are delimited by a newline character.
Overrides the name of the log field used to add the peer host to each event.
If using TCP or UDP, the value is the peer host's address, including the port. For example, 1.2.3.4:9000. If using
UDS, the value is the socket path itself.
Whether or not end-to-end acknowledgements are enabled for this source.
default: null
address
required
string
The socket address to listen for connections on.
It must include a port.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
tls
optional
<oneOf>
Configures the TLS options for incoming/outgoing connections.
Option 1
optional
Configures the TLS options for incoming/outgoing connections.
enabled
optional
boolean,null
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
ca_file
optional
<oneOf>
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
crt_file
optional
<oneOf>
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Option 1
optional
string
A file path.
key_file
optional
<oneOf>
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Option 1
optional
string
A file path.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
version
optional
<oneOf>
Version of the configuration.
Option 1
optional
<oneOf>
Marker type for version two of the configuration for the vector source.