A source is where data is collected and sent to Observability Pipelines. The source component in a configuration defines how Observability Pipelines collects or receives data from the source.
Supports AMQP version 0.9.1
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullThe identifier for the consumer.
Configures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
The namespace to use for logs. This overrides the global setting.
The name of the queue to consume.
URI for the AMQP server.
The URI has the format of
amqp://<user>:<password>@<host>:<port>/<vhost>?timeout=<seconds>.
The default vhost can be specified by using a value of %2f.
To connect over TLS, a scheme of amqps can be specified instead. For example,
amqps://.... Additional TLS settings, such as client certificate verification, can be
configured under the tls section.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
consumer: vector
decoding:
codec: bytes
exchange_key: exchange
framing:
method: bytes
log_namespace: boolean
offset_key: offset
queue: vector
routing_key_field: routing
type: amqp
Configuration for the aws_kinesis_firehose source.
An access key to authenticate requests against.
DEPRECATED: AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If
configured, access_key should be set to the same value. Otherwise, all requests are allowed.
Wrapper for sensitive strings containing credentials
A list of access keys to authenticate requests against.
AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If
configured, access_keys should be set to the same value. Otherwise, all requests are allowed.
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullThe socket address to listen for connections on.
Configures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
The namespace to use for logs. This overrides the global setting.
The compression scheme to use for decompressing records within the Firehose message.
Some services, like AWS CloudWatch Logs, compresses the events with gzip,
before sending them AWS Kinesis Firehose. This option can be used to automatically decompress
them before forwarding them to the next component.
Note that this is different from Content encoding option of the
Firehose HTTP endpoint destination. That option controls the content encoding of the entire HTTP request.
Automatically attempt to determine the compression scheme.
The compression scheme of the object is determined by looking at its file signature, also known
as magic bytes.
If the record fails to decompress with the discovered format, the record is forwarded as is.
Thus, if you know the records are always gzip encoded (for example, if they are coming from AWS CloudWatch Logs),
set gzip in this field so that any records that are not-gzipped are rejected.
Whether or not to store the AWS Firehose Access Key in event secrets.
If set to true, when incoming requests contains an access key sent by AWS Firehose, it is kept in the
event secrets as "aws_kinesis_firehose_access_key".
Configures the TLS options for incoming/outgoing connections.
Configures the TLS options for incoming/outgoing connections.
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
access_key: ''
access_keys: array
acknowledgements:
enabled: null
address: string
decoding:
codec: bytes
framing:
method: bytes
log_namespace: boolean
record_compression: auto
store_access_key: boolean
tls: ''
type: aws_kinesis_firehose
Configuration for the aws_s3 source.
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullConfiguration of the authentication strategy for interacting with AWS services.
The compression scheme used for decompressing objects retrieved from S3.
Automatically attempt to determine the compression scheme.
The compression scheme of the object is determined from its Content-Encoding and
Content-Type metadata, as well as the key suffix (for example, .gz).
It is set to none if the compression scheme cannot be determined.
Configures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
The namespace to use for logs. This overrides the global setting.
Multiline aggregation configuration.
If not specified, multiline aggregation is disabled.
Configuration of multi-line aggregation.
Regular expression pattern that is used to determine whether or not more lines should be read.
This setting must be configured in conjunction with mode.
Aggregation mode.
This setting must be configured in conjunction with condition_pattern.
All consecutive lines matching this pattern are included in the group.
The first line (the line that matched the start pattern) does not need to match the ContinueThrough pattern.
This is useful in cases such as a Java stack trace, where some indicator in the line (such as a leading
whitespace) indicates that it is an extension of the proceeding line.
All consecutive lines matching this pattern, plus one additional line, are included in the group.
This is useful in cases where a log message ends with a continuation marker, such as a backslash, indicating
that the following line is part of the same message.
All consecutive lines not matching this pattern are included in the group.
This is useful where a log line contains a marker indicating that it begins a new message.
All consecutive lines, up to and including the first line matching this pattern, are included in the group.
This is useful where a log line ends with a termination marker, such as a semicolon.
Regular expression pattern that is used to match the start of a new message.
The maximum amount of time to wait for the next additional line, in milliseconds.
Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.
Configuration options for SQS.
SQS configuration options.
Number of concurrent tasks to create for polling the queue for messages.
Defaults to the number of available CPUs on the system.
Should not typically need to be changed, but it can sometimes be beneficial to raise this
value when there is a high rate of messages being pushed into the queue and the objects
being fetched are small. In these cases, system resources may not be fully utilized without
fetching more messages per second, as the SQS message consumption rate affects the S3 object
retrieval rate.
Whether to delete the message once it is processed.
It can be useful to set this to false for debugging or during the initial setup.
How long to wait while polling the queue for new messages, in seconds.
Generally, this should not be changed unless instructed to do so, as if messages are available,
they are always consumed, regardless of the value of poll_secs.
The URL of the SQS queue to poll for bucket notifications.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
The visibility timeout to use for messages, in seconds.
This controls how long a message is left unavailable after it is received. If a message is received, and
takes longer than visibility_timeout_secs to process and delete the message from the queue, it is made available again for another consumer.
This can happen if there is an issue between consuming a message and deleting it.
The strategy to use to consume objects from S3.
Consumes objects by processing bucket notification events sent to an AWS SQS queue.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Custom endpoint for use with AWS-compatible services.
acknowledgements:
enabled: null
assume_role: string
auth:
imds:
connect_timeout_seconds: 1
max_attempts: 4
read_timeout_seconds: 1
load_timeout_secs: null
region: null
compression: ''
decoding:
codec: bytes
framing:
method: newline_delimited
log_namespace: boolean
multiline: ''
sqs: ''
strategy: ''
tls_options: ''
type: aws_s3
Configuration for the aws_sqs source.
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullConfiguration of the authentication strategy for interacting with AWS services.
Number of concurrent tasks to create for polling the queue for messages.
Defaults to the number of available CPUs on the system.
Should not typically need to be changed, but it can sometimes be beneficial to raise this
value when there is a high rate of messages being pushed into the queue and the messages
being fetched are small. In these cases, system resources may not be fully utilized without
fetching more messages per second, as it spends more time fetching the messages than
processing them.
Configures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
Whether to delete the message once it is processed.
It can be useful to set this to false for debugging or during the initial setup.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
The namespace to use for logs. This overrides the global setting.
How long to wait while polling the queue for new messages, in seconds.
Generally, this should not be changed unless instructed to do so, as if messages are available,
they are always consumed, regardless of the value of poll_secs.
The URL of the SQS queue to poll for messages.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
The visibility timeout to use for messages, in seconds.
This controls how long a message is left unavailable after it is received. If a message is received, and
takes longer than visibility_timeout_secs to process and delete the message from the queue, it is made available again for another consumer.
This can happen if there is an issue between consuming a message and deleting it.
Custom endpoint for use with AWS-compatible services.
acknowledgements:
enabled: null
auth:
imds:
connect_timeout_seconds: 1
max_attempts: 4
read_timeout_seconds: 1
load_timeout_secs: null
region: null
client_concurrency: integer
decoding:
codec: bytes
delete_message: true
framing:
method: bytes
log_namespace: boolean
poll_secs: 15
queue_url: string
tls: ''
visibility_timeout_secs: 300
type: aws_sqs
Configuration for the datadog_agent source.
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullThe socket address to accept connections on.
It must include a port.
Configures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
If this is set to true, logs are not accepted by the component.
If this is set to true, metrics are not accepted by the component.
If this is set to true, traces are not accepted by the component.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
The namespace to use for logs. This overrides the global setting.
If this is set to true logs, metrics, and traces are sent to different outputs.
For a source component named agent, the received logs, metrics, and traces can then be
configured as input to other components by specifying agent.logs, agent.metrics, and
agent.traces, respectively.
If this is set to true, when incoming events contain a Datadog API key, it is
stored in the event metadata and used if the event is sent to a Datadog sink.
Configures the TLS options for incoming/outgoing connections.
Configures the TLS options for incoming/outgoing connections.
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
address: string
decoding:
codec: bytes
disable_logs: boolean
disable_metrics: boolean
disable_traces: boolean
framing:
method: bytes
log_namespace: boolean
multiple_outputs: boolean
store_api_key: true
tls: ''
type: datadog_agent
Configuration for the demo_logs source.
The total number of lines to output.
By default, the source continuously prints logs (infinitely).
Configures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
The amount of time, in seconds, to pause between each batch of output lines.
The default is one batch per second. To remove the delay and output batches as quickly as possible, set
interval to 0.0.
The namespace to use for logs. This overrides the global setting.
Output format configuration.
count: 9223372036854776000
decoding:
codec: bytes
framing:
method: bytes
interval: 1
log_namespace: boolean
type: demo_logs
Configuration for the fluent source.
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullThe socket address to listen for connections on, or systemd{#N} to use the Nth socket passed by
systemd socket activation.
If a socket address is used, it must include a port.
The maximum number of TCP connections that are allowed at any given time.
TCP keepalive settings for socket-based components.
TCP keepalive settings for socket-based components.
The time to wait before starting to send TCP keepalive probes on an idle connection.
The namespace to use for logs. This overrides the global setting.
The size of the receive buffer used for each connection.
This generally should not need to be changed.
TlsEnableableConfig for sources, adding metadata from the client certificate.
TlsEnableableConfig for sources, adding metadata from the client certificate.
Event field for client certificate metadata.
An optional path that deserializes an empty string to None.
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
address: string
connection_limit: integer
keepalive: ''
log_namespace: boolean
receive_buffer_bytes: integer
tls: ''
type: fluent
Configuration for the gcp_pubsub source.
The acknowledgement deadline, in seconds, to use for this stream.
DEPRECATED: Messages that are not acknowledged when this deadline expires may be retransmitted.
The acknowledgement deadline, in seconds, to use for this stream.
Messages that are not acknowledged when this deadline expires may be retransmitted.
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullConfigures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
The endpoint from which to pull data.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
The number of messages in a response to mark a stream as
"busy". This is used to determine if more streams should be
started.
The GCP Pub/Sub servers send responses with 100 or more messages when
the subscription is busy.
The amount of time, in seconds, with no received activity
before sending a keepalive request. If this is set larger than
60, you may see periodic errors sent from the server.
The namespace to use for logs. This overrides the global setting.
The maximum number of concurrent stream connections to open at once.
How often to poll the currently active streams to see if they
are all busy and so open a new stream.
The project name from which to pull logs.
DEPRECATED: The amount of time, in seconds, to wait between retry attempts after an error.
The amount of time, in seconds, to wait between retry attempts after an error.
The subscription within the project which is configured to receive logs.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
An API key.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Wrapper for sensitive strings containing credentials
Path to a service account credentials JSON file.
Either an API key or a path to a service account credentials JSON file can be specified.
If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no
filename is named, an attempt is made to fetch an instance service account for the compute instance the program is
running on. If this is not on a GCE instance, then you must define it with an API key or service account
credentials JSON file.
Skip all authentication handling. For use with integration tests only.
ack_deadline_seconds: integer
ack_deadline_secs: 600
acknowledgements:
enabled: null
decoding:
codec: bytes
endpoint: 'https://pubsub.googleapis.com'
framing:
method: bytes
full_response_size: 100
keepalive_secs: 60
log_namespace: boolean
max_concurrency: 10
poll_time_seconds: 2
project: string
retry_delay_seconds: number
retry_delay_secs: 1
subscription: string
tls: ''
type: gcp_pubsub
Configuration for heroku_logs source.
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullThe socket address to listen for connections on.
HTTP Basic authentication configuration.
HTTP Basic authentication configuration.
The password for basic authentication.
The username for basic authentication.
Configures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
The namespace to use for logs. This overrides the global setting.
A list of URL query parameters to include in the log event.
These override any values included in the body with conflicting names.
Configures the TLS options for incoming/outgoing connections.
Configures the TLS options for incoming/outgoing connections.
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
address: string
auth: ''
decoding:
codec: bytes
framing:
method: bytes
log_namespace: boolean
query_parameters: []
tls: ''
type: heroku_logs
Configuration for the host_metrics source.
Options for the cgroups (controller groups) metrics collector.
This collector is only available on Linux systems, and only supports either version 2 or hybrid cgroups.
Options for the cgroups (controller groups) metrics collector.
This collector is only available on Linux systems, and only supports either version 2 or hybrid cgroups.
The base cgroup name to provide metrics for.
default: nullBase cgroup directory, for testing use only
Lists of cgroup name patterns to include or exclude in gathering
usage metrics.
default: {"excludes":null,"includes":null}Any patterns which should be excluded.
The patterns are matched using globbing.
default: nullAny patterns which should be included.
The patterns are matched using globbing.
default: ["*"]The number of levels of the cgroups hierarchy for which to report metrics.
A value of 1 means the root or named cgroup.
default: 100The list of host metric collector services to use.
Defaults to all collectors.
Options for the disk metrics collector.
Lists of device name patterns to include or exclude in gathering
I/O utilization metrics.
default: {"excludes":null,"includes":null}Any patterns which should be excluded.
The patterns are matched using globbing.
default: nullAny patterns which should be included.
The patterns are matched using globbing.
default: ["*"]Options for the filesystem metrics collector.
Lists of device name patterns to include or exclude in gathering
usage metrics.
default: {"excludes":null,"includes":null}Any patterns which should be excluded.
The patterns are matched using globbing.
default: nullAny patterns which should be included.
The patterns are matched using globbing.
default: ["*"]Lists of filesystem name patterns to include or exclude in gathering
usage metrics.
default: {"excludes":null,"includes":null}Any patterns which should be excluded.
The patterns are matched using globbing.
default: nullAny patterns which should be included.
The patterns are matched using globbing.
default: ["*"]Lists of mount point path patterns to include or exclude in gathering
usage metrics.
default: {"excludes":null,"includes":null}Any patterns which should be excluded.
The patterns are matched using globbing.
default: nullAny patterns which should be included.
The patterns are matched using globbing.
default: ["*"]Overrides the default namespace for the metrics emitted by the source.
Options for the network metrics collector.
Lists of device name patterns to include or exclude in gathering
network utilization metrics.
default: {"excludes":null,"includes":null}Any patterns which should be excluded.
The patterns are matched using globbing.
default: nullAny patterns which should be included.
The patterns are matched using globbing.
default: ["*"]The interval between metric gathering, in seconds.
cgroups:
base: null
groups:
excludes: null
includes: null
levels: 100
collectors:
- cpu
- disk
- filesystem
- load
- host
- memory
- network
- cgroups
disk:
devices:
excludes: null
includes: null
filesystem:
devices:
excludes: null
includes: null
filesystems:
excludes: null
includes: null
mountpoints:
excludes: null
includes: null
namespace: host
network:
devices:
excludes: null
includes: null
scrape_interval_secs: 15
type: host_metrics
Configuration for the http source.
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullThe socket address to listen for connections on.
It must include a port.
HTTP Basic authentication configuration.
HTTP Basic authentication configuration.
The password for basic authentication.
The username for basic authentication.
Configures how events are decoded from raw bytes.
Configures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
The expected encoding of received data.
For json and ndjson encodings, the fields of the JSON objects are output as separate fields.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
A list of HTTP headers to include in the log event.
These override any values included in the JSON payload with conflicting names.
The namespace to use for logs. This overrides the global setting.
Specifies the action of the HTTP request.
The URL path on which log event POST requests are sent.
The event key in which the requested URL path used to send the request is stored.
A list of URL query parameters to include in the log event.
These override any values included in the body with conflicting names.
Specifies the HTTP response status code that will be returned on successful requests.
Whether or not to treat the configured path as an absolute path.
If set to true, only requests using the exact URL path specified in path are accepted. Otherwise,
requests sent to a URL path that starts with the value of path are accepted.
With strict_path set to false and path set to "", the configured HTTP source accepts requests from
any URL path.
Configures the TLS options for incoming/outgoing connections.
Configures the TLS options for incoming/outgoing connections.
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
address: string
auth: ''
decoding: ''
encoding: ''
framing: ''
headers: []
log_namespace: boolean
method: POST
path: /
path_key: path
query_parameters: []
response_code: 200
strict_path: true
tls: ''
type: http
Configuration for the http_client source.
Configuration of the authentication strategy for HTTP requests.
HTTP Authentication.
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic authentication.
The username and password are concatenated and encoded via base64.
The basic authentication password.
Basic authentication.
The username and password are concatenated and encoded via base64.
The basic authentication username.
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
Bearer authentication.
The bearer token value (OAuth2, JWT, etc.) is passed as-is.
The bearer authentication token.
Decoder to use on the HTTP responses.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
The HTTP endpoint to collect events from.
The full path must be specified.
Framing configuration.
Framing to use in the decoding.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
Headers to apply to the HTTP requests.
One or more values for the same header can be provided.
The namespace to use for logs. This overrides the global setting.
Specifies the method of the HTTP request.
Custom parameters for the HTTP request query string.
One or more values for the same parameter key can be provided.
The parameters provided in this option are appended to any parameters
manually provided in the endpoint option.
The interval between scrapes. Requests are run concurrently so if a scrape takes longer
than the interval a new scrape will be started. This can take extra resources, set the timeout
to a value lower than the scrape interval to prevent this from happening.
The timeout for each scrape request.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
auth: ''
decoding:
codec: bytes
endpoint: string
framing:
method: bytes
headers: {}
log_namespace: boolean
method: GET
query: {}
scrape_interval_secs: 15
scrape_timeout_secs: 5
tls: ''
type: http_client
Configuration for the http_server source.
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullThe socket address to listen for connections on.
It must include a port.
HTTP Basic authentication configuration.
HTTP Basic authentication configuration.
The password for basic authentication.
The username for basic authentication.
Configures how events are decoded from raw bytes.
Configures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
The expected encoding of received data.
For json and ndjson encodings, the fields of the JSON objects are output as separate fields.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
A list of HTTP headers to include in the log event.
These override any values included in the JSON payload with conflicting names.
The namespace to use for logs. This overrides the global setting.
Specifies the action of the HTTP request.
The URL path on which log event POST requests are sent.
The event key in which the requested URL path used to send the request is stored.
A list of URL query parameters to include in the log event.
These override any values included in the body with conflicting names.
Specifies the HTTP response status code that will be returned on successful requests.
Whether or not to treat the configured path as an absolute path.
If set to true, only requests using the exact URL path specified in path are accepted. Otherwise,
requests sent to a URL path that starts with the value of path are accepted.
With strict_path set to false and path set to "", the configured HTTP source accepts requests from
any URL path.
Configures the TLS options for incoming/outgoing connections.
Configures the TLS options for incoming/outgoing connections.
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
address: string
auth: ''
decoding: ''
encoding: ''
framing: ''
headers: []
log_namespace: boolean
method: POST
path: /
path_key: path
query_parameters: []
response_code: 200
strict_path: true
tls: ''
type: http_server
Configuration for the internal_logs source.
Overrides the name of the log field used to add the current hostname to each event.
By default, the global log_schema.host_key option is used.
Set to "" to suppress this key.
The namespace to use for logs. This overrides the global setting.
Overrides the name of the log field used to add the current process ID to each event.
By default, "pid" is used.
Set to "" to suppress this key.
host_key: host
log_namespace: boolean
pid_key: pid
type: internal_logs
Configuration for the internal_metrics source.
Overrides the default namespace for the metrics emitted by the source.
The interval between metric gathering, in seconds.
Tag configuration for the internal_metrics source.
Overrides the name of the tag used to add the peer host to each metric.
The value is the peer host's address, including the port. For example, 1.2.3.4:9000.
By default, the global log_schema.host_key option is used.
Set to "" to suppress this key.
default: hostSets the name of the tag to use to add the current process ID to each metric.
By default, this is not set and the tag is not automatically added.
default: nullnamespace: vector
scrape_interval_secs: 1
tags:
host_key: host
pid_key: null
type: internal_metrics
Configuration for the kafka source.
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullIf offsets for consumer group do not exist, set them using this strategy.
See the librdkafka documentation for the auto.offset.reset option for further clarification.
A comma-separated list of Kafka bootstrap servers.
These are the servers in a Kafka cluster that a client should use to bootstrap its connection to the cluster,
allowing discovery of all the other hosts in the cluster.
Must be in the form of host:port, and comma-separated.
The frequency that the consumer offsets are committed (written) to offset storage.
Configures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
Maximum time the broker may wait to fill the response.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the byte stream.
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).
Byte frames which are delimited by a chosen character.
Options for the character delimited decoder.
The character that delimits byte sequences.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a chosen character.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.
Byte frames which are delimited by a newline character.
Options for the newline delimited decoder.
The maximum length of the byte buffer.
This length does not include the trailing delimiter.
By default, there is no maximum length enforced. If events are malformed, this can lead to
additional resource usage as events continue to be buffered in memory, and can potentially
lead to memory exhaustion in extreme cases.
If there is a risk of processing malformed data, such as logs with user-controlled input,
consider setting the maximum length to a reasonably large value as a safety net. This
ensures that processing is not actually unbounded.
Byte frames which are delimited by a newline character.
Options for the octet counting decoder.
The maximum length of the byte buffer.
The consumer group name to be used to consume events from Kafka.
Overrides the name of the log field used to add the headers to each event.
The value is the headers of the Kafka message itself.
By default, "headers" is used.
Overrides the name of the log field used to add the message key to each event.
The value is the message key of the Kafka message itself.
By default, "message_key" is used.
The namespace to use for logs. This overrides the global setting.
Expose topic lag metrics for all topics and partitions. Metric names are kafka_consumer_lag.
default: falseOverrides the name of the log field used to add the offset to each event.
The value is the offset of the Kafka message itself.
By default, "offset" is used.
Overrides the name of the log field used to add the partition to each event.
The value is the partition from which the Kafka message was consumed from.
By default, "partition" is used.
The Kafka session timeout.
Timeout for network requests.
Overrides the name of the log field used to add the topic to each event.
The value is the topic from which the Kafka message was consumed from.
By default, "topic" is used.
The Kafka topics names to read events from.
Regular expression syntax is supported if the topic begins with ^.
Configuration for SASL authentication when interacting with Kafka.
Configuration for SASL authentication when interacting with Kafka.
Enables SASL authentication.
Only PLAIN- and SCRAM-based mechanisms are supported when configuring SASL authentication using sasl.*. For
other mechanisms, librdkafka_options.* must be used directly to configure other librdkafka-specific values.
If using sasl.kerberos.* as an example, where * is service.name, principal, kinit.md, etc., then
librdkafka_options.* as a result becomes librdkafka_options.sasl.kerberos.service.name,
librdkafka_options.sasl.kerberos.principal, etc.
See the librdkafka documentation for details.
SASL authentication is not supported on Windows.
The SASL mechanism to use.
Wrapper for sensitive strings containing credentials
Configures the TLS options for incoming/outgoing connections.
Configures the TLS options for incoming/outgoing connections.
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
auto_offset_reset: largest
bootstrap_servers: string
commit_interval_ms: 5000
decoding:
codec: bytes
fetch_wait_max_ms: 100
framing:
method: bytes
group_id: string
headers_key: headers
key_field: message_key
librdkafka_options: object
log_namespace: boolean
metrics:
topic_lag_metric: false
offset_key: offset
partition_key: partition
session_timeout_ms: 10000
socket_timeout_ms: 60000
topic_key: topic
topics: array
type: kafka
Configuration for the logstash source.
Controls how acknowledgements are handled by this source.
DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.
Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.
See End-to-end Acknowledgements for more information on how event acknowledgement is handled.
Whether or not end-to-end acknowledgements are enabled for this source.
default: nullThe socket address to listen for connections on, or systemd{#N} to use the Nth socket passed by
systemd socket activation.
If a socket address is used, it must include a port.
The maximum number of TCP connections that are allowed at any given time.
TCP keepalive settings for socket-based components.
TCP keepalive settings for socket-based components.
The time to wait before starting to send TCP keepalive probes on an idle connection.
The namespace to use for logs. This overrides the global setting.
The size of the receive buffer used for each connection.
TlsEnableableConfig for sources, adding metadata from the client certificate.
TlsEnableableConfig for sources, adding metadata from the client certificate.
Event field for client certificate metadata.
An optional path that deserializes an empty string to None.
Whether or not to require TLS for incoming or outgoing connections.
When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for
more information.
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order
that they are defined.
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
Enables certificate verification.
If enabled, certificates must not be expired and must be issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the
certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and
so on until the verification process reaches a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
acknowledgements:
enabled: null
address: string
connection_limit: integer
keepalive: ''
log_namespace: boolean
receive_buffer_bytes: integer
tls: ''
type: logstash
Configuration for the nats source.
Configuration of the authentication strategy when interacting with NATS.
Configuration of the authentication strategy when interacting with NATS.
Username/password authentication.
Username/password authentication.
Username and password configuration.
Credentials file authentication. (JWT-based)
Credentials file configuration.
Path to credentials file.
Credentials file authentication. (JWT-based)
User.
Conceptually, this is equivalent to a public key.
Seed.
Conceptually, this is equivalent to a private key.
A name assigned to the NATS connection.
Configures how events are decoded from raw bytes.
Uses the raw bytes as-is.
Uses the raw bytes as-is.
Decodes the raw bytes as JSON.
JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as JSON.
Protobuf-specific decoding options.
message type. e.g package.message
Decodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Syslog-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a Syslog message.
Decodes either as the RFC 3164-style format ("old" style) or the
RFC 5424-style format ("new" style, includes structured data).
Vector's native JSON-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
GELF-specific decoding options.
Determines whether or not to replace invalid UTF-8 sequences instead of failing.
When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.
default: trueDecodes the raw bytes as a GELF message.
Framing configuration.
Framing handles how events are separated when encoded in a raw byte form, where each event is
a frame that must be prefixed, or delimited, in a way that marks where an event begins and
ends within the by
