이 페이지는 아직 한국어로 제공되지 않으며 번역 작업 중입니다. 번역에 관한 질문이나 의견이 있으시면 언제든지 저희에게 연락해 주십시오.

Observability Pipelines is not available on the US1-FED Datadog site.

A source is where data is collected and sent to Observability Pipelines. The source component in a configuration defines how Observability Pipelines collects or receives data from the source.

AMQP

Supports AMQP version 0.9.1

항목

required

유형

설명

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

consumer

optional

string

The identifier for the consumer.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

exchange_key

optional

string

The AMQP exchange key.

framing

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

offset_key

optional

string

The AMQP offset key.

queue

optional

string

The name of the queue to consume.

routing_key_field

optional

string

The AMQP routing key.

connection_string

required

string

URI for the AMQP server.

The URI has the format of amqp://<user>:<password>@<host>:<port>/<vhost>?timeout=<seconds>.

The default vhost can be specified by using a value of %2f.

To connect over TLS, a scheme of amqps can be specified instead. For example, amqps://.... Additional TLS settings, such as client certificate verification, can be configured under the tls section.

tls

optional

 <oneOf>

TLS configuration.

Option 1

optional

object

TLS configuration.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

acknowledgements:
  enabled: null
consumer: vector
decoding:
  codec: bytes
exchange_key: exchange
framing:
  method: bytes
log_namespace: boolean
offset_key: offset
queue: vector
routing_key_field: routing
type: amqp

AWS Kinesis Firehose

Configuration for the aws_kinesis_firehose source.

항목

required

유형

설명

access_key

optional

 <oneOf>

An access key to authenticate requests against.

DEPRECATED: AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If configured, access_key should be set to the same value. Otherwise, all requests are allowed.

Option 1

optional

string

Wrapper for sensitive strings containing credentials

access_keys

optional

array,​null

A list of access keys to authenticate requests against.

AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If configured, access_keys should be set to the same value. Otherwise, all requests are allowed.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

address

required

string

The socket address to listen for connections on.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

framing

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

record_compression

optional

 <oneOf>

The compression scheme to use for decompressing records within the Firehose message.

Some services, like AWS CloudWatch Logs, compresses the events with gzip, before sending them AWS Kinesis Firehose. This option can be used to automatically decompress them before forwarding them to the next component.

Note that this is different from Content encoding option of the Firehose HTTP endpoint destination. That option controls the content encoding of the entire HTTP request.

auto

optional

auto

Automatically attempt to determine the compression scheme.

The compression scheme of the object is determined by looking at its file signature, also known as magic bytes.

If the record fails to decompress with the discovered format, the record is forwarded as is. Thus, if you know the records are always gzip encoded (for example, if they are coming from AWS CloudWatch Logs), set gzip in this field so that any records that are not-gzipped are rejected.

none

optional

none

Uncompressed.

gzip

optional

gzip

GZIP.

store_access_key

required

boolean

Whether or not to store the AWS Firehose Access Key in event secrets.

If set to true, when incoming requests contains an access key sent by AWS Firehose, it is kept in the event secrets as "aws_kinesis_firehose_access_key".

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

Configures the TLS options for incoming/outgoing connections.

enabled

optional

boolean,​null

Whether or not to require TLS for incoming or outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

access_key: ''
access_keys: array
acknowledgements:
  enabled: null
address: string
decoding:
  codec: bytes
framing:
  method: bytes
log_namespace: boolean
record_compression: auto
store_access_key: boolean
tls: ''
type: aws_kinesis_firehose

AWS S3

Configuration for the aws_s3 source.

항목

required

유형

설명

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

assume_role

optional

string,​null

The ARN of an [IAM role][iam_role] to assume at startup.

DEPRECATED: [iam_role]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html

auth

optional

Configuration of the authentication strategy for interacting with AWS services.

compression

optional

 <oneOf>

The compression scheme used for decompressing objects retrieved from S3.

auto

optional

auto

Automatically attempt to determine the compression scheme.

The compression scheme of the object is determined from its Content-Encoding and Content-Type metadata, as well as the key suffix (for example, .gz).

It is set to none if the compression scheme cannot be determined.

none

optional

none

Uncompressed.

gzip

optional

gzip

GZIP.

zstd

optional

zstd

ZSTD.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

framing

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

multiline

optional

 <oneOf>

Multiline aggregation configuration.

If not specified, multiline aggregation is disabled.

Option 1

optional

object

Configuration of multi-line aggregation.

condition_pattern

required

string

Regular expression pattern that is used to determine whether or not more lines should be read.

This setting must be configured in conjunction with mode.

mode

required

 <oneOf>

Aggregation mode.

This setting must be configured in conjunction with condition_pattern.

continue_through

optional

continue_through

All consecutive lines matching this pattern are included in the group.

The first line (the line that matched the start pattern) does not need to match the ContinueThrough pattern.

This is useful in cases such as a Java stack trace, where some indicator in the line (such as a leading whitespace) indicates that it is an extension of the proceeding line.

continue_past

optional

continue_past

All consecutive lines matching this pattern, plus one additional line, are included in the group.

This is useful in cases where a log message ends with a continuation marker, such as a backslash, indicating that the following line is part of the same message.

halt_before

optional

halt_before

All consecutive lines not matching this pattern are included in the group.

This is useful where a log line contains a marker indicating that it begins a new message.

halt_with

optional

halt_with

All consecutive lines, up to and including the first line matching this pattern, are included in the group.

This is useful where a log line ends with a termination marker, such as a semicolon.

start_pattern

required

string

Regular expression pattern that is used to match the start of a new message.

timeout_ms

required

integer

The maximum amount of time to wait for the next additional line, in milliseconds.

Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.

sqs

optional

 <oneOf>

Configuration options for SQS.

Option 1

optional

object

SQS configuration options.

client_concurrency

optional

integer,​null

Number of concurrent tasks to create for polling the queue for messages.

Defaults to the number of available CPUs on the system.

Should not typically need to be changed, but it can sometimes be beneficial to raise this value when there is a high rate of messages being pushed into the queue and the objects being fetched are small. In these cases, system resources may not be fully utilized without fetching more messages per second, as the SQS message consumption rate affects the S3 object retrieval rate.

delete_message

optional

boolean

Whether to delete the message once it is processed.

It can be useful to set this to false for debugging or during the initial setup.

poll_secs

optional

integer

How long to wait while polling the queue for new messages, in seconds.

Generally, this should not be changed unless instructed to do so, as if messages are available, they are always consumed, regardless of the value of poll_secs.

queue_url

required

uri

The URL of the SQS queue to poll for bucket notifications.

tls_options

optional

 <oneOf>

TLS configuration.

Option 1

optional

object

TLS configuration.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

visibility_timeout_secs

optional

integer

The visibility timeout to use for messages, in seconds.

This controls how long a message is left unavailable after it is received. If a message is received, and takes longer than visibility_timeout_secs to process and delete the message from the queue, it is made available again for another consumer.

This can happen if there is an issue between consuming a message and deleting it.

strategy

optional

 <oneOf>

The strategy to use to consume objects from S3.

sqs

optional

sqs

Consumes objects by processing bucket notification events sent to an AWS SQS queue.

tls_options

optional

 <oneOf>

TLS configuration.

Option 1

optional

object

TLS configuration.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

endpoint

optional

string,​null

Custom endpoint for use with AWS-compatible services.

region

optional

string,​null

The AWS region of the target service.

acknowledgements:
  enabled: null
assume_role: string
auth:
  imds:
    connect_timeout_seconds: 1
    max_attempts: 4
    read_timeout_seconds: 1
  load_timeout_secs: null
  region: null
compression: ''
decoding:
  codec: bytes
framing:
  method: newline_delimited
log_namespace: boolean
multiline: ''
sqs: ''
strategy: ''
tls_options: ''
type: aws_s3

AWS SQS

Configuration for the aws_sqs source.

항목

required

유형

설명

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

auth

optional

Configuration of the authentication strategy for interacting with AWS services.

client_concurrency

optional

integer,​null

Number of concurrent tasks to create for polling the queue for messages.

Defaults to the number of available CPUs on the system.

Should not typically need to be changed, but it can sometimes be beneficial to raise this value when there is a high rate of messages being pushed into the queue and the messages being fetched are small. In these cases, system resources may not be fully utilized without fetching more messages per second, as it spends more time fetching the messages than processing them.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

delete_message

optional

boolean

Whether to delete the message once it is processed.

It can be useful to set this to false for debugging or during the initial setup.

framing

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

poll_secs

optional

integer

How long to wait while polling the queue for new messages, in seconds.

Generally, this should not be changed unless instructed to do so, as if messages are available, they are always consumed, regardless of the value of poll_secs.

queue_url

required

string

The URL of the SQS queue to poll for messages.

tls

optional

 <oneOf>

TLS configuration.

Option 1

optional

object

TLS configuration.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

visibility_timeout_secs

optional

integer

The visibility timeout to use for messages, in seconds.

This controls how long a message is left unavailable after it is received. If a message is received, and takes longer than visibility_timeout_secs to process and delete the message from the queue, it is made available again for another consumer.

This can happen if there is an issue between consuming a message and deleting it.

endpoint

optional

string,​null

Custom endpoint for use with AWS-compatible services.

region

optional

string,​null

The AWS region of the target service.

acknowledgements:
  enabled: null
auth:
  imds:
    connect_timeout_seconds: 1
    max_attempts: 4
    read_timeout_seconds: 1
  load_timeout_secs: null
  region: null
client_concurrency: integer
decoding:
  codec: bytes
delete_message: true
framing:
  method: bytes
log_namespace: boolean
poll_secs: 15
queue_url: string
tls: ''
visibility_timeout_secs: 300
type: aws_sqs

Datadog Agent

Configuration for the datadog_agent source.

항목

required

유형

설명

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

address

required

string

The socket address to accept connections on.

It must include a port.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

disable_logs

optional

boolean

If this is set to true, logs are not accepted by the component.

disable_metrics

optional

boolean

If this is set to true, metrics are not accepted by the component.

disable_traces

optional

boolean

If this is set to true, traces are not accepted by the component.

framing

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

multiple_outputs

optional

boolean

If this is set to true logs, metrics, and traces are sent to different outputs.

For a source component named agent, the received logs, metrics, and traces can then be configured as input to other components by specifying agent.logs, agent.metrics, and agent.traces, respectively.

store_api_key

optional

boolean

If this is set to true, when incoming events contain a Datadog API key, it is stored in the event metadata and used if the event is sent to a Datadog sink.

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

Configures the TLS options for incoming/outgoing connections.

enabled

optional

boolean,​null

Whether or not to require TLS for incoming or outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

acknowledgements:
  enabled: null
address: string
decoding:
  codec: bytes
disable_logs: boolean
disable_metrics: boolean
disable_traces: boolean
framing:
  method: bytes
log_namespace: boolean
multiple_outputs: boolean
store_api_key: true
tls: ''
type: datadog_agent

Demo Logs

Configuration for the demo_logs source.

항목

required

유형

설명

count

optional

integer

The total number of lines to output.

By default, the source continuously prints logs (infinitely).

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

framing

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

interval

optional

number

The amount of time, in seconds, to pause between each batch of output lines.

The default is one batch per second. To remove the delay and output batches as quickly as possible, set interval to 0.0.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

_metadata

optional

description

optional

Output format configuration.

oneOf

optional

count: 9223372036854776000
decoding:
  codec: bytes
framing:
  method: bytes
interval: 1
log_namespace: boolean
type: demo_logs

Fluent

Configuration for the fluent source.

항목

required

유형

설명

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

address

required

string

The socket address to listen for connections on, or systemd{#N} to use the Nth socket passed by systemd socket activation.

If a socket address is used, it must include a port.

connection_limit

optional

integer,​null

The maximum number of TCP connections that are allowed at any given time.

keepalive

optional

 <oneOf>

TCP keepalive settings for socket-based components.

Option 1

optional

object

TCP keepalive settings for socket-based components.

time_secs

optional

integer,​null

The time to wait before starting to send TCP keepalive probes on an idle connection.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

receive_buffer_bytes

optional

integer,​null

The size of the receive buffer used for each connection.

This generally should not need to be changed.

tls

optional

 <oneOf>

TlsEnableableConfig for sources, adding metadata from the client certificate.

Option 1

optional

TlsEnableableConfig for sources, adding metadata from the client certificate.

client_metadata_key

optional

 <oneOf>

Event field for client certificate metadata.

Option 1

optional

string

An optional path that deserializes an empty string to None.

enabled

optional

boolean,​null

Whether or not to require TLS for incoming or outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

acknowledgements:
  enabled: null
address: string
connection_limit: integer
keepalive: ''
log_namespace: boolean
receive_buffer_bytes: integer
tls: ''
type: fluent

GCP Pub/Sub

Configuration for the gcp_pubsub source.

항목

required

유형

설명

ack_deadline_seconds

optional

integer,​null

The acknowledgement deadline, in seconds, to use for this stream.

DEPRECATED: Messages that are not acknowledged when this deadline expires may be retransmitted.

ack_deadline_secs

optional

integer

The acknowledgement deadline, in seconds, to use for this stream.

Messages that are not acknowledged when this deadline expires may be retransmitted.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

endpoint

optional

string

The endpoint from which to pull data.

framing

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

full_response_size

optional

integer

The number of messages in a response to mark a stream as "busy". This is used to determine if more streams should be started.

The GCP Pub/Sub servers send responses with 100 or more messages when the subscription is busy.

keepalive_secs

optional

number

The amount of time, in seconds, with no received activity before sending a keepalive request. If this is set larger than 60, you may see periodic errors sent from the server.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

max_concurrency

optional

integer

The maximum number of concurrent stream connections to open at once.

poll_time_seconds

optional

number

How often to poll the currently active streams to see if they are all busy and so open a new stream.

project

required

string

The project name from which to pull logs.

retry_delay_seconds

optional

number,​null

DEPRECATED: The amount of time, in seconds, to wait between retry attempts after an error.

retry_delay_secs

optional

number

The amount of time, in seconds, to wait between retry attempts after an error.

subscription

required

string

The subscription within the project which is configured to receive logs.

tls

optional

 <oneOf>

TLS configuration.

Option 1

optional

object

TLS configuration.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

api_key

optional

 <oneOf>

An API key.

Either an API key or a path to a service account credentials JSON file can be specified.

If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no filename is named, an attempt is made to fetch an instance service account for the compute instance the program is running on. If this is not on a GCE instance, then you must define it with an API key or service account credentials JSON file.

Option 1

optional

string

Wrapper for sensitive strings containing credentials

credentials_path

optional

string,​null

Path to a service account credentials JSON file.

Either an API key or a path to a service account credentials JSON file can be specified.

If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no filename is named, an attempt is made to fetch an instance service account for the compute instance the program is running on. If this is not on a GCE instance, then you must define it with an API key or service account credentials JSON file.

skip_authentication

optional

boolean

Skip all authentication handling. For use with integration tests only.

ack_deadline_seconds: integer
ack_deadline_secs: 600
acknowledgements:
  enabled: null
decoding:
  codec: bytes
endpoint: 'https://pubsub.googleapis.com'
framing:
  method: bytes
full_response_size: 100
keepalive_secs: 60
log_namespace: boolean
max_concurrency: 10
poll_time_seconds: 2
project: string
retry_delay_seconds: number
retry_delay_secs: 1
subscription: string
tls: ''
type: gcp_pubsub

Heroku Logs

Configuration for heroku_logs source.

항목

required

유형

설명

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

address

required

string

The socket address to listen for connections on.

auth

optional

 <oneOf>

HTTP Basic authentication configuration.

Option 1

optional

object

HTTP Basic authentication configuration.

password

required

string

The password for basic authentication.

username

required

string

The username for basic authentication.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

framing

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

query_parameters

optional

[string]

A list of URL query parameters to include in the log event.

These override any values included in the body with conflicting names.

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

Configures the TLS options for incoming/outgoing connections.

enabled

optional

boolean,​null

Whether or not to require TLS for incoming or outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

acknowledgements:
  enabled: null
address: string
auth: ''
decoding:
  codec: bytes
framing:
  method: bytes
log_namespace: boolean
query_parameters: []
tls: ''
type: heroku_logs

Host Metrics

Configuration for the host_metrics source.

항목

required

유형

설명

cgroups

optional

 <oneOf>

Options for the cgroups (controller groups) metrics collector.

This collector is only available on Linux systems, and only supports either version 2 or hybrid cgroups.

Option 1

optional

object

Options for the cgroups (controller groups) metrics collector.

This collector is only available on Linux systems, and only supports either version 2 or hybrid cgroups.

base

optional

 <oneOf>

The base cgroup name to provide metrics for.

default: null

Option 1

optional

string

A file path.

base_dir

optional

 <oneOf>

Base cgroup directory, for testing use only

Option 1

optional

string

A file path.

groups

optional

object

Lists of cgroup name patterns to include or exclude in gathering usage metrics.

default: {"excludes":null,"includes":null}

excludes

optional

array,​null

Any patterns which should be excluded.

The patterns are matched using globbing.

default: null

includes

optional

array,​null

Any patterns which should be included.

The patterns are matched using globbing.

default: ["*"]

levels

optional

integer

The number of levels of the cgroups hierarchy for which to report metrics.

A value of 1 means the root or named cgroup.

default: 100

collectors

optional

array,​null

The list of host metric collector services to use.

Defaults to all collectors.

disk

optional

object

Options for the disk metrics collector.

devices

optional

object

Lists of device name patterns to include or exclude in gathering I/O utilization metrics.

default: {"excludes":null,"includes":null}

excludes

optional

array,​null

Any patterns which should be excluded.

The patterns are matched using globbing.

default: null

includes

optional

array,​null

Any patterns which should be included.

The patterns are matched using globbing.

default: ["*"]

filesystem

optional

object

Options for the filesystem metrics collector.

devices

optional

object

Lists of device name patterns to include or exclude in gathering usage metrics.

default: {"excludes":null,"includes":null}

excludes

optional

array,​null

Any patterns which should be excluded.

The patterns are matched using globbing.

default: null

includes

optional

array,​null

Any patterns which should be included.

The patterns are matched using globbing.

default: ["*"]

filesystems

optional

object

Lists of filesystem name patterns to include or exclude in gathering usage metrics.

default: {"excludes":null,"includes":null}

excludes

optional

array,​null

Any patterns which should be excluded.

The patterns are matched using globbing.

default: null

includes

optional

array,​null

Any patterns which should be included.

The patterns are matched using globbing.

default: ["*"]

mountpoints

optional

object

Lists of mount point path patterns to include or exclude in gathering usage metrics.

default: {"excludes":null,"includes":null}

excludes

optional

array,​null

Any patterns which should be excluded.

The patterns are matched using globbing.

default: null

includes

optional

array,​null

Any patterns which should be included.

The patterns are matched using globbing.

default: ["*"]

namespace

optional

string,​null

Overrides the default namespace for the metrics emitted by the source.

network

optional

object

Options for the network metrics collector.

devices

optional

object

Lists of device name patterns to include or exclude in gathering network utilization metrics.

default: {"excludes":null,"includes":null}

excludes

optional

array,​null

Any patterns which should be excluded.

The patterns are matched using globbing.

default: null

includes

optional

array,​null

Any patterns which should be included.

The patterns are matched using globbing.

default: ["*"]

scrape_interval_secs

optional

integer

The interval between metric gathering, in seconds.

cgroups:
  base: null
  groups:
    excludes: null
    includes: null
  levels: 100
collectors:
  - cpu
  - disk
  - filesystem
  - load
  - host
  - memory
  - network
  - cgroups
disk:
  devices:
    excludes: null
    includes: null
filesystem:
  devices:
    excludes: null
    includes: null
  filesystems:
    excludes: null
    includes: null
  mountpoints:
    excludes: null
    includes: null
namespace: host
network:
  devices:
    excludes: null
    includes: null
scrape_interval_secs: 15
type: host_metrics

HTTP

Configuration for the http source.

항목

required

유형

설명

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

address

required

string

The socket address to listen for connections on.

It must include a port.

auth

optional

 <oneOf>

HTTP Basic authentication configuration.

Option 1

optional

object

HTTP Basic authentication configuration.

password

required

string

The password for basic authentication.

username

required

string

The username for basic authentication.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Option 1

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

encoding

optional

 <oneOf>

The expected encoding of received data.

For json and ndjson encodings, the fields of the JSON objects are output as separate fields.

Option 1

optional

 <oneOf>

Content encoding.

text

optional

text

Plaintext.

ndjson

optional

ndjson

Newline-delimited JSON.

json

optional

json

JSON.

binary

optional

binary

Binary.

framing

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Option 1

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

headers

optional

[string]

A list of HTTP headers to include in the log event.

These override any values included in the JSON payload with conflicting names.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

method

optional

 <oneOf>

Specifies the action of the HTTP request.

HEAD

optional

HEAD

HTTP HEAD method.

GET

optional

GET

HTTP GET method.

POST

optional

POST

HTTP POST method.

PUT

optional

PUT

HTTP Put method.

PATCH

optional

PATCH

HTTP PATCH method.

DELETE

optional

DELETE

HTTP DELETE method.

path

optional

string

The URL path on which log event POST requests are sent.

path_key

optional

string

The event key in which the requested URL path used to send the request is stored.

query_parameters

optional

[string]

A list of URL query parameters to include in the log event.

These override any values included in the body with conflicting names.

response_code

optional

integer

Specifies the HTTP response status code that will be returned on successful requests.

strict_path

optional

boolean

Whether or not to treat the configured path as an absolute path.

If set to true, only requests using the exact URL path specified in path are accepted. Otherwise, requests sent to a URL path that starts with the value of path are accepted.

With strict_path set to false and path set to "", the configured HTTP source accepts requests from any URL path.

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

Configures the TLS options for incoming/outgoing connections.

enabled

optional

boolean,​null

Whether or not to require TLS for incoming or outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

acknowledgements:
  enabled: null
address: string
auth: ''
decoding: ''
encoding: ''
framing: ''
headers: []
log_namespace: boolean
method: POST
path: /
path_key: path
query_parameters: []
response_code: 200
strict_path: true
tls: ''
type: http

HTTP Client

Configuration for the http_client source.

항목

required

유형

설명

auth

optional

 <oneOf>

Configuration of the authentication strategy for HTTP requests.

HTTP Authentication.

Option 1

optional

 <oneOf>

Configuration of the authentication strategy for HTTP requests.

HTTP authentication should be used with HTTPS only, as the authentication credentials are passed as an HTTP header without any additional encryption beyond what is provided by the transport itself.

Basic

optional

object

Basic authentication.

The username and password are concatenated and encoded via base64.

password

required

string

The basic authentication password.

strategy

required

basic

Basic authentication.

The username and password are concatenated and encoded via base64.

user

required

string

The basic authentication username.

Bearer

optional

object

Bearer authentication.

The bearer token value (OAuth2, JWT, etc.) is passed as-is.

strategy

required

bearer

Bearer authentication.

The bearer token value (OAuth2, JWT, etc.) is passed as-is.

token

required

string

The bearer authentication token.

decoding

optional

 <oneOf>

Decoder to use on the HTTP responses.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

endpoint

required

string

The HTTP endpoint to collect events from.

The full path must be specified.

framing

optional

 <oneOf>

Framing configuration.

Framing to use in the decoding.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

headers

optional

object

Headers to apply to the HTTP requests.

One or more values for the same header can be provided.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

method

optional

 <oneOf>

Specifies the method of the HTTP request.

HEAD

optional

HEAD

HTTP HEAD method.

GET

optional

GET

HTTP GET method.

POST

optional

POST

HTTP POST method.

PUT

optional

PUT

HTTP Put method.

PATCH

optional

PATCH

HTTP PATCH method.

DELETE

optional

DELETE

HTTP DELETE method.

query

optional

object

Custom parameters for the HTTP request query string.

One or more values for the same parameter key can be provided.

The parameters provided in this option are appended to any parameters manually provided in the endpoint option.

scrape_interval_secs

optional

integer

The interval between scrapes. Requests are run concurrently so if a scrape takes longer than the interval a new scrape will be started. This can take extra resources, set the timeout to a value lower than the scrape interval to prevent this from happening.

scrape_timeout_secs

optional

number

The timeout for each scrape request.

tls

optional

 <oneOf>

TLS configuration.

Option 1

optional

object

TLS configuration.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

auth: ''
decoding:
  codec: bytes
endpoint: string
framing:
  method: bytes
headers: {}
log_namespace: boolean
method: GET
query: {}
scrape_interval_secs: 15
scrape_timeout_secs: 5
tls: ''
type: http_client

HTTP Server

Configuration for the http_server source.

항목

required

유형

설명

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

address

required

string

The socket address to listen for connections on.

It must include a port.

auth

optional

 <oneOf>

HTTP Basic authentication configuration.

Option 1

optional

object

HTTP Basic authentication configuration.

password

required

string

The password for basic authentication.

username

required

string

The username for basic authentication.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Option 1

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

encoding

optional

 <oneOf>

The expected encoding of received data.

For json and ndjson encodings, the fields of the JSON objects are output as separate fields.

Option 1

optional

 <oneOf>

Content encoding.

text

optional

text

Plaintext.

ndjson

optional

ndjson

Newline-delimited JSON.

json

optional

json

JSON.

binary

optional

binary

Binary.

framing

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Option 1

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

headers

optional

[string]

A list of HTTP headers to include in the log event.

These override any values included in the JSON payload with conflicting names.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

method

optional

 <oneOf>

Specifies the action of the HTTP request.

HEAD

optional

HEAD

HTTP HEAD method.

GET

optional

GET

HTTP GET method.

POST

optional

POST

HTTP POST method.

PUT

optional

PUT

HTTP Put method.

PATCH

optional

PATCH

HTTP PATCH method.

DELETE

optional

DELETE

HTTP DELETE method.

path

optional

string

The URL path on which log event POST requests are sent.

path_key

optional

string

The event key in which the requested URL path used to send the request is stored.

query_parameters

optional

[string]

A list of URL query parameters to include in the log event.

These override any values included in the body with conflicting names.

response_code

optional

integer

Specifies the HTTP response status code that will be returned on successful requests.

strict_path

optional

boolean

Whether or not to treat the configured path as an absolute path.

If set to true, only requests using the exact URL path specified in path are accepted. Otherwise, requests sent to a URL path that starts with the value of path are accepted.

With strict_path set to false and path set to "", the configured HTTP source accepts requests from any URL path.

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

Configures the TLS options for incoming/outgoing connections.

enabled

optional

boolean,​null

Whether or not to require TLS for incoming or outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

acknowledgements:
  enabled: null
address: string
auth: ''
decoding: ''
encoding: ''
framing: ''
headers: []
log_namespace: boolean
method: POST
path: /
path_key: path
query_parameters: []
response_code: 200
strict_path: true
tls: ''
type: http_server

Internal Logs

Configuration for the internal_logs source.

항목

required

유형

설명

host_key

optional

string

Overrides the name of the log field used to add the current hostname to each event.

By default, the global log_schema.host_key option is used.

Set to "" to suppress this key.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

pid_key

optional

string

Overrides the name of the log field used to add the current process ID to each event.

By default, "pid" is used.

Set to "" to suppress this key.

host_key: host
log_namespace: boolean
pid_key: pid
type: internal_logs

Internal Metrics

Configuration for the internal_metrics source.

항목

required

유형

설명

namespace

optional

string

Overrides the default namespace for the metrics emitted by the source.

scrape_interval_secs

optional

number

The interval between metric gathering, in seconds.

tags

optional

object

Tag configuration for the internal_metrics source.

host_key

optional

string

Overrides the name of the tag used to add the peer host to each metric.

The value is the peer host's address, including the port. For example, 1.2.3.4:9000.

By default, the global log_schema.host_key option is used.

Set to "" to suppress this key.

default: host

pid_key

optional

string,​null

Sets the name of the tag to use to add the current process ID to each metric.

By default, this is not set and the tag is not automatically added.

default: null
namespace: vector
scrape_interval_secs: 1
tags:
  host_key: host
  pid_key: null
type: internal_metrics

Kafka

Configuration for the kafka source.

항목

required

유형

설명

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

auto_offset_reset

optional

string

If offsets for consumer group do not exist, set them using this strategy.

See the librdkafka documentation for the auto.offset.reset option for further clarification.

bootstrap_servers

required

string

A comma-separated list of Kafka bootstrap servers.

These are the servers in a Kafka cluster that a client should use to bootstrap its connection to the cluster, allowing discovery of all the other hosts in the cluster.

Must be in the form of host:port, and comma-separated.

commit_interval_ms

optional

integer

The frequency that the consumer offsets are committed (written) to offset storage.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

Decodes the raw bytes as JSON.

json

optional

object

JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

json

Decodes the raw bytes as JSON.

Protobuf

optional

Decodes the raw bytes as protobuf.

protobuf

optional

object

Protobuf-specific decoding options.

desc_file

required

string

Path to desc file

message_type

required

string

message type. e.g package.message

codec

required

protobuf

Decodes the raw bytes as protobuf.

Syslog

optional

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

syslog

optional

object

Syslog-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Decodes either as the RFC 3164-style format ("old" style) or the RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

native_json

optional

object

Vector's native JSON-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

Decodes the raw bytes as a GELF message.

gelf

optional

object

GELF-specific decoding options.

lossy

optional

boolean

Determines whether or not to replace invalid UTF-8 sequences instead of failing.

When true, invalid UTF-8 sequences are replaced with the U+FFFD REPLACEMENT CHARACTER.

default: true

codec

required

gelf

Decodes the raw bytes as a GELF message.

fetch_wait_max_ms

optional

integer

Maximum time the broker may wait to fill the response.

framing

optional

 <oneOf>

Framing configuration.

Framing handles how events are separated when encoded in a raw byte form, where each event is a frame that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

CharacterDelimited

optional

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases.

If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This ensures that processing is not actually unbounded.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

group_id

required

string

The consumer group name to be used to consume events from Kafka.

headers_key

optional

string

Overrides the name of the log field used to add the headers to each event.

The value is the headers of the Kafka message itself.

By default, "headers" is used.

key_field

optional

string

Overrides the name of the log field used to add the message key to each event.

The value is the message key of the Kafka message itself.

By default, "message_key" is used.

librdkafka_options

optional

object,​null

Advanced options set directly on the underlying librdkafka client.

See the librdkafka documentation for details.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

metrics

optional

object

Metrics configuration.

topic_lag_metric

required

boolean

Expose topic lag metrics for all topics and partitions. Metric names are kafka_consumer_lag.

default: false

offset_key

optional

string

Overrides the name of the log field used to add the offset to each event.

The value is the offset of the Kafka message itself.

By default, "offset" is used.

partition_key

optional

string

Overrides the name of the log field used to add the partition to each event.

The value is the partition from which the Kafka message was consumed from.

By default, "partition" is used.

session_timeout_ms

optional

integer

The Kafka session timeout.

socket_timeout_ms

optional

integer

Timeout for network requests.

topic_key

optional

string

Overrides the name of the log field used to add the topic to each event.

The value is the topic from which the Kafka message was consumed from.

By default, "topic" is used.

topics

required

[string]

The Kafka topics names to read events from.

Regular expression syntax is supported if the topic begins with ^.

sasl

optional

 <oneOf>

Configuration for SASL authentication when interacting with Kafka.

Option 1

optional

object

Configuration for SASL authentication when interacting with Kafka.

enabled

optional

boolean,​null

Enables SASL authentication.

Only PLAIN- and SCRAM-based mechanisms are supported when configuring SASL authentication using sasl.*. For other mechanisms, librdkafka_options.* must be used directly to configure other librdkafka-specific values. If using sasl.kerberos.* as an example, where * is service.name, principal, kinit.md, etc., then librdkafka_options.* as a result becomes librdkafka_options.sasl.kerberos.service.name, librdkafka_options.sasl.kerberos.principal, etc.

See the librdkafka documentation for details.

SASL authentication is not supported on Windows.

mechanism

optional

string,​null

The SASL mechanism to use.

password

optional

 <oneOf>

The SASL password.

Option 1

optional

string

Wrapper for sensitive strings containing credentials

username

optional

string,​null

The SASL username.

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

Configures the TLS options for incoming/outgoing connections.

enabled

optional

boolean,​null

Whether or not to require TLS for incoming or outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. They are prioritized in the order that they are defined.

ca_file

optional

 <oneOf>

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

crt_file

optional

 <oneOf>

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

Option 1

optional

string

A file path.

key_file

optional

 <oneOf>

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

Option 1

optional

string

A file path.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

acknowledgements:
  enabled: null
auto_offset_reset: largest
bootstrap_servers: string
commit_interval_ms: 5000
decoding:
  codec: bytes
fetch_wait_max_ms: 100
framing:
  method: bytes
group_id: string
headers_key: headers
key_field: message_key
librdkafka_options: object
log_namespace: boolean
metrics:
  topic_lag_metric: false
offset_key: offset
partition_key: partition
session_timeout_ms: 10000
socket_timeout_ms: 60000
topic_key: topic
topics: array
type: kafka

Logstash

Configuration for the logstash source.

항목

required

유형

설명

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

DEPRECATED: This setting is deprecated in favor of enabling acknowledgements at the global or sink level.

Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

address

required

string

The socket address to listen for connections on, or systemd{#N} to use the Nth socket passed by systemd socket activation.

If a socket address is used, it must include a port.

connection_limit

optional

integer,​null

The maximum number of TCP connections that are allowed at any given time.

keepalive

optional

 <oneOf>

TCP keepalive settings for socket-based components.

Option 1

optional

object

TCP keepalive settings for socket-based components.

time_secs

optional

integer,​null

The time to wait before starting to send TCP keepalive probes on an idle connection.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

receive_buffer_bytes

optional

integer,​null

The size of the receive buffer used for each connection.