Sources

A source is where data is collected and sent to Observability Pipelines. The source component in a configuration defines how Observability Pipelines collects or receives data from the source.

AMQP

Supports AMQP version 0.9.1

Field

required

Type

Description

queue

optional

string

The name of the queue to consume.

consumer

optional

string

The identifier for the consumer.

routing_key_field

optional

string

The AMQP routing key.

exchange_key

optional

string

The AMQP exchange key.

offset_key

optional

string

The AMQP offset key.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

connection_string

required

string

URI for the AMQP server.

The URI has the format of amqp://<user>:<password>@<host>:<port>/<vhost>?timeout=<seconds>.

The default vhost can be specified by using a value of %2f.

In order to connect over TLS, a scheme of amqps can be specified instead i.e. amqps://.... Additional TLS settings, such as client certificate verification, etc, can be configured under the tls section.

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

queue: vector
consumer: vector
routing_key_field: routing
exchange_key: exchange
offset_key: offset
log_namespace: boolean
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
type: amqp
queue: vector
consumer: vector
routing_key_field: routing
exchange_key: exchange
offset_key: offset
log_namespace: boolean
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
connection_string: string
tls: object
type: amqp

Apache HTTP Server (HTTPD) Metrics

Configuration for the apache_metrics source.

Field

required

Type

Description

endpoints

required

[string]

The list of mod_status endpoints to scrape metrics from.

scrape_interval_secs

optional

integer

The interval between scrapes.

namespace

optional

string

The namespace of the metric.

Disabled if empty.

endpoints: array
scrape_interval_secs: 15
namespace: apache
type: apache_metrics
endpoints: array
scrape_interval_secs: 15
namespace: apache
type: apache_metrics

AWS ECS Metrics

Configuration for the aws_ecs_metrics source.

Field

required

Type

Description

endpoint

optional

string

Base URI of the task metadata endpoint.

If empty, the URI will be automatically discovered based on the latest version detected.

By default:

  • The version 4 endpoint base URI is stored in the environment variable ECS_CONTAINER_METADATA_URI_V4.
  • The version 3 endpoint base URI is stored in the environment variable ECS_CONTAINER_METADATA_URI.
  • The version 2 endpoint base URI is 169.254.170.2/v2/.

version

optional

 <oneOf>

The version of the task metadata endpoint to use.

If empty, the version is automatically discovered based on environment variables.

By default:

  • Version 4 is used if the environment variable ECS_CONTAINER_METADATA_URI_V4 is defined.
  • Version 3 is used if the environment variable ECS_CONTAINER_METADATA_URI_V4 is not defined, but the environment variable ECS_CONTAINER_METADATA_URI is defined.
  • Version 2 is used if neither of the environment variables ECS_CONTAINER_METADATA_URI_V4 or ECS_CONTAINER_METADATA_URI are defined.

v2

optional

v2

Version 2.

More information about version 2 of the task metadata endpoint can be found here.

v3

optional

v3

Version 3.

More information about version 3 of the task metadata endpoint can be found here.

v4

optional

v4

Version 4.

More information about version 4 of the task metadata endpoint can be found here.

scrape_interval_secs

optional

integer

The interval between scrapes, in seconds.

namespace

optional

string

The namespace of the metric.

Disabled if empty.

endpoint: 'http://169.254.170.2/v2'
version: v2
scrape_interval_secs: 15
namespace: awsecs
type: aws_ecs_metrics
endpoint: 'http://169.254.170.2/v2'
version: v2
scrape_interval_secs: 15
namespace: awsecs
type: aws_ecs_metrics

AWS Kinesis Firehose

Configuration for the aws_kinesis_firehose source.

Field

required

Type

Description

address

required

string

The address to listen for connections on.

access_key

optional

string,​null

An optional access key to authenticate requests against.

DEPRECATED: AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If configured, access_key should be set to the same value. Otherwise, all requests will be allowed.

This option has been deprecated, the access_keys option should be used instead.

access_keys

optional

array,​null

An optional list of access keys to authenticate requests against.

AWS Kinesis Firehose can be configured to pass along a user-configurable access key with each request. If configured, access_keys should be set to the same value. Otherwise, all requests will be allowed.

store_access_key

required

boolean

Whether or not to store the AWS Firehose Access Key in event secrets.

If set to true, when incoming requests contains an Access Key sent by AWS Firehose, it will be kept in the event secrets as "aws_kinesis_firehose_access_key".

record_compression

optional

 <oneOf>

The compression scheme to use for decompressing records within the Firehose message.

Some services, like AWS CloudWatch Logs, will compress the events with gzip, before sending them AWS Kinesis Firehose. This option can be used to automatically decompress them before forwarding them to the next component.

Note that this is different from Content encoding option of the Firehose HTTP endpoint destination. That option controls the content encoding of the entire HTTP request.

auto

optional

auto

Automatically attempt to determine the compression scheme.

The compression scheme of the object is determined by looking at its file signature, also known as magic bytes.

If the record fails to decompress with the discovered format, the record is forwarded as is. Thus, if you know the records are always gzip encoded (for example, if they are coming from AWS CloudWatch Logs), set gzip in this field so that any records that are not-gzipped are rejected.

none

optional

none

Uncompressed.

gzip

optional

gzip

GZIP.

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

enabled

optional

boolean,​null

Whether or not to require TLS for incoming/outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

address: string
access_key: string
access_keys: array
store_access_key: boolean
record_compression: auto
tls: ''
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
log_namespace: boolean
type: aws_kinesis_firehose
address: string
access_key: string
access_keys: array
store_access_key: boolean
record_compression: auto
tls: ''
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
log_namespace: boolean
type: aws_kinesis_firehose

AWS S3

Configuration for the aws_s3 source.

Field

required

Type

Description

compression

optional

 <oneOf>

The compression scheme used for decompressing objects retrieved from S3.

auto

optional

auto

Automatically attempt to determine the compression scheme.

The compression scheme of the object is determined from its Content-Encoding and Content-Type metadata, as well as the key suffix (for example, .gz).

It is set to 'none' if the compression scheme cannot be determined.

none

optional

none

Uncompressed.

gzip

optional

gzip

GZIP.

zstd

optional

zstd

ZSTD.

strategy

optional

 <oneOf>

The strategy to use to consume objects from S3.

sqs

optional

sqs

Consumes objects by processing bucket notification events sent to an AWS SQS queue.

sqs

optional

object,​null

Configuration options for SQS.

Only relevant when strategy = "sqs".

queue_url

required

string

The URL of the SQS queue to poll for bucket notifications.

poll_secs

optional

integer

How long to wait while polling the queue for new messages, in seconds.

Generally should not be changed unless instructed to do so, as if messages are available, they will always be consumed, regardless of the value of poll_secs.

visibility_timeout_secs

optional

integer

The visibility timeout to use for messages, in seconds.

This controls how long a message is left unavailable after it is received. If a message is received, and takes longer than visibility_timeout_secs to process and delete the message from the queue, it is made available again for another consumer.

This can happen if there is an issue between consuming a message and deleting it.

delete_message

optional

boolean

Whether to delete the message once it is processed.

It can be useful to set this to false for debugging or during the initial setup.

client_concurrency

optional

integer,​null

Number of concurrent tasks to create for polling the queue for messages.

Defaults to the number of available CPUs on the system.

Should not typically need to be changed, but it can sometimes be beneficial to raise this value when there is a high rate of messages being pushed into the queue and the objects being fetched are small. In these cases, System resources may not be fully utilized without fetching more messages per second, as the SQS message consumption rate affects the S3 object retrieval rate.

tls_options

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

assume_role

optional

string,​null

The ARN of an [IAM role][iam_role] to assume at startup.

DEPRECATED: [iam_role]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html

auth

optional

 <oneOf>

Configuration of the authentication strategy for interacting with AWS services.

AccessKey

optional

object

Authenticate using a fixed access key and secret pair.

access_key_id

required

string

The AWS access key ID.

secret_access_key

required

string

The AWS secret access key.

File

optional

object

Authenticate using credentials stored in a file.

Additionally, the specific credential profile to use can be set.

credentials_file

required

string

Path to the credentials file.

profile

optional

string,​null

The credentials profile to use.

Role

optional

object

Assume the given role ARN.

assume_role

required

string

The ARN of the role to assume.

load_timeout_secs

optional

integer,​null

Timeout for assuming the role, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

region

optional

string,​null

The AWS region to send STS requests to.

If not set, this will default to the configured region for the service itself.

Default

optional

object

Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.

load_timeout_secs

optional

integer,​null

Timeout for successfully loading any credentials, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

multiline

optional

object,​null

Multiline aggregation configuration.

If not specified, multiline aggregation is disabled.

start_pattern

required

string

Regular expression pattern that is used to match the start of a new message.

condition_pattern

required

string

Regular expression pattern that is used to determine whether or not more lines should be read.

This setting must be configured in conjunction with mode.

mode

required

 <oneOf>

Aggregation mode.

This setting must be configured in conjunction with condition_pattern.

continue_through

optional

continue_through

All consecutive lines matching this pattern are included in the group.

The first line (the line that matched the start pattern) does not need to match the ContinueThrough pattern.

This is useful in cases such as a Java stack trace, where some indicator in the line (such as leading whitespace) indicates that it is an extension of the proceeding line.

continue_past

optional

continue_past

All consecutive lines matching this pattern, plus one additional line, are included in the group.

This is useful in cases where a log message ends with a continuation marker, such as a backslash, indicating that the following line is part of the same message.

halt_before

optional

halt_before

All consecutive lines not matching this pattern are included in the group.

This is useful where a log line contains a marker indicating that it begins a new message.

halt_with

optional

halt_with

All consecutive lines, up to and including the first line matching this pattern, are included in the group.

This is useful where a log line ends with a termination marker, such as a semicolon.

timeout_ms

required

integer

The maximum amount of time to wait for the next additional line, in milliseconds.

Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

tls_options

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

region

optional

string,​null

The AWS region to use.

endpoint

optional

string,​null

The API endpoint of the service.

compression: ''
strategy: ''
sqs: object
assume_role: string
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
multiline: object
acknowledgements:
  enabled: null
tls_options: object
log_namespace: boolean
type: aws_s3
compression: ''
strategy: ''
sqs: object
assume_role: string
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
multiline: object
acknowledgements:
  enabled: null
tls_options: object
log_namespace: boolean
region: string
endpoint: string
type: aws_s3

AWS SQS

Configuration for the aws_sqs source.

Field

required

Type

Description

auth

optional

 <oneOf>

Configuration of the authentication strategy for interacting with AWS services.

AccessKey

optional

object

Authenticate using a fixed access key and secret pair.

access_key_id

required

string

The AWS access key ID.

secret_access_key

required

string

The AWS secret access key.

File

optional

object

Authenticate using credentials stored in a file.

Additionally, the specific credential profile to use can be set.

credentials_file

required

string

Path to the credentials file.

profile

optional

string,​null

The credentials profile to use.

Role

optional

object

Assume the given role ARN.

assume_role

required

string

The ARN of the role to assume.

load_timeout_secs

optional

integer,​null

Timeout for assuming the role, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

region

optional

string,​null

The AWS region to send STS requests to.

If not set, this will default to the configured region for the service itself.

Default

optional

object

Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.

load_timeout_secs

optional

integer,​null

Timeout for successfully loading any credentials, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

queue_url

required

string

The URL of the SQS queue to poll for messages.

poll_secs

optional

integer

How long to wait while polling the queue for new messages, in seconds.

Generally should not be changed unless instructed to do so, as if messages are available, they will always be consumed, regardless of the value of poll_secs.

visibility_timeout_secs

optional

integer

The visibility timeout to use for messages, in seconds.

This controls how long a message is left unavailable after it is received. If a message is received, and takes longer than visibility_timeout_secs to process and delete the message from the queue, it is made available again for another consumer.

This can happen if there is an issue between consuming a message and deleting it.

delete_message

optional

boolean

Whether to delete the message once it is processed.

It can be useful to set this to false for debugging or during the initial setup.

client_concurrency

optional

integer,​null

Number of concurrent tasks to create for polling the queue for messages.

Defaults to the number of available CPUs on the system.

Should not typically need to be changed, but it can sometimes be beneficial to raise this value when there is a high rate of messages being pushed into the queue and the messages being fetched are small. In these cases, System resources may not be fully utilized without fetching more messages per second, as it spends more time fetching the messages than processing them.

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

region

optional

string,​null

The AWS region to use.

endpoint

optional

string,​null

The API endpoint of the service.

auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
queue_url: string
poll_secs: 15
visibility_timeout_secs: 300
delete_message: true
client_concurrency: integer
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
tls: object
log_namespace: boolean
type: aws_sqs
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
queue_url: string
poll_secs: 15
visibility_timeout_secs: 300
delete_message: true
client_concurrency: integer
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
tls: object
log_namespace: boolean
region: string
endpoint: string
type: aws_sqs

Datadog Agent

Configuration for the datadog_agent source.

Field

required

Type

Description

address

required

string

The address to accept connections on.

The address must include a port.

store_api_key

optional

boolean

When incoming events contain a Datadog API key, if this setting is set to true the key will kept in the event metadata and will be used if the event is sent to a Datadog sink.

disable_logs

optional

boolean

If this settings is set to true, logs won't be accepted by the component.

disable_metrics

optional

boolean

If this settings is set to true, metrics won't be accepted by the component.

disable_traces

optional

boolean

If this settings is set to true, traces won't be accepted by the component.

multiple_outputs

optional

boolean

If this setting is set to true logs, metrics and traces will be sent to different outputs.

For a source component named agent the received logs, metrics, and traces can then be accessed by specifying agent.logs, agent.metrics, and agent.traces, respectively, as the input to another component.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

enabled

optional

boolean,​null

Whether or not to require TLS for incoming/outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null
address: string
store_api_key: true
disable_logs: boolean
disable_metrics: boolean
disable_traces: boolean
multiple_outputs: boolean
log_namespace: boolean
tls: ''
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
type: datadog_agent
address: string
store_api_key: true
disable_logs: boolean
disable_metrics: boolean
disable_traces: boolean
multiple_outputs: boolean
log_namespace: boolean
tls: ''
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
type: datadog_agent

Demo logs

Configuration for the demo_logs source.

Field

required

Type

Description

interval

optional

number

The amount of time, in seconds, to pause between each batch of output lines.

The default is one batch per second. In order to remove the delay and output batches as quickly as possible, set interval to 0.0.

count

optional

integer

The total number of lines to output.

By default, the source continuously prints logs (infinitely).

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

_metadata

optional

description

optional

Output format configuration.

oneOf

optional

interval: 1
count: 9223372036854776000
framing:
  method: bytes
decoding:
  codec: bytes
log_namespace: boolean
type: demo_logs
interval: 1
count: 9223372036854776000
framing:
  method: bytes
decoding:
  codec: bytes
log_namespace: boolean
type: demo_logs

DNSTAP

Configuration for the dnstap source.

Field

required

Type

Description

max_frame_length

optional

integer

Maximum length, in bytes, that a frame can be.

host_key

optional

string,​null

Overrides the name of the log field used to add the source path to each event.

The value will be the socket path itself.

By default, the global log_schema.host_key option is used.

socket_path

required

string

Absolute path to the socket file to read DNSTAP data from.

The DNS server must be configured to send its DNSTAP data to this socket file. The socket file will be created, if it doesn't already exist, when the source first starts.

raw_data_only

optional

boolean,​null

Whether or not to skip parsing/decoding of DNSTAP frames.

If set to true, frames will not be parsed/decoded. The raw frame data will be set as a field on the event (called rawData) and encoded as a base64 string.

multithreaded

optional

boolean,​null

Whether or not to concurrently process DNSTAP frames.

max_frame_handling_tasks

optional

integer,​null

Maximum number of frames that can be processed concurrently.

socket_file_mode

optional

integer,​null

Unix file mode bits to be applied to the unix socket file as its designated file permissions.

Note that the file mode value can be specified in any numeric format supported by your configuration language, but it is most intuitive to use an octal number.

socket_receive_buffer_size

optional

integer,​null

The size, in bytes, of the receive buffer used for the socket.

This should not typically needed to be changed.

socket_send_buffer_size

optional

integer,​null

The size, in bytes, of the send buffer used for the socket.

This should not typically needed to be changed.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global settings.

max_frame_length: 102400
host_key: string
socket_path: string
raw_data_only: boolean
multithreaded: boolean
max_frame_handling_tasks: integer
socket_file_mode: integer
socket_receive_buffer_size: integer
socket_send_buffer_size: integer
log_namespace: boolean
type: dnstap
max_frame_length: 102400
host_key: string
socket_path: string
raw_data_only: boolean
multithreaded: boolean
max_frame_handling_tasks: integer
socket_file_mode: integer
socket_receive_buffer_size: integer
socket_send_buffer_size: integer
log_namespace: boolean
type: dnstap

Docker Logs

Configuration for the docker_logs source.

Field

required

Type

Description

host_key

optional

string

Overrides the name of the log field used to add the current hostname to each event.

By default, the global log_schema.host_key option is used.

docker_host

optional

string,​null

Docker host to connect to.

Use an HTTPS URL to enable TLS encryption.

If absent, the DOCKER_HOST environment variable is used. If DOCKER_HOST is also absent, the default Docker local socket (/var/run/docker.sock on Unix platforms, //./pipe/docker_engine on Windows) is used.

exclude_containers

optional

array,​null

A list of container IDs or names of containers to exclude from log collection.

Matching is prefix first, so specifying a value of foo would match any container named foo as well as any container whose name started with foo. This applies equally whether matching container IDs or names.

By default, the source will collect logs for all containers. If exclude_containers is configured, any container that matches a configured exclusion will be excluded even if it is also included via include_containers, so care should be taken when utilizing prefix matches as they cannot be overridden by a corresponding entry in include_containers e.g. excluding foo by attempting to include foo-specific-id.

This can be used in conjunction with include_containers.

include_containers

optional

array,​null

A list of container IDs or names of containers to include in log collection.

Matching is prefix first, so specifying a value of foo would match any container named foo as well as any container whose name started with foo. This applies equally whether matching container IDs or names.

By default, the source will collect logs for all containers. If include_containers is configured, only containers that match a configured inclusion and are also not excluded will be matched.

This can be used in conjunction with include_containers.

include_labels

optional

array,​null

A list of container object labels to match against when filtering running containers.

Labels should follow the syntax described in the Docker object labels documentation.

include_images

optional

array,​null

A list of image names to match against.

If not provided, all images will be included.

partial_event_marker_field

optional

string,​null

Overrides the name of the log field used to mark an event as partial.

If auto_partial_merge is disabled, partial events will be emitted with a log field, controlled by this configuration value, is set, indicating that the event is not complete.

By default, "_partial" is used.

auto_partial_merge

optional

boolean

Enables automatic merging of partial events.

retry_backoff_secs

optional

integer

The amount of time, in seconds, to wait before retrying after an error.

multiline

optional

object,​null

Multiline aggregation configuration.

If not specified, multiline aggregation is disabled.

start_pattern

required

string

Regular expression pattern that is used to match the start of a new message.

condition_pattern

required

string

Regular expression pattern that is used to determine whether or not more lines should be read.

This setting must be configured in conjunction with mode.

mode

required

 <oneOf>

Aggregation mode.

This setting must be configured in conjunction with condition_pattern.

continue_through

optional

continue_through

All consecutive lines matching this pattern are included in the group.

The first line (the line that matched the start pattern) does not need to match the ContinueThrough pattern.

This is useful in cases such as a Java stack trace, where some indicator in the line (such as leading whitespace) indicates that it is an extension of the proceeding line.

continue_past

optional

continue_past

All consecutive lines matching this pattern, plus one additional line, are included in the group.

This is useful in cases where a log message ends with a continuation marker, such as a backslash, indicating that the following line is part of the same message.

halt_before

optional

halt_before

All consecutive lines not matching this pattern are included in the group.

This is useful where a log line contains a marker indicating that it begins a new message.

halt_with

optional

halt_with

All consecutive lines, up to and including the first line matching this pattern, are included in the group.

This is useful where a log line ends with a termination marker, such as a semicolon.

timeout_ms

required

integer

The maximum amount of time to wait for the next additional line, in milliseconds.

Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.

tls

optional

object,​null

Configuration of TLS when connecting to the Docker daemon.

Only relevant when connecting to Docker via an HTTPS URL.

If not configured, the environment variable DOCKER_CERT_PATH is used. If DOCKER_CERT_PATH is absent, then DOCKER_CONFIG is used. If both environment variables are absent, the certificates in ~/.docker/ are read.

ca_file

required

string

Path to the CA certificate file.

crt_file

required

string

Path to the TLS certificate file.

key_file

required

string

Path to the TLS key file.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

host_key: host
docker_host: string
exclude_containers: array
include_containers: array
include_labels: array
include_images: array
partial_event_marker_field: string
auto_partial_merge: boolean
retry_backoff_secs: integer
multiline: object
tls: object
log_namespace: boolean
type: docker_logs
host_key: host
docker_host: string
exclude_containers: array
include_containers: array
include_labels: array
include_images: array
partial_event_marker_field: string
auto_partial_merge: boolean
retry_backoff_secs: integer
multiline: object
tls: object
log_namespace: boolean
type: docker_logs

EventStoreDB Metrics

Configuration for the eventstoredb_metrics source.

Field

required

Type

Description

endpoint

optional

string

Endpoint to scrape stats from.

scrape_interval_secs

optional

integer

The interval between scrapes, in seconds.

default_namespace

optional

string,​null

Overrides the default namespace for the metrics emitted by the source.

By default, eventstoredb is used.

endpoint: 'https://localhost:2113/stats'
scrape_interval_secs: 15
default_namespace: string
type: eventstoredb_metrics
endpoint: 'https://localhost:2113/stats'
scrape_interval_secs: 15
default_namespace: string
type: eventstoredb_metrics

Exec

Configuration for the exec source.

Field

required

Type

Description

mode

required

 <oneOf>

Mode of operation for running the command.

scheduled

optional

scheduled

The command is run on a schedule.

streaming

optional

streaming

The command is run until it exits, potentially being restarted.

scheduled

optional

object,​null

Configuration options for scheduled commands.

exec_interval_secs

optional

integer

The interval, in seconds, between scheduled command runs.

If the command takes longer than exec_interval_secs to run, it will be killed.

streaming

optional

object,​null

Configuration options for streaming commands.

respawn_on_exit

optional

boolean

Whether or not the command should be rerun if the command exits.

respawn_interval_secs

optional

integer

The amount of time, in seconds, that Vector will wait before rerunning a streaming command that exited.

command

required

[string]

The command to be run, plus any arguments required.

working_directory

optional

string,​null

The directory in which to run the command.

include_stderr

optional

boolean

Whether or not the output from stderr should be included when generating events.

maximum_buffer_size_bytes

optional

integer

The maximum buffer size allowed before a log event will be generated.

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

mode: ''
scheduled: object
streaming: object
command: array
working_directory: string
include_stderr: true
maximum_buffer_size_bytes: 1000000
framing: ''
decoding:
  codec: bytes
log_namespace: boolean
type: exec
mode: ''
scheduled: object
streaming: object
command: array
working_directory: string
include_stderr: true
maximum_buffer_size_bytes: 1000000
framing: ''
decoding:
  codec: bytes
log_namespace: boolean
type: exec

File

Configuration for the file source.

Field

required

Type

Description

include

required

[string]

Array of file patterns to include. Globbing is supported.

exclude

optional

[string]

Array of file patterns to exclude. Globbing is supported.

Takes precedence over the include option. Note: The exclude patterns are applied after the attempt to glob everything in include. This means that all files are first matched by include and then filtered by the exclude patterns. This can be impactful if include contains directories with contents that are not accessible.

file_key

optional

string,​null

Overrides the name of the log field used to add the file path to each event.

The value will be the full path to the file where the event was read message.

By default, file is used.

start_at_beginning

optional

boolean,​null

Whether or not to start reading from the beginning of a new file.

DEPRECATED: DEPRECATED: This is a deprecated option -- replaced by ignore_checkpoints/read_from -- and should be removed.

ignore_checkpoints

optional

boolean,​null

Whether or not to ignore existing checkpoints when determining where to start reading a file.

Checkpoints are still written normally.

read_from

optional

 <oneOf>

File position to use when reading a new file.

beginning

optional

beginning

Read from the beginning of the file.

end

optional

end

Start reading from the current end of the file.

ignore_older_secs

optional

integer,​null

Ignore files with a data modification date older than the specified number of seconds.

max_line_bytes

optional

integer

The maximum number of bytes a line can contain before being discarded.

This protects against malformed lines or tailing incorrect files.

host_key

optional

string,​null

Overrides the name of the log field used to add the current hostname to each event.

The value is the current hostname.

By default, the global log_schema.host_key option is used.

data_dir

optional

string,​null

The directory used to persist file checkpoint positions.

By default, the global data_dir option is used. Make sure the running user has write permissions to this directory.

offset_key

optional

string,​null

Enables adding the file offset to each event and sets the name of the log field used.

The value will be the byte offset of the start of the line within the file.

Off by default, the offset is only added to the event if this is set.

glob_minimum_cooldown_ms

optional

integer

Delay between file discovery calls, in milliseconds.

This controls the interval at which files are searched. A higher value results in greater chances of some short-lived files being missed between searches, but a lower value increases the performance impact of file discovery.

fingerprint

optional

 <oneOf>

Configuration for how files should be identified.

This is important for checkpointing when file rotation is used.

Checksum

optional

object

Read lines from the beginning of the file and compute a checksum over them.

bytes

optional

integer,​null

Maximum number of bytes to use, from the lines that are read, for generating the checksum.

ignored_header_bytes

required

integer

The number of bytes to skip ahead (or ignore) when reading the data used for generating the checksum.

This can be helpful if all files share a common header that should be skipped.

lines

optional

integer

The number of lines to read for generating the checksum.

If your files share a common header that is not always a fixed size,

If the file has less than this amount of lines, it won’t be read at all.

strategy

required

checksum

Read lines from the beginning of the file and compute a checksum over them.

DevInode

optional

object

Use the device and inode as the identifier.

strategy

required

device_and_inode

Use the device and inode as the identifier.

ignore_not_found

optional

boolean

Ignore missing files when fingerprinting.

This may be useful when used with source directories containing dangling symlinks.

message_start_indicator

optional

string,​null

String value used to identify the start of a multi-line message.

DEPRECATED: DEPRECATED: This is a deprecated option -- replaced by multiline -- and should be removed.

multi_line_timeout

optional

integer

How long to wait for more data when aggregating a multi-line message, in milliseconds.

DEPRECATED: DEPRECATED: This is a deprecated option -- replaced by multiline -- and should be removed.

multiline

optional

object,​null

Multiline aggregation configuration.

If not specified, multiline aggregation is disabled.

start_pattern

required

string

Regular expression pattern that is used to match the start of a new message.

condition_pattern

required

string

Regular expression pattern that is used to determine whether or not more lines should be read.

This setting must be configured in conjunction with mode.

mode

required

 <oneOf>

Aggregation mode.

This setting must be configured in conjunction with condition_pattern.

continue_through

optional

continue_through

All consecutive lines matching this pattern are included in the group.

The first line (the line that matched the start pattern) does not need to match the ContinueThrough pattern.

This is useful in cases such as a Java stack trace, where some indicator in the line (such as leading whitespace) indicates that it is an extension of the proceeding line.

continue_past

optional

continue_past

All consecutive lines matching this pattern, plus one additional line, are included in the group.

This is useful in cases where a log message ends with a continuation marker, such as a backslash, indicating that the following line is part of the same message.

halt_before

optional

halt_before

All consecutive lines not matching this pattern are included in the group.

This is useful where a log line contains a marker indicating that it begins a new message.

halt_with

optional

halt_with

All consecutive lines, up to and including the first line matching this pattern, are included in the group.

This is useful where a log line ends with a termination marker, such as a semicolon.

timeout_ms

required

integer

The maximum amount of time to wait for the next additional line, in milliseconds.

Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.

max_read_bytes

optional

integer

An approximate limit on the amount of data read from a single file at a given time.

oldest_first

optional

boolean

Instead of balancing read capacity fairly across all watched files, prioritize draining the oldest files before moving on to read data from younger files.

remove_after_secs

optional

integer,​null

Timeout from reaching EOF after which file will be removed from filesystem, unless new data is written in the meantime.

If not specified, files will not be removed.

line_delimiter

optional

string

String sequence used to separate one file line from another.

encoding

optional

object,​null

Character set encoding.

charset

required

string

Encoding of the source messages.

Takes one of the encoding label strings defined as part of the Encoding Standard.

When set, the messages are transcoded from the specified encoding to UTF-8, which is the encoding that is assumed internally for string-like data. Enable this transcoding operation if you need your data to be in UTF-8 for further processing. At the time of transcoding, any malformed sequences (that can't be mapped to UTF-8) is replaced with the Unicode REPLACEMENT CHARACTER and warnings are logged.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

include: array
exclude: []
file_key: file
start_at_beginning: boolean
ignore_checkpoints: boolean
read_from: ''
ignore_older_secs: integer
max_line_bytes: 102400
host_key: string
data_dir: string
offset_key: string
glob_minimum_cooldown_ms: 1000
fingerprint:
  strategy: checksum
  bytes: null
  ignored_header_bytes: 0
  lines: 1
ignore_not_found: boolean
message_start_indicator: string
multi_line_timeout: 1000
multiline: object
max_read_bytes: 2048
oldest_first: boolean
remove_after_secs: integer
line_delimiter: |+

encoding: object
acknowledgements:
  enabled: null
log_namespace: boolean
type: file
include: array
exclude: []
file_key: file
start_at_beginning: boolean
ignore_checkpoints: boolean
read_from: ''
ignore_older_secs: integer
max_line_bytes: 102400
host_key: string
data_dir: string
offset_key: string
glob_minimum_cooldown_ms: 1000
fingerprint:
  strategy: checksum
  bytes: null
  ignored_header_bytes: 0
  lines: 1
ignore_not_found: boolean
message_start_indicator: string
multi_line_timeout: 1000
multiline: object
max_read_bytes: 2048
oldest_first: boolean
remove_after_secs: integer
line_delimiter: |+

encoding: object
acknowledgements:
  enabled: null
log_namespace: boolean
type: file

File descriptor

Configuration for the file_descriptor source.

Field

required

Type

Description

max_length

optional

integer

The maximum buffer size, in bytes, of incoming messages.

Messages larger than this are truncated.

host_key

optional

string,​null

Overrides the name of the log field used to add the current hostname to each event.

By default, the global host_key option is used.

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

fd

required

integer

The file descriptor number to read from.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

max_length: 102400
host_key: string
framing: ''
decoding:
  codec: bytes
fd: integer
log_namespace: boolean
type: file_descriptor
max_length: 102400
host_key: string
framing: ''
decoding:
  codec: bytes
fd: integer
log_namespace: boolean
type: file_descriptor

Fluent

Configuration for the fluent source.

Field

required

Type

Description

address

required

 <oneOf>

The address to listen for connections on.

SocketAddr

optional

string

An IPv4/IPv6 address and port.

SystemdFd

optional

integer

A file descriptor identifier that is given from, and managed by, the socket activation feature of systemd.

connection_limit

optional

integer,​null

The maximum number of TCP connections that will be allowed at any given time.

keepalive

optional

object,​null

TCP keepalive settings for socket-based components.

time_secs

optional

integer,​null

The time to wait, in seconds, before starting to send TCP keepalive probes on an idle connection.

receive_buffer_bytes

optional

integer,​null

The size, in bytes, of the receive buffer used for each connection.

This should not typically needed to be changed.

tls

optional

 <oneOf>

TlsEnableableConfig for sources, adding metadata from the client certificate

Option 1

optional

client_metadata_key

optional

string,​null

Event field for client certificate metadata.

enabled

optional

boolean,​null

Whether or not to require TLS for incoming/outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

address: ''
connection_limit: integer
keepalive: object
receive_buffer_bytes: integer
tls: ''
acknowledgements:
  enabled: null
log_namespace: boolean
type: fluent
address: ''
connection_limit: integer
keepalive: object
receive_buffer_bytes: integer
tls: ''
acknowledgements:
  enabled: null
log_namespace: boolean
type: fluent

GCP Pub/Sub

Configuration for the gcp_pubsub source.

Field

required

Type

Description

project

required

string

The project name from which to pull logs.

subscription

required

string

The subscription within the project which is configured to receive logs.

endpoint

optional

string

The endpoint from which to pull data.

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

max_concurrency

optional

integer

The maximum number of concurrent stream connections to open at once.

full_response_size

optional

integer

The number of messages in a response to mark a stream as "busy". This is used to determine if more streams should be started.

The GCP Pub/Sub servers send responses with 100 or more messages when the subscription is busy.

poll_time_seconds

optional

number

How often to poll the currently active streams to see if they are all busy and so open a new stream.

ack_deadline_secs

optional

integer

The acknowledgement deadline, in seconds, to use for this stream.

Messages that are not acknowledged when this deadline expires may be retransmitted.

ack_deadline_seconds

optional

integer,​null

DEPRECATED: Deprecated, old name of ack_deadline_secs.

retry_delay_secs

optional

number

The amount of time, in seconds, to wait between retry attempts after an error.

retry_delay_seconds

optional

number,​null

DEPRECATED: Deprecated, old name of retry_delay_secs.

keepalive_secs

optional

number

The amount of time, in seconds, with no received activity before sending a keepalive request. If this is set larger than 60, you may see periodic errors sent from the server.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

api_key

optional

string,​null

An API key. (documentation)

Either an API key, or a path to a service account credentials JSON file can be specified.

If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no filename is named, an attempt is made to fetch an instance service account for the compute instance the program is running on. If this is not on a GCE instance, then you must define it with an API key or service account credentials JSON file.

credentials_path

optional

string,​null

Path to a service account credentials JSON file. (documentation)

Either an API key, or a path to a service account credentials JSON file can be specified.

If both are unset, the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked for a filename. If no filename is named, an attempt is made to fetch an instance service account for the compute instance the program is running on. If this is not on a GCE instance, then you must define it with an API key or service account credentials JSON file.

skip_authentication

optional

boolean

Skip all authentication handling. For use with integration tests only.

project: string
subscription: string
endpoint: 'https://pubsub.googleapis.com'
tls: object
max_concurrency: 10
full_response_size: 100
poll_time_seconds: 2
ack_deadline_secs: 600
ack_deadline_seconds: integer
retry_delay_secs: 1
retry_delay_seconds: number
keepalive_secs: 60
log_namespace: boolean
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
type: gcp_pubsub
project: string
subscription: string
endpoint: 'https://pubsub.googleapis.com'
tls: object
max_concurrency: 10
full_response_size: 100
poll_time_seconds: 2
ack_deadline_secs: 600
ack_deadline_seconds: integer
retry_delay_secs: 1
retry_delay_seconds: number
keepalive_secs: 60
log_namespace: boolean
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
api_key: string
credentials_path: string
skip_authentication: boolean
type: gcp_pubsub

Heroku Logs

Configuration for heroku_logs source.

Field

required

Type

Description

address

required

string

The socket address to listen for connections on.

query_parameters

optional

[string]

A list of URL query parameters to include in the log event.

These will override any values included in the body with conflicting names.

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

enabled

optional

boolean,​null

Whether or not to require TLS for incoming/outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

auth

optional

object,​null

HTTP Basic authentication configuration.

username

required

string

The username for basic authentication.

password

required

string

The password for basic authentication.

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

address: string
query_parameters: []
tls: ''
auth: object
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
log_namespace: boolean
type: heroku_logs
address: string
query_parameters: []
tls: ''
auth: object
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
log_namespace: boolean
type: heroku_logs

Host Metrics

Configuration for the host_metrics source.

Field

required

Type

Description

scrape_interval_secs

optional

number

The interval between metric gathering, in seconds.

collectors

optional

array,​null

The list of host metric collector services to use.

Defaults to all collectors.

namespace

optional

string,​null

Overrides the default namespace for the metrics emitted by the source.

By default, host is used.

cgroups

optional

object

Options for the “cgroups” (controller groups) metrics collector.

This collector is only available on Linux systems, and only supports either version 2 or hybrid cgroups.

levels

optional

integer

The number of levels of the cgroups hierarchy for which to report metrics.

A value of 1 means just the root or named cgroup.

default: 100

base

optional

string,​null

The base cgroup name to provide metrics for.

default: null

groups

optional

object

Lists of group name patterns to include or exclude.

default: {"includes":null,"excludes":null}

includes

optional

array,​null

Any patterns which should be included.

excludes

optional

array,​null

Any patterns which should be excluded.

base_dir

optional

string,​null

Base cgroup directory, for testing use only

disk

optional

object

Options for the “disk” metrics collector.

devices

optional

object

Lists of device name patterns to include or exclude.

default: {"includes":null,"excludes":null}

includes

optional

array,​null

Any patterns which should be included.

default: null

excludes

optional

array,​null

Any patterns which should be excluded.

default: null

filesystem

optional

object

Options for the “filesystem” metrics collector.

devices

optional

object

Lists of device name patterns to include or exclude.

default: {"includes":null,"excludes":null}

includes

optional

array,​null

Any patterns which should be included.

default: null

excludes

optional

array,​null

Any patterns which should be excluded.

default: null

filesystems

optional

object

Lists of filesystem name patterns to include or exclude.

default: {"includes":null,"excludes":null}

includes

optional

array,​null

Any patterns which should be included.

default: null

excludes

optional

array,​null

Any patterns which should be excluded.

default: null

mountpoints

optional

object

Lists of mount point path patterns to include or exclude.

default: {"includes":null,"excludes":null}

includes

optional

array,​null

Any patterns which should be included.

default: null

excludes

optional

array,​null

Any patterns which should be excluded.

default: null

network

optional

object

Options for the “network” metrics collector.

devices

optional

object

Lists of device name patterns to include or exclude.

default: {"includes":null,"excludes":null}

includes

optional

array,​null

Any patterns which should be included.

default: null

excludes

optional

array,​null

Any patterns which should be excluded.

default: null
scrape_interval_secs: 15
collectors: array
namespace: host
cgroups:
  levels: 100
  base: null
  groups:
    includes: null
    excludes: null
disk:
  devices:
    includes: null
    excludes: null
filesystem:
  devices:
    includes: null
    excludes: null
  filesystems:
    includes: null
    excludes: null
  mountpoints:
    includes: null
    excludes: null
network:
  devices:
    includes: null
    excludes: null
type: host_metrics
scrape_interval_secs: 15
collectors: array
namespace: host
cgroups:
  levels: 100
  base: null
  groups:
    includes: null
    excludes: null
disk:
  devices:
    includes: null
    excludes: null
filesystem:
  devices:
    includes: null
    excludes: null
  filesystems:
    includes: null
    excludes: null
  mountpoints:
    includes: null
    excludes: null
network:
  devices:
    includes: null
    excludes: null
type: host_metrics

HTTP

Configuration for the http source.

Field

required

Type

Description

address

required

string

The address to listen for connections on.

encoding

optional

 <oneOf>

The expected encoding of received data.

Note that for json and ndjson encodings, the fields of the JSON objects are output as separate fields.

text

optional

text

Plaintext.

ndjson

optional

ndjson

Newline-delimited JSON.

json

optional

json

JSON.

binary

optional

binary

Binary.

headers

optional

[string]

A list of HTTP headers to include in the log event.

These will override any values included in the JSON payload with conflicting names.

query_parameters

optional

[string]

A list of URL query parameters to include in the log event.

These will override any values included in the body with conflicting names.

auth

optional

object,​null

HTTP Basic authentication configuration.

username

required

string

The username for basic authentication.

password

required

string

The password for basic authentication.

strict_path

optional

boolean

Whether or not to treat the configured path as an absolute path.

If set to true, only requests using the exact URL path specified in path will be accepted. Otherwise, requests sent to a URL path that starts with the value of path will be accepted.

With strict_path set to false and path set to "", the configured HTTP source will accept requests from any URL path.

path

optional

string

The URL path on which log event POST requests shall be sent.

path_key

optional

string

The event key in which the requested URL path used to send the request will be stored.

method

optional

 <oneOf>

Specifies the action of the HTTP request.

HEAD

optional

HEAD

HTTP HEAD method.

GET

optional

GET

HTTP GET method.

POST

optional

POST

HTTP POST method.

PUT

optional

PUT

HTTP Put method.

PATCH

optional

PATCH

HTTP PATCH method.

DELETE

optional

DELETE

HTTP DELETE method.

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

enabled

optional

boolean,​null

Whether or not to require TLS for incoming/outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

address: string
encoding: ''
headers: []
query_parameters: []
auth: object
strict_path: true
path: /
path_key: path
method: POST
tls: ''
framing: ''
decoding: ''
acknowledgements:
  enabled: null
log_namespace: boolean
type: http
address: string
encoding: ''
headers: []
query_parameters: []
auth: object
strict_path: true
path: /
path_key: path
method: POST
tls: ''
framing: ''
decoding: ''
acknowledgements:
  enabled: null
log_namespace: boolean
type: http

HTTP Client

Configuration for the http_client source.

Field

required

Type

Description

endpoint

required

string

Endpoint to collect events from. The full path must be specified. Example: "http://127.0.0.1:9898/logs"

scrape_interval_secs

optional

integer

The interval between calls, in seconds.

query

optional

object

Custom parameters for the HTTP request query string.

One or more values for the same parameter key can be provided. The parameters provided in this option are appended to any parameters manually provided in the endpoint option.

decoding

optional

 <oneOf>

Decoder to use on the HTTP responses.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

framing

optional

 <oneOf>

Framing configuration.

Framing to use in the decoding.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

headers

optional

object

Headers to apply to the HTTP requests.

One or more values for the same header can be provided.

method

optional

 <oneOf>

Specifies the action of the HTTP request.

HEAD

optional

HEAD

HTTP HEAD method.

GET

optional

GET

HTTP GET method.

POST

optional

POST

HTTP POST method.

PUT

optional

PUT

HTTP Put method.

PATCH

optional

PATCH

HTTP PATCH method.

DELETE

optional

DELETE

HTTP DELETE method.

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

auth

optional

 <oneOf>

Configuration of the authentication strategy for HTTP requests.

HTTP Authentication.

Basic

optional

object

Basic authentication.

The username and password are concatenated and encoded via base64.

user

required

string

The username to send.

password

required

string

The password to send.

strategy

required

basic

Basic authentication.

The username and password are concatenated and encoded via base64.

Bearer

optional

object

Bearer authentication.

The bearer token value (OAuth2, JWT, etc) is passed as-is.

token

required

string

The bearer token to send.

strategy

required

bearer

Bearer authentication.

The bearer token value (OAuth2, JWT, etc) is passed as-is.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

endpoint: string
scrape_interval_secs: 15
query: {}
decoding:
  codec: bytes
framing:
  method: bytes
headers: {}
method: GET
tls: object
auth: ''
log_namespace: boolean
type: http_client
endpoint: string
scrape_interval_secs: 15
query: {}
decoding:
  codec: bytes
framing:
  method: bytes
headers: {}
method: GET
tls: object
auth: ''
log_namespace: boolean
type: http_client

HTTP Server

Configuration for the http_server source.

Field

required

Type

Description

address

required

string

The address to listen for connections on.

encoding

optional

 <oneOf>

The expected encoding of received data.

Note that for json and ndjson encodings, the fields of the JSON objects are output as separate fields.

text

optional

text

Plaintext.

ndjson

optional

ndjson

Newline-delimited JSON.

json

optional

json

JSON.

binary

optional

binary

Binary.

headers

optional

[string]

A list of HTTP headers to include in the log event.

These will override any values included in the JSON payload with conflicting names.

query_parameters

optional

[string]

A list of URL query parameters to include in the log event.

These will override any values included in the body with conflicting names.

auth

optional

object,​null

HTTP Basic authentication configuration.

username

required

string

The username for basic authentication.

password

required

string

The password for basic authentication.

strict_path

optional

boolean

Whether or not to treat the configured path as an absolute path.

If set to true, only requests using the exact URL path specified in path will be accepted. Otherwise, requests sent to a URL path that starts with the value of path will be accepted.

With strict_path set to false and path set to "", the configured HTTP source will accept requests from any URL path.

path

optional

string

The URL path on which log event POST requests shall be sent.

path_key

optional

string

The event key in which the requested URL path used to send the request will be stored.

method

optional

 <oneOf>

Specifies the action of the HTTP request.

HEAD

optional

HEAD

HTTP HEAD method.

GET

optional

GET

HTTP GET method.

POST

optional

POST

HTTP POST method.

PUT

optional

PUT

HTTP Put method.

PATCH

optional

PATCH

HTTP PATCH method.

DELETE

optional

DELETE

HTTP DELETE method.

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

enabled

optional

boolean,​null

Whether or not to require TLS for incoming/outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

address: string
encoding: ''
headers: []
query_parameters: []
auth: object
strict_path: true
path: /
path_key: path
method: POST
tls: ''
framing: ''
decoding: ''
acknowledgements:
  enabled: null
log_namespace: boolean
type: http_server
address: string
encoding: ''
headers: []
query_parameters: []
auth: object
strict_path: true
path: /
path_key: path
method: POST
tls: ''
framing: ''
decoding: ''
acknowledgements:
  enabled: null
log_namespace: boolean
type: http_server

Internal Logs

Configuration for the internal_logs source.

Field

required

Type

Description

host_key

optional

string

Overrides the name of the log field used to add the current hostname to each event.

By default, the global log_schema.host_key option is used.

Set to "" to suppress this key.

pid_key

optional

string

Overrides the name of the log field used to add the current process ID to each event.

By default, "pid" is used.

Set to "" to suppress this key.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

host_key: host
pid_key: pid
log_namespace: boolean
type: internal_logs
host_key: host
pid_key: pid
log_namespace: boolean
type: internal_logs

Internal Metrics

Configuration for the internal_metrics source.

Field

required

Type

Description

scrape_interval_secs

optional

number

The interval between metric gathering, in seconds.

tags

optional

object

Tag configuration for the internal_metrics source.

host_key

optional

string,​null

Sets the name of the tag to use to add the current hostname to each metric.

By default, the [global log_schema.host_key option][global_host_key] is used.

default: null

pid_key

optional

string,​null

Sets the name of the tag to use to add the current process ID to each metric.

By default, this is not set and the tag will not be automatically added.

default: null

namespace

optional

string,​null

Overrides the default namespace for the metrics emitted by the source.

Overrides the default namespace.

scrape_interval_secs: number
tags:
  host_key: null
  pid_key: null
namespace: string
type: internal_metrics
scrape_interval_secs: number
tags:
  host_key: null
  pid_key: null
namespace: string
type: internal_metrics

Journald

Configuration for the journald source.

Field

required

Type

Description

since_now

optional

boolean,​null

Only include entries that appended to the journal after the entries have been read.

current_boot_only

optional

boolean,​null

Only include entries that occurred after the current boot of the system.

units

optional

[string]

The list of unit names to monitor.

DEPRECATED: If empty or not present, all units are accepted. Unit names lacking a "." will have ".service" appended to make them a valid service unit name.

include_units

optional

[string]

A list of unit names to monitor.

If empty or not present, all units are accepted. Unit names lacking a "." will have ".service" appended to make them a valid service unit name.

exclude_units

optional

[string]

A list of unit names to exclude from monitoring.

Unit names lacking a "." will have ".service" appended to make them a valid service unit name.

include_matches

optional

object

A list of sets of field/value pairs to monitor.

If empty or not present, all journal fields are accepted. If include_units is specified, it will be merged into this list.

exclude_matches

optional

object

A list of sets of field/value pairs that, if any are present in a journal entry, will cause the entry to be excluded from this source.

If exclude_units is specified, it will be merged into this list.

data_dir

optional

string,​null

The directory used to persist file checkpoint positions.

By default, the global data_dir option is used. Make sure the running user has write permissions to this directory.

batch_size

optional

integer,​null

The systemd journal is read in batches, and a checkpoint is set at the end of each batch. This option limits the size of the batch.

journalctl_path

optional

string,​null

The full path of the journalctl executable.

If not set, a search is done for the journalctl` path.

journal_directory

optional

string,​null

The full path of the journal directory.

If not set, journalctl will use the default system journal paths.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

remap_priority

optional

boolean

Enables remapping the PRIORITY field from an integer to string value.

DEPRECATED: Has no effect unless the value of the field is already an integer.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

since_now: boolean
current_boot_only: boolean
units: array
include_units: array
exclude_units: array
include_matches: object
exclude_matches: object
data_dir: string
batch_size: integer
journalctl_path: string
journal_directory: string
acknowledgements:
  enabled: null
remap_priority: boolean
log_namespace: boolean
type: journald
since_now: boolean
current_boot_only: boolean
units: array
include_units: array
exclude_units: array
include_matches: object
exclude_matches: object
data_dir: string
batch_size: integer
journalctl_path: string
journal_directory: string
acknowledgements:
  enabled: null
remap_priority: boolean
log_namespace: boolean
type: journald

Kafka

Configuration for the kafka source.

Field

required

Type

Description

bootstrap_servers

required

string

A comma-separated list of Kafka bootstrap servers.

These are the servers in a Kafka cluster that a client should use to "bootstrap" its connection to the cluster, allowing discovering all other hosts in the cluster.

Must be in the form of host:port, and comma-separated.

topics

required

[string]

The Kafka topics names to read events from.

Regular expression syntax is supported if the topic begins with ^.

group_id

required

string

The consumer group name to be used to consume events from Kafka.

group_instance_id

optional

string,​null

Override dynamic membership and broker assignment behavior with static membership, using a group instance (member) id.

auto_offset_reset

optional

string

If offsets for consumer group do not exist, set them using this strategy.

See the librdkafka documentation for the auto.offset.reset option for further clarification.

session_timeout_ms

optional

integer

The Kafka session timeout.

socket_timeout_ms

optional

integer

Timeout for network requests.

fetch_wait_max_ms

optional

integer

Maximum time the broker may wait to fill the response.

commit_interval_ms

optional

integer

The frequency that the consumer offsets are committed (written) to offset storage.

key_field

optional

string

Overrides the name of the log field used to add the message key to each event.

The value will be the message key of the Kafka message itself.

By default, "message_key" is used.

topic_key

optional

string

Overrides the name of the log field used to add the topic to each event.

The value will be the topic from which the Kafka message was consumed from.

By default, "topic" is used.

partition_key

optional

string

Overrides the name of the log field used to add the partition to each event.

The value will be the partition from which the Kafka message was consumed from.

By default, "partition" is used.

offset_key

optional

string

Overrides the name of the log field used to add the offset to each event.

The value will be the offset of the Kafka message itself.

By default, "offset" is used.

headers_key

optional

string

Overrides the name of the log field used to add the headers to each event.

The value will be the headers of the Kafka message itself.

By default, "headers" is used.

librdkafka_options

optional

object,​null

Advanced options set directly on the underlying librdkafka client.

See the librdkafka documentation for details.

framing

optional

 <oneOf>

Framing configuration.

Framing deals with how events are separated when encoded in a raw byte form, where each event is a "frame" that must be prefixed, or delimited, in a way that marks where an event begins and ends within the byte stream.

Bytes

optional

object

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

method

required

bytes

Byte frames are passed through as-is according to the underlying I/O boundaries (e.g. split between messages or stream segments).

CharacterDelimited

optional

object

Byte frames which are delimited by a chosen character.

character_delimited

required

object

Options for the character delimited decoder.

delimiter

required

integer

The character that delimits byte sequences.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

character_delimited

Byte frames which are delimited by a chosen character.

LengthDelimited

optional

object

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

method

required

length_delimited

Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

NewlineDelimited

optional

object

Byte frames which are delimited by a newline character.

newline_delimited

optional

object

Options for the newline delimited decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

This length does not include the trailing delimiter.

method

required

newline_delimited

Byte frames which are delimited by a newline character.

OctetCounting

optional

object

Byte frames according to the octet counting format.

octet_counting

optional

object

Options for the octet counting decoder.

max_length

optional

integer,​null

The maximum length of the byte buffer.

method

required

octet_counting

Byte frames according to the octet counting format.

decoding

optional

 <oneOf>

Configures how events are decoded from raw bytes.

Bytes

optional

object

Uses the raw bytes as-is.

codec

required

bytes

Uses the raw bytes as-is.

Json

optional

object

Decodes the raw bytes as JSON.

codec

required

json

Decodes the raw bytes as JSON.

Syslog

optional

object

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

codec

required

syslog

Decodes the raw bytes as a Syslog message.

Will decode either as the RFC 3164-style format ("old" style) or the more modern RFC 5424-style format ("new" style, includes structured data).

Native

optional

object

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Decodes the raw bytes as Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Decodes the raw bytes as Vector’s native JSON format.

This codec is experimental.

Gelf

optional

object

Decodes the raw bytes as a GELF message.

codec

required

gelf

Decodes the raw bytes as a GELF message.

acknowledgements

optional

object

Controls how acknowledgements are handled by this source.

This setting is deprecated in favor of enabling acknowledgements at the global or sink level. Enabling or disabling acknowledgements at the source level has no effect on acknowledgement behavior.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled for this source.

default: null

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

sasl

optional

object,​null

Configuration for SASL authentication when interacting with Kafka.

enabled

optional

boolean,​null

Enables SASL authentication.

Only PLAIN and SCRAM-based mechanisms are supported when configuring SASL authentication via sasl.*. For other mechanisms, librdkafka_options.* must be used directly to configure other librdkafka-specific values i.e. sasl.kerberos.* and so on.

See the librdkafka documentation for details.

SASL authentication is not supported on Windows.

username

optional

string,​null

The SASL username.

password

optional

string,​null

The SASL password.

mechanism

optional

string,​null

The SASL mechanism to use.

tls

optional

 <oneOf>

Configures the TLS options for incoming/outgoing connections.

Option 1

optional

enabled

optional

boolean,​null

Whether or not to require TLS for incoming/outgoing connections.

When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

bootstrap_servers: string
topics: array
group_id: string
group_instance_id: string
auto_offset_reset: largest
session_timeout_ms: 10000
socket_timeout_ms: 60000
fetch_wait_max_ms: 100
commit_interval_ms: 5000
key_field: message_key
topic_key: topic
partition_key: partition
offset_key: offset
headers_key: headers
librdkafka_options: object
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
log_namespace: boolean
type: kafka
bootstrap_servers: string
topics: array
group_id: string
group_instance_id: string
auto_offset_reset: largest
session_timeout_ms: 10000
socket_timeout_ms: 60000
fetch_wait_max_ms: 100
commit_interval_ms: 5000
key_field: message_key
topic_key: topic
partition_key: partition
offset_key: offset
headers_key: headers
librdkafka_options: object
framing:
  method: bytes
decoding:
  codec: bytes
acknowledgements:
  enabled: null
log_namespace: boolean
sasl: object
tls: ''
type: kafka

Kubernetes Logs

Configuration for the kubernetes_logs source.

Field

required

Type

Description

extra_label_selector

optional

string

Specifies the label selector to filter Pods with, to be used in addition to the built-in exclude filter.

extra_namespace_label_selector

optional

string

Specifies the label selector to filter Namespaces with, to be used in addition to the built-in exclude filter.

self_node_name

optional

string

The name of the Kubernetes Node that is running.

Configured to use an environment var by default, to be evaluated to a value provided by Kubernetes at Pod deploy time.

extra_field_selector

optional

string

Specifies the field selector to filter Pods with, to be used in addition to the built-in Node filter.

auto_partial_merge

optional

boolean

Whether or not to automatically merge partial events.

data_dir

optional

string,​null

The directory used to persist file checkpoint positions.

By default, the global data_dir option is used. Make sure the running user has write permissions to this directory.

pod_annotation_fields

optional

object

Configuration for how the events are annotated with Pod metadata.

pod_name

optional

string

Event field for Pod name.

default: .kubernetes.pod_name

pod_namespace

optional

string

Event field for Pod namespace.

default: .kubernetes.pod_namespace

pod_uid

optional

string

Event field for Pod uid.

default: .kubernetes.pod_uid

pod_ip

optional

string

Event field for Pod IPv4 address.

default: .kubernetes.pod_ip

pod_ips

optional

string

Event field for Pod IPv4 and IPv6 addresses.

default: .kubernetes.pod_ips

pod_labels

optional

string

Event field for Pod labels.

default: .kubernetes.pod_labels

pod_annotations

optional

string

Event field for Pod annotations.

default: .kubernetes.pod_annotations

pod_node_name

optional

string

Event field for Pod node_name.

default: .kubernetes.pod_node_name

pod_owner

optional

string

Event field for Pod owner reference.

default: .kubernetes.pod_owner

container_name

optional

string

Event field for container name.

default: .kubernetes.container_name

container_id

optional

string

Event field for container ID.

default: .kubernetes.container_id

container_image

optional

string

Event field for container image.

default: .kubernetes.container_image

namespace_annotation_fields

optional

object

Configuration for how the events are annotated with Namespace metadata.

namespace_labels

optional

string

Event field for Namespace labels.

default: .kubernetes.namespace_labels

node_annotation_fields

optional

object

Configuration for how the events are annotated with Node metadata.

node_labels

optional

string

Event field for Node labels.

default: .kubernetes.node_labels

exclude_paths_glob_patterns

optional

[string]

A list of glob patterns to exclude from reading the files.

read_from

optional

 <oneOf>

File position to use when reading a new file.

beginning

optional

beginning

Read from the beginning of the file.

end

optional

end

Start reading from the current end of the file.

ignore_older_secs

optional

integer,​null

Ignore files with a data modification date older than the specified number of seconds.

max_read_bytes

optional

integer

Max amount of bytes to read from a single file before switching over to the next file. This allows distributing the reads more or less evenly across the files.

max_line_bytes

optional

integer

The maximum number of bytes a line can contain before being discarded. This protects against malformed lines or tailing incorrect files.

fingerprint_lines

optional

integer

How many first lines in a file are used for fingerprinting.

glob_minimum_cooldown_ms

optional

integer

This value specifies not exactly the globbing, but interval between the polling the files to watch from the paths_provider. This is quite efficient, yet might still create some load of the file system; in addition, it is currently coupled with chechsum dumping in the underlying file server, so setting it too low may introduce a significant overhead.

ingestion_timestamp_field

optional

string,​null

Overrides the name of the log field used to add the ingestion timestamp to each event.

This is useful to compute the latency between important event processing stages. For example, the time delta between when a log line was written and when it was processed by the kubernetes_logs source.

By default, the global log_schema.timestamp_key option is used.

timezone

optional

 <oneOf>

Timezone reference.

The default time zone for timestamps without an explicit zone.

local

optional

local

System local timezone.

Named

optional

string

A named timezone.

Must be a valid name in the TZ database.

kube_config_file

optional

string,​null

Optional path to a readable kubeconfig file. If not set, a connection to Kubernetes is made using the in-cluster configuration.

delay_deletion_ms

optional

integer

How long to delay removing entries from our map when we receive a deletion event from the watched stream.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

extra_label_selector: string
extra_namespace_label_selector: string
self_node_name: string
extra_field_selector: string
auto_partial_merge: boolean
data_dir: string
pod_annotation_fields:
  pod_name: .kubernetes.pod_name
  pod_namespace: .kubernetes.pod_namespace
  pod_uid: .kubernetes.pod_uid
  pod_ip: .kubernetes.pod_ip
  pod_ips: .kubernetes.pod_ips
  pod_labels: .kubernetes.pod_labels
  pod_annotations: .kubernetes.pod_annotations
  pod_node_name: .kubernetes.pod_node_name
  pod_owner: .kubernetes.pod_owner
  container_name: .kubernetes.container_name
  container_id: .kubernetes.container_id
  container_image: .kubernetes.container_image
namespace_annotation_fields:
  namespace_labels: .kubernetes.namespace_labels
node_annotation_fields:
  node_labels: .kubernetes.node_labels
exclude_paths_glob_patterns: array
read_from: beginning
ignore_older_secs: integer
max_read_bytes: integer
max_line_bytes: integer
fingerprint_lines: integer
glob_minimum_cooldown_ms: integer
ingestion_timestamp_field: string
timezone: ''
kube_config_file: string
delay_deletion_ms: integer
log_namespace: boolean
type: kubernetes_logs
extra_label_selector: string
extra_namespace_label_selector: string
self_node_name: string
extra_field_selector: string
auto_partial_merge: boolean
data_dir: string
pod_annotation_fields:
  pod_name: .kubernetes.pod_name
  pod_namespace: .kubernetes.pod_namespace