Sinks

A sink is a destination for events. Each sink’s design and transmission method is determined by the downstream service with which it interacts. For example, the socket sink streams individual events, while the aws_s3 sink buffers and flushes data.

AMQP

Supports AMQP version 0.9.1

Field

required

Type

Description

exchange

required

string

A templated field.

The exchange to publish messages to.

routing_key

optional

string,​null

A templated field.

Template used to generate a routing key which corresponds to a queue binding.

encoding

required

Configures how events are encoded into raw bytes.

Avro

optional

object

Encodes an event as an Apache Avro message.

avro

required

object

Apache Avro-specific encoder options.

schema

required

string

The Avro schema.

codec

required

avro

Encodes an event as an Apache Avro message.

Gelf

optional

object

Encodes an event as a GELF message.

codec

required

gelf

Encodes an event as a GELF message.

Json

optional

Encodes an event as JSON.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

json

Encodes an event as JSON.

Logfmt

optional

object

Encodes an event as a logfmt message.

codec

required

logfmt

Encodes an event as a logfmt message.

Native

optional

object

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Encodes an event in Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Encodes an event in Vector’s native JSON format.

This codec is experimental.

RawMessage

optional

object

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

codec

required

raw_message

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

Text

optional

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

text

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

only_fields

optional

array,​null

List of fields that will be included in the encoded event.

except_fields

optional

array,​null

List of fields that will be excluded from the encoded event.

timestamp_format

optional

 <oneOf>

Format used for timestamp fields.

unix

optional

unix

Represent the timestamp as a Unix timestamp.

rfc3339

optional

rfc3339

Represent the timestamp as a RFC 3339 timestamp.

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null

connection_string

required

string

URI for the AMQP server.

The URI has the format of amqp://<user>:<password>@<host>:<port>/<vhost>?timeout=<seconds>.

The default vhost can be specified by using a value of %2f.

In order to connect over TLS, a scheme of amqps can be specified instead i.e. amqps://.... Additional TLS settings, such as client certificate verification, etc, can be configured under the tls section.

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

exchange: string
routing_key: string
encoding: ''
acknowledgements:
  enabled: null
type: amqp
exchange: string
routing_key: string
encoding: ''
acknowledgements:
  enabled: null
connection_string: string
tls: object
type: amqp

Apex Logs

Configuration for the apex sink.

Field

required

Type

Description

uri

required

string

The base URI of the Apex instance.

/add_events is appended to this.

project_id

required

string

The ID of the project to associate reported logs with.

api_token

required

string

The API token to use to authenticate with Apex.

batch

optional

object

Event batching behavior.

max_bytes

optional

integer,​null

The maximum size of a batch that will be processed by a sink.

This is based on the uncompressed size of the batched events, before they are serialized / compressed.

default: null

max_events

optional

integer,​null

The maximum size of a batch, in events, before it is flushed.

default: null

timeout_secs

optional

number,​null

The maximum age of a batch, in seconds, before it is flushed.

default: null

request

optional

object

Middleware settings for outbound requests.

Various settings can be configured, such as concurrency and rate limits, timeouts, etc.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

default: 60

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

default: 1

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

default: 9223372036854776000

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

default: 9223372036854776000

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

default: 3600

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

default: 1

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"rtt_deviation_scale":2.5}

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null
uri: string
project_id: string
api_token: string
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
acknowledgements:
  enabled: null
type: apex
uri: string
project_id: string
api_token: string
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
acknowledgements:
  enabled: null
type: apex

AWS CloudWatch Logs

Configuration for the aws_cloudwatch_logs sink.

Field

required

Type

Description

group_name

required

string

The group name of the target CloudWatch Logs stream.

stream_name

required

string

The stream name of the target CloudWatch Logs stream.

There can only be one writer to a log stream at a time. If you have multiple instances writing to the same log group, you must include an identifier in the stream name that is guaranteed to be unique per instance.

For example, you might choose host.

create_missing_group

optional

boolean,​null

Dynamically create a log group if it does not already exist.

This will ignore create_missing_stream directly after creating the group and will create the first stream.

create_missing_stream

optional

boolean,​null

Dynamically create a log stream if it does not already exist.

encoding

required

Configures how events are encoded into raw bytes.

Avro

optional

object

Encodes an event as an Apache Avro message.

avro

required

object

Apache Avro-specific encoder options.

schema

required

string

The Avro schema.

codec

required

avro

Encodes an event as an Apache Avro message.

Gelf

optional

object

Encodes an event as a GELF message.

codec

required

gelf

Encodes an event as a GELF message.

Json

optional

Encodes an event as JSON.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

json

Encodes an event as JSON.

Logfmt

optional

object

Encodes an event as a logfmt message.

codec

required

logfmt

Encodes an event as a logfmt message.

Native

optional

object

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Encodes an event in Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Encodes an event in Vector’s native JSON format.

This codec is experimental.

RawMessage

optional

object

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

codec

required

raw_message

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

Text

optional

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

text

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

only_fields

optional

array,​null

List of fields that will be included in the encoded event.

except_fields

optional

array,​null

List of fields that will be excluded from the encoded event.

timestamp_format

optional

 <oneOf>

Format used for timestamp fields.

unix

optional

unix

Represent the timestamp as a Unix timestamp.

rfc3339

optional

rfc3339

Represent the timestamp as a RFC 3339 timestamp.

compression

optional

 <oneOf>

Compression configuration.

All compression algorithms use the default compression level unless otherwise specified.

Option 1

optional

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

Option 2

optional

object

algorithm

required

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

level

optional

enum

Compression level. Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9

batch

optional

object

Event batching behavior.

max_bytes

optional

integer,​null

The maximum size of a batch that will be processed by a sink.

This is based on the uncompressed size of the batched events, before they are serialized / compressed.

default: null

max_events

optional

integer,​null

The maximum size of a batch, in events, before it is flushed.

default: null

timeout_secs

optional

number,​null

The maximum age of a batch, in seconds, before it is flushed.

default: null

request

optional

Outbound HTTP request settings.

headers

optional

object

Additional HTTP headers to add to every HTTP request.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

assume_role

optional

string,​null

The ARN of an [IAM role][iam_role] to assume at startup.

DEPRECATED: [iam_role]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html

auth

optional

 <oneOf>

Configuration of the authentication strategy for interacting with AWS services.

AccessKey

optional

object

Authenticate using a fixed access key and secret pair.

access_key_id

required

string

The AWS access key ID.

secret_access_key

required

string

The AWS secret access key.

File

optional

object

Authenticate using credentials stored in a file.

Additionally, the specific credential profile to use can be set.

credentials_file

required

string

Path to the credentials file.

profile

optional

string,​null

The credentials profile to use.

Role

optional

object

Assume the given role ARN.

assume_role

required

string

The ARN of the role to assume.

load_timeout_secs

optional

integer,​null

Timeout for assuming the role, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

region

optional

string,​null

The AWS region to send STS requests to.

If not set, this will default to the configured region for the service itself.

Default

optional

object

Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.

load_timeout_secs

optional

integer,​null

Timeout for successfully loading any credentials, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null

region

optional

string,​null

The AWS region to use.

endpoint

optional

string,​null

The API endpoint of the service.

group_name: string
stream_name: string
create_missing_group: boolean
create_missing_stream: boolean
encoding: ''
compression: none
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
  headers: {}
tls: object
assume_role: string
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
acknowledgements:
  enabled: null
type: aws_cloudwatch_logs
group_name: string
stream_name: string
create_missing_group: boolean
create_missing_stream: boolean
encoding: ''
compression: none
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
  headers: {}
tls: object
assume_role: string
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
acknowledgements:
  enabled: null
region: string
endpoint: string
type: aws_cloudwatch_logs

AWS CloudWatch Metrics

Configuration for the aws_cloudwatch_metrics sink.

Field

required

Type

Description

default_namespace

required

string

The default namespace to use for metrics that do not have one.

Metrics with the same name can only be differentiated by their namespace, and not all metrics have their own namespace.

compression

optional

 <oneOf>

Compression configuration.

All compression algorithms use the default compression level unless otherwise specified.

Option 1

optional

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

Option 2

optional

object

algorithm

required

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

level

optional

enum

Compression level. Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9

batch

optional

object

Event batching behavior.

max_bytes

optional

integer,​null

The maximum size of a batch that will be processed by a sink.

This is based on the uncompressed size of the batched events, before they are serialized / compressed.

default: null

max_events

optional

integer,​null

The maximum size of a batch, in events, before it is flushed.

default: null

timeout_secs

optional

number,​null

The maximum age of a batch, in seconds, before it is flushed.

default: null

request

optional

object

Middleware settings for outbound requests.

Various settings can be configured, such as concurrency and rate limits, timeouts, etc.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

default: 60

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

default: 1

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

default: 9223372036854776000

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

default: 9223372036854776000

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

default: 3600

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

default: 1

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"rtt_deviation_scale":2.5}

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

assume_role

optional

string,​null

The ARN of an [IAM role][iam_role] to assume at startup.

DEPRECATED: [iam_role]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html

auth

optional

 <oneOf>

Configuration of the authentication strategy for interacting with AWS services.

AccessKey

optional

object

Authenticate using a fixed access key and secret pair.

access_key_id

required

string

The AWS access key ID.

secret_access_key

required

string

The AWS secret access key.

File

optional

object

Authenticate using credentials stored in a file.

Additionally, the specific credential profile to use can be set.

credentials_file

required

string

Path to the credentials file.

profile

optional

string,​null

The credentials profile to use.

Role

optional

object

Assume the given role ARN.

assume_role

required

string

The ARN of the role to assume.

load_timeout_secs

optional

integer,​null

Timeout for assuming the role, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

region

optional

string,​null

The AWS region to send STS requests to.

If not set, this will default to the configured region for the service itself.

Default

optional

object

Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.

load_timeout_secs

optional

integer,​null

Timeout for successfully loading any credentials, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null

region

optional

string,​null

The AWS region to use.

endpoint

optional

string,​null

The API endpoint of the service.

default_namespace: string
compression: none
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
assume_role: string
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
acknowledgements:
  enabled: null
type: aws_cloudwatch_metrics
default_namespace: string
compression: none
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
assume_role: string
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
acknowledgements:
  enabled: null
region: string
endpoint: string
type: aws_cloudwatch_metrics

AWS Kinesis Firehose

Configuration for the aws_kinesis_firehose sink.

Field

required

Type

Description

batch

optional

object

Event batching behavior.

max_bytes

optional

integer,​null

The maximum size of a batch that will be processed by a sink.

This is based on the uncompressed size of the batched events, before they are serialized / compressed.

default: null

max_events

optional

integer,​null

The maximum size of a batch, in events, before it is flushed.

default: null

timeout_secs

optional

number,​null

The maximum age of a batch, in seconds, before it is flushed.

default: null

stream_name

optional

string

The stream name of the target Kinesis Firehose delivery stream.

encoding

optional

Configures how events are encoded into raw bytes.

Avro

optional

object

Encodes an event as an Apache Avro message.

avro

required

object

Apache Avro-specific encoder options.

schema

required

string

The Avro schema.

codec

required

avro

Encodes an event as an Apache Avro message.

Gelf

optional

object

Encodes an event as a GELF message.

codec

required

gelf

Encodes an event as a GELF message.

Json

optional

Encodes an event as JSON.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

json

Encodes an event as JSON.

Logfmt

optional

object

Encodes an event as a logfmt message.

codec

required

logfmt

Encodes an event as a logfmt message.

Native

optional

object

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Encodes an event in Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Encodes an event in Vector’s native JSON format.

This codec is experimental.

RawMessage

optional

object

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

codec

required

raw_message

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

Text

optional

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

text

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

only_fields

optional

array,​null

List of fields that will be included in the encoded event.

except_fields

optional

array,​null

List of fields that will be excluded from the encoded event.

timestamp_format

optional

 <oneOf>

Format used for timestamp fields.

unix

optional

unix

Represent the timestamp as a Unix timestamp.

rfc3339

optional

rfc3339

Represent the timestamp as a RFC 3339 timestamp.

compression

optional

 <oneOf>

Compression configuration.

All compression algorithms use the default compression level unless otherwise specified.

Option 1

optional

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

Option 2

optional

object

algorithm

required

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

level

optional

enum

Compression level. Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9

request

optional

object

Middleware settings for outbound requests.

Various settings can be configured, such as concurrency and rate limits, timeouts, etc.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

default: 60

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

default: 1

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

default: 9223372036854776000

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

default: 9223372036854776000

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

default: 3600

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

default: 1

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"rtt_deviation_scale":2.5}

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

auth

optional

 <oneOf>

Configuration of the authentication strategy for interacting with AWS services.

AccessKey

optional

object

Authenticate using a fixed access key and secret pair.

access_key_id

required

string

The AWS access key ID.

secret_access_key

required

string

The AWS secret access key.

File

optional

object

Authenticate using credentials stored in a file.

Additionally, the specific credential profile to use can be set.

credentials_file

required

string

Path to the credentials file.

profile

optional

string,​null

The credentials profile to use.

Role

optional

object

Assume the given role ARN.

assume_role

required

string

The ARN of the role to assume.

load_timeout_secs

optional

integer,​null

Timeout for assuming the role, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

region

optional

string,​null

The AWS region to send STS requests to.

If not set, this will default to the configured region for the service itself.

Default

optional

object

Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.

load_timeout_secs

optional

integer,​null

Timeout for successfully loading any credentials, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null

region

optional

string,​null

The AWS region to use.

endpoint

optional

string,​null

The API endpoint of the service.

batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
type: aws_kinesis_firehose
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
stream_name: string
encoding: ''
compression: none
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
acknowledgements:
  enabled: null
type: aws_kinesis_firehose

AWS Kinesis Streams

Configuration for the aws_kinesis_streams sink.

Field

required

Type

Description

partition_key_field

optional

string,​null

The log field used as the Kinesis record’s partition key value.

If not specified, a unique partition key will be generated for each Kinesis record.

batch

optional

object

Event batching behavior.

max_bytes

optional

integer,​null

The maximum size of a batch that will be processed by a sink.

This is based on the uncompressed size of the batched events, before they are serialized / compressed.

default: null

max_events

optional

integer,​null

The maximum size of a batch, in events, before it is flushed.

default: null

timeout_secs

optional

number,​null

The maximum age of a batch, in seconds, before it is flushed.

default: null

stream_name

optional

string

The stream name of the target Kinesis Firehose delivery stream.

encoding

optional

Configures how events are encoded into raw bytes.

Avro

optional

object

Encodes an event as an Apache Avro message.

avro

required

object

Apache Avro-specific encoder options.

schema

required

string

The Avro schema.

codec

required

avro

Encodes an event as an Apache Avro message.

Gelf

optional

object

Encodes an event as a GELF message.

codec

required

gelf

Encodes an event as a GELF message.

Json

optional

Encodes an event as JSON.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

json

Encodes an event as JSON.

Logfmt

optional

object

Encodes an event as a logfmt message.

codec

required

logfmt

Encodes an event as a logfmt message.

Native

optional

object

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Encodes an event in Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Encodes an event in Vector’s native JSON format.

This codec is experimental.

RawMessage

optional

object

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

codec

required

raw_message

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

Text

optional

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

text

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

only_fields

optional

array,​null

List of fields that will be included in the encoded event.

except_fields

optional

array,​null

List of fields that will be excluded from the encoded event.

timestamp_format

optional

 <oneOf>

Format used for timestamp fields.

unix

optional

unix

Represent the timestamp as a Unix timestamp.

rfc3339

optional

rfc3339

Represent the timestamp as a RFC 3339 timestamp.

compression

optional

 <oneOf>

Compression configuration.

All compression algorithms use the default compression level unless otherwise specified.

Option 1

optional

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

Option 2

optional

object

algorithm

required

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

level

optional

enum

Compression level. Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9

request

optional

object

Middleware settings for outbound requests.

Various settings can be configured, such as concurrency and rate limits, timeouts, etc.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

default: 60

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

default: 1

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

default: 9223372036854776000

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

default: 9223372036854776000

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

default: 3600

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

default: 1

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"rtt_deviation_scale":2.5}

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

auth

optional

 <oneOf>

Configuration of the authentication strategy for interacting with AWS services.

AccessKey

optional

object

Authenticate using a fixed access key and secret pair.

access_key_id

required

string

The AWS access key ID.

secret_access_key

required

string

The AWS secret access key.

File

optional

object

Authenticate using credentials stored in a file.

Additionally, the specific credential profile to use can be set.

credentials_file

required

string

Path to the credentials file.

profile

optional

string,​null

The credentials profile to use.

Role

optional

object

Assume the given role ARN.

assume_role

required

string

The ARN of the role to assume.

load_timeout_secs

optional

integer,​null

Timeout for assuming the role, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

region

optional

string,​null

The AWS region to send STS requests to.

If not set, this will default to the configured region for the service itself.

Default

optional

object

Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.

load_timeout_secs

optional

integer,​null

Timeout for successfully loading any credentials, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null

region

optional

string,​null

The AWS region to use.

endpoint

optional

string,​null

The API endpoint of the service.

partition_key_field: string
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
type: aws_kinesis_streams
partition_key_field: string
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
stream_name: string
encoding: ''
compression: none
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
acknowledgements:
  enabled: null
type: aws_kinesis_streams

AWS S3

Configuration for the aws_s3 sink.

Field

required

Type

Description

bucket

required

string

The S3 bucket name.

This must not include a leading s3:// or a trailing /.

key_prefix

optional

string,​null

A prefix to apply to all object keys.

Prefixes are useful for partitioning objects, such as by creating an object key that stores objects under a particular "directory". If using a prefix for this purpose, it must end in / to act as a directory path. A trailing / is not automatically added.

filename_time_format

optional

string,​null

The timestamp format for the time component of the object key.

By default, object keys are appended with a timestamp that reflects when the objects are sent to S3, such that the resulting object key is functionally equivalent to joining the key prefix with the formatted timestamp, such as date=2022-07-18/1658176486.

This would represent a key_prefix set to date=%F/ and the timestamp of Mon Jul 18 2022 20:34:44 GMT+0000, with the filename_time_format being set to %s, which renders timestamps in seconds since the Unix epoch.

Supports the common strftime specifiers found in most languages.

When set to an empty string, no timestamp will be appended to the key prefix.

filename_append_uuid

optional

boolean,​null

Whether or not to append a UUID v4 token to the end of the object key.

The UUID is appended to the timestamp portion of the object key, such that if the object key being generated was date=2022-07-18/1658176486, setting this field to true would result in an object key that looked like date=2022-07-18/1658176486-30f6652c-71da-4f9f-800d-a1189c47c547.

This ensures there are no name collisions, and can be useful in high-volume workloads where object keys must be unique.

filename_extension

optional

string,​null

The filename extension to use in the object key.

compression

optional

 <oneOf>

Compression configuration.

All compression algorithms use the default compression level unless otherwise specified.

Option 1

optional

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

Option 2

optional

object

algorithm

required

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

level

optional

enum

Compression level. Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9

batch

optional

object

Event batching behavior.

max_bytes

optional

integer,​null

The maximum size of a batch that will be processed by a sink.

This is based on the uncompressed size of the batched events, before they are serialized / compressed.

default: null

max_events

optional

integer,​null

The maximum size of a batch, in events, before it is flushed.

default: null

timeout_secs

optional

number,​null

The maximum age of a batch, in seconds, before it is flushed.

default: null

request

optional

object

Middleware settings for outbound requests.

Various settings can be configured, such as concurrency and rate limits, timeouts, etc.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

default: 60

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

default: 1

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

default: 9223372036854776000

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

default: 9223372036854776000

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

default: 3600

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

default: 1

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"rtt_deviation_scale":2.5}

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

auth

optional

 <oneOf>

Configuration of the authentication strategy for interacting with AWS services.

AccessKey

optional

object

Authenticate using a fixed access key and secret pair.

access_key_id

required

string

The AWS access key ID.

secret_access_key

required

string

The AWS secret access key.

File

optional

object

Authenticate using credentials stored in a file.

Additionally, the specific credential profile to use can be set.

credentials_file

required

string

Path to the credentials file.

profile

optional

string,​null

The credentials profile to use.

Role

optional

object

Assume the given role ARN.

assume_role

required

string

The ARN of the role to assume.

load_timeout_secs

optional

integer,​null

Timeout for assuming the role, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

region

optional

string,​null

The AWS region to send STS requests to.

If not set, this will default to the configured region for the service itself.

Default

optional

object

Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.

load_timeout_secs

optional

integer,​null

Timeout for successfully loading any credentials, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null

acl

optional

 <oneOf>

Canned ACL to apply to the created objects.

For more information, see Canned ACL.

private

optional

private

Bucket/object are private.

The bucket/object owner is granted the FULL_CONTROL permission, and no one else has access.

This is the default.

public-read

optional

public-read

Bucket/object can be read publically.

The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the AllUsers grantee group is granted the READ permission.

public-read-write

optional

public-read-write

Bucket/object can be read and written publically.

The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the AllUsers grantee group is granted the READ and WRITE permissions.

This is generally not recommended.

aws-exec-read

optional

aws-exec-read

Bucket/object are private, and readable by EC2.

The bucket/object owner is granted the FULL_CONTROL permission, and the AWS EC2 service is granted the READ permission for the purpose of reading Amazon Machine Image (AMI) bundles from the given bucket.

authenticated-read

optional

authenticated-read

Bucket/object can be read by authenticated users.

The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the AuthenticatedUsers grantee group is granted the READ permission.

bucket-owner-read

optional

bucket-owner-read

Object is private, except to the bucket owner.

The object owner is granted the FULL_CONTROL permission, and the bucket owner is granted the READ permission.

Only relevant when specified for an object: this canned ACL is otherwise ignored when specified for a bucket.

bucket-owner-full-control

optional

bucket-owner-full-control

Object is semi-private.

Both the object owner and bucket owner are granted the FULL_CONTROL permission.

Only relevant when specified for an object: this canned ACL is otherwise ignored when specified for a bucket.

log-delivery-write

optional

log-delivery-write

Bucket can have logs written.

The LogDelivery grantee group is granted WRITE and READ_ACP permissions.

Only relevant when specified for a bucket: this canned ACL is otherwise ignored when specified for an object.

grant_full_control

optional

string,​null

Grants READ, READ_ACP, and WRITE_ACP permissions on the created objects to the named grantee.

This allows the grantee to read the created objects and their metadata, as well as read and modify the ACL on the created objects.

grant_read

optional

string,​null

Grants READ permissions on the created objects to the named grantee.

This allows the grantee to read the created objects and their metadata.

grant_read_acp

optional

string,​null

Grants READ_ACP permissions on the created objects to the named grantee.

This allows the grantee to read the ACL on the created objects.

grant_write_acp

optional

string,​null

Grants WRITE_ACP permissions on the created objects to the named grantee.

This allows the grantee to modify the ACL on the created objects.

server_side_encryption

optional

 <oneOf>

AWS S3 Server-Side Encryption algorithms.

The Server-side Encryption algorithm used when storing these objects.

AES256

optional

AES256

Each object is encrypted with AES-256 using a unique key.

This corresponds to the SSE-S3 option.

aws:kms

optional

aws:kms

Each object is encrypted with AES-256 using keys managed by AWS KMS.

Depending on whether or not a KMS key ID is specified, this will correspond either to the SSE-KMS option (keys generated/managed by KMS) or the SSE-C option (keys generated by the customer, managed by KMS).

ssekms_key_id

optional

string,​null

Specifies the ID of the AWS Key Management Service (AWS KMS) symmetrical customer managed customer master key (CMK) that will used for the created objects.

Only applies when server_side_encryption is configured to use KMS.

If not specified, Amazon S3 uses the AWS managed CMK in AWS to protect the data.

storage_class

optional

 <oneOf>

The storage class for the created objects.

See the S3 Storage Classes for more details.

STANDARD

optional

STANDARD

Standard Redundancy.

This is the default.

REDUCED_REDUNDANCY

optional

REDUCED_REDUNDANCY

Reduced Redundancy.

INTELLIGENT_TIERING

optional

INTELLIGENT_TIERING

Intelligent Tiering.

STANDARD_IA

optional

STANDARD_IA

Infrequently Accessed.

ONEZONE_IA

optional

ONEZONE_IA

Infrequently Accessed (single Availability zone).

GLACIER

optional

GLACIER

Glacier Flexible Retrieval.

DEEP_ARCHIVE

optional

DEEP_ARCHIVE

Glacier Deep Archive.

tags

optional

object,​null

The tag-set for the object.

content_encoding

optional

string,​null

Specifies what content encoding has been applied to the object.

Directly comparable to the Content-Encoding HTTP header.

By default, the compression scheme used dictates this value.

content_type

optional

string,​null

Specifies the MIME type of the object.

Directly comparable to the Content-Type HTTP header.

By default, text/x-log is used.

region

optional

string,​null

The AWS region to use.

endpoint

optional

string,​null

The API endpoint of the service.

framing

optional

 <oneOf>

Framing configuration.

Bytes

optional

object

Event data is not delimited at all.

method

required

bytes

Event data is not delimited at all.

CharacterDelimited

optional

object

Event data is delimited by a single ASCII (7-bit) character.

character_delimited

required

object

Options for the character delimited encoder.

delimiter

required

integer

The ASCII (7-bit) character that delimits byte sequences.

method

required

character_delimited

Event data is delimited by a single ASCII (7-bit) character.

LengthDelimited

optional

object

Event data is prefixed with its length in bytes.

The prefix is a 32-bit unsigned integer, little endian.

method

required

length_delimited

Event data is prefixed with its length in bytes.

The prefix is a 32-bit unsigned integer, little endian.

NewlineDelimited

optional

object

Event data is delimited by a newline (LF) character.

method

required

newline_delimited

Event data is delimited by a newline (LF) character.

encoding

required

Configures how events are encoded into raw bytes.

Avro

optional

object

Encodes an event as an Apache Avro message.

avro

required

object

Apache Avro-specific encoder options.

schema

required

string

The Avro schema.

codec

required

avro

Encodes an event as an Apache Avro message.

Gelf

optional

object

Encodes an event as a GELF message.

codec

required

gelf

Encodes an event as a GELF message.

Json

optional

Encodes an event as JSON.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

json

Encodes an event as JSON.

Logfmt

optional

object

Encodes an event as a logfmt message.

codec

required

logfmt

Encodes an event as a logfmt message.

Native

optional

object

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Encodes an event in Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Encodes an event in Vector’s native JSON format.

This codec is experimental.

RawMessage

optional

object

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

codec

required

raw_message

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

Text

optional

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

text

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

only_fields

optional

array,​null

List of fields that will be included in the encoded event.

except_fields

optional

array,​null

List of fields that will be excluded from the encoded event.

timestamp_format

optional

 <oneOf>

Format used for timestamp fields.

unix

optional

unix

Represent the timestamp as a Unix timestamp.

rfc3339

optional

rfc3339

Represent the timestamp as a RFC 3339 timestamp.

bucket: string
key_prefix: string
filename_time_format: string
filename_append_uuid: boolean
filename_extension: string
compression: gzip
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
acknowledgements:
  enabled: null
type: aws_s3
bucket: string
key_prefix: string
filename_time_format: string
filename_append_uuid: boolean
filename_extension: string
compression: gzip
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
acknowledgements:
  enabled: null
acl: ''
grant_full_control: string
grant_read: string
grant_read_acp: string
grant_write_acp: string
server_side_encryption: ''
ssekms_key_id: string
storage_class: ''
tags: object
content_encoding: string
content_type: string
region: string
endpoint: string
framing: ''
encoding: ''
type: aws_s3

AWS SQS

Configuration for the aws_sqs sink.

Field

required

Type

Description

queue_url

required

uri

The URL of the Amazon SQS queue to which messages are sent.

encoding

required

Configures how events are encoded into raw bytes.

Avro

optional

object

Encodes an event as an Apache Avro message.

avro

required

object

Apache Avro-specific encoder options.

schema

required

string

The Avro schema.

codec

required

avro

Encodes an event as an Apache Avro message.

Gelf

optional

object

Encodes an event as a GELF message.

codec

required

gelf

Encodes an event as a GELF message.

Json

optional

Encodes an event as JSON.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

json

Encodes an event as JSON.

Logfmt

optional

object

Encodes an event as a logfmt message.

codec

required

logfmt

Encodes an event as a logfmt message.

Native

optional

object

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Encodes an event in Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Encodes an event in Vector’s native JSON format.

This codec is experimental.

RawMessage

optional

object

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

codec

required

raw_message

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

Text

optional

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

text

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

only_fields

optional

array,​null

List of fields that will be included in the encoded event.

except_fields

optional

array,​null

List of fields that will be excluded from the encoded event.

timestamp_format

optional

 <oneOf>

Format used for timestamp fields.

unix

optional

unix

Represent the timestamp as a Unix timestamp.

rfc3339

optional

rfc3339

Represent the timestamp as a RFC 3339 timestamp.

message_group_id

optional

string,​null

The tag that specifies that a message belongs to a specific message group.

Can be applied only to FIFO queues.

message_deduplication_id

optional

string,​null

The message deduplication ID value to allow AWS to identify duplicate messages.

This value is a template which should result in a unique string for each event. See the AWS documentation for more about how AWS does message deduplication.

request

optional

object

Middleware settings for outbound requests.

Various settings can be configured, such as concurrency and rate limits, timeouts, etc.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

default: 60

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

default: 1

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

default: 9223372036854776000

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

default: 9223372036854776000

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

default: 3600

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

default: 1

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"rtt_deviation_scale":2.5}

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

assume_role

optional

string,​null

The ARN of an [IAM role][iam_role] to assume at startup.

DEPRECATED: [iam_role]: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html

auth

optional

 <oneOf>

Configuration of the authentication strategy for interacting with AWS services.

AccessKey

optional

object

Authenticate using a fixed access key and secret pair.

access_key_id

required

string

The AWS access key ID.

secret_access_key

required

string

The AWS secret access key.

File

optional

object

Authenticate using credentials stored in a file.

Additionally, the specific credential profile to use can be set.

credentials_file

required

string

Path to the credentials file.

profile

optional

string,​null

The credentials profile to use.

Role

optional

object

Assume the given role ARN.

assume_role

required

string

The ARN of the role to assume.

load_timeout_secs

optional

integer,​null

Timeout for assuming the role, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

region

optional

string,​null

The AWS region to send STS requests to.

If not set, this will default to the configured region for the service itself.

Default

optional

object

Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.

load_timeout_secs

optional

integer,​null

Timeout for successfully loading any credentials, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null

region

optional

string,​null

The AWS region to use.

endpoint

optional

string,​null

The API endpoint of the service.

queue_url: string
encoding: ''
message_group_id: string
message_deduplication_id: string
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
assume_role: string
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
acknowledgements:
  enabled: null
type: aws_sqs
queue_url: string
encoding: ''
message_group_id: string
message_deduplication_id: string
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
assume_role: string
auth:
  load_timeout_secs: null
  imds:
    max_attempts: 4
    connect_timeout_seconds: 1
    read_timeout_seconds: 1
acknowledgements:
  enabled: null
region: string
endpoint: string
type: aws_sqs

Axiom

Configuration for the axiom sink.

Field

required

Type

Description

url

optional

uri

URI of the Axiom endpoint to send data to.

Only required if not using Axiom Cloud.

org_id

optional

string,​null

The Axiom organization ID.

Only required when using personal tokens.

token

required

string

The Axiom API token.

dataset

required

string

The Axiom dataset to write to.

request

optional

Outbound HTTP request settings.

headers

optional

object

Additional HTTP headers to add to every HTTP request.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

compression

optional

 <oneOf>

Compression configuration.

All compression algorithms use the default compression level unless otherwise specified.

Option 1

optional

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

Option 2

optional

object

algorithm

required

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

level

optional

enum

Compression level. Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null
url: string
org_id: string
token: string
dataset: string
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
  headers: {}
compression: none
tls: object
acknowledgements:
  enabled: null
type: axiom
url: string
org_id: string
token: string
dataset: string
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
  headers: {}
compression: none
tls: object
acknowledgements:
  enabled: null
type: axiom

Azure Blob Storage

Configuration for the azure_blob sink.

Field

required

Type

Description

connection_string

optional

string,​null

The Azure Blob Storage Account connection string.

Authentication with access key is the only supported authentication method.

Either storage_account, or this field, must be specified.

storage_account

optional

string,​null

The Azure Blob Storage Account name.

Attempts to load credentials for the account in the following ways, in order:

Either connection_string, or this field, must be specified.

endpoint

optional

string,​null

The Azure Blob Storage Endpoint URL.

This is used to override the default blob storage endpoint URL in cases where you are using credentials read from the environment/managed identities or access tokens without using an explicit connection_string (which already explicitly supports overriding the blob endpoint URL).

This may only be used with storage_account and will be ignored when used with connection_string.

container_name

required

string

The Azure Blob Storage Account container name.

blob_prefix

optional

string,​null

A prefix to apply to all blob keys.

Prefixes are useful for partitioning objects, such as by creating an blob key that stores blobs under a particular "directory". If using a prefix for this purpose, it must end in / to act as a directory path. A trailing / is not automatically added.

blob_time_format

optional

string,​null

The timestamp format for the time component of the blob key.

By default, blob keys are appended with a timestamp that reflects when the blob are sent to Azure Blob Storage, such that the resulting blob key is functionally equivalent to joining the blob prefix with the formatted timestamp, such as date=2022-07-18/1658176486.

This would represent a blob_prefix set to date=%F/ and the timestamp of Mon Jul 18 2022 20:34:44 GMT+0000, with the filename_time_format being set to %s, which renders timestamps in seconds since the Unix epoch.

Supports the common strftime specifiers found in most languages.

When set to an empty string, no timestamp will be appended to the blob prefix.

blob_append_uuid

optional

boolean,​null

Whether or not to append a UUID v4 token to the end of the blob key.

The UUID is appended to the timestamp portion of the object key, such that if the blob key being generated was date=2022-07-18/1658176486, setting this field to true would result in an blob key that looked like date=2022-07-18/1658176486-30f6652c-71da-4f9f-800d-a1189c47c547.

This ensures there are no name collisions, and can be useful in high-volume workloads where blob keys must be unique.

compression

optional

 <oneOf>

Compression configuration.

All compression algorithms use the default compression level unless otherwise specified.

Option 1

optional

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

Option 2

optional

object

algorithm

required

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

level

optional

enum

Compression level. Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9

batch

optional

object

Event batching behavior.

max_bytes

optional

integer,​null

The maximum size of a batch that will be processed by a sink.

This is based on the uncompressed size of the batched events, before they are serialized / compressed.

default: null

max_events

optional

integer,​null

The maximum size of a batch, in events, before it is flushed.

default: null

timeout_secs

optional

number,​null

The maximum age of a batch, in seconds, before it is flushed.

default: null

request

optional

object

Middleware settings for outbound requests.

Various settings can be configured, such as concurrency and rate limits, timeouts, etc.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

default: 60

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

default: 1

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

default: 9223372036854776000

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

default: 9223372036854776000

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

default: 3600

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

default: 1

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"rtt_deviation_scale":2.5}

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null

framing

optional

 <oneOf>

Framing configuration.

Bytes

optional

object

Event data is not delimited at all.

method

required

bytes

Event data is not delimited at all.

CharacterDelimited

optional

object

Event data is delimited by a single ASCII (7-bit) character.

character_delimited

required

object

Options for the character delimited encoder.

delimiter

required

integer

The ASCII (7-bit) character that delimits byte sequences.

method

required

character_delimited

Event data is delimited by a single ASCII (7-bit) character.

LengthDelimited

optional

object

Event data is prefixed with its length in bytes.

The prefix is a 32-bit unsigned integer, little endian.

method

required

length_delimited

Event data is prefixed with its length in bytes.

The prefix is a 32-bit unsigned integer, little endian.

NewlineDelimited

optional

object

Event data is delimited by a newline (LF) character.

method

required

newline_delimited

Event data is delimited by a newline (LF) character.

encoding

required

Configures how events are encoded into raw bytes.

Avro

optional

object

Encodes an event as an Apache Avro message.

avro

required

object

Apache Avro-specific encoder options.

schema

required

string

The Avro schema.

codec

required

avro

Encodes an event as an Apache Avro message.

Gelf

optional

object

Encodes an event as a GELF message.

codec

required

gelf

Encodes an event as a GELF message.

Json

optional

Encodes an event as JSON.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

json

Encodes an event as JSON.

Logfmt

optional

object

Encodes an event as a logfmt message.

codec

required

logfmt

Encodes an event as a logfmt message.

Native

optional

object

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Encodes an event in Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Encodes an event in Vector’s native JSON format.

This codec is experimental.

RawMessage

optional

object

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

codec

required

raw_message

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

Text

optional

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

text

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

only_fields

optional

array,​null

List of fields that will be included in the encoded event.

except_fields

optional

array,​null

List of fields that will be excluded from the encoded event.

timestamp_format

optional

 <oneOf>

Format used for timestamp fields.

unix

optional

unix

Represent the timestamp as a Unix timestamp.

rfc3339

optional

rfc3339

Represent the timestamp as a RFC 3339 timestamp.

connection_string: string
storage_account: string
endpoint: string
container_name: string
blob_prefix: string
blob_time_format: string
blob_append_uuid: boolean
compression: gzip
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
acknowledgements:
  enabled: null
type: azure_blob
connection_string: string
storage_account: string
endpoint: string
container_name: string
blob_prefix: string
blob_time_format: string
blob_append_uuid: boolean
compression: gzip
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
acknowledgements:
  enabled: null
framing: ''
encoding: ''
type: azure_blob

Azure Monitor Logs

Configuration for the azure_monitor_logs sink.

Field

required

Type

Description

customer_id

required

string

The unique identifier for the Log Analytics workspace.

shared_key

required

string

The primary or the secondary key for the Log Analytics workspace.

log_type

required

string

The record type of the data that is being submitted.

Can only contain letters, numbers, and underscores (_), and may not exceed 100 characters.

azure_resource_id

optional

string,​null

The Resource ID of the Azure resource the data should be associated with.

host

optional

string

Alternative host for dedicated Azure regions.

encoding

optional

object

Transformations to prepare an event for serialization.

only_fields

optional

array,​null

List of fields that will be included in the encoded event.

except_fields

optional

array,​null

List of fields that will be excluded from the encoded event.

timestamp_format

optional

 <oneOf>

Format used for timestamp fields.

unix

optional

unix

Represent the timestamp as a Unix timestamp.

rfc3339

optional

rfc3339

Represent the timestamp as a RFC 3339 timestamp.

batch

optional

object

Event batching behavior.

max_bytes

optional

integer,​null

The maximum size of a batch that will be processed by a sink.

This is based on the uncompressed size of the batched events, before they are serialized / compressed.

default: null

max_events

optional

integer,​null

The maximum size of a batch, in events, before it is flushed.

default: null

timeout_secs

optional

number,​null

The maximum age of a batch, in seconds, before it is flushed.

default: null

request

optional

object

Middleware settings for outbound requests.

Various settings can be configured, such as concurrency and rate limits, timeouts, etc.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

default: 60

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

default: 1

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

default: 9223372036854776000

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

default: 9223372036854776000

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

default: 3600

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

default: 1

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"rtt_deviation_scale":2.5}

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null
customer_id: string
shared_key: string
log_type: string
azure_resource_id: string
host: ods.opinsights.azure.com
encoding: {}
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
acknowledgements:
  enabled: null
type: azure_monitor_logs
customer_id: string
shared_key: string
log_type: string
azure_resource_id: string
host: ods.opinsights.azure.com
encoding: {}
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
acknowledgements:
  enabled: null
type: azure_monitor_logs

Blackhole

Configuration for the blackhole sink.

Field

required

Type

Description

print_interval_secs

optional

integer

The number of seconds between reporting a summary of activity.

Set to 0 to disable reporting.

rate

optional

integer,​null

The number of events, per second, that the sink is allowed to consume.

By default, there is no limit.

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null
print_interval_secs: 1
rate: integer
acknowledgements:
  enabled: null
type: blackhole
print_interval_secs: 1
rate: integer
acknowledgements:
  enabled: null
type: blackhole

Clickhouse

Configuration for the clickhouse sink.

Field

required

Type

Description

endpoint

required

string

The URI component of a request.

The endpoint of the Clickhouse server.

table

required

string

The table that data will be inserted into.

database

optional

string,​null

The database that contains the table that data will be inserted into.

skip_unknown_fields

optional

boolean

Sets input_format_skip_unknown_fields, allowing Clickhouse to discard fields not present in the table schema.

date_time_best_effort

optional

boolean

Sets date_time_input_format to best_effort, allowing ClickHouse to properly parse RFC3339/ISO 8601.

compression

optional

 <oneOf>

Compression configuration.

All compression algorithms use the default compression level unless otherwise specified.

Option 1

optional

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

Option 2

optional

object

algorithm

required

 <oneOf>

Compression algorithm.

none

optional

none

No compression.

gzip

optional

gzip

Gzip compression.

zlib

optional

zlib

Zlib]zlib compression.

level

optional

enum

Compression level. Allowed enum values: none,fast,best,default,0,1,2,3,4,5,6,7,8,9

encoding

optional

object

Transformations to prepare an event for serialization.

only_fields

optional

array,​null

List of fields that will be included in the encoded event.

except_fields

optional

array,​null

List of fields that will be excluded from the encoded event.

timestamp_format

optional

 <oneOf>

Format used for timestamp fields.

unix

optional

unix

Represent the timestamp as a Unix timestamp.

rfc3339

optional

rfc3339

Represent the timestamp as a RFC 3339 timestamp.

batch

optional

object

Event batching behavior.

max_bytes

optional

integer,​null

The maximum size of a batch that will be processed by a sink.

This is based on the uncompressed size of the batched events, before they are serialized / compressed.

default: null

max_events

optional

integer,​null

The maximum size of a batch, in events, before it is flushed.

default: null

timeout_secs

optional

number,​null

The maximum age of a batch, in seconds, before it is flushed.

default: null

auth

optional

 <oneOf>

Configuration of the authentication strategy for HTTP requests.

HTTP authentication should almost always be used with HTTPS only, as the authentication credentials are passed as an HTTP header without any additional encryption beyond what is provided by the transport itself.

Basic

optional

object

Basic authentication.

The username and password are concatenated and encoded via base64.

user

required

string

The username to send.

password

required

string

The password to send.

strategy

required

basic

Basic authentication.

The username and password are concatenated and encoded via base64.

Bearer

optional

object

Bearer authentication.

The bearer token value (OAuth2, JWT, etc) is passed as-is.

token

required

string

The bearer token to send.

strategy

required

bearer

Bearer authentication.

The bearer token value (OAuth2, JWT, etc) is passed as-is.

request

optional

object

Middleware settings for outbound requests.

Various settings can be configured, such as concurrency and rate limits, timeouts, etc.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

default: 60

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

default: 1

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

default: 9223372036854776000

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

default: 9223372036854776000

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

default: 3600

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

default: 1

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"rtt_deviation_scale":2.5}

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

tls

optional

object,​null

TLS configuration.

verify_certificate

optional

boolean,​null

Enables certificate verification.

If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and so on until reaching a root certificate.

Relevant for both incoming and outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

verify_hostname

optional

boolean,​null

Enables hostname verification.

If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.

Only relevant for outgoing connections.

Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

alpn_protocols

optional

array,​null

Sets the list of supported ALPN protocols.

Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order they are defined.

ca_file

optional

string,​null

Absolute path to an additional CA certificate file.

The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

crt_file

optional

string,​null

Absolute path to a certificate file used to identify this server.

The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format.

If this is set, and is not a PKCS#12 archive, key_file must also be set.

key_file

optional

string,​null

Absolute path to a private key file used to identify this server.

The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

key_pass

optional

string,​null

Passphrase used to unlock the encrypted key file.

This has no effect unless key_file is set.

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null
endpoint: string
table: string
database: string
skip_unknown_fields: boolean
date_time_best_effort: boolean
compression: gzip
encoding: {}
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
auth: ''
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
acknowledgements:
  enabled: null
type: clickhouse
endpoint: string
table: string
database: string
skip_unknown_fields: boolean
date_time_best_effort: boolean
compression: gzip
encoding: {}
batch:
  max_bytes: null
  max_events: null
  timeout_secs: null
auth: ''
request:
  timeout_secs: 60
  rate_limit_duration_secs: 1
  rate_limit_num: 9223372036854776000
  retry_attempts: 9223372036854776000
  retry_max_duration_secs: 3600
  retry_initial_backoff_secs: 1
  adaptive_concurrency:
    decrease_ratio: 0.9
    ewma_alpha: 0.4
    rtt_deviation_scale: 2.5
tls: object
acknowledgements:
  enabled: null
type: clickhouse

Console

Configuration for the console sink.

Field

required

Type

Description

target

optional

 <oneOf>

Output target.

stdout

optional

stdout

Standard output.

stderr

optional

stderr

Standard error.

acknowledgements

optional

object

Controls how acknowledgements are handled for this sink.

See End-to-end Acknowledgements for more information on how event acknowledgement is handled.

enabled

optional

boolean,​null

Whether or not end-to-end acknowledgements are enabled.

When enabled for a sink, any source connected to that sink, where the source supports end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink before acknowledging them at the source.

Enabling or disabling acknowledgements at the sink level takes precedence over any global acknowledgements configuration.

default: null

framing

optional

 <oneOf>

Framing configuration.

Bytes

optional

object

Event data is not delimited at all.

method

required

bytes

Event data is not delimited at all.

CharacterDelimited

optional

object

Event data is delimited by a single ASCII (7-bit) character.

character_delimited

required

object

Options for the character delimited encoder.

delimiter

required

integer

The ASCII (7-bit) character that delimits byte sequences.

method

required

character_delimited

Event data is delimited by a single ASCII (7-bit) character.

LengthDelimited

optional

object

Event data is prefixed with its length in bytes.

The prefix is a 32-bit unsigned integer, little endian.

method

required

length_delimited

Event data is prefixed with its length in bytes.

The prefix is a 32-bit unsigned integer, little endian.

NewlineDelimited

optional

object

Event data is delimited by a newline (LF) character.

method

required

newline_delimited

Event data is delimited by a newline (LF) character.

encoding

required

Configures how events are encoded into raw bytes.

Avro

optional

object

Encodes an event as an Apache Avro message.

avro

required

object

Apache Avro-specific encoder options.

schema

required

string

The Avro schema.

codec

required

avro

Encodes an event as an Apache Avro message.

Gelf

optional

object

Encodes an event as a GELF message.

codec

required

gelf

Encodes an event as a GELF message.

Json

optional

Encodes an event as JSON.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

json

Encodes an event as JSON.

Logfmt

optional

object

Encodes an event as a logfmt message.

codec

required

logfmt

Encodes an event as a logfmt message.

Native

optional

object

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

codec

required

native

Encodes an event in Vector’s native Protocol Buffers format.

This codec is experimental.

NativeJson

optional

object

Encodes an event in Vector’s native JSON format.

This codec is experimental.

codec

required

native_json

Encodes an event in Vector’s native JSON format.

This codec is experimental.

RawMessage

optional

object

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

codec

required

raw_message

No encoding.

This "encoding" simply uses the message field of a log event.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

Text

optional

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments.

single

optional

single

Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.

full

optional

full

All tags will be exposed as arrays of either string or null values.

codec

required

text

Plain text encoding.

This "encoding" simply uses the message field of a log event. For metrics, it uses an encoding that resembles the Prometheus export format.

Users should take care if they're modifying their log events (such as by using a remap transform, etc) and removing the message field while doing additional parsing on it, as this could lead to the encoding emitting empty strings for the given event.

only_fields

optional

array,​null

List of fields that will be included in the encoded event.

except_fields

optional

array,​null

List of fields that will be excluded from the encoded event.

timestamp_format

optional

 <oneOf>

Format used for timestamp fields.

unix

optional

unix

Represent the timestamp as a Unix timestamp.

rfc3339

optional

rfc3339

Represent the timestamp as a RFC 3339 timestamp.

target: stdout
acknowledgements:
  enabled: null
type: console
target: stdout
acknowledgements:
  enabled: null
framing: ''
encoding: ''
type: console

Datadog Archives

Configuration for the datadog_archives sink.

Field

required

Type

Description

service

required

string

The name of the object storage service to use.

bucket

required

string

The name of the bucket to store the archives in.

key_prefix

optional

string,​null

A prefix to apply to all object keys.

Prefixes are useful for partitioning objects, such as by creating an object key that stores objects under a particular "directory". If using a prefix for this purpose, it must end in / to act as a directory path. A trailing / is not automatically added.

request

optional

object

Middleware settings for outbound requests.

Various settings can be configured, such as concurrency and rate limits, timeouts, etc.

concurrency

optional

 <oneOf>

Configuration for outbound request concurrency.

none

optional

none

A fixed concurrency of 1.

Only one request can be outstanding at any given time.

adaptive

optional

adaptive

Concurrency will be managed by Vector's Adaptive Request Concurrency feature.

Fixed

optional

integer

A fixed amount of concurrency will be allowed.

timeout_secs

optional

integer,​null

The maximum time a request can take before being aborted.

It is highly recommended that you do not lower this value below the service’s internal timeout, as this could create orphaned requests, pile on retries, and result in duplicate data downstream.

default: 60

rate_limit_duration_secs

optional

integer,​null

The time window, in seconds, used for the rate_limit_num option.

default: 1

rate_limit_num

optional

integer,​null

The maximum number of requests allowed within the rate_limit_duration_secs time window.

default: 9223372036854776000

retry_attempts

optional

integer,​null

The maximum number of retries to make for failed requests.

The default, for all intents and purposes, represents an infinite number of retries.

default: 9223372036854776000

retry_max_duration_secs

optional

integer,​null

The maximum amount of time, in seconds, to wait between retries.

default: 3600

retry_initial_backoff_secs

optional

integer,​null

The amount of time to wait before attempting the first retry for a failed request.

After the first retry has failed, the fibonacci sequence will be used to select future backoffs.

default: 1

adaptive_concurrency

optional

object

Configuration of adaptive concurrency parameters.

These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or unstable performance and sink behavior. Proceed with caution.

default: {"decrease_ratio":0.9,"ewma_alpha":0.4,"rtt_deviation_scale":2.5}

decrease_ratio

optional

number

The fraction of the current value to set the new concurrency limit when decreasing the limit.

Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly when latency increases.

Note that the new limit is rounded down after applying this ratio.

default: 0.9

ewma_alpha

optional

number

The weighting of new measurements compared to older measurements.

Valid values are greater than 0 and less than 1.

ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has unusually high response variability.

default: 0.4

rtt_deviation_scale

optional

number

Scale of RTT deviations which are not considered anomalous.

Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.

When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable those values are. We use that deviation when comparing the past RTT average to the current measurements, so we can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.

default: 2.5

aws_s3

optional

 <oneOf>

S3-specific configuration options.

Option 1

optional

auth

optional

 <oneOf>

Configuration of the authentication strategy for interacting with AWS services.

AccessKey

optional

object

Authenticate using a fixed access key and secret pair.

access_key_id

required

string

The AWS access key ID.

secret_access_key

required

string

The AWS secret access key.

File

optional

object

Authenticate using credentials stored in a file.

Additionally, the specific credential profile to use can be set.

credentials_file

required

string

Path to the credentials file.

profile

optional

string,​null

The credentials profile to use.

Role

optional

object

Assume the given role ARN.

assume_role

required

string

The ARN of the role to assume.

load_timeout_secs

optional

integer,​null

Timeout for assuming the role, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

region

optional

string,​null

The AWS region to send STS requests to.

If not set, this will default to the configured region for the service itself.

Default

optional

object

Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.

load_timeout_secs

optional

integer,​null

Timeout for successfully loading any credentials, in seconds.

imds

optional

object

Configuration for authenticating with AWS through IMDS.

max_attempts

optional

integer

Number of IMDS retries for fetching tokens and metadata.

default: 4

connect_timeout_seconds

optional

integer

Connect timeout for IMDS.

default: 1

read_timeout_seconds

optional

integer

Read timeout for IMDS.

default: 1

acl

optional

 <oneOf>

Canned ACL to apply to the created objects.

For more information, see Canned ACL.

private

optional

private

Bucket/object are private.

The bucket/object owner is granted the FULL_CONTROL permission, and no one else has access.

This is the default.

public-read

optional

public-read

Bucket/object can be read publically.

The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the AllUsers grantee group is granted the READ permission.

public-read-write

optional

public-read-write

Bucket/object can be read and written publically.

The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the AllUsers grantee group is granted the READ and WRITE permissions.

This is generally not recommended.

aws-exec-read

optional

aws-exec-read

Bucket/object are private, and readable by EC2.

The bucket/object owner is granted the FULL_CONTROL permission, and the AWS EC2 service is granted the READ permission for the purpose of reading Amazon Machine Image (AMI) bundles from the given bucket.

authenticated-read

optional

authenticated-read

Bucket/object can be read by authenticated users.

The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the AuthenticatedUsers grantee group is granted the READ permission.

bucket-owner-read

optional

bucket-owner-read

Object is private, except to the bucket owner.

The object owner is granted the FULL_CONTROL permission, and the bucket owner is granted the READ permission.

Only relevant when specified for an object: this canned ACL is otherwise ignored when specified for a bucket.

bucket-owner-full-control

optional

bucket-owner-full-control

Object is semi-private.

Both the object owner and bucket owner are granted the FULL_CONTROL permission.

Only relevant when specified for an object: this canned ACL is otherwise ignored when specified for a bucket.

log-delivery-write

optional

log-delivery-write

Bucket can have logs written.

The LogDelivery grantee group is granted WRITE and READ_ACP permissions.

Only relevant when specified for a bucket: this canned ACL is otherwise ignored when specified for an object.

grant_full_control

optional

string,​null

Grants READ, READ_ACP, and WRITE_ACP permissions on the created objects to the named grantee.

This allows the grantee to read the created objects and their metadata, as well as read and modify the ACL on the created objects.

grant_read

optional

string,​null

Grants READ permissions on the created objects to the named grantee.

This allows the grantee to read the created objects and their metadata.

grant_read_acp

optional

string,​null

Grants READ_ACP permissions on the created objects to the named grantee.

This allows the grantee to read the ACL on the created objects.

grant_write_acp

optional

string,​null

Grants WRITE_ACP permissions on the created objects to the named grantee.

This allows the grantee to modify the ACL on the created objects.

server_side_encryption

optional

 <oneOf>

AWS S3 Server-Side Encryption algorithms.

The Server-side Encryption algorithm used when storing these objects.

AES256

optional

AES256

Each object is encrypted with AES-256 using a unique key.

This corresponds to the SSE-S3 option.

aws:kms

optional

aws:kms

Each object is encrypted with AES-256 using keys managed by AWS KMS.

Depending on whether or not a KMS key ID is specified, this will correspond either to the SSE-KMS option (keys generated/managed by KMS) or the SSE-C option (keys generated by the customer, managed by KMS).

ssekms_key_id

optional

string,​null

Specifies the ID of the AWS Key Management Service (AWS KMS) symmetrical customer managed customer master key (CMK) that will used for the created objects.

Only applies when server_side_encryption is configured to use KMS.

If not specified, Amazon S3 uses the AWS managed CMK in AWS to protect the data.

storage_class

optional

 <oneOf>

The storage class for the created objects.

For more information, see Using Amazon S3 storage classes.

STANDARD

optional

STANDARD

Standard Redundancy.

This is the default.

REDUCED_REDUNDANCY

optional

REDUCED_REDUNDANCY

Reduced Redundancy.

INTELLIGENT_TIERING

optional

INTELLIGENT_TIERING

Intelligent Tiering.

STANDARD_IA

optional

STANDARD_IA

Infrequently Accessed.

ONEZONE_IA

optional

ONEZONE_IA

Infrequently Accessed (single Availability zone).

GLACIER

optional

GLACIER

Glacier Flexible Retrieval.

DEEP_ARCHIVE

optional

DEEP_ARCHIVE

Glacier Deep Archive.

tags

optional

object,​null

The tag-set for the object.

region

optional

string,​null

The AWS region to use.

default: null

endpoint

optional

string,​null

The API endpoint of the service.

default: null

azure_blob

optional

object,​null

ABS-specific configuration options.

connection_string

required

string

The Azure Blob Storage Account connection string.

Authentication with access key is the only supported authentication method.

gcp_cloud_storage

optional

 <oneOf>

GCS-specific configuration options.

Option 1

optional

acl

optional

 <oneOf>

GCS Predefined ACLs.

For more information, see Predefined ACLs.

authenticated-read

optional

authenticated-read

Bucket/object can be read by authenticated users.

The bucket/object owner is granted the OWNER permission, and anyone authenticated Google account holder is granted the READER permission.

bucket-owner-full-control

optional

bucket-owner-full-control

Object is semi-private.

Both the object owner and bucket owner are granted the OWNER permission.

Only relevant when specified for an object: this predefined ACL is otherwise ignored when specified for a bucket.

bucket-owner-read

optional

bucket-owner-read

Object is private, except to the bucket owner.

The object owner is granted the OWNER permission, and the bucket owner is granted the READER permission.

Only relevant when specified for an object: this predefined ACL is otherwise ignored when specified for a bucket.

private

optional

private

Bucket/object are private.

The bucket/object owner is granted the OWNER permission, and no one else has access.

project-private

optional

project-private

Bucket/object are private within the project.

Project owners and project editors are granted the OWNER permission, and anyone who is part of the project team is granted the READER permission.

This is the default.

public-read

optional

public-read

Bucket/object can be read publically.

The bucket/object owner is granted the OWNER permission, and all other users, whether authenticated or anonymous, are granted the READER permission.