A sink is a destination for events. Each sink’s design and transmission method is determined by the downstream service with which it interacts. For example, the socket sink streams individual events, while the aws_s3 sink buffers and flushes data.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
only_fields
optional
array,null
List of fields that will be included in the encoded event.
except_fields
optional
array,null
List of fields that will be excluded from the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
connection_string
required
string
URI for the AMQP server.
The URI has the format of
amqp://<user>:<password>@<host>:<port>/<vhost>?timeout=<seconds>.
The default vhost can be specified by using a value of %2f.
In order to connect over TLS, a scheme of amqps can be specified instead i.e.
amqps://.... Additional TLS settings, such as client certificate verification, etc, can be
configured under the tls section.
tls
optional
object,null
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the
certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and
so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order
they are defined.
ca_file
optional
string,null
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
crt_file
optional
string,null
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
key_file
optional
string,null
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
default: 3600
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
default: 1
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
The group name of the target CloudWatch Logs stream.
stream_name
required
string
The stream name of the target CloudWatch Logs stream.
There can only be one writer to a log stream at a time. If you have multiple
instances writing to the same log group, you must include an identifier in the
stream name that is guaranteed to be unique per instance.
For example, you might choose host.
create_missing_group
optional
boolean,null
Dynamically create a log group if it does not already exist.
This will ignore create_missing_stream directly after creating the group and will create
the first stream.
create_missing_stream
optional
boolean,null
Dynamically create a log stream if it does not already exist.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
only_fields
optional
array,null
List of fields that will be included in the encoded event.
except_fields
optional
array,null
List of fields that will be excluded from the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
tls
optional
object,null
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the
certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and
so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order
they are defined.
ca_file
optional
string,null
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
crt_file
optional
string,null
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
key_file
optional
string,null
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
assume_role
optional
string,null
The ARN of an [IAM role][iam_role] to assume at startup.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
default: 3600
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
default: 1
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
tls
optional
object,null
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the
certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and
so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order
they are defined.
ca_file
optional
string,null
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
crt_file
optional
string,null
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
key_file
optional
string,null
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
assume_role
optional
string,null
The ARN of an [IAM role][iam_role] to assume at startup.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
only_fields
optional
array,null
List of fields that will be included in the encoded event.
except_fields
optional
array,null
List of fields that will be excluded from the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
default: 3600
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
default: 1
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
tls
optional
object,null
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the
certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and
so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order
they are defined.
ca_file
optional
string,null
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
crt_file
optional
string,null
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
key_file
optional
string,null
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
auth
optional
<oneOf>
Configuration of the authentication strategy for interacting with AWS services.
AccessKey
optional
object
Authenticate using a fixed access key and secret pair.
access_key_id
required
string
The AWS access key ID.
secret_access_key
required
string
The AWS secret access key.
File
optional
object
Authenticate using credentials stored in a file.
Additionally, the specific credential profile to use can be set.
credentials_file
required
string
Path to the credentials file.
profile
optional
string,null
The credentials profile to use.
Role
optional
object
Assume the given role ARN.
assume_role
required
string
The ARN of the role to assume.
load_timeout_secs
optional
integer,null
Timeout for assuming the role, in seconds.
imds
optional
object
Configuration for authenticating with AWS through IMDS.
max_attempts
optional
integer
Number of IMDS retries for fetching tokens and metadata.
default: 4
connect_timeout_seconds
optional
integer
Connect timeout for IMDS.
default: 1
read_timeout_seconds
optional
integer
Read timeout for IMDS.
default: 1
region
optional
string,null
The AWS region to send STS requests to.
If not set, this will default to the configured region
for the service itself.
Default
optional
object
Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.
load_timeout_secs
optional
integer,null
Timeout for successfully loading any credentials, in seconds.
imds
optional
object
Configuration for authenticating with AWS through IMDS.
max_attempts
optional
integer
Number of IMDS retries for fetching tokens and metadata.
default: 4
connect_timeout_seconds
optional
integer
Connect timeout for IMDS.
default: 1
read_timeout_seconds
optional
integer
Read timeout for IMDS.
default: 1
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
only_fields
optional
array,null
List of fields that will be included in the encoded event.
except_fields
optional
array,null
List of fields that will be excluded from the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
default: 3600
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
default: 1
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
tls
optional
object,null
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the
certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and
so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order
they are defined.
ca_file
optional
string,null
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
crt_file
optional
string,null
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
key_file
optional
string,null
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
auth
optional
<oneOf>
Configuration of the authentication strategy for interacting with AWS services.
AccessKey
optional
object
Authenticate using a fixed access key and secret pair.
access_key_id
required
string
The AWS access key ID.
secret_access_key
required
string
The AWS secret access key.
File
optional
object
Authenticate using credentials stored in a file.
Additionally, the specific credential profile to use can be set.
credentials_file
required
string
Path to the credentials file.
profile
optional
string,null
The credentials profile to use.
Role
optional
object
Assume the given role ARN.
assume_role
required
string
The ARN of the role to assume.
load_timeout_secs
optional
integer,null
Timeout for assuming the role, in seconds.
imds
optional
object
Configuration for authenticating with AWS through IMDS.
max_attempts
optional
integer
Number of IMDS retries for fetching tokens and metadata.
default: 4
connect_timeout_seconds
optional
integer
Connect timeout for IMDS.
default: 1
read_timeout_seconds
optional
integer
Read timeout for IMDS.
default: 1
region
optional
string,null
The AWS region to send STS requests to.
If not set, this will default to the configured region
for the service itself.
Default
optional
object
Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.
load_timeout_secs
optional
integer,null
Timeout for successfully loading any credentials, in seconds.
imds
optional
object
Configuration for authenticating with AWS through IMDS.
max_attempts
optional
integer
Number of IMDS retries for fetching tokens and metadata.
default: 4
connect_timeout_seconds
optional
integer
Connect timeout for IMDS.
default: 1
read_timeout_seconds
optional
integer
Read timeout for IMDS.
default: 1
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
This must not include a leading s3:// or a trailing /.
key_prefix
optional
string,null
A prefix to apply to all object keys.
Prefixes are useful for partitioning objects, such as by creating an object key that
stores objects under a particular "directory". If using a prefix for this purpose, it must end
in / to act as a directory path. A trailing / is not automatically added.
filename_time_format
optional
string,null
The timestamp format for the time component of the object key.
By default, object keys are appended with a timestamp that reflects when the objects are
sent to S3, such that the resulting object key is functionally equivalent to joining the key
prefix with the formatted timestamp, such as date=2022-07-18/1658176486.
This would represent a key_prefix set to date=%F/ and the timestamp of Mon Jul 18 2022
20:34:44 GMT+0000, with the filename_time_format being set to %s, which renders
timestamps in seconds since the Unix epoch.
Supports the common strftime specifiers found in most
languages.
When set to an empty string, no timestamp will be appended to the key prefix.
filename_append_uuid
optional
boolean,null
Whether or not to append a UUID v4 token to the end of the object key.
The UUID is appended to the timestamp portion of the object key, such that if the object key
being generated was date=2022-07-18/1658176486, setting this field to true would result
in an object key that looked like date=2022-07-18/1658176486-30f6652c-71da-4f9f-800d-a1189c47c547.
This ensures there are no name collisions, and can be useful in high-volume workloads where
object keys must be unique.
filename_extension
optional
string,null
The filename extension to use in the object key.
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
default: 3600
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
default: 1
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
tls
optional
object,null
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the
certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and
so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order
they are defined.
ca_file
optional
string,null
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
crt_file
optional
string,null
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
key_file
optional
string,null
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
auth
optional
<oneOf>
Configuration of the authentication strategy for interacting with AWS services.
AccessKey
optional
object
Authenticate using a fixed access key and secret pair.
access_key_id
required
string
The AWS access key ID.
secret_access_key
required
string
The AWS secret access key.
File
optional
object
Authenticate using credentials stored in a file.
Additionally, the specific credential profile to use can be set.
credentials_file
required
string
Path to the credentials file.
profile
optional
string,null
The credentials profile to use.
Role
optional
object
Assume the given role ARN.
assume_role
required
string
The ARN of the role to assume.
load_timeout_secs
optional
integer,null
Timeout for assuming the role, in seconds.
imds
optional
object
Configuration for authenticating with AWS through IMDS.
max_attempts
optional
integer
Number of IMDS retries for fetching tokens and metadata.
default: 4
connect_timeout_seconds
optional
integer
Connect timeout for IMDS.
default: 1
read_timeout_seconds
optional
integer
Read timeout for IMDS.
default: 1
region
optional
string,null
The AWS region to send STS requests to.
If not set, this will default to the configured region
for the service itself.
Default
optional
object
Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.
load_timeout_secs
optional
integer,null
Timeout for successfully loading any credentials, in seconds.
imds
optional
object
Configuration for authenticating with AWS through IMDS.
max_attempts
optional
integer
Number of IMDS retries for fetching tokens and metadata.
default: 4
connect_timeout_seconds
optional
integer
Connect timeout for IMDS.
default: 1
read_timeout_seconds
optional
integer
Read timeout for IMDS.
default: 1
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
The bucket/object owner is granted the FULL_CONTROL permission, and no one else has
access.
This is the default.
public-read
optional
public-read
Bucket/object can be read publically.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AllUsers grantee group is granted the READ permission.
public-read-write
optional
public-read-write
Bucket/object can be read and written publically.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AllUsers grantee group is granted the READ and WRITE permissions.
This is generally not recommended.
aws-exec-read
optional
aws-exec-read
Bucket/object are private, and readable by EC2.
The bucket/object owner is granted the FULL_CONTROL permission, and the AWS EC2 service is
granted the READ permission for the purpose of reading Amazon Machine Image (AMI) bundles
from the given bucket.
authenticated-read
optional
authenticated-read
Bucket/object can be read by authenticated users.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AuthenticatedUsers grantee group is granted the READ permission.
bucket-owner-read
optional
bucket-owner-read
Object is private, except to the bucket owner.
The object owner is granted the FULL_CONTROL permission, and the bucket owner is granted the READ permission.
Only relevant when specified for an object: this canned ACL is otherwise ignored when
specified for a bucket.
bucket-owner-full-control
optional
bucket-owner-full-control
Object is semi-private.
Both the object owner and bucket owner are granted the FULL_CONTROL permission.
Only relevant when specified for an object: this canned ACL is otherwise ignored when
specified for a bucket.
log-delivery-write
optional
log-delivery-write
Bucket can have logs written.
The LogDelivery grantee group is granted WRITE and READ_ACP permissions.
Only relevant when specified for a bucket: this canned ACL is otherwise ignored when
specified for an object.
grant_full_control
optional
string,null
Grants READ, READ_ACP, and WRITE_ACP permissions on the created objects to the named grantee.
This allows the grantee to read the created objects and their metadata, as well as read and
modify the ACL on the created objects.
grant_read
optional
string,null
Grants READ permissions on the created objects to the named grantee.
This allows the grantee to read the created objects and their metadata.
grant_read_acp
optional
string,null
Grants READ_ACP permissions on the created objects to the named grantee.
This allows the grantee to read the ACL on the created objects.
grant_write_acp
optional
string,null
Grants WRITE_ACP permissions on the created objects to the named grantee.
This allows the grantee to modify the ACL on the created objects.
server_side_encryption
optional
<oneOf>
AWS S3 Server-Side Encryption algorithms.
The Server-side Encryption algorithm used when storing these objects.
AES256
optional
AES256
Each object is encrypted with AES-256 using a unique key.
This corresponds to the SSE-S3 option.
aws:kms
optional
aws:kms
Each object is encrypted with AES-256 using keys managed by AWS KMS.
Depending on whether or not a KMS key ID is specified, this will correspond either to the
SSE-KMS option (keys generated/managed by KMS) or the SSE-C option (keys generated by
the customer, managed by KMS).
ssekms_key_id
optional
string,null
Specifies the ID of the AWS Key Management Service (AWS KMS) symmetrical customer managed
customer master key (CMK) that will used for the created objects.
Only applies when server_side_encryption is configured to use KMS.
If not specified, Amazon S3 uses the AWS managed CMK in AWS to protect the data.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
only_fields
optional
array,null
List of fields that will be included in the encoded event.
except_fields
optional
array,null
List of fields that will be excluded from the encoded event.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
only_fields
optional
array,null
List of fields that will be included in the encoded event.
except_fields
optional
array,null
List of fields that will be excluded from the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
message_group_id
optional
string,null
The tag that specifies that a message belongs to a specific message group.
Can be applied only to FIFO queues.
message_deduplication_id
optional
string,null
The message deduplication ID value to allow AWS to identify duplicate messages.
This value is a template which should result in a unique string for each event. See the AWS
documentation for more about how AWS does message deduplication.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
default: 3600
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
default: 1
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
tls
optional
object,null
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the
certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and
so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order
they are defined.
ca_file
optional
string,null
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
crt_file
optional
string,null
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
key_file
optional
string,null
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
assume_role
optional
string,null
The ARN of an [IAM role][iam_role] to assume at startup.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
decrease_ratio
optional
number
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the
certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and
so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order
they are defined.
ca_file
optional
string,null
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
crt_file
optional
string,null
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
key_file
optional
string,null
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Either connection_string, or this field, must be specified.
endpoint
optional
string,null
The Azure Blob Storage Endpoint URL.
This is used to override the default blob storage endpoint URL in cases where you are using
credentials read from the environment/managed identities or access tokens without using an
explicit connection_string (which already explicitly supports overriding the blob endpoint
URL).
This may only be used with storage_account and will be ignored when used with
connection_string.
container_name
required
string
The Azure Blob Storage Account container name.
blob_prefix
optional
string,null
A prefix to apply to all blob keys.
Prefixes are useful for partitioning objects, such as by creating an blob key that
stores blobs under a particular "directory". If using a prefix for this purpose, it must end
in / to act as a directory path. A trailing / is not automatically added.
blob_time_format
optional
string,null
The timestamp format for the time component of the blob key.
By default, blob keys are appended with a timestamp that reflects when the blob are sent to
Azure Blob Storage, such that the resulting blob key is functionally equivalent to joining
the blob prefix with the formatted timestamp, such as date=2022-07-18/1658176486.
This would represent a blob_prefix set to date=%F/ and the timestamp of Mon Jul 18 2022
20:34:44 GMT+0000, with the filename_time_format being set to %s, which renders
timestamps in seconds since the Unix epoch.
Supports the common strftime specifiers found in most
languages.
When set to an empty string, no timestamp will be appended to the blob prefix.
blob_append_uuid
optional
boolean,null
Whether or not to append a UUID v4 token to the end of the blob key.
The UUID is appended to the timestamp portion of the object key, such that if the blob key
being generated was date=2022-07-18/1658176486, setting this field to true would result
in an blob key that looked like
date=2022-07-18/1658176486-30f6652c-71da-4f9f-800d-a1189c47c547.
This ensures there are no name collisions, and can be useful in high-volume workloads where
blob keys must be unique.
compression
optional
<oneOf>
Compression configuration.
All compression algorithms use the default compression level unless otherwise specified.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
default: 3600
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
default: 1
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
framing
optional
<oneOf>
Framing configuration.
Bytes
optional
object
Event data is not delimited at all.
method
required
bytes
Event data is not delimited at all.
CharacterDelimited
optional
object
Event data is delimited by a single ASCII (7-bit) character.
character_delimited
required
object
Options for the character delimited encoder.
delimiter
required
integer
The ASCII (7-bit) character that delimits byte sequences.
method
required
character_delimited
Event data is delimited by a single ASCII (7-bit) character.
LengthDelimited
optional
object
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
method
required
length_delimited
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
NewlineDelimited
optional
object
Event data is delimited by a newline (LF) character.
method
required
newline_delimited
Event data is delimited by a newline (LF) character.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
only_fields
optional
array,null
List of fields that will be included in the encoded event.
except_fields
optional
array,null
List of fields that will be excluded from the encoded event.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
default: 3600
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
default: 1
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
tls
optional
object,null
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the
certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and
so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order
they are defined.
ca_file
optional
string,null
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
crt_file
optional
string,null
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
key_file
optional
string,null
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Transformations to prepare an event for serialization.
only_fields
optional
array,null
List of fields that will be included in the encoded event.
except_fields
optional
array,null
List of fields that will be excluded from the encoded event.
timestamp_format
optional
<oneOf>
Format used for timestamp fields.
unix
optional
unix
Represent the timestamp as a Unix timestamp.
rfc3339
optional
rfc3339
Represent the timestamp as a RFC 3339 timestamp.
batch
optional
object
Event batching behavior.
max_bytes
optional
integer,null
The maximum size of a batch that will be processed by a sink.
This is based on the uncompressed size of the batched events, before they are
serialized / compressed.
default: null
max_events
optional
integer,null
The maximum size of a batch, in events, before it is flushed.
default: null
timeout_secs
optional
number,null
The maximum age of a batch, in seconds, before it is flushed.
default: null
auth
optional
<oneOf>
Configuration of the authentication strategy for HTTP requests.
HTTP authentication should almost always be used with HTTPS only, as the authentication credentials are passed as an
HTTP header without any additional encryption beyond what is provided by the transport itself.
Basic
optional
object
Basic authentication.
The username and password are concatenated and encoded via base64.
user
required
string
The username to send.
password
required
string
The password to send.
strategy
required
basic
Basic authentication.
The username and password are concatenated and encoded via base64.
Bearer
optional
object
Bearer authentication.
The bearer token value (OAuth2, JWT, etc) is passed as-is.
token
required
string
The bearer token to send.
strategy
required
bearer
Bearer authentication.
The bearer token value (OAuth2, JWT, etc) is passed as-is.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
default: 3600
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
default: 1
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
tls
optional
object,null
TLS configuration.
verify_certificate
optional
boolean,null
Enables certificate verification.
If enabled, certificates must be valid in terms of not being expired, as well as being issued by a trusted
issuer. This verification operates in a hierarchical manner, checking that not only the leaf certificate (the
certificate presented by the client/server) is valid, but also that the issuer of that certificate is valid, and
so on until reaching a root certificate.
Relevant for both incoming and outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
verify_hostname
optional
boolean,null
Enables hostname verification.
If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by
the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension.
Only relevant for outgoing connections.
Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
alpn_protocols
optional
array,null
Sets the list of supported ALPN protocols.
Declare the supported ALPN protocols, which are used during negotiation with peer. Prioritized in the order
they are defined.
ca_file
optional
string,null
Absolute path to an additional CA certificate file.
The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.
crt_file
optional
string,null
Absolute path to a certificate file used to identify this server.
The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as
an inline string in PEM format.
If this is set, and is not a PKCS#12 archive, key_file must also be set.
key_file
optional
string,null
Absolute path to a private key file used to identify this server.
The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
key_pass
optional
string,null
Passphrase used to unlock the encrypted key file.
This has no effect unless key_file is set.
acknowledgements
optional
object
Controls how acknowledgements are handled for this sink.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
Whether or not end-to-end acknowledgements are enabled.
When enabled for a sink, any source connected to that sink, where the source supports
end-to-end acknowledgements as well, will wait for events to be acknowledged by the sink
before acknowledging them at the source.
Enabling or disabling acknowledgements at the sink level takes precedence over any global
acknowledgements configuration.
default: null
framing
optional
<oneOf>
Framing configuration.
Bytes
optional
object
Event data is not delimited at all.
method
required
bytes
Event data is not delimited at all.
CharacterDelimited
optional
object
Event data is delimited by a single ASCII (7-bit) character.
character_delimited
required
object
Options for the character delimited encoder.
delimiter
required
integer
The ASCII (7-bit) character that delimits byte sequences.
method
required
character_delimited
Event data is delimited by a single ASCII (7-bit) character.
LengthDelimited
optional
object
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
method
required
length_delimited
Event data is prefixed with its length in bytes.
The prefix is a 32-bit unsigned integer, little endian.
NewlineDelimited
optional
object
Event data is delimited by a newline (LF) character.
method
required
newline_delimited
Event data is delimited by a newline (LF) character.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
codec
required
raw_message
No encoding.
This "encoding" simply uses the message field of a log event.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
Text
optional
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config
option. Tags with multiple values will show the last assigned value, and null values will be
ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
codec
required
text
Plain text encoding.
This "encoding" simply uses the message field of a log event. For metrics, it uses an
encoding that resembles the Prometheus export format.
Users should take care if they're modifying their log events (such as by using a remap
transform, etc) and removing the message field while doing additional parsing on it, as this
could lead to the encoding emitting empty strings for the given event.
only_fields
optional
array,null
List of fields that will be included in the encoded event.
except_fields
optional
array,null
List of fields that will be excluded from the encoded event.
Prefixes are useful for partitioning objects, such as by creating an object key that
stores objects under a particular "directory". If using a prefix for this purpose, it must end
in / to act as a directory path. A trailing / is not automatically added.
request
optional
object
Middleware settings for outbound requests.
Various settings can be configured, such as concurrency and rate limits, timeouts, etc.
concurrency
optional
<oneOf>
Configuration for outbound request concurrency.
none
optional
none
A fixed concurrency of 1.
Only one request can be outstanding at any given time.
The maximum time a request can take before being aborted.
It is highly recommended that you do not lower this value below the service’s internal timeout, as this could
create orphaned requests, pile on retries, and result in duplicate data downstream.
default: 60
rate_limit_duration_secs
optional
integer,null
The time window, in seconds, used for the rate_limit_num option.
default: 1
rate_limit_num
optional
integer,null
The maximum number of requests allowed within the rate_limit_duration_secs time window.
default: 9223372036854776000
retry_attempts
optional
integer,null
The maximum number of retries to make for failed requests.
The default, for all intents and purposes, represents an infinite number of retries.
default: 9223372036854776000
retry_max_duration_secs
optional
integer,null
The maximum amount of time, in seconds, to wait between retries.
default: 3600
retry_initial_backoff_secs
optional
integer,null
The amount of time to wait before attempting the first retry for a failed request.
After the first retry has failed, the fibonacci sequence will be used to select future backoffs.
default: 1
adaptive_concurrency
optional
object
Configuration of adaptive concurrency parameters.
These parameters typically do not require changes from the default, and incorrect values can lead to meta-stable or
unstable performance and sink behavior. Proceed with caution.
The fraction of the current value to set the new concurrency limit when decreasing the limit.
Valid values are greater than 0 and less than 1. Smaller values cause the algorithm to scale back rapidly
when latency increases.
Note that the new limit is rounded down after applying this ratio.
default: 0.9
ewma_alpha
optional
number
The weighting of new measurements compared to older measurements.
Valid values are greater than 0 and less than 1.
ARC uses an exponentially weighted moving average (EWMA) of past RTT measurements as a reference to compare with
the current RTT. Smaller values cause this reference to adjust more slowly, which may be useful if a service has
unusually high response variability.
default: 0.4
rtt_deviation_scale
optional
number
Scale of RTT deviations which are not considered anomalous.
Valid values are greater than or equal to 0, and we expect reasonable values to range from 1.0 to 3.0.
When calculating the past RTT average, we also compute a secondary “deviation” value that indicates how variable
those values are. We use that deviation when comparing the past RTT average to the current measurements, so we
can ignore increases in RTT that are within an expected range. This factor is used to scale up the deviation to
an appropriate range. Larger values cause the algorithm to ignore larger increases in the RTT.
default: 2.5
aws_s3
optional
<oneOf>
S3-specific configuration options.
Option 1
optional
auth
optional
<oneOf>
Configuration of the authentication strategy for interacting with AWS services.
AccessKey
optional
object
Authenticate using a fixed access key and secret pair.
access_key_id
required
string
The AWS access key ID.
secret_access_key
required
string
The AWS secret access key.
File
optional
object
Authenticate using credentials stored in a file.
Additionally, the specific credential profile to use can be set.
credentials_file
required
string
Path to the credentials file.
profile
optional
string,null
The credentials profile to use.
Role
optional
object
Assume the given role ARN.
assume_role
required
string
The ARN of the role to assume.
load_timeout_secs
optional
integer,null
Timeout for assuming the role, in seconds.
imds
optional
object
Configuration for authenticating with AWS through IMDS.
max_attempts
optional
integer
Number of IMDS retries for fetching tokens and metadata.
default: 4
connect_timeout_seconds
optional
integer
Connect timeout for IMDS.
default: 1
read_timeout_seconds
optional
integer
Read timeout for IMDS.
default: 1
region
optional
string,null
The AWS region to send STS requests to.
If not set, this will default to the configured region
for the service itself.
Default
optional
object
Default authentication strategy which tries a variety of substrategies in a one-after-the-other fashion.
load_timeout_secs
optional
integer,null
Timeout for successfully loading any credentials, in seconds.
imds
optional
object
Configuration for authenticating with AWS through IMDS.
max_attempts
optional
integer
Number of IMDS retries for fetching tokens and metadata.
The bucket/object owner is granted the FULL_CONTROL permission, and no one else has
access.
This is the default.
public-read
optional
public-read
Bucket/object can be read publically.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AllUsers grantee group is granted the READ permission.
public-read-write
optional
public-read-write
Bucket/object can be read and written publically.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AllUsers grantee group is granted the READ and WRITE permissions.
This is generally not recommended.
aws-exec-read
optional
aws-exec-read
Bucket/object are private, and readable by EC2.
The bucket/object owner is granted the FULL_CONTROL permission, and the AWS EC2 service is
granted the READ permission for the purpose of reading Amazon Machine Image (AMI) bundles
from the given bucket.
authenticated-read
optional
authenticated-read
Bucket/object can be read by authenticated users.
The bucket/object owner is granted the FULL_CONTROL permission, and anyone in the
AuthenticatedUsers grantee group is granted the READ permission.
bucket-owner-read
optional
bucket-owner-read
Object is private, except to the bucket owner.
The object owner is granted the FULL_CONTROL permission, and the bucket owner is granted the READ permission.
Only relevant when specified for an object: this canned ACL is otherwise ignored when
specified for a bucket.
bucket-owner-full-control
optional
bucket-owner-full-control
Object is semi-private.
Both the object owner and bucket owner are granted the FULL_CONTROL permission.
Only relevant when specified for an object: this canned ACL is otherwise ignored when
specified for a bucket.
log-delivery-write
optional
log-delivery-write
Bucket can have logs written.
The LogDelivery grantee group is granted WRITE and READ_ACP permissions.
Only relevant when specified for a bucket: this canned ACL is otherwise ignored when
specified for an object.
grant_full_control
optional
string,null
Grants READ, READ_ACP, and WRITE_ACP permissions on the created objects to the named grantee.
This allows the grantee to read the created objects and their metadata, as well as read and
modify the ACL on the created objects.
grant_read
optional
string,null
Grants READ permissions on the created objects to the named grantee.
This allows the grantee to read the created objects and their metadata.
grant_read_acp
optional
string,null
Grants READ_ACP permissions on the created objects to the named grantee.
This allows the grantee to read the ACL on the created objects.
grant_write_acp
optional
string,null
Grants WRITE_ACP permissions on the created objects to the named grantee.
This allows the grantee to modify the ACL on the created objects.
server_side_encryption
optional
<oneOf>
AWS S3 Server-Side Encryption algorithms.
The Server-side Encryption algorithm used when storing these objects.
AES256
optional
AES256
Each object is encrypted with AES-256 using a unique key.
This corresponds to the SSE-S3 option.
aws:kms
optional
aws:kms
Each object is encrypted with AES-256 using keys managed by AWS KMS.
Depending on whether or not a KMS key ID is specified, this will correspond either to the
SSE-KMS option (keys generated/managed by KMS) or the SSE-C option (keys generated by
the customer, managed by KMS).
ssekms_key_id
optional
string,null
Specifies the ID of the AWS Key Management Service (AWS KMS) symmetrical customer managed
customer master key (CMK) that will used for the created objects.
Only applies when server_side_encryption is configured to use KMS.
If not specified, Amazon S3 uses the AWS managed CMK in AWS to protect the data.
