A transform allows you to parse, structure, and enrich your data. You can use multiple transforms in your pipeline.
Configuration for the aggregate transform.
Field
required
Type
Description
interval_ms
optional
integer
The interval between flushes, in milliseconds.
Over this period metrics with the same series data (name, namespace, tags, …) will be aggregated.
interval_ms: 10000
type: aggregate
interval_ms: 10000
type: aggregate
Configuration for the aws_ec2_metadata transform.
Field
required
Type
Description
endpoint
optional
string
Overrides the default EC2 metadata endpoint.
namespace
optional
string,null
Sets a prefix for all event fields added by the transform.
refresh_interval_secs
optional
integer
The interval between querying for updated metadata, in seconds.
fields
optional
[string]
A list of metadata fields to include in each transformed event.
tags
optional
[string]
A list of instance tags to include in each transformed event.
refresh_timeout_secs
optional
integer
The timeout for querying the EC2 metadata endpoint, in seconds.
proxy
optional
object
Proxy configuration.
Configure to proxy traffic through an HTTP(S) proxy when making external requests. Similar to common proxy configuration convention, users can set different proxies to use based on the type of traffic being proxied, as well as set specific hosts that should not be proxied.
enabled
optional
boolean
Enables proxying support.
default: truehttp
optional
uri
Proxy endpoint to use when proxying HTTP traffic.
Must be a valid URI string.
default: nullhttps
optional
uri
Proxy endpoint to use when proxying HTTPS traffic.
Must be a valid URI string.
default: nullno_proxy
optional
[string]
A list of hosts to avoid proxying.
Multiple patterns are allowed:
| Pattern | Example match |
|---|---|
| Domain names | example.com matches requests to example.com |
| Wildcard domains | .example.com matches requests to example.com and its subdomains |
| IP addresses | 127.0.0.1 matches requests to 127.0.0.1 |
| CIDR blocks | 192.168.0.0/16 matches requests to any IP addresses in this range |
| Splat | * matches all hosts |
required
optional
boolean
Requires the transform to be able to successfully query the EC2 metadata before starting to process the data.
endpoint: 'http://169.254.169.254'
namespace: string
refresh_interval_secs: 10
fields:
- ami-id
- availability-zone
- instance-id
- instance-type
- local-hostname
- local-ipv4
- public-hostname
- public-ipv4
- region
- subnet-id
- vpc-id
- role-name
tags: []
refresh_timeout_secs: 1
proxy:
enabled: true
http: null
https: null
required: true
type: aws_ec2_metadata
endpoint: 'http://169.254.169.254'
namespace: string
refresh_interval_secs: 10
fields:
- ami-id
- availability-zone
- instance-id
- instance-type
- local-hostname
- local-ipv4
- public-hostname
- public-ipv4
- region
- subnet-id
- vpc-id
- role-name
tags: []
refresh_timeout_secs: 1
proxy:
enabled: true
http: null
https: null
required: true
type: aws_ec2_metadata
Configuration for the dedupe transform.
Field
required
Type
Description
fields
optional
<oneOf>
Options to control what fields to match against.
When no field matching configuration is specified, events are matched using the timestamp,
host, and message fields from an event. The specific field names used will be those set in
the global log schema configuration.
MatchFields
optional
object
Matches events using only the specified fields.
match
required
[string]
IgnoreFields
optional
object
Matches events using all fields except for the ignored ones.
ignore
required
[string]
cache
optional
object
Caching configuration for deduplication.
num_events
required
integer
Number of events to cache and use for comparing incoming events to previously seen events.
default: 5000fields: ''
cache:
num_events: 5000
type: dedupe
fields: ''
cache:
num_events: 5000
type: dedupe
Configuration for the filter transform.
Field
required
Type
Description
condition
required
<oneOf>
The condition that every input event is matched against.
If an event is matched by the condition, it is forwarded. Otherwise, the event is dropped.
Map
optional
<oneOf>
An event matching condition.
A fully-specified condition.
IsLog
optional
object
Matches an event if it is a log.
type
required
is_log
Matches an event if it is a log.
IsMetric
optional
object
Matches an event if it is a metric.
type
required
is_metric
Matches an event if it is a metric.
IsTrace
optional
object
Matches an event if it is a trace.
type
required
is_trace
Matches an event if it is a trace.
source
required
string
The VRL boolean expression.
runtime
optional
<oneOf>
Available VRL runtimes.
ast
optional
ast
Tree-walking runtime.
This is the only, and default, runtime.
source
required
string
The query string.
condition: ''
type: filter
condition: ''
type: filter
Configuration for the log_to_metric transform.
Field
required
Type
Description
metrics
required
[]
A list of metrics to generate.
metrics: array
type: log_to_metric
metrics: array
type: log_to_metric
Configuration for the lua transform.
Field
required
Type
Description
Option 1
optional
Configuration for version one.
version
optional
<oneOf>
Transform API version.
Specifying this version ensures that backward compatibility is not broken.
1
optional
1
Lua transform API version 1.
This version is deprecated and will be removed in a future version.
source
required
string
The Lua program to execute for each event.
search_dirs
optional
[string]
A list of directories to search when loading a Lua file via the require function.
If not specified, the modules are looked up in the configuration directories.
Option 2
optional
Configuration for version two.
version
required
<oneOf>
Transform API version.
Specifying this version ensures that backward compatibility is not broken.
2
optional
2
Lua transform API version 2.
source
optional
string,null
The Lua program to initialize the transform with.
The program can be used to to import external dependencies, as well as define the functions used for the various lifecycle hooks. However, it's not strictly required, as the lifecycle hooks can be configured directly with inline Lua source for each respective hook.
search_dirs
optional
[string]
A list of directories to search when loading a Lua file via the require function.
If not specified, the modules are looked up in the configuration directories.
hooks
required
object
Lifecycle hooks.
These hooks can be set to perform additional processing during the lifecycle of the transform.
init
optional
string,null
The function called when the first event comes in, before hooks.process is called.
It can produce new events using the emit function.
This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both
cases, the closure/function takes a single parameter, emit, which is a reference to a function for emitting events.
process
required
string
The function called for each incoming event.
It can produce new events using the emit function.
This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both
cases, the closure/function takes two parameters. The first parameter, event, is the event being processed,
while the second parameter, emit, is a reference to a function for emitting events.
shutdown
optional
string,null
The function called when the transform is stopped.
It can produce new events using the emit function.
This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both
cases, the closure/function takes a single parameter, emit, which is a reference to a function for emitting events.
timers
optional
[object]
A list of timers which should be configured and executed periodically.
interval_seconds
required
integer
The interval to execute the handler, in seconds.
handler
required
string
The handler function which is called when the timer ticks.
It can produce new events using the emit function.
This can either be inline Lua that defines a closure to use, or the name of the Lua function
to call. In both cases, the closure/function takes a single parameter, emit, which is a
reference to a function for emitting events.
metric_tag_values
optional
<oneOf>
When set to single, metric tag values will be exposed as single strings, the
same as they were before this config option. Tags with multiple values will show the last assigned value, and null values
will be ignored.
When set to full, all metric tags will be exposed as arrays of either string or null
values.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
type: lua
type: lua
Configuration for the metric_to_log transform.
Field
required
Type
Description
host_tag
optional
string,null
Name of the tag in the metric to use for the source host.
If present, the value of the tag is set on the generated log event in the "host" field,
where the field key will use the global host_key option.
timezone
optional
<oneOf>
The name of the timezone to apply to timestamp conversions that do not contain an explicit time zone.
This overrides the global timezone option. The time zone name may be
any name in the TZ database, or local to indicate system local time.
local
optional
local
System local timezone.
log_namespace
optional
boolean,null
The namespace to use for logs. This overrides the global setting.
metric_tag_values
optional
<oneOf>
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags will be displayed with the
metric. When set to full, all metric tags will be exposed as separate assignments as
described by [the native_json codec][vector_native_json].
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
host_tag: string
timezone: ''
log_namespace: boolean
metric_tag_values: single
type: metric_to_log
host_tag: string
timezone: ''
log_namespace: boolean
metric_tag_values: single
type: metric_to_log
Configuration for the reduce transform.
Field
required
Type
Description
expire_after_ms
optional
integer
The maximum period of time to wait after the last event is received, in milliseconds, before a combined event should be considered complete.
flush_period_ms
optional
integer
The interval to check for and flush any expired events, in milliseconds.
group_by
optional
[string]
An ordered list of fields by which to group events.
Each group with matching values for the specified keys is reduced independently, allowing you to keep independent event streams separate. When no fields are specified, all events will be combined in a single group.
For example, if group_by = ["host", "region"], then all incoming events that have the same
host and region will be grouped together before being reduced.
merge_strategies
optional
object
A map of field names to custom merge strategies.
For each field specified, the given strategy will be used for combining events rather than the default behavior.
The default behavior is as follows:
[field-name]_end is added with
the last received timestamp value.ends_when
optional
<oneOf>
A condition used to distinguish the final event of a transaction.
If this condition resolves to true for an event, the current transaction is immediately
flushed with this event.
Map
optional
<oneOf>
An event matching condition.
A fully-specified condition.
IsLog
optional
object
Matches an event if it is a log.
type
required
is_log
Matches an event if it is a log.
IsMetric
optional
object
Matches an event if it is a metric.
type
required
is_metric
Matches an event if it is a metric.
IsTrace
optional
object
Matches an event if it is a trace.
type
required
is_trace
Matches an event if it is a trace.
source
required
string
The VRL boolean expression.
runtime
optional
<oneOf>
Available VRL runtimes.
ast
optional
ast
Tree-walking runtime.
This is the only, and default, runtime.
source
required
string
The query string.
starts_when
optional
<oneOf>
A condition used to distinguish the first event of a transaction.
If this condition resolves to true for an event, the previous transaction is flushed
(without this event) and a new transaction is started.
Map
optional
<oneOf>
An event matching condition.
A fully-specified condition.
IsLog
optional
object
Matches an event if it is a log.
type
required
is_log
Matches an event if it is a log.
IsMetric
optional
object
Matches an event if it is a metric.
type
required
is_metric
Matches an event if it is a metric.
IsTrace
optional
object
Matches an event if it is a trace.
type
required
is_trace
Matches an event if it is a trace.
source
required
string
The VRL boolean expression.
runtime
optional
<oneOf>
Available VRL runtimes.
ast
optional
ast
Tree-walking runtime.
This is the only, and default, runtime.
source
required
string
The query string.
expire_after_ms: 30000
flush_period_ms: 1000
group_by: []
merge_strategies: {}
ends_when: ''
starts_when: ''
type: reduce
expire_after_ms: 30000
flush_period_ms: 1000
group_by: []
merge_strategies: {}
ends_when: ''
starts_when: ''
type: reduce
Configuration for the remap transform.
Field
required
Type
Description
source
optional
string,null
The Vector Remap Language (VRL) program to execute for each event.
Required if file is missing.
file
optional
string,null
File path to the Vector Remap Language (VRL) program to execute for each event.
If a relative path is provided, its root is the current working directory.
Required if source is missing.
metric_tag_values
optional
<oneOf>
When set to single, metric tag values will be exposed as single strings, the
same as they were before this config option. Tags with multiple values will show the last assigned value, and null values
will be ignored.
When set to full, all metric tags will be exposed as arrays of either string or null
values.
single
optional
single
Tag values will be exposed as single strings, the same as they were before this config option. Tags with multiple values will show the last assigned value, and null values will be ignored.
full
optional
full
All tags will be exposed as arrays of either string or null values.
timezone
optional
<oneOf>
The name of the timezone to apply to timestamp conversions that do not contain an explicit time zone.
This overrides the global timezone option. The time zone name may be
any name in the TZ database, or local to indicate system local time.
local
optional
local
System local timezone.
drop_on_error
optional
boolean
Drops any event that encounters an error during processing.
Normally, if a VRL program encounters an error when processing an event, the original,
unmodified event will be sent downstream. In some cases, you may not wish to send the event
any further, such as if certain transformation or enrichment is strictly required. Setting
drop_on_error to true allows you to ensure these events do not get processed any
further.
Additionally, dropped events can potentially be diverted to a specially named output for
further logging and analysis by setting reroute_dropped.
drop_on_abort
optional
boolean
Drops any event that is manually aborted during processing.
Normally, if a VRL program is manually aborted (via abort) when
processing an event, the original, unmodified event will be sent downstream. In some cases,
you may not wish to send the event any further, such as if certain transformation or
enrichment is strictly required. Setting drop_on_abort to true allows you to ensure
these events do not get processed any further.
Additionally, dropped events can potentially be diverted to a specially-named output for
further logging and analysis by setting reroute_dropped.
reroute_dropped
optional
boolean
Reroutes dropped events to a named output instead of halting processing on them.
When using drop_on_error or drop_on_abort, events that are "dropped" are processed no
further. In some cases, it may be desirable to keep the events around for further analysis,
debugging, or retrying.
In these cases, reroute_dropped can be set to true which will forward the original event
to a specially-named output, dropped. The original event will be annotated with additional
fields describing why the event was dropped.
runtime
optional
<oneOf>
Available VRL runtimes.
ast
optional
ast
Tree-walking runtime.
This is the only, and default, runtime.
source: string
file: string
metric_tag_values: single
timezone: ''
drop_on_error: boolean
drop_on_abort: true
reroute_dropped: boolean
runtime: ast
type: remap
source: string
file: string
metric_tag_values: single
timezone: ''
drop_on_error: boolean
drop_on_abort: true
reroute_dropped: boolean
runtime: ast
type: remap
Configuration for the route transform.
Field
required
Type
Description
route
optional
object
A table of route identifiers to logical conditions representing the filter of the route.
Each route can then be referenced as an input by other components with the name
<transform_name>.<route_id>. If an event doesn’t match any route, it will be sent to the
<transform_name>._unmatched output.
Both _unmatched, as well as _default, are reserved output names and thus cannot be used
as a route name.
route: object
type: route
route: object
type: route
Configuration for the sample transform.
Field
required
Type
Description
rate
required
integer
The rate at which events will be forwarded, expressed as 1/N.
For example, rate = 10 means 1 out of every 10 events will be forwarded and the rest will
be dropped.
key_field
optional
string,null
The name of the log field whose value will be hashed to determine if the event should be passed.
Consistently samples the same events. Actual rate of sampling may differ from the configured
one if values in the field are not uniformly distributed. If left unspecified, or if the
event doesn’t have key_field, events will be count rated.
exclude
optional
<oneOf>
An event matching condition.
A logical condition used to exclude events from sampling.
Map
optional
<oneOf>
An event matching condition.
A fully-specified condition.
IsLog
optional
object
Matches an event if it is a log.
type
required
is_log
Matches an event if it is a log.
IsMetric
optional
object
Matches an event if it is a metric.
type
required
is_metric
Matches an event if it is a metric.
IsTrace
optional
object
Matches an event if it is a trace.
type
required
is_trace
Matches an event if it is a trace.
source
required
string
The VRL boolean expression.
runtime
optional
<oneOf>
Available VRL runtimes.
ast
optional
ast
Tree-walking runtime.
This is the only, and default, runtime.
source
required
string
The query string.
rate: integer
key_field: string
exclude: ''
type: sample
rate: integer
key_field: string
exclude: ''
type: sample
Configuration for the tag_cardinality_limit transform.
Field
required
Type
Description
value_limit: 500
limit_exceeded_action: drop_tag
type: tag_cardinality_limit
value_limit: 500
limit_exceeded_action: drop_tag
type: tag_cardinality_limit
Configuration for the throttle transform.
Field
required
Type
Description
threshold
required
integer
The number of events allowed for a given bucket per configured window_secs.
Each unique key will have its own threshold.
window_secs
required
number
The time window in which the configured threshold is applied, in seconds.
key_field
optional
string,null
The name of the log field whose value will be hashed to determine if the event should be rate limited.
Each unique key will create a bucket of related events to be rate limited separately. If
left unspecified, or if the event doesn’t have key_field, the event be will not be rate
limited separately.
exclude
optional
<oneOf>
An event matching condition.
A logical condition used to exclude events from sampling.
Map
optional
<oneOf>
An event matching condition.
A fully-specified condition.
IsLog
optional
object
Matches an event if it is a log.
type
required
is_log
Matches an event if it is a log.
IsMetric
optional
object
Matches an event if it is a metric.
type
required
is_metric
Matches an event if it is a metric.
IsTrace
optional
object
Matches an event if it is a trace.
type
required
is_trace
Matches an event if it is a trace.
source
required
string
The VRL boolean expression.
runtime
optional
<oneOf>
Available VRL runtimes.
ast
optional
ast
Tree-walking runtime.
This is the only, and default, runtime.
source
required
string
The query string.
threshold: integer
window_secs: number
key_field: string
exclude: ''
type: throttle
threshold: integer
window_secs: number
key_field: string
exclude: ''
type: throttle