Transforms

A transform allows you to parse, structure, and enrich your data. You can use multiple transforms in your pipeline.

Aggregate

Configuration for the aggregate transform.

Field

required

Type

Description

interval_ms

optional

integer

The interval between flushes, in milliseconds.

During this time frame, metrics with the same series data (name, namespace, tags, and so on) are aggregated.

interval_ms: 10000
type: aggregate
interval_ms: 10000
type: aggregate

AWS EC2 Metadata

Configuration for the aws_ec2_metadata transform.

Field

required

Type

Description

endpoint

optional

string

Overrides the default EC2 metadata endpoint.

namespace

optional

 <oneOf>

Sets a prefix for all event fields added by the transform.

Option 1

optional

string

An optional path that deserializes an empty string to None.

refresh_interval_secs

optional

integer

The interval between querying for updated metadata, in seconds.

fields

optional

[string]

A list of metadata fields to include in each transformed event.

tags

optional

[string]

A list of instance tags to include in each transformed event.

refresh_timeout_secs

optional

integer

The timeout for querying the EC2 metadata endpoint, in seconds.

proxy

optional

object

Proxy configuration.

Configure to proxy traffic through an HTTP(S) proxy when making external requests.

Similar to common proxy configuration convention, users can set different proxies to use based on the type of traffic being proxied, as well as set specific hosts that should not be proxied.

enabled

optional

boolean

Enables proxying support.

default: true

http

optional

uri

Proxy endpoint to use when proxying HTTP traffic.

Must be a valid URI string.

default: null

https

optional

uri

Proxy endpoint to use when proxying HTTPS traffic.

Must be a valid URI string.

default: null

no_proxy

optional

[string]

A list of hosts to avoid proxying.

Multiple patterns are allowed:

PatternExample match
Domain namesexample.com matches requests to example.com
Wildcard domains.example.com matches requests to example.com and its subdomains
IP addresses127.0.0.1 matches requests to 127.0.0.1
CIDR blocks192.168.0.0/16 matches requests to any IP addresses in this range
Splat* matches all hosts

required

optional

boolean

Requires the transform to be able to successfully query the EC2 metadata before starting to process the data.

endpoint: 'http://169.254.169.254'
namespace: ''
refresh_interval_secs: 10
fields:
  - ami-id
  - availability-zone
  - instance-id
  - instance-type
  - local-hostname
  - local-ipv4
  - public-hostname
  - public-ipv4
  - region
  - subnet-id
  - vpc-id
  - role-name
tags: []
refresh_timeout_secs: 1
proxy:
  enabled: true
  http: null
  https: null
required: true
type: aws_ec2_metadata
endpoint: 'http://169.254.169.254'
namespace: ''
refresh_interval_secs: 10
fields:
  - ami-id
  - availability-zone
  - instance-id
  - instance-type
  - local-hostname
  - local-ipv4
  - public-hostname
  - public-ipv4
  - region
  - subnet-id
  - vpc-id
  - role-name
tags: []
refresh_timeout_secs: 1
proxy:
  enabled: true
  http: null
  https: null
required: true
type: aws_ec2_metadata

Dedupe

Configuration for the dedupe transform.

Field

required

Type

Description

fields

optional

 <oneOf>

Options to control what fields to match against.

When no field matching configuration is specified, events are matched using the timestamp, host, and message fields from an event. The specific field names used are those set in the global log schema configuration.

Option 1

optional

 <oneOf>

Options to control what fields to match against.

When no field matching configuration is specified, events are matched using the timestamp, host, and message fields from an event. The specific field names used are those set in the global log schema configuration.

MatchFields

optional

object

Matches events using only the specified fields.

match

required

[string]

IgnoreFields

optional

object

Matches events using all fields except for the ignored ones.

ignore

required

[string]

cache

optional

object

Caching configuration for deduplication.

num_events

required

integer

Number of events to cache and use for comparing incoming events to previously seen events.

default: 5000
fields: ''
cache:
  num_events: 5000
type: dedupe
fields: ''
cache:
  num_events: 5000
type: dedupe

Filter

Configuration for the filter transform.

Field

required

Type

Description

condition

required

 <oneOf>

The condition that every input event is matched against.

If an event is matched by the condition, it is forwarded. Otherwise, the event is dropped.

String

optional

string

Map

optional

 <oneOf>

An event matching condition.

A fully-specified condition.

IsLog

optional

object

Matches an event if it is a log.

type

required

is_log

Matches an event if it is a log.

IsMetric

optional

object

Matches an event if it is a metric.

type

required

is_metric

Matches an event if it is a metric.

IsTrace

optional

object

Matches an event if it is a trace.

type

required

is_trace

Matches an event if it is a trace.

Vrl

optional

Matches an event with a Vector Remap Language (VRL) boolean expression.

source

required

string

The VRL boolean expression.

runtime

optional

string

The runtime to use for executing VRL code.

type

required

vrl

Matches an event with a Vector Remap Language (VRL) boolean expression.

DatadogSearch

optional

Matches an event with a Datadog Search query.

source

required

string

The query string.

type

required

datadog_search

Matches an event with a Datadog Search query.

condition: ''
type: filter
condition: ''
type: filter

Log To Metric

Configuration for the log_to_metric transform.

Field

required

Type

Description

metrics

required

[]

A list of metrics to generate.

metrics: array
type: log_to_metric
metrics: array
type: log_to_metric

Lua

Configuration for the lua transform.

Field

required

Type

Description

Option 1

optional

Configuration for version one.

version

optional

 <oneOf>

Transform API version.

Specifying this version ensures that backward compatibility is not broken.

Option 1

optional

 <oneOf>

Marker type for the version one of the configuration for the lua transform.

1

optional

1

Lua transform API version 1.

This version is deprecated and will be removed in a future version.

source

required

string

The Lua program to execute for each event.

search_dirs

optional

[string]

A list of directories to search when loading a Lua file via the require function.

If not specified, the modules are looked up in the configuration directories.

Option 2

optional

Configuration for version two.

version

required

 <oneOf>

Transform API version.

Specifying this version ensures that backward compatibility is not broken.

2

optional

2

Lua transform API version 2.

source

optional

string,​null

The Lua program to initialize the transform with.

The program can be used to to import external dependencies, as well as define the functions used for the various lifecycle hooks. However, it's not strictly required, as the lifecycle hooks can be configured directly with inline Lua source for each respective hook.

search_dirs

optional

[string]

A list of directories to search when loading a Lua file via the require function.

If not specified, the modules are looked up in the configuration directories.

hooks

required

object

Lifecycle hooks.

These hooks can be set to perform additional processing during the lifecycle of the transform.

init

optional

string,​null

The function called when the first event comes in, before hooks.process is called.

It can produce new events using the emit function.

This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both cases, the closure/function takes a single parameter, emit, which is a reference to a function for emitting events.

process

required

string

The function called for each incoming event.

It can produce new events using the emit function.

This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both cases, the closure/function takes two parameters. The first parameter, event, is the event being processed, while the second parameter, emit, is a reference to a function for emitting events.

shutdown

optional

string,​null

The function called when the transform is stopped.

It can produce new events using the emit function.

This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both cases, the closure/function takes a single parameter, emit, which is a reference to a function for emitting events.

timers

optional

[object]

A list of timers which should be configured and executed periodically.

interval_seconds

required

integer

The interval to execute the handler, in seconds.

handler

required

string

The handler function which is called when the timer ticks.

It can produce new events using the emit function.

This can either be inline Lua that defines a closure to use, or the name of the Lua function to call. In both cases, the closure/function takes a single parameter, emit, which is a reference to a function for emitting events.

metric_tag_values

optional

 <oneOf>

When set to single, metric tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.

When set to full, all metric tags are exposed as arrays of either string or null values.

single

optional

single

Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.

full

optional

full

All tags are exposed as arrays of either string or null values.

type: lua
type: lua

Metric To Log

Configuration for the metric_to_log transform.

Field

required

Type

Description

host_tag

optional

string,​null

Name of the tag in the metric to use for the source host.

If present, the value of the tag is set on the generated log event in the host field, where the field key uses the global host_key option.

timezone

optional

 <oneOf>

The name of the time zone to apply to timestamp conversions that do not contain an explicit time zone.

This overrides the global timezone option. The time zone name may be any name in the TZ database or local to indicate system local time.

Option 1

optional

 <oneOf>

Timezone reference.

This can refer to any valid timezone as defined in the TZ database, or "local" which refers to the system local timezone.

local

optional

local

System local timezone.

Named

optional

string

A named timezone.

Must be a valid name in the TZ database.

log_namespace

optional

boolean,​null

The namespace to use for logs. This overrides the global setting.

metric_tag_values

optional

 <oneOf>

Controls how metric tag values are encoded.

When set to single, only the last non-bare value of tags are displayed with the metric. When set to full, all metric tags are exposed as separate assignments as described by the native_json codec.

single

optional

single

Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.

full

optional

full

All tags are exposed as arrays of either string or null values.

host_tag: string
timezone: ''
log_namespace: boolean
metric_tag_values: single
type: metric_to_log
host_tag: string
timezone: ''
log_namespace: boolean
metric_tag_values: single
type: metric_to_log

Reduce

Configuration for the reduce transform.

Field

required

Type

Description

expire_after_ms

optional

integer

The maximum period of time to wait after the last event is received, in milliseconds, before a combined event should be considered complete.

flush_period_ms

optional

integer

The interval to check for and flush any expired events, in milliseconds.

max_events

optional

integer,​null

The maximum number of events to group together.

group_by

optional

[string]

An ordered list of fields by which to group events.

Each group with matching values for the specified keys is reduced independently, allowing you to keep independent event streams separate. When no fields are specified, all events are combined in a single group.

For example, if group_by = ["host", "region"], then all incoming events that have the same host and region are grouped together before being reduced.

merge_strategies

optional

object

A map of field names to custom merge strategies.

For each field specified, the given strategy is used for combining events rather than the default behavior.

The default behavior is as follows:

  • The first value of a string field is kept and subsequent values are discarded.
  • For timestamp fields the first is kept and a new field [field-name]_end is added with the last received timestamp value.
  • Numeric values are summed.

ends_when

optional

 <oneOf>

A condition used to distinguish the final event of a transaction.

If this condition resolves to true for an event, the current transaction is immediately flushed with this event.

Option 1

optional

 <oneOf>

An event matching condition.

Many methods exist for matching events, such as using a VRL expression, a Datadog Search query string, or hard-coded matchers like "must be a metric" or "fields A, B, and C must match these constraints".

As VRL is the most common way to apply conditions to events, this type provides a shortcut to define VRL expressions directly in the configuration by passing the VRL expression as a string:

condition = '.message == "hooray"'

When other condition types are required, they can be specified with an enum-style notation:

condition.type = 'datadog_search'
condition.source = 'NOT "foo"'

String

optional

string

Map

optional

 <oneOf>

An event matching condition.

A fully-specified condition.

IsLog

optional

object

Matches an event if it is a log.

type

required

is_log

Matches an event if it is a log.

IsMetric

optional

object

Matches an event if it is a metric.

type

required

is_metric

Matches an event if it is a metric.

IsTrace

optional

object

Matches an event if it is a trace.

type

required

is_trace

Matches an event if it is a trace.

Vrl

optional

Matches an event with a Vector Remap Language (VRL) boolean expression.

source

required

string

The VRL boolean expression.

runtime

optional

string

The runtime to use for executing VRL code.

type

required

vrl

Matches an event with a Vector Remap Language (VRL) boolean expression.

DatadogSearch

optional

Matches an event with a Datadog Search query.

source

required

string

The query string.

type

required

datadog_search

Matches an event with a Datadog Search query.

starts_when

optional

 <oneOf>

A condition used to distinguish the first event of a transaction.

If this condition resolves to true for an event, the previous transaction is flushed (without this event) and a new transaction is started.

Option 1

optional

 <oneOf>

An event matching condition.

Many methods exist for matching events, such as using a VRL expression, a Datadog Search query string, or hard-coded matchers like "must be a metric" or "fields A, B, and C must match these constraints".

As VRL is the most common way to apply conditions to events, this type provides a shortcut to define VRL expressions directly in the configuration by passing the VRL expression as a string:

condition = '.message == "hooray"'

When other condition types are required, they can be specified with an enum-style notation:

condition.type = 'datadog_search'
condition.source = 'NOT "foo"'

String

optional

string

Map

optional

 <oneOf>

An event matching condition.

A fully-specified condition.

IsLog

optional

object

Matches an event if it is a log.

type

required

is_log

Matches an event if it is a log.

IsMetric

optional

object

Matches an event if it is a metric.

type

required

is_metric

Matches an event if it is a metric.

IsTrace

optional

object

Matches an event if it is a trace.

type

required

is_trace

Matches an event if it is a trace.

Vrl

optional

Matches an event with a Vector Remap Language (VRL) boolean expression.

source

required

string

The VRL boolean expression.

runtime

optional

string

The runtime to use for executing VRL code.

type

required

vrl

Matches an event with a Vector Remap Language (VRL) boolean expression.

DatadogSearch

optional

Matches an event with a Datadog Search query.

source

required

string

The query string.

type

required

datadog_search

Matches an event with a Datadog Search query.

expire_after_ms: 30000
flush_period_ms: 1000
max_events: integer
group_by: []
merge_strategies: {}
ends_when: ''
starts_when: ''
type: reduce
expire_after_ms: 30000
flush_period_ms: 1000
max_events: integer
group_by: []
merge_strategies: {}
ends_when: ''
starts_when: ''
type: reduce

Remap

Configuration for the remap transform.

Field

required

Type

Description

source

optional

string,​null

The Vector Remap Language (VRL) program to execute for each event.

Required if file is missing.

file

optional

 <oneOf>

File path to the Vector Remap Language (VRL) program to execute for each event.

If a relative path is provided, its root is the current working directory.

Required if source is missing.

Option 1

optional

string

A file path.

metric_tag_values

optional

 <oneOf>

When set to single, metric tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.

When set to full, all metric tags are exposed as arrays of either string or null values.

single

optional

single

Tag values are exposed as single strings, the same as they were before this config option. Tags with multiple values show the last assigned value, and null values are ignored.

full

optional

full

All tags are exposed as arrays of either string or null values.

timezone

optional

 <oneOf>

The name of the timezone to apply to timestamp conversions that do not contain an explicit time zone.

This overrides the global timezone option. The time zone name may be any name in the TZ database, or local to indicate system local time.

Option 1

optional

 <oneOf>

Timezone reference.

This can refer to any valid timezone as defined in the TZ database, or "local" which refers to the system local timezone.

local

optional

local

System local timezone.

Named

optional

string

A named timezone.

Must be a valid name in the TZ database.

drop_on_error

optional

boolean

Drops any event that encounters an error during processing.

Normally, if a VRL program encounters an error when processing an event, the original, unmodified event is sent downstream. In some cases, you may not want to send the event any further, such as if certain transformation or enrichment is strictly required. Setting drop_on_error to true allows you to ensure these events do not get processed any further.

Additionally, dropped events can potentially be diverted to a specially named output for further logging and analysis by setting reroute_dropped.

drop_on_abort

optional

boolean

Drops any event that is manually aborted during processing.

Normally, if a VRL program is manually aborted (using abort) when processing an event, the original, unmodified event is sent downstream. In some cases, you may not wish to send the event any further, such as if certain transformation or enrichment is strictly required. Setting drop_on_abort to true allows you to ensure these events do not get processed any further.

Additionally, dropped events can potentially be diverted to a specially-named output for further logging and analysis by setting reroute_dropped.

reroute_dropped

optional

boolean

Reroutes dropped events to a named output instead of halting processing on them.

When using drop_on_error or drop_on_abort, events that are "dropped" are processed no further. In some cases, it may be desirable to keep the events around for further analysis, debugging, or retrying.

In these cases, reroute_dropped can be set to true which forwards the original event to a specially-named output, dropped. The original event is annotated with additional fields describing why the event was dropped.

runtime

optional

string

The runtime to use for executing VRL code.

source: string
file: ''
metric_tag_values: single
timezone: ''
drop_on_error: boolean
drop_on_abort: true
reroute_dropped: boolean
runtime: ast
type: remap
source: string
file: ''
metric_tag_values: single
timezone: ''
drop_on_error: boolean
drop_on_abort: true
reroute_dropped: boolean
runtime: ast
type: remap

Route

Configuration for the route transform.

Field

required

Type

Description

route

optional

object

A table of route identifiers to logical conditions representing the filter of the route.

Each route can then be referenced as an input by other components with the name <transform_name>.<route_id>. If an event doesn’t match any route, it is sent to the <transform_name>._unmatched output.

Both _unmatched, as well as _default, are reserved output names and thus cannot be used as a route name.

route: object
type: route
route: object
type: route

Sample

Configuration for the sample transform.

Field

required

Type

Description

rate

required

integer

The rate at which events are forwarded, expressed as 1/N.

For example, rate = 10 means 1 out of every 10 events are forwarded and the rest are dropped.

key_field

optional

string,​null

The name of the log field whose value is hashed to determine if the event should be passed.

Consistently samples the same events. Actual rate of sampling may differ from the configured one if values in the field are not uniformly distributed. If left unspecified, or if the event doesn't have key_field, then events are count rated.

exclude

optional

 <oneOf>

An event matching condition.

A logical condition used to exclude events from sampling.

Option 1

optional

 <oneOf>

An event matching condition.

Many methods exist for matching events, such as using a VRL expression, a Datadog Search query string, or hard-coded matchers like "must be a metric" or "fields A, B, and C must match these constraints".

As VRL is the most common way to apply conditions to events, this type provides a shortcut to define VRL expressions directly in the configuration by passing the VRL expression as a string:

condition = '.message == "hooray"'

When other condition types are required, they can be specified with an enum-style notation:

condition.type = 'datadog_search'
condition.source = 'NOT "foo"'

String

optional

string

Map

optional

 <oneOf>

An event matching condition.

A fully-specified condition.

IsLog

optional

object

Matches an event if it is a log.

type

required

is_log

Matches an event if it is a log.

IsMetric

optional

object

Matches an event if it is a metric.

type

required

is_metric

Matches an event if it is a metric.

IsTrace

optional

object

Matches an event if it is a trace.

type

required

is_trace

Matches an event if it is a trace.

Vrl

optional

Matches an event with a Vector Remap Language (VRL) boolean expression.

source

required

string

The VRL boolean expression.

runtime

optional

string

The runtime to use for executing VRL code.

type

required

vrl

Matches an event with a Vector Remap Language (VRL) boolean expression.

DatadogSearch

optional

Matches an event with a Datadog Search query.

source

required

string

The query string.

type

required

datadog_search

Matches an event with a Datadog Search query.

rate: integer
key_field: string
exclude: ''
type: sample
rate: integer
key_field: string
exclude: ''
type: sample

Tag Cardinality Limit

Configuration for the tag_cardinality_limit transform.

Field

required

Type

Description

value_limit

optional

integer

How many distinct values to accept for any given key.

limit_exceeded_action

optional

 <oneOf>

Possible actions to take when an event arrives that would exceed the cardinality limit for one or more of its tags.

drop_tag

optional

drop_tag

Drop the tag(s) that would exceed the configured limit.

drop_event

optional

drop_event

Drop the entire event itself.

description

optional

Controls the approach taken for tracking tag cardinality.

oneOf

optional

_metadata

optional

value_limit: 500
limit_exceeded_action: drop_tag
type: tag_cardinality_limit
value_limit: 500
limit_exceeded_action: drop_tag
type: tag_cardinality_limit

Throttle

Configuration for the throttle transform.

Field

required

Type

Description

threshold

required

integer

The number of events allowed for a given bucket per configured window_secs.

Each unique key has its own threshold.

window_secs

required

number

The time window in which the configured threshold is applied, in seconds.

key_field

optional

 <oneOf>

The name of the log field whose value is hashed to determine if the event should be rate limited.

Each unique key creates a bucket of related events to be rate limited separately. If left unspecified, or if the event doesn't have key_field, then the event is not rate limited separately.

Option 1

optional

string

A templated field.

In many cases, components can be configured so that part of the component's functionality can be customized on a per-event basis. For example, you have a sink that writes events to a file and you want to specify which file an event should go to by using an event field as part of the input to the filename used.

By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}} is the key's value when the template is rendered into a string.

exclude

optional

 <oneOf>

An event matching condition.

A logical condition used to exclude events from sampling.

Option 1

optional

 <oneOf>

An event matching condition.

Many methods exist for matching events, such as using a VRL expression, a Datadog Search query string, or hard-coded matchers like "must be a metric" or "fields A, B, and C must match these constraints".

As VRL is the most common way to apply conditions to events, this type provides a shortcut to define VRL expressions directly in the configuration by passing the VRL expression as a string:

condition = '.message == "hooray"'

When other condition types are required, they can be specified with an enum-style notation:

condition.type = 'datadog_search'
condition.source = 'NOT "foo"'

String

optional

string

Map

optional

 <oneOf>

An event matching condition.

A fully-specified condition.

IsLog

optional

object

Matches an event if it is a log.

type

required

is_log

Matches an event if it is a log.

IsMetric

optional

object

Matches an event if it is a metric.

type

required

is_metric

Matches an event if it is a metric.

IsTrace

optional

object

Matches an event if it is a trace.

type

required

is_trace

Matches an event if it is a trace.

Vrl

optional

Matches an event with a Vector Remap Language (VRL) boolean expression.

source

required

string

The VRL boolean expression.

runtime

optional

string

The runtime to use for executing VRL code.

type

required

vrl

Matches an event with a Vector Remap Language (VRL) boolean expression.

DatadogSearch

optional

Matches an event with a Datadog Search query.

source

required

string

The query string.

type

required

datadog_search

Matches an event with a Datadog Search query.

threshold: integer
window_secs: number
key_field: ''
exclude: ''
type: throttle
threshold: integer
window_secs: number
key_field: ''
exclude: ''
type: throttle