Observability Pipelines is not available on the US1-FED Datadog site.
A transform allows you to parse, structure, and enrich your data. You can use multiple transforms in your pipeline.
Configuration for the aggregate transform.
The interval between flushes, in milliseconds.
During this time frame, metrics with the same series data (name, namespace, tags, and so on) are aggregated.
interval_ms: 10000
type: aggregate
Configuration for the aws_ec2_metadata transform.
Configuration for the dedupe transform.
Caching configuration for deduplication.
Number of events to cache and use for comparing incoming events to previously seen events.
default: 5000Options to control what fields to match against.
When no field matching configuration is specified, events are matched using the timestamp,
host, and message fields from an event. The specific field names used are those set in
the global log schema configuration.
Options to control what fields to match against.
When no field matching configuration is specified, events are matched using the timestamp,
host, and message fields from an event. The specific field names used are those set in
the global log schema configuration.
Matches events using only the specified fields.
A wrapper around OwnedTargetPath that allows it to be used in Vector config
with prefix default to PathPrefix::Event
Matches events using all fields except for the ignored ones.
A wrapper around OwnedTargetPath that allows it to be used in Vector config
with prefix default to PathPrefix::Event
cache:
num_events: 5000
fields: ''
type: dedupe
Configuration for the filter transform.
The condition that every input event is matched against.
If an event is matched by the condition, it is forwarded. Otherwise, the event is dropped.
An event matching condition.
A fully-specified condition.
Matches an event if it is a log.
Matches an event if it is a log.
Matches an event if it is a metric.
Matches an event if it is a metric.
Matches an event if it is a trace.
Matches an event if it is a trace.
The runtime to use for executing VRL code.
The VRL boolean expression.
condition: ''
type: filter
Configuration for the metric_to_log transform.
Name of the tag in the metric to use for the source host.
If present, the value of the tag is set on the generated log event in the host field,
where the field key uses the global host_key option.
The namespace to use for logs. This overrides the global setting.
Controls how metric tag values are encoded.
When set to single, only the last non-bare value of tags are displayed with the
metric. When set to full, all metric tags are exposed as separate assignments as
described by the native_json codec.
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
All tags are exposed as arrays of either string or null values.
The name of the time zone to apply to timestamp conversions that do not contain an explicit
time zone.
This overrides the global timezone option. The time zone name may be
any name in the TZ database or local to indicate system local time.
Timezone reference.
This can refer to any valid timezone as defined in the TZ database, or "local" which refers to the system local timezone.
A named timezone.
Must be a valid name in the TZ database.
host_tag: string
log_namespace: boolean
metric_tag_values: single
timezone: ''
type: metric_to_log
Configuration for the reduce transform.
A condition used to distinguish the final event of a transaction.
If this condition resolves to true for an event, the current transaction is immediately
flushed with this event.
An event matching condition.
Many methods exist for matching events, such as using a VRL expression, a Datadog Search query string,
or hard-coded matchers like "must be a metric" or "fields A, B, and C must match these constraints".
As VRL is the most common way to apply conditions to events, this type provides a shortcut to define VRL expressions
directly in the configuration by passing the VRL expression as a string:
condition = '.message == "hooray"'
When other condition types are required, they can be specified with an enum-style notation:
condition.type = 'datadog_search'
condition.source = 'NOT "foo"'
An event matching condition.
A fully-specified condition.
Matches an event if it is a log.
Matches an event if it is a log.
Matches an event if it is a metric.
Matches an event if it is a metric.
Matches an event if it is a trace.
Matches an event if it is a trace.
The runtime to use for executing VRL code.
The VRL boolean expression.
The maximum period of time to wait after the last event is received, in milliseconds, before
a combined event should be considered complete.
The interval to check for and flush any expired events, in milliseconds.
An ordered list of fields by which to group events.
Each group with matching values for the specified keys is reduced independently, allowing
you to keep independent event streams separate. When no fields are specified, all events
are combined in a single group.
For example, if group_by = ["host", "region"], then all incoming events that have the same
host and region are grouped together before being reduced.
The maximum number of events to group together.
A map of field names to custom merge strategies.
For each field specified, the given strategy is used for combining events rather than
the default behavior.
The default behavior is as follows:
- The first value of a string field is kept and subsequent values are discarded.
- For timestamp fields the first is kept and a new field
[field-name]_end is added with
the last received timestamp value. - Numeric values are summed.
A condition used to distinguish the first event of a transaction.
If this condition resolves to true for an event, the previous transaction is flushed
(without this event) and a new transaction is started.
An event matching condition.
Many methods exist for matching events, such as using a VRL expression, a Datadog Search query string,
or hard-coded matchers like "must be a metric" or "fields A, B, and C must match these constraints".
As VRL is the most common way to apply conditions to events, this type provides a shortcut to define VRL expressions
directly in the configuration by passing the VRL expression as a string:
condition = '.message == "hooray"'
When other condition types are required, they can be specified with an enum-style notation:
condition.type = 'datadog_search'
condition.source = 'NOT "foo"'
An event matching condition.
A fully-specified condition.
Matches an event if it is a log.
Matches an event if it is a log.
Matches an event if it is a metric.
Matches an event if it is a metric.
Matches an event if it is a trace.
Matches an event if it is a trace.
The runtime to use for executing VRL code.
The VRL boolean expression.
ends_when: ''
expire_after_ms: 30000
flush_period_ms: 1000
group_by: []
max_events: integer
merge_strategies: {}
starts_when: ''
type: reduce
Configuration for the remap transform.
Drops any event that is manually aborted during processing.
Normally, if a VRL program is manually aborted (using abort) when
processing an event, the original, unmodified event is sent downstream. In some cases,
you may not wish to send the event any further, such as if certain transformation or
enrichment is strictly required. Setting drop_on_abort to true allows you to ensure
these events do not get processed any further.
Additionally, dropped events can potentially be diverted to a specially-named output for
further logging and analysis by setting reroute_dropped.
Drops any event that encounters an error during processing.
Normally, if a VRL program encounters an error when processing an event, the original,
unmodified event is sent downstream. In some cases, you may not want to send the event
any further, such as if certain transformation or enrichment is strictly required. Setting
drop_on_error to true allows you to ensure these events do not get processed any
further.
Additionally, dropped events can potentially be diverted to a specially named output for
further logging and analysis by setting reroute_dropped.
File path to the Vector Remap Language (VRL) program to execute for each event.
If a relative path is provided, its root is the current working directory.
Required if source is missing.
When set to single, metric tag values are exposed as single strings, the
same as they were before this config option. Tags with multiple values show the last assigned value, and null values
are ignored.
When set to full, all metric tags are exposed as arrays of either string or null
values.
Tag values are exposed as single strings, the same as they were before this config
option. Tags with multiple values show the last assigned value, and null values
are ignored.
All tags are exposed as arrays of either string or null values.
Reroutes dropped events to a named output instead of halting processing on them.
When using drop_on_error or drop_on_abort, events that are "dropped" are processed no
further. In some cases, it may be desirable to keep the events around for further analysis,
debugging, or retrying.
In these cases, reroute_dropped can be set to true which forwards the original event
to a specially-named output, dropped. The original event is annotated with additional
fields describing why the event was dropped.
The runtime to use for executing VRL code.
The name of the timezone to apply to timestamp conversions that do not contain an explicit
time zone.
This overrides the global timezone option. The time zone name may be
any name in the TZ database, or local to indicate system local time.
Timezone reference.
This can refer to any valid timezone as defined in the TZ database, or "local" which refers to the system local timezone.
A named timezone.
Must be a valid name in the TZ database.
drop_on_abort: true
drop_on_error: boolean
file: ''
metric_tag_values: single
reroute_dropped: boolean
runtime: ast
source: string
timezone: ''
type: remap
Configuration for the route transform.
Reroutes unmatched events to a named output instead of silently discarding them.
Normally, if an event doesn't match any defined route, it is sent to the <transform_name>._unmatched
output for further processing. In some cases, you may want to simply discard unmatched events and not
process them any further.
In these cases, reroute_unmatched can be set to false to disable the <transform_name>._unmatched
output and instead silently discard any unmatched events.
A table of route identifiers to logical conditions representing the filter of the route.
Each route can then be referenced as an input by other components with the name
<transform_name>.<route_id>. If an event doesn’t match any route, and if reroute_unmatched
is set to true (the default), it is sent to the <transform_name>._unmatched output.
Otherwise, the unmatched event is instead silently discarded.
Both _unmatched, as well as _default, are reserved output names and thus cannot be used
as a route name.
reroute_unmatched: true
route: object
type: route
Configuration for the sample transform.
An event matching condition.
A logical condition used to exclude events from sampling.
An event matching condition.
Many methods exist for matching events, such as using a VRL expression, a Datadog Search query string,
or hard-coded matchers like "must be a metric" or "fields A, B, and C must match these constraints".
As VRL is the most common way to apply conditions to events, this type provides a shortcut to define VRL expressions
directly in the configuration by passing the VRL expression as a string:
condition = '.message == "hooray"'
When other condition types are required, they can be specified with an enum-style notation:
condition.type = 'datadog_search'
condition.source = 'NOT "foo"'
An event matching condition.
A fully-specified condition.
Matches an event if it is a log.
Matches an event if it is a log.
Matches an event if it is a metric.
Matches an event if it is a metric.
Matches an event if it is a trace.
Matches an event if it is a trace.
The runtime to use for executing VRL code.
The VRL boolean expression.
The name of the field whose value is hashed to determine if the event should be
sampled.
Each unique value for the key creates a bucket of related events to be sampled together
and the rate is applied to the buckets themselves to sample 1/N buckets. The overall rate
of sampling may differ from the configured one if values in the field are not uniformly
distributed. If left unspecified, or if the event doesn’t have key_field, then the
event is sampled independently.
This can be useful to, for example, ensure that all logs for a given transaction are
sampled together, but that overall 1/N transactions are sampled.
The rate at which events are forwarded, expressed as 1/N.
For example, rate = 10 means 1 out of every 10 events are forwarded and the rest are
dropped.
exclude: ''
key_field: string
rate: integer
type: sample
Configuration for the sensitive_data_scanner transform.
Optional description of this scanner.
Array of rules containing all the scanning rules inside the scanning group.
The name of this scanning rule.
Define action on match
Scrubbing and hashing are irreversible changes. Please be sure you want to replace the
matching values with the replacement text or hashed value.
Redact to replace all matching values entirely with a single token of your choice.
Partially redact to replace a specific portion of all matching values.
Hash to replace all matching values entirely with a non-reversible unique identifier.
Redact the matched data by replacing it with a fixed string.
Redact the matched data by replacing it with a fixed string.
Hash the matched data with the farmhash algorithm.
Hash the matched data with the farmhash algorithm.
Replace part of the matched data with asterisks.
Replace part of the matched data with asterisks.
How many characters should be redacted?
Which part of the matched data should be redacted?
Redact the first characters.
Redact the last characters.
The sensitive data pattern to match in the events.
Use a custom regex rule to scan this pattern.
The custom rule regex pattern.
The log content is checked against the regular expression below.
Use a custom regex rule to scan this pattern.
Look up a rule name in the library of pre-configured patterns.
The name of the library pattern.
Look up a rule name in the library of pre-configured patterns.
Scan entire event or portion of it
By default, the regex is checked on all log attributes. For finer grained processing,
identify an inclusion or exclusion list of attributes with a comma-separated list, such as
admin.email, recipient.
The list of field names to exclude.
The list of field names to include.
An optional array of tags to associated with matched events.
description: string
rules: array
type: sensitive_data_scanner
Configuration for the tag_cardinality_limit transform.
Possible actions to take when an event arrives that would exceed the cardinality limit for one
or more of its tags.
Drop the tag(s) that would exceed the configured limit.
Drop the entire event itself.
How many distinct values to accept for any given key.
Controls the approach taken for tracking tag cardinality.
limit_exceeded_action: drop_tag
value_limit: 500
type: tag_cardinality_limit
Configuration for the throttle transform.
An event matching condition.
A logical condition used to exclude events from sampling.
An event matching condition.
Many methods exist for matching events, such as using a VRL expression, a Datadog Search query string,
or hard-coded matchers like "must be a metric" or "fields A, B, and C must match these constraints".
As VRL is the most common way to apply conditions to events, this type provides a shortcut to define VRL expressions
directly in the configuration by passing the VRL expression as a string:
condition = '.message == "hooray"'
When other condition types are required, they can be specified with an enum-style notation:
condition.type = 'datadog_search'
condition.source = 'NOT "foo"'
An event matching condition.
A fully-specified condition.
Matches an event if it is a log.
Matches an event if it is a log.
Matches an event if it is a metric.
Matches an event if it is a metric.
Matches an event if it is a trace.
Matches an event if it is a trace.
The runtime to use for executing VRL code.
The VRL boolean expression.
The value to group events into separate buckets to be rate limited independently.
If left unspecified, or if the event doesn't have key_field, then the event is not rate
limited separately.
A templated field.
In many cases, components can be configured so that part of the component's functionality can be
customized on a per-event basis. For example, you have a sink that writes events to a file and you want to
specify which file an event should go to by using an event field as part of the
input to the filename used.
By using Template, users can specify either fixed strings or templated strings. Templated strings use a common syntax to
refer to fields in an event that is used as the input data when rendering the template. An example of a fixed string
is my-file.log. An example of a template string is my-file-{{key}}.log, where {{key}}
is the key's value when the template is rendered into a string.
The number of events allowed for a given bucket per configured window_secs.
Each unique key has its own threshold.
The time window in which the configured threshold is applied, in seconds.
exclude: ''
key_field: ''
threshold: integer
window_secs: number
type: throttle
