Cette page n'est pas encore disponible en français, sa traduction est en cours.
Si vous avez des questions ou des retours sur notre projet de traduction actuel, n'hésitez pas à nous contacter.

Observability Pipelines is not available on the US1-FED Datadog site.

Datadog Processing Language (DPL), or Vector Remap Language (VRL), contains built-in functions for transforming your data.


Array Functions

append

Appends each item in the items array to the end of the value array.

argument

Type

Description

default

required

value

array

The initial array.

N/A

yes

items

array

The items to append.

N/A

yes

Examples

Append to an array

Source:

append([1, 2], [3, 4])

Return:

[1,2,3,4]

chunks

Chunks value into slices of length chunk_size bytes.

argument

Type

Description

default

required

value

array, string

The array of bytes to split.

N/A

yes

chunk_size

integer

The desired length of each chunk in bytes. This may be constrained by the host platform architecture.

N/A

yes

Errors

  • chunk_size must be at least 1 byte.
  • chunk_size is too large.

Examples

Split a string into chunks

Source:

chunks("abcdefgh", 4)

Return:

["abcd","efgh"]
Chunks do not respect unicode code point boundaries

Source:

chunks("ab你好", 4)

Return:

["ab�","�好"]

push

Adds the item to the end of the value array.

argument

Type

Description

default

required

value

array

The target array.

N/A

yes

item

any

The item to push.

N/A

yes

Examples

Push an item onto an array

Source:

push([1, 2], 3)

Return:

[1,2,3]

Codec Functions

decode_base16

Decodes the value (a Base16 string) into its original string.

argument

Type

Description

default

required

value

string

The Base16 data to decode.

N/A

yes

Errors

  • value isn’t a valid encoded Base16 string.

Examples

Decode Base16 data

Source:

decode_base16!("796f752068617665207375636365737366756c6c79206465636f646564206d65")

Return:

"you have successfully decoded me"

decode_base64

Decodes the value (a Base64 string) into its original string.

argument

Type

Description

default

required

value

string

The Base64 data to decode.

N/A

yes

charset

string

The character set to use when decoding the data.

standard

no

Errors

  • value isn’t a valid encoded Base64 string.

Examples

Decode Base64 data (default)

Source:

decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")

Return:

"you have successfully decoded me"
Decode Base64 data (URL safe)

Source:

decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")

Return:

"you can't make your heart feel something it won't"

decode_gzip

Decodes the value (a Gzip string) into its original string.

argument

Type

Description

default

required

value

string

The Gzip data to decode.

N/A

yes

Errors

  • value isn’t a valid encoded Gzip string.

Examples

Decode Gzip data

Source:

encoded_text = decode_base64!("H4sIAHEAymMAA6vML1XISCxLVSguTU5OLS5OK83JqVRISU3OT0lNUchNBQD7BGDaIAAAAA==")
decode_gzip!(encoded_text)

Return:

"you have successfully decoded me"

decode_mime_q

Replaces q-encoded or base64-encoded encoded-word substrings in the value with their original string.

argument

Type

Description

default

required

value

string

The string with encoded-words to decode.

N/A

yes

Errors

Examples

Decode single encoded-word

Source:

decode_mime_q!("=?utf-8?b?SGVsbG8sIFdvcmxkIQ==?=")

Return:

"Hello, World!"
Embedded

Source:

decode_mime_q!("From: =?utf-8?b?SGVsbG8sIFdvcmxkIQ==?= <=?utf-8?q?hello=5Fworld=40example=2ecom?=>")

Return:

"From: Hello, World! <hello_world@example.com>"
Without charset

Source:

decode_mime_q!("?b?SGVsbG8sIFdvcmxkIQ==")

Return:

"Hello, World!"

decode_percent

Decodes a percent-encoded value like a URL.

argument

Type

Description

default

required

value

string

The string to decode.

N/A

yes

Examples

Percent decode a value

Source:

decode_percent("foo%20bar%3F")

Return:

"foo bar?"

decode_punycode

Decodes a punycode encoded value, like an internationalized domain name (IDN).

argument

Type

Description

default

required

value

string

The string to decode.

N/A

yes

validate

boolean

If enabled, checks if the input string is a valid domain name.

true

no

Errors

  • value is not valid punycode

Examples

Decode a punycode encoded internationalized domain name

Source:

decode_punycode!("www.xn--caf-dma.com")

Return:

"www.café.com"
Decode an ASCII only string

Source:

decode_punycode!("www.cafe.com")

Return:

"www.cafe.com"
Ignore validation

Source:

decode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)

Return:

"١٠.٦٦.٣٠.٥."

decode_snappy

Decodes the value (a Snappy string) into its original string.

argument

Type

Description

default

required

value

string

The Snappy data to decode.

N/A

yes

Errors

  • value isn’t a valid encoded Snappy string.

Examples

Decode Snappy data

Source:

encoded_text = decode_base64!("LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==")
decode_snappy!(encoded_text)

Return:

"The quick brown fox jumps over 13 lazy dogs."

decode_zlib

Decodes the value (a Zlib string) into its original string.

argument

Type

Description

default

required

value

string

The Zlib data to decode.

N/A

yes

Errors

  • value isn’t a valid encoded Zlib string.

Examples

Decode Zlib data

Source:

encoded_text = decode_base64!("eJwNy4ENwCAIBMCNXIlQ/KqplUSgCdvXAS41qPMHshCB2R1zJlWIVlR6UURX2+wx2YcuK3kAb9C1wd6dn7Fa+QH9gRxr")
decode_zlib!(encoded_text)

Return:

"you_have_successfully_decoded_me.congratulations.you_are_breathtaking."

decode_zstd

Decodes the value (a Zstandard string) into its original string.

argument

Type

Description

default

required

value

string

The Zstandard data to decode.

N/A

yes

Errors

  • value isn’t a valid encoded Zstd string.

Examples

Decode Zstd data

Source:

encoded_text = decode_base64!("KLUv/QBY/QEAYsQOFKClbQBedqXsb96EWDax/f/F/z+gNU4ZTInaUeAj82KqPFjUzKqhcfDqAIsLvAsnY1bI/N2mHzDixRQA")
decode_zstd!(encoded_text)

Return:

"you_have_successfully_decoded_me.congratulations.you_are_breathtaking."

encode_base16

Encodes the value to Base16.

argument

Type

Description

default

required

value

string

The string to encode.

N/A

yes

Examples

Encode to Base16

Source:

encode_base16("please encode me")

Return:

"706c6561736520656e636f6465206d65"

encode_base64

Encodes the value to Base64.

argument

Type

Description

default

required

value

string

The string to encode.

N/A

yes

padding

boolean

Whether the Base64 output is padded.

true

no

charset

string

The character set to use when encoding the data.

standard

no

Examples

Encode to Base64 (default)

Source:

encode_base64("please encode me")

Return:

"cGxlYXNlIGVuY29kZSBtZQ=="
Encode to Base64 (without padding)

Source:

encode_base64("please encode me, no padding though", padding: false)

Return:

"cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g"
Encode to Base64 (URL safe)

Source:

encode_base64("please encode me, but safe for URLs", charset: "url_safe")

Return:

"cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM="

encode_gzip

Encodes the value to Gzip.

argument

Type

Description

default

required

value

string

The string to encode.

N/A

yes

compression_level

integer

The default compression level.

6

no

Examples

Encode to Gzip

Source:

encoded_text = encode_gzip("please encode me")
encode_base64(encoded_text)

Return:

"H4sIAAAAAAAA/yvISU0sTlVIzUvOT0lVyE0FAI4R4vcQAAAA"

encode_json

Encodes the value to JSON.

argument

Type

Description

default

required

value

any

The value to convert to a JSON string.

N/A

yes

pretty

boolean

Whether to pretty print the JSON string or not.

N/A

no

Examples

Encode to JSON

Source:

.payload = encode_json({"hello": "world"})

Return:

"{\"hello\":\"world\"}"

encode_key_value

Encodes the value into key-value format with customizable delimiters. Default delimiters match the logfmt format.

argument

Type

Description

default

required

value

object

The value to convert to a string.

N/A

yes

fields_ordering

array

The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.

N/A

no

key_value_delimiter

string

The string that separates the key from the value.

=

no

field_delimiter

string

The string that separates each key-value pair.

N/A

no

flatten_boolean

boolean

Whether to encode key-value with a boolean value as a standalone key if true and nothing if false.

false

no

Errors

  • fields_ordering contains a non-string element.

Examples

Encode with default delimiters (no ordering)

Source:

encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})

Return:

"lvl=info msg=\"This is a message\" ts=2021-06-05T17:20:00Z"
Encode with default delimiters (fields ordering)

Source:

encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])

Return:

"ts=2021-06-05T17:20:00Z lvl=info msg=\"This is a message\" log_id=12345"
Encode with default delimiters (nested fields)

Source:

encode_key_value({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})

Return:

"agent.name=foo event=log log.file.path=my.log"
Encode with default delimiters (nested fields ordering)

Source:

encode_key_value!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])

Return:

"event=log log.file.path=my.log agent.name=foo"
Encode with custom delimiters (no ordering)

Source:

encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
	field_delimiter: ",",
	key_value_delimiter: ":"
)

Return:

"lvl:info,msg:\"This is a message\",ts:2021-06-05T17:20:00Z"
Encode with custom delimiters and flatten boolean

Source:

encode_key_value(
	{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
	field_delimiter: ",",
	key_value_delimiter: ":",
	flatten_boolean: true
)

Return:

"beta,lvl:info,msg:\"This is a message\",ts:2021-06-05T17:20:00Z"

encode_logfmt

Encodes the value to logfmt.

argument

Type

Description

default

required

value

object

The value to convert to a logfmt string.

N/A

yes

fields_ordering

array

The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.

N/A

no

Errors

  • fields_ordering contains a non-string element.

Examples

Encode to logfmt (no ordering)

Source:

encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})

Return:

"lvl=info msg=\"This is a message\" ts=2021-06-05T17:20:00Z"
Encode to logfmt (fields ordering)

Source:

encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])

Return:

"ts=2021-06-05T17:20:00Z lvl=info msg=\"This is a message\" log_id=12345"
Encode to logfmt (nested fields)

Source:

encode_logfmt({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})

Return:

"agent.name=foo event=log log.file.path=my.log"
Encode to logfmt (nested fields ordering)

Source:

encode_logfmt!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])

Return:

"event=log log.file.path=my.log agent.name=foo"

encode_percent

Encodes a value with percent encoding to safely be used in URLs.

argument

Type

Description

default

required

value

string

The string to encode.

N/A

yes

ascii_set

string

The ASCII set to use when encoding the data.

NON_ALPHANUMERIC

no

Examples

Percent encode all non-alphanumeric characters (default)

Source:

encode_percent("foo bar?")

Return:

"foo%20bar%3F"
Percent encode only control characters

Source:

encode_percent("foo 	bar", ascii_set: "CONTROLS")

Return:

"foo %09bar"

encode_proto

Encodes the value into a protocol buffer payload.

argument

Type

Description

default

required

value

object

The object to convert to a protocol buffer payload.

N/A

yes

desc_file

string

The path to the protobuf descriptor set file. Must be a literal string.

This file is the output of protoc -o

N/A

yes

message_type

string

The name of the message type to use for serializing.

Must be a literal string.

N/A

yes

Errors

  • desc_file file does not exist.
  • message_type message type does not exist in the descriptor file.

Examples

Encode to proto

Source:

.payload = encode_base64(encode_proto!({"name": "someone", "phones": [{"number": "123456"}]}, "resources/protobuf_descriptor_set.desc", "test_protobuf.Person"))

Return:

"Cgdzb21lb25lIggKBjEyMzQ1Ng=="

encode_punycode

Encodes a value to punycode. Useful for internationalized domain names (IDN).

argument

Type

Description

default

required

value

string

The string to encode.

N/A

yes

validate

boolean

Whether to validate the input string to check if it is a valid domain name.

true

no

Errors

  • value can not be encoded to punycode

Examples

Encode an internationalized domain name

Source:

encode_punycode!("www.café.com")

Return:

"www.xn--caf-dma.com"
Encode an internationalized domain name with mixed case

Source:

encode_punycode!("www.CAFé.com")

Return:

"www.xn--caf-dma.com"
Encode an ASCII only string

Source:

encode_punycode!("www.cafe.com")

Return:

"www.cafe.com"
Ignore validation

Source:

encode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)

Return:

"xn--8hbb.xn--fiba.xn--8hbf.xn--eib."

encode_snappy

Encodes the value to Snappy.

argument

Type

Description

default

required

value

string

The string to encode.

N/A

yes

Errors

  • value cannot be encoded into a Snappy string.

Examples

Encode to Snappy

Source:

encoded_text = encode_snappy!("The quick brown fox jumps over 13 lazy dogs.")
encode_base64(encoded_text)

Return:

"LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg=="

encode_zlib

Encodes the value to Zlib.

argument

Type

Description

default

required

value

string

The string to encode.

N/A

yes

compression_level

integer

The default compression level.

6

no

Examples

Encode to Zlib

Source:

encoded_text = encode_zlib("please encode me")
encode_base64(encoded_text)

Return:

"eJwryElNLE5VSM1Lzk9JVchNBQA0RQX7"

encode_zstd

Encodes the value to Zstandard.

argument

Type

Description

default

required

value

string

The string to encode.

N/A

yes

compression_level

integer

The default compression level.

3

no

Examples

Encode to Zstd

Source:

encoded_text = encode_zstd("please encode me")
encode_base64(encoded_text)

Return:

"KLUv/QBYgQAAcGxlYXNlIGVuY29kZSBtZQ=="

Coerce Functions

to_bool

Coerces the value into a boolean.

argument

Type

Description

default

required

value

boolean, integer, float, null, string

The value to convert to a Boolean.

N/A

yes

Errors

  • value is not a supported boolean representation.

Examples

Coerce to a Boolean (string)

Source:

to_bool!("yes")

Return:

true
Coerce to a Boolean (float)

Source:

to_bool(0.0)

Return:

false
Coerce to a Boolean (int)

Source:

to_bool(0)

Return:

false
Coerce to a Boolean (null)

Source:

to_bool(null)

Return:

false
Coerce to a Boolean (Boolean)

Source:

to_bool(true)

Return:

true

to_float

Coerces the value into a float.

argument

Type

Description

default

required

value

integer, float, boolean, string, timestamp

The value to convert to a float. Must be convertible to a float, otherwise an error is raised.

N/A

yes

Errors

  • value is not a supported float representation.

Examples

Coerce to a float

Source:

to_float!("3.145")

Return:

3.145
Coerce to a float (timestamp)

Source:

to_float(t'2020-12-30T22:20:53.824727Z')

Return:

1609366853.824727

to_int

Coerces the value into an integer.

argument

Type

Description

default

required

value

integer, float, boolean, string, timestamp

The value to convert to an integer.

N/A

yes

Errors

  • value is a string but the text is not an integer.
  • value is not a string, int, or timestamp.

Examples

Coerce to an int (string)

Source:

to_int!("2")

Return:

2
Coerce to an int (timestamp)

Source:

to_int(t'2020-12-30T22:20:53.824727Z')

Return:

1609366853

to_regex

Coerces the value into a regex.

argument

Type

Description

default

required

value

string

The value to convert to a regex.

N/A

yes

Errors

  • value is not a string.

Examples

Coerce to a regex

Source:

to_regex("^foo$") ?? r''

Return:

"^foo$"

to_string

Coerces the value into a string.

argument

Type

Description

default

required

value

integer, float, boolean, string, timestamp, null

The value to convert to a string.

N/A

yes

Errors

  • value is not an integer, float, boolean, string, timestamp, or null.

Examples

Coerce to a string (Boolean)

Source:

to_string(true)

Return:

"true"
Coerce to a string (int)

Source:

to_string(52)

Return:

"52"
Coerce to a string (float)

Source:

to_string(52.2)

Return:

"52.2"

Convert Functions

from_unix_timestamp

Converts the value integer from a Unix timestamp to a DPL timestamp.

Converts from the number of seconds since the Unix epoch by default. To convert from milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.

argument

Type

Description

default

required

value

integer

The Unix timestamp to convert.

N/A

yes

unit

string

The time unit.

seconds

no

Examples

Convert from a Unix timestamp (seconds)

Source:

from_unix_timestamp!(5)

Return:

"1970-01-01T00:00:05Z"
Convert from a Unix timestamp (milliseconds)

Source:

from_unix_timestamp!(5000, unit: "milliseconds")

Return:

"1970-01-01T00:00:05Z"
Convert from a Unix timestamp (nanoseconds)

Source:

from_unix_timestamp!(5000, unit: "nanoseconds")

Return:

"1970-01-01T00:00:00.000005Z"

to_syslog_facility

Converts the value, a Syslog facility code, into its corresponding Syslog keyword. For example, 0 into "kern", 1 into "user", etc.

argument

Type

Description

default

required

value

integer

The facility code.

N/A

yes

Errors

Examples

Coerce to a Syslog facility

Source:

to_syslog_facility!(4)

Return:

"auth"

to_syslog_level

Converts the value, a Syslog severity level, into its corresponding keyword, i.e. 0 into "emerg", 1 into "alert", etc.

argument

Type

Description

default

required

value

integer

The severity level.

N/A

yes

Errors

Examples

Coerce to a Syslog level

Source:

to_syslog_level!(5)

Return:

"notice"

to_syslog_severity

Converts the value, a Syslog log level keyword, into a Syslog integer severity level (0 to 7).

argument

Type

Description

default

required

value

string

The Syslog level keyword to convert.

N/A

yes

Errors

  • value is not a valid Syslog level keyword.

Examples

Coerce to Syslog severity

Source:

to_syslog_severity!("alert")

Return:

1

to_unix_timestamp

Converts the value timestamp into a Unix timestamp.

Returns the number of seconds since the Unix epoch by default. To return the number in milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.

argument

Type

Description

default

required

value

timestamp

The timestamp to convert into a Unix timestamp.

N/A

yes

unit

string

The time unit.

seconds

no

Examples

Convert to a Unix timestamp (seconds)

Source:

to_unix_timestamp(t'2021-01-01T00:00:00+00:00')

Return:

1609459200
Convert to a Unix timestamp (milliseconds)

Source:

to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")

Return:

1609459200000
Convert to a Unix timestamp (nanoseconds)

Source:

to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")

Return:

1609459200000000000

Cryptography Functions

decrypt

Decrypts a string with a symmetric encryption algorithm.

Supported Algorithms:

  • AES-256-CFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-CFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-CFB (key = 16 bytes, iv = 16 bytes)
  • AES-256-OFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-OFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-OFB (key = 16 bytes, iv = 16 bytes)
  • Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
  • Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
  • Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
  • CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
  • XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
  • XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)

argument

Type

Description

default

required

ciphertext

string

The string in raw bytes (not encoded) to decrypt.

N/A

yes

algorithm

string

The algorithm to use.

N/A

yes

key

string

The key in raw bytes (not encoded) for decryption. The length must match the algorithm requested.

N/A

yes

iv

string

The IV in raw bytes (not encoded) for decryption. The length must match the algorithm requested. A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value. The value should match the one used during encryption.

N/A

yes

Errors

  • algorithm is not a supported algorithm.
  • key length does not match the key size required for the algorithm specified.
  • iv length does not match the iv size required for the algorithm specified.

Examples

Decrypt value

Source:

ciphertext = decode_base64!("5fLGcu1VHdzsPcGNDio7asLqE1P43QrVfPfmP4i4zOU=")
iv = decode_base64!("fVEIRkIiczCRWNxaarsyxA==")
key = "16_byte_keyxxxxx"
decrypt!(ciphertext, "AES-128-CBC-PKCS7", key, iv: iv)

Return:

"super_secret_message"

encrypt

Encrypts a string with a symmetric encryption algorithm.

Supported Algorithms:

  • AES-256-CFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-CFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-CFB (key = 16 bytes, iv = 16 bytes)
  • AES-256-OFB (key = 32 bytes, iv = 16 bytes)
  • AES-192-OFB (key = 24 bytes, iv = 16 bytes)
  • AES-128-OFB (key = 16 bytes, iv = 16 bytes)
  • Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
  • Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
  • Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
  • AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
  • AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
  • AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
  • AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
  • AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
  • CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
  • XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
  • XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)

argument

Type

Description

default

required

plaintext

string

The string to encrypt.

N/A

yes

algorithm

string

The algorithm to use.

N/A

yes

key

string

The key in raw bytes (not encoded) for encryption. The length must match the algorithm requested.

N/A

yes

iv

string

The IV in raw bytes (not encoded) for encryption. The length must match the algorithm requested. A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value.

N/A

yes

Errors

  • algorithm is not a supported algorithm.
  • key length does not match the key size required for the algorithm specified.
  • iv length does not match the iv size required for the algorithm specified.

Examples

Encrypt value

Source:

plaintext = "super secret message"
iv = "1234567890123456" # typically you would call random_bytes(16)
key = "16_byte_keyxxxxx"
encrypted_message = encrypt!(plaintext, "AES-128-CBC-PKCS7", key, iv: iv)
encode_base64(encrypted_message)

Return:

"GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE="

hmac

Calculates a HMAC of the value using the given key. The hashing algorithm used can be optionally specified.

For most use cases, the resulting bytestream should be encoded into a hex or base64 string using either encode_base16 or encode_base64.

This function is infallible if either the default algorithm value or a recognized-valid compile-time algorithm string literal is used. Otherwise, it is fallible.

argument

Type

Description

default

required

value

string

The string to calculate the HMAC for.

N/A

yes

key

string

The string to use as the cryptographic key.

N/A

yes

algorithm

string

The hashing algorithm to use.

SHA-256

no

Examples

Calculate message HMAC (defaults: SHA-256), encoding to a base64 string

Source:

encode_base64(hmac("Hello there", "super-secret-key"))

Return:

"eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI="
Calculate message HMAC using SHA-224, encoding to a hex-encoded string

Source:

encode_base16(hmac("Hello there", "super-secret-key", algorithm: "SHA-224"))

Return:

"42fccbc2b7d22a143b92f265a8046187558a94d11ddbb30622207e90"
Calculate message HMAC using a variable hash algorithm

Source:

.hash_algo = "SHA-256"
hmac_bytes, err = hmac("Hello there", "super-secret-key", algorithm: .hash_algo)
if err == null {
	.hmac = encode_base16(hmac_bytes)
}

Return:

"78b184f1832f8aff3934f5e0212454671b2d04d494e3b25075c5e45167029662"

md5

Calculates an md5 hash of the value.

argument

Type

Description

default

required

value

string

The string to calculate the hash for.

N/A

yes

Examples

Create md5 hash

Source:

md5("foo")

Return:

"acbd18db4cc2f85cedef654fccc4a4d8"

seahash

Calculates a Seahash hash of the value. Note: Due to limitations in the underlying DPL data types, this function converts the unsigned 64-bit integer SeaHash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values.

argument

Type

Description

default

required

value

string

The string to calculate the hash for.

N/A

yes

Examples

Calculate seahash

Source:

seahash("foobar")

Return:

5348458858952426000
Calculate negative seahash

Source:

seahash("bar")

Return:

-2796170501982571500

sha1

Calculates a SHA-1 hash of the value.

argument

Type

Description

default

required

value

string

The string to calculate the hash for.

N/A

yes

Examples

Calculate sha1 hash

Source:

sha1("foo")

Return:

"0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33"

sha2

Calculates a SHA-2 hash of the value.

argument

Type

Description

default

required

value

string

The string to calculate the hash for.

N/A

yes

variant

string

The variant of the algorithm to use.

SHA-512/256

no

Examples

Calculate sha2 hash

Source:

sha2("foo", variant: "SHA-512/224")

Return:

"d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be"

sha3

Calculates a SHA-3 hash of the value.

argument

Type

Description

default

required

value

string

The string to calculate the hash for.

N/A

yes

variant

string

The variant of the algorithm to use.

SHA3-512

no

Examples

Calculate sha3 hash

Source:

sha3("foo", variant: "SHA3-224")

Return:

"f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a"

Debug Functions

assert

Asserts the condition, which must be a Boolean expression. The program is aborted with message if the condition evaluates to false.

argument

Type

Description

default

required

condition

boolean

The condition to check.

N/A

yes

message

string

An optional custom error message. If the equality assertion fails, message is appended to the default message prefix. See the examples below for a fully formed log message sample.

N/A

no

Errors

  • condition evaluates to false.

Examples

Assertion (true)

Source:

assert!("foo" == "foo", message: "\"foo\" must be \"foo\"!")

Return:

true
Assertion (false)

Source:

assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!")

assert_eq

Asserts that two expressions, left and right, have the same value. The program is aborted with message if they do not have the same value.

argument

Type

Description

default

required

left

any

The value to check for equality against right.

N/A

yes

right

any

The value to check for equality against left.

N/A

yes

message

string

An optional custom error message. If the equality assertion fails, message is appended to the default message prefix. See the examples below for a fully formed log message sample.

N/A

no

Examples

Successful assertion

Source:

assert_eq!(1, 1)

Return:

true
Unsuccessful assertion

Source:

assert_eq!(127, [1, 2, 3])
Unsuccessful assertion with custom log message

Source:

 assert_eq!(1, 0, message: "Unequal integers")

log

Logs the value to stdout at the specified level.

argument

Type

Description

default

required

value

any

The value to log.

N/A

yes

level

string

The log level.

info

no

rate_limit_secs

integer

Specifies that the log message is output no more than once per the given number of seconds. Use a value of 0 to turn rate limiting off.

1

no

Examples

Log a message

Source:

log("Hello, World!", level: "info", rate_limit_secs: 60)

Return:

null
Log an error

Source:

_, err = to_int(.field)
if err != null {
	log(err, level: "error")
}

Return:

null

Enrichment Functions

find_enrichment_table_records

Searches an enrichment table for rows that match the provided condition.

For file enrichment tables, this condition needs to be a DPL object in which the key-value pairs indicate a field to search mapped to a value to search in that field. This function returns the rows that match the provided condition(s). All fields need to match for rows to be returned; if any fields do not match, then no rows are returned.

There are currently two forms of search criteria:

  1. Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the case_sensitive argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.

  2. Date range search. The given field must be greater than or equal to the from date and less than or equal to the to date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.

For geoip and mmdb enrichment tables, this condition needs to be a DPL object with a single key-value pair whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected and without a value, null is used. This table can return the following fields:

  • ISP databases:

    • autonomous_system_number
    • autonomous_system_organization
    • isp
    • organization
  • City databases:

    • city_name
    • continent_code
    • country_code
    • country_name
    • region_code
    • region_name
    • metro_code
    • latitude
    • longitude
    • postal_code
    • timezone
  • Connection-Type databases:

    • connection_type

To use this function, you need to update your configuration to include an enrichment_tables parameter.

argument

Type

Description

default

required

table

string

The enrichment table to search.

N/A

yes

condition

object

The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined.

N/A

yes

select

array

A subset of fields from the enrichment table to return. If not specified, all fields are returned.

N/A

no

case_sensitive

boolean

Whether text fields need to match cases exactly.

true

no

Examples

Exact match

Source:

find_enrichment_table_records!("test",
  {
	"surname": "smith",
  },
  case_sensitive: false)

Return:

[{"id":1,"firstname":"Bob","surname":"Smith"},{"id":2,"firstname":"Fred","surname":"Smith"}]

Source:

find_enrichment_table_records!("test",
  {
	"surname": "Smith",
	"date_of_birth": {
	  "from": t'1985-01-01T00:00:00Z',
	  "to": t'1985-12-31T00:00:00Z'
	}
  })

Return:

[{"id":1,"firstname":"Bob","surname":"Smith"},{"id":2,"firstname":"Fred","surname":"Smith"}]

get_enrichment_table_record

Searches an enrichment table for a row that matches the provided condition. A single row must be matched. If no rows are found or more than one row is found, an error is returned.

For file enrichment tables, this condition needs to be a DPL object in which the key-value pairs indicate a field to search mapped to a value to search in that field. This function returns the rows that match the provided condition(s). All fields need to match for rows to be returned; if any fields do not match, then no rows are returned.

There are currently two forms of search criteria:

  1. Exact match search. The given field must match the value exactly. Case sensitivity can be specified using the case_sensitive argument. An exact match search can use an index directly into the dataset, which should make this search fairly “cheap” from a performance perspective.

  2. Date range search. The given field must be greater than or equal to the from date and less than or equal to the to date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the only criteria when the enrichment data set is very small.

For geoip and mmdb enrichment tables, this condition needs to be a DPL object with a single key-value pair whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected and without a value, null is used. This table can return the following fields:

  • ISP databases:

    • autonomous_system_number
    • autonomous_system_organization
    • isp
    • organization
  • City databases:

    • city_name
    • continent_code
    • country_code
    • country_name
    • region_code
    • region_name
    • metro_code
    • latitude
    • longitude
    • postal_code
    • timezone
  • Connection-Type databases:

    • connection_type

To use this function, you need to update your configuration to include an enrichment_tables parameter.

argument

Type

Description

default

required

table

string

The enrichment table to search.

N/A

yes

condition

object

The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined.

N/A

yes

select

array

A subset of fields from the enrichment table to return. If not specified, all fields are returned.

N/A

no

case_sensitive

boolean

Whether the text fields match the case exactly.

true

no

Errors

  • The row is not found.
  • Multiple rows are found that match the condition.

Examples

Exact match

Source:

get_enrichment_table_record!("test",
  {
    "surname": "bob",
    "firstname": "John"
  },
  case_sensitive: false)

Return:

{"id":1,"firstname":"Bob","surname":"Smith"}

Source:

get_enrichment_table_record!("test",
  {
    "surname": "Smith",
    "date_of_birth": {
      "from": t'1985-01-01T00:00:00Z',
      "to": t'1985-12-31T00:00:00Z'
    }
  })

Return:

{"id":1,"firstname":"Bob","surname":"Smith"}

Enumerate Functions

compact

Compacts the value by removing empty values, where empty values are defined using the available parameters.

argument

Type

Description

default

required

value

array, object

The object or array to compact.

N/A

yes

recursive

boolean

Whether the compaction be recursive.

true

no

null

boolean

Whether null should be treated as an empty value.

true

no

string

boolean

Whether an empty string should be treated as an empty value.

true

no

object

boolean

Whether an empty object should be treated as an empty value.

true

no

array

boolean

Whether an empty array should be treated as an empty value.

true

no

nullish

boolean

Tests whether the value is “nullish” as defined by the is_nullish function.

false

no

Examples

Compact an array

Source:

compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true)

Return:

["foo","bar","buzz"]
Compact an object

Source:

compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true)

Return:

{"field1":1}

filter

Filter elements from a collection.

This function currently does not support recursive iteration.

The function uses the function closure syntax to allow reading the key-value or index-value combination for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

argument

Type

Description

default

required

value

array, object

The array or object to filter.

N/A

yes

Examples

Filter elements

Source:

filter(array!(.tags)) -> |_index, value| {
    # keep any elements that aren't equal to "foo"
    value != "foo"
}

Return:

["bar","baz"]

flatten

Flattens the value into a single-level representation.

argument

Type

Description

default

required

value

array, object

The array or object to flatten.

N/A

yes

separator

string

The separator to be used to join nested keys

.

no

Examples

Flatten array

Source:

flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])

Return:

[1,2,3,4,5,6,7,8,9]
Flatten object

Source:

flatten({
	"parent1": {
		"child1": 1,
		"child2": 2
	},
	"parent2": {
		"child3": 3
	}
})

Return:

{"parent1.child1":1,"parent1.child2":2,"parent2.child3":3}

for_each

Iterate over a collection.

This function currently does not support recursive iteration.

The function uses the “function closure syntax” to allow reading the key/value or index/value combination for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

argument

Type

Description

default

required

value

array, object

The array or object to iterate.

N/A

yes

Examples

Tally elements

Source:

tally = {}
for_each(array!(.tags)) -> |_index, value| {
    # Get the current tally for the `value`, or
    # set to `0`.
    count = int(get!(tally, [value])) ?? 0

    # Increment the tally for the value by `1`.
    tally = set!(tally, [value], count + 1)
}

tally

Return:

{"foo":2,"bar":1,"baz":1}

includes

Determines whether the value array includes the specified item.

argument

Type

Description

default

required

value

array

The array.

N/A

yes

item

any

The item to check.

N/A

yes

Examples

Array includes

Source:

includes(["apple", "orange", "banana"], "banana")

Return:

true

keys

Returns the keys from the object passed into the function.

argument

Type

Description

default

required

value

object

The object to extract keys from.

N/A

yes

Examples

Get keys from the object

Source:

keys({"key1": "val1", "key2": "val2"})

Return:

["key1","key2"]

length

Returns the length of the value.

  • If value is an array, returns the number of elements.
  • If value is an object, returns the number of top-level keys.
  • If value is a string, returns the number of bytes in the string. If you want the number of characters, see strlen.

argument

Type

Description

default

required

value

array, object, string

The array or object.

N/A

yes

Examples

Length (object)

Source:

length({
	"portland": "Trail Blazers",
	"seattle":  "Supersonics"
})

Return:

2
Length (nested object)

Source:

length({
	"home": {
		"city":  "Portland",
		"state": "Oregon"
	},
	"name": "Trail Blazers",
	"mascot": {
		"name": "Blaze the Trail Cat"
	}
})

Return:

3
Length (array)

Source:

length(["Trail Blazers", "Supersonics", "Grizzlies"])

Return:

3
Length (string)

Source:

length("The Planet of the Apes Musical")

Return:

30

map_keys

Map the keys within an object.

If recursive is enabled, the function iterates into nested objects, using the following rules:

  1. Iteration starts at the root.
  2. For every nested object type:
    • First return the key of the object type itself.
    • Then recurse into the object, and loop back to item (1) in this list.
    • Any mutation done on a nested object before recursing into it, are preserved.
  3. For every nested array type:
    • First return the key of the array type itself.
    • Then find all objects within the array, and apply item (2) to each individual object.

The above rules mean that map_keys with recursive enabled finds all keys in the target, regardless of whether nested objects are nested inside arrays.

The function uses the function closure syntax to allow reading the key for each item in the object.

The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

See the examples below to learn about the closure syntax.

argument

Type

Description

default

required

value

object

The object to iterate.

N/A

yes

recursive

boolean

Whether to recursively iterate the collection.

false

no

Examples

Upcase keys

Source:

map_keys(.) -> |key| { upcase(key) }

Return:

{"FOO":"foo","BAR":"bar"}
De-dot keys

Source:

map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }

Return:

{"labels":{"app_kubernetes_io/name":"mysql"}}

map_values

Map the values within a collection.

If recursive is enabled, the function iterates into nested collections, using the following rules:

  1. Iteration starts at the root.
  2. For every nested collection type:
    • First return the collection type itself.
    • Then recurse into the collection, and loop back to item (1) in the list
    • Any mutation done on a collection before recursing into it, are preserved.

The function uses the function closure syntax to allow mutating the value for each item in the collection.

The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block.

Check out the examples below to learn about the closure syntax.

argument

Type

Description

default

required

value

array, object

The object or array to iterate.

N/A

yes

recursive

boolean

Whether to recursively iterate the collection.

false

no

Examples

Upcase values

Source:

map_values(.) -> |value| { upcase!(value) }

Return:

{"foo":"FOO","bar":"BAR"}

match_array

Determines whether the elements in the value array matches the pattern. By default, it checks that at least one element matches, but can be set to determine if all the elements match.

argument

Type

Description

default

required

value

array

The array.

N/A

yes

pattern

regex

The regular expression pattern to match against.

N/A

yes

all

boolean

Whether to match on all elements of value.

false

no

Examples

Match at least one element

Source:

match_array(["foobar", "bazqux"], r'foo')

Return:

true
Match all elements

Source:

match_array(["foo", "foobar", "barfoo"], r'foo', all: true)

Return:

true
No matches

Source:

match_array(["bazqux", "xyz"], r'foo')

Return:

false
Not all elements match

Source:

match_array(["foo", "foobar", "baz"], r'foo', all: true)

Return:

false

strlen

Returns the number of UTF-8 characters in value. This differs from length which counts the number of bytes of a string.

Note: This is the count of Unicode scalar values which can sometimes differ from Unicode code points.

argument

Type

Description

default

required

value

string

The string.

N/A

yes

Examples

strlen

Source:

strlen("ñandú")

Return:

5

unique

Returns the unique values for an array.

The first occurrence of each element is kept.

argument

Type

Description

default

required

value

array

The array to return unique elements from.

N/A

yes

Examples

Unique

Source:

unique(["foo", "bar", "foo", "baz"])

Return:

["foo","bar","baz"]

values

Returns the values from the object passed into the function.

argument

Type

Description

default

required

value

object

The object to extract values from.

N/A

yes

Examples

Get values from the object

Source:

values({"key1": "val1", "key2": "val2"})

Return:

["val1","val2"]

Event Functions

get_secret

Returns the value of the given secret from an event.

argument

Type

Description

default

required

key

string

The name of the secret.

N/A

yes

Examples

Get the Datadog API key from the event metadata

Source:

get_secret("datadog_api_key")

Return:

"secret value"

remove_secret

Removes a secret from an event.

argument

Type

Description

default

required

key

string

The name of the secret to remove.

N/A

yes

Examples

Removes the Datadog API key from the event

Source:

remove_secret("datadog_api_key")

Return:

null

set_secret

Sets the given secret in the event.

argument

Type

Description

default

required

key

string

The name of the secret.

N/A

yes

secret

string

The secret value.

N/A

yes

Examples

Set the Datadog API key to the given value

Source:

set_secret("datadog_api_key", "abc122")

Return:

null

set_semantic_meaning

Sets a semantic meaning for an event. Note: This function assigns meaning at startup, and has no runtime behavior. It is suggested to put all calls to this function at the beginning of a DPL function. The function cannot be conditionally called. For example, using an if statement cannot stop the meaning from being assigned.

argument

Type

Description

default

required

target

path

The path of the value that is assigned a meaning.

N/A

yes

meaning

string

The name of the meaning to assign.

N/A

yes

Examples

Sets custom field semantic meaning

Source:

set_semantic_meaning(.foo, "bar")

Return:

null

IP Functions

ip_aton

Converts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer.

This behavior mimics inet_aton.

argument

Type

Description

default

required

value

string

The IP address to convert to binary.

N/A

yes

Errors

  • value is not a valid IPv4 address.

Examples

IPv4 to integer

Source:

ip_aton!("1.2.3.4")

Return:

16909060

ip_cidr_contains

Determines whether the ip is contained in the block referenced by the cidr.

argument

Type

Description

default

required

cidr

string

The CIDR mask (v4 or v6).

N/A

yes

ip

string

The IP address (v4 or v6).

N/A

yes

Errors

  • cidr is not a valid CIDR.
  • ip is not a valid IP address.

Examples

IPv4 contains CIDR

Source:

ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")

Return:

true
IPv6 contains CIDR

Source:

ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")

Return:

true

ip_ntoa

Converts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation.

This behavior mimics inet_ntoa.

argument

Type

Description

default

required

value

string

The integer representation of an IPv4 address.

N/A

yes

Errors

  • value cannot fit in an unsigned 32-bit integer.

Examples

Integer to IPv4

Source:

ip_ntoa!(16909060)

Return:

"1.2.3.4"

ip_ntop

Converts IPv4 and IPv6 addresses from binary to text form.

This behavior mimics inet_ntop.

argument

Type

Description

default

required

value

string

The binary data to convert from. For IPv4 addresses, it must be 4 bytes (32 bits) long. For IPv6 addresses, it must be 16 bytes (128 bits) long.

N/A

yes

Errors

  • value must be of length 4 or 16 bytes.

Examples

Convert IPv4 address from bytes after decoding from Base64

Source:

ip_ntop!(decode_base64!("wKgAAQ=="))

Return:

"192.168.0.1"
Convert IPv6 address from bytes after decoding from Base64

Source:

ip_ntop!(decode_base64!("IAENuIWjAAAAAIouA3BzNA=="))

Return:

"2001:db8:85a3::8a2e:370:7334"

ip_pton

Converts IPv4 and IPv6 addresses from text to binary form.

  • The binary form of IPv4 addresses is 4 bytes (32 bits) long.
  • The binary form of IPv6 addresses is 16 bytes (128 bits) long.

This behavior mimics inet_pton.

argument

Type

Description

default

required

value

string

The IP address (v4 or v6) to convert to binary form.

N/A

yes

Errors

  • value is not a valid IP (v4 or v6) address in text form.

Examples

Convert IPv4 address to bytes and encode to Base64

Source:

encode_base64(ip_pton!("192.168.0.1"))

Return:

"wKgAAQ=="
Convert IPv6 address to bytes and encode to Base64

Source:

encode_base64(ip_pton!("2001:db8:85a3::8a2e:370:7334"))

Return:

"IAENuIWjAAAAAIouA3BzNA=="

ip_subnet

Extracts the subnet address from the ip using the supplied subnet.

argument

Type

Description

default

required

ip

string

The IP address (v4 or v6).

N/A

yes

subnet

string

The subnet to extract from the IP address. This can be either a prefix length like /8 or a net mask like 255.255.0.0. The net mask can be either an IPv4 or IPv6 address.

N/A

yes

Errors

  • ip is not a valid IP address.
  • subnet is not a valid subnet.

Examples

IPv4 subnet

Source:

ip_subnet!("192.168.10.32", "255.255.255.0")

Return:

"192.168.10.0"
IPv6 subnet

Source:

ip_subnet!("2404:6800:4003:c02::64", "/32")

Return:

"2404:6800::"

ip_to_ipv6

Converts the ip to an IPv6 address.

argument

Type

Description

default

required

ip

string

The IP address to convert to IPv6.

N/A

yes

Errors

  • ip is not a valid IP address.

Examples

IPv4 to IPv6

Source:

ip_to_ipv6!("192.168.10.32")

Return:

"::ffff:192.168.10.32"

ipv6_to_ipv4

Converts the ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.

argument

Type

Description

default

required

ip

string

The IPv4-mapped IPv6 address to convert.

N/A

yes

Errors

  • ip is not a valid IP address.
  • ip is an IPv6 address that is not compatible with IPv4.

Examples

IPv6 to IPv4

Source:

ipv6_to_ipv4!("::ffff:192.168.0.1")

Return:

"192.168.0.1"

is_ipv4

Check if the string is a valid IPv4 address or not.

An IPv4-mapped or IPv4-compatible IPv6 address is not considered valid for the purpose of this function.

argument

Type

Description

default

required

value

string

The IP address to check

N/A

yes

Examples

Valid IPv4 address

Source:

is_ipv4("10.0.102.37")

Return:

true
Valid IPv6 address

Source:

is_ipv4("2001:0db8:85a3:0000:0000:8a2e:0370:7334")

Return:

false
Arbitrary string

Source:

is_ipv4("foobar")

Return:

false

is_ipv6

Check if the string is a valid IPv6 address or not.

argument

Type

Description

default

required

value

string

The IP address to check

N/A

yes

Examples

Valid IPv6 address

Source:

is_ipv6("2001:0db8:85a3:0000:0000:8a2e:0370:7334")

Return:

true
Valid IPv4 address

Source:

is_ipv6("10.0.102.37")

Return:

false
Arbitrary string

Source:

is_ipv6("foobar")

Return:

false

Number Functions

abs

Computes the absolute value of value.

argument

Type

Description

default

required

value

integer, float

The number to calculate the absolute value.

N/A

yes

Examples

Computes the absolute value of the integer

Source:

abs(-42)

Return:

42
Computes the absolute value of the float

Source:

abs(-42.2)

Return:

42.2

ceil

Rounds the value up to the specified precision.

argument

Type

Description

default

required

value

integer, float

The number to round up.

N/A

yes

precision

integer

The number of decimal places to round to.

0

no

Examples

Round a number up (without precision)

Source:

ceil(4.345)

Return:

5
Round a number up (with precision)

Source:

ceil(4.345, precision: 2)

Return:

4.35

floor

Rounds the value down to the specified precision.

argument

Type

Description

default

required

value

integer, float

The number to round down.

N/A

yes

precision

integer

The number of decimal places to round to.

0

no

Examples

Round a number down (without precision)

Source:

floor(4.345)

Return:

4
Round a number down (with precision)

Source:

floor(4.345, precision: 2)

Return:

4.34

format_int

Formats the integer value into a string representation using the given base/radix.

argument

Type

Description

default

required

value

integer

The number to format.

N/A

yes

base

integer

The base to format the number in. Must be between 2 and 36 (inclusive).

10

no

Errors

  • The base is not between 2 and 36.

Examples

Format as a hexadecimal integer

Source:

format_int!(42, 16)

Return:

"2a"
Format as a negative hexadecimal integer

Source:

format_int!(-42, 16)

Return:

"-2a"

format_number

Formats the value into a string representation of the number.

argument

Type

Description

default

required

value

integer, float

The number to format as a string.

N/A

yes

scale

integer

The number of decimal places to display.

N/A

no

decimal_separator

string

The character to use between the whole and decimal parts of the number.

.

no

grouping_separator

string

The character to use between each thousands part of the number.

,

no

Examples

Format a number (3 decimals)

Source:

format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")

Return:

"1,234,567.890"

mod

Calculates the remainder of value divided by modulus.

argument

Type

Description

default

required

value

integer, float

The value the modulus is applied to.

N/A

yes

modulus

integer, float

The modulus value.

N/A

yes

Errors

  • value is not an integer or float.
  • modulus is not an integer or float.
  • modulus is equal to 0.

Examples

Calculate the remainder of two integers

Source:

mod(5, 2)

Return:

1

round

Rounds the value to the specified precision.

argument

Type

Description

default

required

value

integer, float

The number to round.

N/A

yes

precision

integer

The number of decimal places to round to.

0

no

Examples

Round a number (without precision)

Source:

round(4.345)

Return:

4
Round a number (with precision)

Source:

round(4.345, precision: 2)

Return:

4.35

Object Functions

match_datadog_query

Matches an object against a Datadog Search Syntax query.

argument

Type

Description

default

required

value

object

The object.

N/A

yes

query

string

The Datadog Search Syntax query.

N/A

yes

Examples

OR query

Source:

match_datadog_query({"message": "contains this and that"}, "this OR that")

Return:

true
AND query

Source:

match_datadog_query({"message": "contains only this"}, "this AND that")

Return:

false
Facet wildcard

Source:

match_datadog_query({"custom": {"name": "foo"}}, "@name:foo*")

Return:

true
Tag range

Source:

match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]')

Return:

true

merge

Merges the from object into the to object.

argument

Type

Description

default

required

to

object

The object to merge into.

N/A

yes

from

object

The object to merge from.

N/A

yes

deep

boolean

A deep merge is performed if true, otherwise only top-level fields are merged.

false

no

Examples

Object merge (shallow)

Source:

merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	}
)

Return:

{"parent1":{"child2":4,"child5":5},"parent2":{"child3":3}}
Object merge (deep)

Source:

merge(
	{
		"parent1": {
			"child1": 1,
			"child2": 2
		},
		"parent2": {
			"child3": 3
		}
	},
	{
		"parent1": {
			"child2": 4,
			"child5": 5
		}
	},
	deep: true
)

Return:

{"parent1":{"child1":1,"child2":4,"child5":5},"parent2":{"child3":3}}

unnest

Unnest an array field from an object to create an array of objects using that field; keeping all other fields.

Assigning the array result of this to . results in multiple events being emitted from remap. See the remap transform docs for more details.

This is also referred to as explode in some languages.

argument

Type

Description

default

required

path

path

The path of the field to unnest.

N/A

yes

Errors

  • The field path referred to is not an array.

Examples

Unnest an array field

Source:

. = unnest!(.messages)
Unnest nested an array field

Source:

. = unnest!(.event.messages)

Parse Functions

parse_apache_log

Parses Apache access and error log lines. Lines can be in common, combined, or the default error format.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

timestamp_format

string

The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp does not specify a timezone.

%d/%b/%Y:%T %z

no

format

string

The format to use for parsing the log.

N/A

yes

Errors

  • value does not match the specified format.
  • timestamp_format is not a valid format string.
  • The timestamp in value fails to parse using the provided timestamp_format.

Examples

Parse using Apache log format (common)

Source:

parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")

Return:

{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}
Parse using Apache log format (combined)

Source:

parse_apache_log!(
	s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
	"combined",
)

Return:

{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326,"referrer":"http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth","agent":"Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"}
Parse using Apache log format (error)

Source:

parse_apache_log!(
	s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
	"error"
)

Return:

{"client":"147.159.108.175","message":"I will bypass the haptic COM bandwidth, that should matrix the CSS driver!","module":"ab","pid":4803,"port":24259,"severity":"alert","thread":"3814","timestamp":"2021-03-01T12:00:19Z"}

parse_aws_alb_log

Parses value in the Elastic Load Balancer Access format.

argument

Type

Description

default

required

value

string

Access log of the Application Load Balancer.

N/A

yes

Errors

  • value is not a properly formatted AWS ALB log.

Examples

Parse AWS ALB log

Source:

parse_aws_alb_log!(
	"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)

Return:

{"type":"http","timestamp":"2018-11-30T22:23:00.186641Z","elb":"app/my-loadbalancer/50dc6c495c0c9188","client_host":"192.168.131.39:2817","target_host":null,"request_processing_time":0,"target_processing_time":0.001,"response_processing_time":0,"elb_status_code":"200","target_status_code":"200","received_bytes":34,"sent_bytes":366,"request_method":"GET","request_url":"http://www.example.com:80/","request_protocol":"HTTP/1.1","user_agent":"curl/7.46.0","ssl_cipher":null,"ssl_protocol":null,"target_group_arn":"arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067","trace_id":"Root=1-58337364-23a8c76965a2ef7629b185e3","traceability_id":null,"domain_name":null,"chosen_cert_arn":null,"matched_rule_priority":"0","request_creation_time":"2018-11-30T22:22:48.364000Z","actions_executed":"forward","redirect_url":null,"error_reason":null,"target_port_list":[],"target_status_code_list":[],"classification":null,"classification_reason":null}

parse_aws_cloudwatch_log_subscription_message

Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the aws_kinesis_firehose source.

argument

Type

Description

default

required

value

string

The string representation of the message to parse.

N/A

yes

Errors

  • value is not a properly formatted AWS CloudWatch Log subscription message.

Examples

Parse AWS Cloudwatch Log subscription message

Source:

parse_aws_cloudwatch_log_subscription_message!(.message)

Return:

{"owner":"111111111111","message_type":"DATA_MESSAGE","log_group":"test","log_stream":"test","subscription_filters":["Destination"],"log_events":[{"id":"35683658089614582423604394983260738922885519999578275840","message":"{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}","timestamp":"2020-09-14T19:09:29.039Z"}]}

parse_aws_vpc_flow_log

Parses value in the VPC Flow Logs format.

argument

Type

Description

default

required

value

string

VPC Flow Log.

N/A

yes

format

string

VPC Flow Log format.

N/A

no

Errors

  • value is not a properly formatted AWS VPC Flow log.

Examples

Parse AWS VPC Flow log (default format)

Source:

parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")

Return:

{"version":2,"account_id":"123456789010","interface_id":"eni-1235b8ca123456789","srcaddr":null,"dstaddr":null,"srcport":null,"dstport":null,"protocol":null,"packets":null,"bytes":null,"start":1431280876,"end":1431280934,"action":null,"log_status":"NODATA"}
Parse AWS VPC Flow log (custom format)

Source:

parse_aws_vpc_flow_log!(
	"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
	"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)

Return:

{"instance_id":null,"interface_id":"eni-1235b8ca123456789","srcaddr":"10.0.1.5","dstaddr":"10.0.0.220","pkt_srcaddr":"10.0.1.5","pkt_dstaddr":"203.0.113.5"}
Parse AWS VPC Flow log including v5 fields

Source:

parse_aws_vpc_flow_log!("5 52.95.128.179 10.0.0.71 80 34210 6 1616729292 1616729349 IPv4 14 15044 123456789012 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-0c50d5961bcb2d47b eni-1235b8ca123456789 ap-southeast-2 apse2-az3 - - ACCEPT 19 52.95.128.179 10.0.0.71 S3 - - ingress OK",
format: "version srcaddr dstaddr srcport dstport protocol start end type packets bytes account_id vpc_id subnet_id instance_id interface_id region az_id sublocation_type sublocation_id action tcp_flags pkt_srcaddr pkt_dstaddr pkt_src_aws_service pkt_dst_aws_service traffic_path flow_direction log_status")

Return:

{"account_id":"123456789012","action":"ACCEPT","az_id":"apse2-az3","bytes":15044,"dstaddr":"10.0.0.71","dstport":34210,"end":1616729349,"flow_direction":"ingress","instance_id":"i-0c50d5961bcb2d47b","interface_id":"eni-1235b8ca123456789","log_status":"OK","packets":14,"pkt_dst_aws_service":null,"pkt_dstaddr":"10.0.0.71","pkt_src_aws_service":"S3","pkt_srcaddr":"52.95.128.179","protocol":6,"region":"ap-southeast-2","srcaddr":"52.95.128.179","srcport":80,"start":1616729292,"sublocation_id":null,"sublocation_type":null,"subnet_id":"subnet-aaaaaaaa012345678","tcp_flags":19,"traffic_path":null,"type":"IPv4","version":5,"vpc_id":"vpc-abcdefab012345678"}

parse_cef

Parses the value in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

translate_custom_fields

boolean

Toggles translation of custom field pairs to key:value.

N/A

no

Errors

  • value is not a properly formatted CEF string.

Examples

Parse output generated by PTA

Source:

parse_cef!(
	"CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None"
)

Return:

{"cefVersion":"0","deviceVendor":"CyberArk","deviceProduct":"PTA","deviceVersion":"12.6","deviceEventClassId":"1","name":"Suspected credentials theft","severity":"8","suser":"mike2@prod1.domain.com","shost":"prod1.domain.com","src":"1.1.1.1","duser":"andy@dev1.domain.com","dhost":"dev1.domain.com","dst":"2.2.2.2","cs1Label":"ExtraData","cs1":"None","cs2Label":"EventID","cs2":"52b06812ec3500ed864c461e","deviceCustomDate1Label":"detectionDate","deviceCustomDate1":"1388577900000","cs3Label":"PTAlink","cs3":"https://1.1.1.1/incidents/52b06812ec3500ed864c461e","cs4Label":"ExternalLink","cs4":"None"}
Ignore syslog header

Source:

parse_cef!(
	"Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232"
)

Return:

{"cefVersion":"1","deviceVendor":"Security","deviceProduct":"threatmanager","deviceVersion":"1.0","deviceEventClassId":"100","name":"worm successfully stopped","severity":"10","src":"10.0.0.1","dst":"2.1.2.2","spt":"1232"}
Translate custom fields

Source:

parse_cef!(
	"CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address",
	translate_custom_fields: true
)

Return:

{"cefVersion":"0","deviceVendor":"Dev","deviceProduct":"firewall","deviceVersion":"2.2","deviceEventClassId":"1","name":"Connection denied","severity":"5","Device IPv6 Address":"2345:0425:2CA1:0000:0000:0567:5673:23b5"}

parse_common_log

Parses the value using the Common Log Format (CLF).

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

timestamp_format

string

The date/time format to use for encoding the timestamp.

%d/%b/%Y:%T %z

no

Errors

  • value does not match the Common Log Format.
  • timestamp_format is not a valid format string.
  • The timestamp in value fails to parse using the provided timestamp_format.

Examples

Parse using Common Log Format (with default timestamp format)

Source:

parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")

Return:

{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}
Parse using Common Log Format (with custom timestamp format)

Source:

parse_common_log!(
	"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
	"%+"
)

Return:

{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}

parse_csv

Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

delimiter

string

The field delimiter to use when parsing. Must be a single-byte utf8 character.

,

no

Errors

  • The delimiter must be a single-byte UTF-8 character.
  • value is not a valid CSV string.

Examples

Parse a single CSV formatted row

Source:

parse_csv!("foo,bar,\"foo \"\", bar\"")

Return:

["foo","bar","foo \", bar"]
Parse a single CSV formatted row with custom delimiter

Source:

parse_csv!("foo bar", delimiter: " ")

Return:

["foo","bar"]

parse_duration

Parses the value into a human-readable duration format specified by unit.

argument

Type

Description

default

required

value

string

The string of the duration.

N/A

yes

unit

string

The output units for the duration.

N/A

yes

Errors

  • value is not a properly formatted duration.

Examples

Parse duration (milliseconds)

Source:

parse_duration!("1005ms", unit: "s")

Return:

1.005

parse_etld

Parses the eTLD from value representing domain name.

argument

Type

Description

default

required

value

string

The domain string.

N/A

yes

plus_parts

integer

Can be provided to get additional parts of the domain name. When 1 is passed, eTLD+1 will be returned, which represents a domain registrable by a single organization. Higher numbers will return subdomains.

false

no

psl

string

Can be provided to use a different public suffix list.

By default, https://publicsuffix.org/list/public_suffix_list.dat is used.

false

no

Errors

  • unable to determine eTLD for value

Examples

Parse eTLD

Source:

parse_etld!("sub.sussex.ac.uk")

Return:

{"etld":"ac.uk","etld_plus":"ac.uk","known_suffix":true}
Parse eTLD+1

Source:

parse_etld!("sub.sussex.ac.uk", plus_parts: 1)

Return:

{"etld":"ac.uk","etld_plus":"sussex.ac.uk","known_suffix":true}
Parse eTLD with unknown suffix

Source:

parse_etld!("vector.acmecorp")

Return:

{"etld":"acmecorp","etld_plus":"acmecorp","known_suffix":false}
Parse eTLD with custom PSL

Source:

parse_etld!("vector.acmecorp", psl: "resources/public_suffix_list.dat")

Return:

{"etld":"acmecorp","etld_plus":"acmecorp","known_suffix":false}

parse_glog

Parses the value using the glog (Google Logging Library) format.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

Errors

  • value does not match the glog format.

Examples

Parse using glog

Source:

parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")

Return:

{"level":"info","timestamp":"2021-01-31T14:48:54.411655Z","id":15520,"file":"main.c++","line":9,"message":"Hello world!"}

parse_grok

Parses the value using the grok format. All patterns listed here are supported.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

pattern

string

N/A

yes

Errors

  • value fails to parse using the provided pattern.

Examples

Parse using Grok

Source:

parse_grok!(
	"2020-10-02T23:22:12.223222Z info Hello world",
	"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
)

Return:

{"timestamp":"2020-10-02T23:22:12.223222Z","level":"info","message":"Hello world"}

parse_groks

Parses the value using multiple grok patterns. All patterns listed here are supported.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

patterns

array

The Grok patterns, which are tried in order until the first match.

N/A

yes

aliases

object

The shared set of grok aliases that can be referenced in the patterns to simplify them.

true

no

Errors

  • value fails to parse using the provided pattern.

Examples

Parse using multiple Grok patterns

Source:

parse_groks!(
	"2020-10-02T23:22:12.223222Z info Hello world",
	patterns: [
		"%{common_prefix} %{_status} %{_message}",
		"%{common_prefix} %{_message}",
	],
	aliases: {
		"common_prefix": "%{_timestamp} %{_loglevel}",
		"_timestamp": "%{TIMESTAMP_ISO8601:timestamp}",
		"_loglevel": "%{LOGLEVEL:level}",
		"_status": "%{POSINT:status}",
		"_message": "%{GREEDYDATA:message}"
	}
)

Return:

{"timestamp":"2020-10-02T23:22:12.223222Z","level":"info","message":"Hello world"}

parse_int

Parses the string value representing a number in an optional base/radix to an integer.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

base

integer

The base the number is in. Must be between 2 and 36 (inclusive).

If unspecified, the string prefix is used to determine the base: “0b”, 8 for “0” or “0o”, 16 for “0x”, and 10 otherwise.

N/A

no

Errors

  • The base is not between 2 and 36.
  • The number cannot be parsed in the base.

Examples

Parse decimal

Source:

parse_int!("-42")

Return:

-42
Parse binary

Source:

parse_int!("0b1001")

Return:

9
Parse octal

Source:

parse_int!("0o42")

Return:

34
Parse hexadecimal

Source:

parse_int!("0x2a")

Return:

42
Parse explicit base

Source:

parse_int!("2a", 17)

Return:

44

parse_json

Parses the value as JSON.

argument

Type

Description

default

required

value

string

The string representation of the JSON to parse.

N/A

yes

max_depth

integer

Number of layers to parse for nested JSON-formatted documents. The value must be in the range of 1 to 128.

N/A

no

Errors

  • value is not a valid JSON-formatted payload.

Examples

Parse JSON

Source:

parse_json!("{\"key\": \"val\"}")

Return:

{"key":"val"}
Parse JSON with max_depth

Source:

parse_json!("{\"top_level\":{\"key\": \"val\"}}", max_depth: 1)

Return:

{"top_level":"{\"key\": \"val\"}"}

parse_key_value

Parses the value in key-value format. Also known as logfmt.

  • Keys and values can be wrapped with ".
  • " characters can be escaped using \.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

key_value_delimiter

string

The string that separates the key from the value.

=

no

field_delimiter

string

The string that separates each key-value pair.

N/A

no

whitespace

string

Defines the acceptance of unnecessary whitespace surrounding the configured key_value_delimiter.

lenient

no

accept_standalone_key

boolean

Whether a standalone key should be accepted, the resulting object associates such keys with the boolean value true.

true

no

Errors

  • value is not a properly formatted key-value string.

Examples

Parse logfmt log

Source:

parse_key_value!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)

Return:

{"@timestamp":"Sun Jan 10 16:47:39 EST 2021","level":"info","msg":"Stopping all fetchers","tag#production":"stopping_fetchers","id":"ConsumerFetcherManager-1382721708341","module":"kafka.consumer.ConsumerFetcherManager"}
Parse comma delimited log

Source:

parse_key_value!(
	"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
	field_delimiter: ",",
	key_value_delimiter: ":"
)

Return:

{"path":"/cart_link","host":"store.app.com","fwd":"102.30.171.16","dyno":"web.1","connect":"0ms","service":"87ms","status":"304","bytes":"632","protocol":"https"}
Parse comma delimited log with standalone keys

Source:

parse_key_value!(
	"env:prod,service:backend,region:eu-east1,beta",
	field_delimiter: ",",
	key_value_delimiter: ":",
)

Return:

{"env":"prod","service":"backend","region":"eu-east1","beta":true}
Parse duplicate keys

Source:

parse_key_value!(
	"at=info,method=GET,path=\"/index\",status=200,tags=dev,tags=dummy",
	field_delimiter: ",",
	key_value_delimiter: "=",
)

Return:

{"at":"info","method":"GET","path":"/index","status":"200","tags":["dev","dummy"]}

parse_klog

Parses the value using the klog format used by Kubernetes components.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

Errors

  • value does not match the klog format.

Examples

Parse using klog

Source:

parse_klog!("I0505 17:59:40.692994   28133 klog.go:70] hello from klog")

Return:

{"file":"klog.go","id":28133,"level":"info","line":70,"message":"hello from klog","timestamp":"2024-05-05T17:59:40.692994Z"}

parse_linux_authorization

Parses Linux authorization logs usually found under either /var/log/auth.log (for Debian-based systems) or /var/log/secure (for RedHat-based systems) according to Syslog format.

argument

Type

Description

default

required

value

string

The text containing the message to parse.

N/A

yes

Errors

  • value is not a properly formatted Syslog message.

Examples

Parse Linux authorization event

Source:

parse_linux_authorization!(
	s'Mar 23 2023 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)

Return:

{"appname":"sshd","hostname":"localhost","message":"Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar","procid":1111,"timestamp":"2023-03-23T01:49:58Z"}

parse_logfmt

Parses the value in logfmt.

  • Keys and values can be wrapped using the " character.
  • " characters can be escaped by the \ character.
  • As per this logfmt specification, the parse_logfmt function accepts standalone keys and assigns them a Boolean value of true.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

Errors

  • value is not a properly formatted key-value string

Examples

Parse logfmt log

Source:

parse_logfmt!(
	"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)

Return:

{"@timestamp":"Sun Jan 10 16:47:39 EST 2021","level":"info","msg":"Stopping all fetchers","tag#production":"stopping_fetchers","id":"ConsumerFetcherManager-1382721708341","module":"kafka.consumer.ConsumerFetcherManager"}

parse_nginx_log

Parses Nginx access and error log lines. Lines can be in combined, ingress_upstreaminfo, or error format.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

timestamp_format

string

The date/time format to use for encoding the timestamp. The time is parsed in local time if the timestamp doesn’t specify a timezone. The default format is %d/%b/%Y:%T %z for combined logs and %Y/%m/%d %H:%M:%S for error logs.

%d/%b/%Y:%T %z

no

format

string

The format to use for parsing the log.

N/A

yes

Errors

  • value does not match the specified format.
  • timestamp_format is not a valid format string.
  • The timestamp in value fails to parse using the provided timestamp_format.

Examples

Parse via Nginx log format (combined)

Source:

parse_nginx_log!(
    s'172.17.0.1 - alice [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
    "combined",
)

Return:

{"agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36","client":"172.17.0.1","compression":"2.75","referer":"http://localhost/somewhere","request":"POST /not-found HTTP/1.1","size":153,"status":404,"timestamp":"2021-04-01T12:02:31Z","user":"alice"}
Parse via Nginx log format (error)

Source:

parse_nginx_log!(
    s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
    "error"
)

Return:

{"timestamp":"2021-04-01T13:02:31Z","severity":"error","pid":31,"tid":31,"cid":1,"message":"open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)","client":"172.17.0.1","server":"localhost","request":"POST /not-found HTTP/1.1","host":"localhost:8081"}
Parse via Nginx log format (ingress_upstreaminfo)

Source:

parse_nginx_log!(
    s'0.0.0.0 - bob [18/Mar/2023:15:00:00 +0000] "GET /some/path HTTP/2.0" 200 12312 "https://10.0.0.1/some/referer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 462 0.050 [some-upstream-service-9000] [some-other-upstream-5000] 10.0.50.80:9000 19437 0.049 200 752178adb17130b291aefd8c386279e7',
    "ingress_upstreaminfo"
)

Return:

{"body_bytes_size":12312,"http_referer":"https://10.0.0.1/some/referer","http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36","proxy_alternative_upstream_name":"some-other-upstream-5000","proxy_upstream_name":"some-upstream-service-9000","remote_addr":"0.0.0.0","remote_user":"bob","req_id":"752178adb17130b291aefd8c386279e7","request":"GET /some/path HTTP/2.0","request_length":462,"request_time":0.05,"status":200,"timestamp":"2023-03-18T15:00:00Z","upstream_addr":"10.0.50.80:9000","upstream_response_length":19437,"upstream_response_time":0.049,"upstream_status":200}

parse_proto

Parses the value as a protocol buffer payload.

argument

Type

Description

default

required

value

string

The protocol buffer payload to parse.

N/A

yes

desc_file

string

The path to the protobuf descriptor set file. Must be a literal string.

This file is the output of protoc -o

N/A

yes

message_type

string

The name of the message type to use for serializing.

Must be a literal string.

N/A

yes

Errors

  • value is not a valid proto payload.
  • desc_file file does not exist.
  • message_type message type does not exist in the descriptor file.

Examples

Parse proto

Source:

parse_proto!(decode_base64!("Cgdzb21lb25lIggKBjEyMzQ1Ng=="), "resources/protobuf_descriptor_set.desc", "test_protobuf.Person")

Return:

{"name":"someone","phones":[{"number":"123456"}]}

parse_query_string

Parses the value as a query string.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

Examples

Parse query string

Source:

parse_query_string("foo=%2B1&bar=2&bar=3&xyz")

Return:

{"foo":"+1","bar":["2","3"],"xyz":""}
Parse Ruby on Rails’ query string

Source:

parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")

Return:

{"foo[]":["1","2"]}

parse_regex

Parses the value using the provided Regex pattern.

This function differs from the parse_regex_all function in that it returns only the first match.

argument

Type

Description

default

required

value

string

The string to search.

N/A

yes

pattern

regex

The regular expression pattern to search against.

N/A

yes

numeric_groups

regex

If true, the index of each group in the regular expression is also captured. Index 0 contains the whole match.

false

no

Errors

  • value fails to parse using the provided pattern.

Examples

Parse using Regex (with capture groups)

Source:

parse_regex!("first group and second group.", r'(?P<number>.*?) group')

Return:

{"number":"first"}
Parse using Regex (without capture groups)

Source:

parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)

Return:

{"0":"first group","1":"first"}

parse_regex_all

Parses the value using the provided Regex pattern.

This function differs from the parse_regex function in that it returns all matches, not just the first.

argument

Type

Description

default

required

value

string

The string to search.

N/A

yes

pattern

regex

The regular expression pattern to search against.

N/A

yes

numeric_groups

regex

If true, the index of each group in the regular expression is also captured. Index 0 contains the whole match.

false

no

Errors

  • value fails to parse using the provided pattern.

Examples

Parse using Regex (all matches)

Source:

parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)

Return:

[{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}]

parse_ruby_hash

Parses the value as ruby hash.

argument

Type

Description

default

required

value

string

The string representation of the ruby hash to parse.

N/A

yes

Errors

  • value is not a valid ruby hash formatted payload.

Examples

Parse ruby hash

Source:

parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }')

Return:

{"test":"value","testNum":0.2,"testObj":{"testBool":true,"testNull":null}}

parse_syslog

Parses the value in Syslog format.

argument

Type

Description

default

required

value

string

The text containing the Syslog message to parse.

N/A

yes

Errors

  • value is not a properly formatted Syslog message.

Examples

Parse Syslog log (5424)

Source:

parse_syslog!(
	s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!'
)

Return:

{"severity":"notice","facility":"user","timestamp":"2020-03-13T20:45:38.119Z","hostname":"dynamicwireless.name","appname":"non","procid":2426,"msgid":"ID931","message":"Try to override the THX port, maybe it will reboot the neural interface!","exampleSDID@32473":{"eventID":"1011","eventSource":"Application","iut":"3"},"version":1}

parse_timestamp

Parses the value in strptime format.

argument

Type

Description

default

required

value

string

The text of the timestamp.

N/A

yes

format

string

The strptime format.

N/A

yes

Errors

  • value fails to parse using the provided format.

Examples

Parse timestamp

Source:

parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z")

Return:

"2020-10-10T16:00:00Z"

parse_tokens

Parses the value in token format. A token is considered to be one of the following:

  • A word surrounded by whitespace.
  • Text delimited by double quotes: "..". Quotes can be included in the token if they are escaped by a backslash (\).
  • Text delimited by square brackets: [..]. Closing square brackets can be included in the token if they are escaped by a backslash (\).

argument

Type

Description

default

required

value

string

The string to tokenize.

N/A

yes

Errors

  • value is not a properly formatted tokenized string.

Examples

Parse tokens

Source:

parse_tokens(
	"A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]"
)

Return:

["A","sentence","with \\\"a\\\" sentence inside","and","some brackets"]

parse_url

Parses the value in URL format.

argument

Type

Description

default

required

value

string

The text of the URL.

N/A

yes

default_known_ports

boolean

If true and the port number is not specified in the input URL string (or matches the default port for the scheme), it is populated from well-known ports for the following schemes: http, https, ws, wss, and ftp.

false

no

Errors

  • value is not a properly formatted URL.

Examples

Parse URL

Source:

parse_url!("ftp://foo:bar@example.com:4343/foobar?hello=world#123")

Return:

{"scheme":"ftp","username":"foo","password":"bar","host":"example.com","port":4343,"path":"/foobar","query":{"hello":"world"},"fragment":"123"}
Parse URL with default port

Source:

parse_url!("https://example.com", default_known_ports: true)

Return:

{"scheme":"https","username":"","password":"","host":"example.com","port":443,"path":"/","query":{},"fragment":null}
Parse URL with internationalized domain name

Source:

parse_url!("https://www.café.com")

Return:

{"scheme":"https","username":"","password":"","host":"www.xn--caf-dma.com","port":null,"path":"/","query":{},"fragment":null}
Parse URL with mixed case internationalized domain name

Source:

parse_url!("https://www.CAFé.com")

Return:

{"scheme":"https","username":"","password":"","host":"www.xn--caf-dma.com","port":null,"path":"/","query":{},"fragment":null}

parse_user_agent

Parses the value as a user agent string, which has a loosely defined format so this parser only provides best effort guarantee.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

mode

string

Determines performance and reliability characteristics.

fast

no

Examples

Fast mode

Source:

parse_user_agent(
	"Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1"
)

Return:

{"browser":{"family":"Firefox","version":"1.0.1"},"device":{"category":"pc"},"os":{"family":"Linux","version":null}}
Reliable mode

Source:

parse_user_agent(
	"Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
	mode: "reliable"
)

Return:

{"browser":{"family":"Internet Explorer","version":"7.66"},"device":{"category":"pc"},"os":{"family":"Windows XP","version":"NT 5.1"}}
Enriched mode

Source:

parse_user_agent(
	"Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54",
	mode: "enriched"
)

Return:

{"browser":{"family":"Opera Mini","major":"4","minor":"3","patch":"24214","version":"10.54"},"device":{"brand":"Apple","category":"smartphone","family":"iPhone","model":"iPhone"},"os":{"family":"iOS","major":"4","minor":"2","patch":"1","patch_minor":null,"version":"4.2.1"}}

parse_xml

Parses the value as XML.

argument

Type

Description

default

required

value

string

The string representation of the XML document to parse.

N/A

yes

include_attr

boolean

Include XML tag attributes in the returned object.

true

no

attr_prefix

string

String prefix to use for XML tag attribute keys.

@

no

text_key

string

Key name to use for expanded text nodes.

text

no

always_use_text_key

boolean

Always return text nodes as {"<text_key>": "value"}.

false

no

parse_bool

boolean

Parse “true” and “false” as boolean.

true

no

parse_null

boolean

Parse “null” as null.

true

no

parse_number

boolean

Parse numbers as integers/floats.

true

no

Errors

  • value is not a valid XML document.

Examples

Parse XML

Source:

value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';

parse_xml!(value, text_key: "value", parse_number: false)

Return:

{"book":{"@category":"CHILDREN","author":"J K. Rowling","title":{"@lang":"en","value":"Harry Potter"},"year":"2005"}}

Path Functions

del

Removes the field specified by the static path from the target.

For dynamic path deletion, see the remove function.

argument

Type

Description

default

required

path

path

The path of the field to delete.

N/A

yes

compact

boolean

After deletion, if compact is true and there is an empty object or array left, the empty object or array is also removed, cascading up to the root. This only applies to the path being deleted, and any parent paths.

false

no

Examples

Delete a field

Source:

del(.field1)
Rename a field

Source:

.new_field = del(.old_field)

exists

Checks whether the path exists for the target.

This function distinguishes between a missing path and a path with a null value. A regular path lookup, such as .foo, cannot distinguish between the two cases since it always returns null if the path doesn’t exist.

argument

Type

Description

default

required

path

path

The path of the field to check.

N/A

yes

Examples

Exists (field)

Source:

exists(.field)

Return:

true
Exists (array element)

Source:

exists(.array[2])

Return:

true

get

Dynamically get the value of a given path.

If you know the path you want to look up, use static paths such as .foo.bar[1] to get the value of that path. However, if you do not know the path names, use the dynamic get function to get the requested value.

argument

Type

Description

default

required

value

object, array

The object or array to query.

N/A

yes

path

array

An array of path segments to look for the value.

N/A

yes

Errors

  • The path segment must be a string or an integer.

Examples

single-segment top-level field

Source:

get!(value: { "foo": "bar" }, path: ["foo"])

Return:

"bar"
multi-segment nested field

Source:

get!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])

Return:

"baz"
array indexing

Source:

get!(value: ["foo", "bar", "baz"], path: [-2])

Return:

"bar"

remove

Dynamically remove the value for a given path.

If you know the path you want to remove, use the del function and static paths such as del(.foo.bar[1]) to remove the value at that path. The del function returns the deleted value, and is more performant than remove. However, if you do not know the path names, use the dynamic remove function to remove the value at the provided path.

argument

Type

Description

default

required

value

object, array

The object or array to remove data from.

N/A

yes

path

array

An array of path segments to remove the value from.

N/A

yes

compact

boolean

After deletion, if compact is true, any empty objects or arrays left are also removed.

false

no

Errors

  • The path segment must be a string or an integer.

Examples

single-segment top-level field

Source:

remove!(value: { "foo": "bar" }, path: ["foo"])

Return:

{}
multi-segment nested field

Source:

remove!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])

Return:

{"foo":{}}
array indexing

Source:

remove!(value: ["foo", "bar", "baz"], path: [-2])

Return:

["foo","baz"]
compaction

Source:

remove!(value: { "foo": { "bar": [42], "baz": true } }, path: ["foo", "bar", 0], compact: true)

Return:

{"foo":{"baz":true}}

set

Dynamically insert data into the path of a given object or array.

If you know the path you want to assign a value to, use static path assignments such as .foo.bar[1] = true for improved performance and readability. However, if you do not know the path names, use the dynamic set function to insert the data into the object or array.

argument

Type

Description

default

required

value

object, array

The object or array to insert data into.

N/A

yes

path

array

An array of path segments to insert the value into.

N/A

yes

data

any

The data to be inserted.

N/A

yes

Errors

  • The path segment must be a string or an integer.

Examples

single-segment top-level field

Source:

set!(value: { "foo": "bar" }, path: ["foo"], data: "baz")

Return:

{"foo":"baz"}
multi-segment nested field

Source:

set!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"], data: "qux")

Return:

{"foo":{"bar":"qux"}}
array

Source:

set!(value: ["foo", "bar", "baz"], path: [-2], data: 42)

Return:

["foo",42,"baz"]

Random Functions

random_bool

Returns a random boolean.

Examples

Random boolean

Source:

is_boolean(random_bool())

Return:

true

random_bytes

A cryptographically secure random number generator. Returns a string value containing the number of random bytes requested.

argument

Type

Description

default

required

length

integer

The number of bytes to generate. Must not be larger than 64k.

N/A

yes

Errors

  • length is negative.
  • length is larger than the maximum value (64k).

Examples

Generate random base 64 encoded bytes

Source:

encode_base64(random_bytes(16))

Return:

"LNu0BBgUbh7XAlXbjSOomQ=="

random_float

Returns a random float between [min, max).

argument

Type

Description

default

required

min

float

Minimum value (inclusive).

N/A

yes

max

float

Maximum value (exclusive).

N/A

yes

Errors

  • max is not greater than min.

Examples

Random float from 0.0 to 10.0, not including 10.0

Source:

f = random_float(0.0, 10.0)
f >= 0 && f < 10

Return:

true

random_int

Returns a random integer between [min, max).

argument

Type

Description

default

required

min

integer

Minimum value (inclusive).

N/A

yes

max

integer

Maximum value (exclusive).

N/A

yes

Errors

  • max is not greater than min.

Examples

Random integer from 0 to 10, not including 10

Source:

i = random_int(0, 10)
i >= 0 && i < 10

Return:

true

uuid_from_friendly_id

Convert a Friendly ID (base62 encoding a 128-bit word) to a UUID.

argument

Type

Description

default

required

value

timestamp

A string that is a Friendly ID

N/A

yes

Errors

  • value is a string but the text uses characters outside of class [0-9A-Za-z].
  • value is a base62 encoding of an integer, but the integer is greater than or equal to 2^128.

Examples

Convert a Friendly ID to a UUID

Source:

uuid_from_friendly_id!("3s87yEvnmkiPBMHsj8bwwc")

Return:

"7f41deed-d5e2-8b5e-7a13-ab4ff93cfad2"

uuid_v4

Generates a random UUIDv4 string.

Examples

Create a UUIDv4

Source:

uuid_v4()

Return:

"1d262f4f-199b-458d-879f-05fd0a5f0683"

uuid_v7

Generates a random UUIDv7 string.

argument

Type

Description

default

required

timestamp

timestamp

The timestamp used to generate the UUIDv7.

now()

no

Examples

Create a UUIDv7 with implicit now()

Source:

uuid_v7()

Return:

"06338364-8305-7b74-8000-de4963503139"
Create a UUIDv7 with explicit now()

Source:

uuid_v7(now())

Return:

"018e29b3-0bea-7f78-8af3-d32ccb1b93c1"
Create a UUIDv7 with custom timestamp

Source:

uuid_v7(t'2020-12-30T22:20:53.824727Z')

Return:

"0176b5bd-5d19-7394-bb60-c21028c6152b"

String Functions

community_id

Generates an ID based on the Community ID Spec.

argument

Type

Description

default

required

source_ip

string

The source IP address.

N/A

yes

destination_ip

string

The destination IP address.

N/A

yes

protocol

integer

The protocol number.

N/A

yes

source_port

integer

The source port.

N/A

no

destination_port

integer

The destination port.

N/A

no

seed

integer

The custom seed number.

N/A

no

Examples

TCP

Source:

community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", source_port: 1122, destination_port: 3344, protocol: 6)

Return:

"1:wCb3OG7yAFWelaUydu0D+125CLM="

contains

Determines whether the value string contains the specified substring.

argument

Type

Description

default

required

value

string

The text to search.

N/A

yes

substring

string

The substring to search for in value.

N/A

yes

case_sensitive

boolean

Whether the match should be case sensitive.

true

no

Examples

String contains (case sensitive)

Source:

contains("The Needle In The Haystack", "Needle")

Return:

true
String contains (case insensitive)

Source:

contains("The Needle In The Haystack", "needle", case_sensitive: false)

Return:

true

contains_all

Determines whether the value string contains all the specified substrings.

argument

Type

Description

default

required

value

string

The text to search.

N/A

yes

substrings

array

An array of substrings to search for in value.

N/A

yes

case_sensitive

boolean

Whether the match should be case sensitive.

N/A

no

Examples

String contains all

Source:

contains_all("The Needle In The Haystack", ["Needle", "Haystack"])

Return:

true
String contains all (case sensitive)

Source:

contains_all("the NEEDLE in the haystack", ["needle", "haystack"])

Return:

false

downcase

Downcases the value string, where downcase is defined according to the Unicode Derived Core Property Lowercase.

argument

Type

Description

default

required

value

string

The string to convert to lowercase.

N/A

yes

Examples

Downcase a string

Source:

downcase("Hello, World!")

Return:

"hello, world!"

ends_with

Determines whether the value string ends with the specified substring.

argument

Type

Description

default

required

value

string

The string to search.

N/A

yes

substring

string

The substring with which value must end.

N/A

yes

case_sensitive

boolean

Whether the match should be case sensitive.

true

no

Examples

String ends with (case sensitive)

Source:

ends_with("The Needle In The Haystack", "The Haystack")

Return:

true
String ends with (case insensitive)

Source:

ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false)

Return:

true

find

Determines from left to right the start position of the first found element in value that matches pattern. Returns -1 if not found.

argument

Type

Description

default

required

value

string

The string to find the pattern in.

N/A

yes

pattern

regex, string

The regular expression or string pattern to match against.

N/A

yes

from

integer

Offset to start searching.

0

no

Examples

Match text

Source:

find("foobar", "foo")

Return:

0
Match regex

Source:

find("foobar", r'b.r')

Return:

3
No matches

Source:

find("foobar", "baz")

Return:

-1
With an offset

Source:

find("foobarfoobarfoo", "bar", 4)

Return:

9

join

Joins each string in the value array into a single string, with items optionally separated from one another by a separator.

argument

Type

Description

default

required

value

array

The array of strings to join together.

N/A

yes

separator

string

The string separating each original element when joined.

N/A

no

Examples

Join array (no separator)

Source:

join!(["bring", "us", "together"])

Return:

"bringustogether"
Join array (comma separator)

Source:

join!(["sources", "transforms", "sinks"], separator: ", ")

Return:

"sources, transforms, sinks"

match

Determines whether the value matches the pattern.

argument

Type

Description

default

required

value

string

The value to match.

N/A

yes

pattern

regex

The regular expression pattern to match against.

N/A

yes

Examples

Regex match on a string

Source:

match("I'm a little teapot", r'teapot')

Return:

true
String does not match the regular expression

Source:

match("I'm a little teapot", r'.*balloon')

Return:

false

match_any

Determines whether value matches any of the given patterns. All patterns are checked in a single pass over the target string, giving this function a potential performance advantage over the multiple calls in the match function.

argument

Type

Description

default

required

value

string

The value to match.

N/A

yes

patterns

array

The array of regular expression patterns to match against.

N/A

yes

Examples

Regex match on a string

Source:

match_any("I'm a little teapot", [r'frying pan', r'teapot'])

Return:

true

parse_float

Parses the string value representing a floating point number in base 10 to a float.

argument

Type

Description

default

required

value

string

The string to parse.

N/A

yes

Examples

Parse negative integer

Source:

parse_float!("-42")

Return:

-42
Parse negative integer

Source:

parse_float!("42.38")

Return:

42.38
Scientific notation

Source:

parse_float!("2.5e3")

Return:

2500

redact

Redact sensitive data in value such as:

This can help achieve compliance by ensuring sensitive data does not leave your network.

argument

Type

Description

default

required

value

string, object, array

The value to redact sensitive data from.

The function’s behavior depends on value’s type:

  • For strings, the sensitive data is redacted and a new string is returned.
  • For arrays, the sensitive data is redacted in each string element.
  • For objects, the sensitive data in each string value is masked, but the keys are not masked.

For arrays and objects, the function recurses into any nested arrays or objects. Any non-string elements are skipped.

Redacted text is replaced with [REDACTED].

N/A

yes

filters

array

List of filters applied to value.

Each filter can be specified in the following ways:

  • As a regular expression, which is used to redact text that match it.
  • As an object with a type key that corresponds to a named filter and additional keys for customizing that filter.
  • As a named filter, if it has no required parameters.

Named filters can be a:

  • pattern: Redacts text matching any regular expressions specified in the patterns key, which is required. This is the expanded version of just passing a regular expression as a filter.
  • us_social_security_number: Redacts US social security card numbers.

See examples for more details.

This parameter must be a static expression so that the argument can be validated at compile-time to avoid runtime errors. You cannot use variables or other dynamic expressions with it.

N/A

yes

redactor

string, object

Specifies what to replace the redacted strings with.

It is given as an object with a “type” key specifying the type of redactor to use and additional keys depending on the type. The following types are supported:

  • full: The default. Replace with the string “[REDACTED]”.
  • text: Replace with a custom string. The replacement key is required, and must contain the string that is used as a replacement.
  • sha2: Hash the redacted text with SHA-2 as with sha2. Supports two optional parameters:
    • variant: The variant of the algorithm to use. Defaults to SHA-512/256.
    • encoding: How to encode the hash as text. Can be base16 or base64. Defaults to base64.
  • sha3: Hash the redacted text with SHA-3 as with sha3. Supports two optional parameters:
    • variant: The variant of the algorithm to use. Defaults to SHA3-512.
    • encoding: How to encode the hash as text. Can be base16 or base64. Defaults to base64.

As a convenience you can use a string as a shorthand for common redactor patterns:

  • "full" is equivalent to {"type": "full"}
  • "sha2" is equivalent to {"type": "sha2", "variant": "SHA-512/256", "encoding": "base64"}
  • "sha3" is equivalent to {"type": "sha3", "variant": "SHA3-512", "encoding": "base64"}

This parameter must be a static expression so that the argument can be validated at compile-time to avoid runtime errors. You cannot use variables or other dynamic expressions with it.

N/A

no

Examples

Replace text using a regex

Source:

redact("my id is 123456", filters: [r'\d+'])

Return:

"my id is [REDACTED]"
Replace us social security numbers in any field

Source:

redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"])

Return:

{"name":"John Doe","ssn":"[REDACTED]"}
Replace with custom text

Source:

redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "text", "replacement": "***"})

Return:

"my id is ***"
Replace with SHA-2 hash

Source:

redact("my id is 123456", filters: [r'\d+'], redactor: "sha2")

Return:

"my id is GEtTedW1p6tC094dDKH+3B8P+xSnZz69AmpjaXRd63I="
Replace with SHA-3 hash

Source:

redact("my id is 123456", filters: [r'\d+'], redactor: "sha3")

Return:

"my id is ZNCdmTDI7PeeUTFnpYjLdUObdizo+bIupZdl8yqnTKGdLx6X3JIqPUlUWUoFBikX+yTR+OcvLtAqWO11NPlNJw=="
Replace with SHA-256 hash using hex encoding

Source:

redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})

Return:

"my id is 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92"

replace

Replaces all matching instances of pattern in value.

The pattern argument accepts regular expression capture groups. Note: Use $$foo instead of $foo, which is interpreted in a configuration file.

argument

Type

Description

default

required

value

string

The original string.

N/A

yes

pattern

regex, string

Replace all matches of this pattern. Can be a static string or a regular expression.

N/A

yes

with

string

The string that the matches are replaced with.

N/A

yes

count

integer

The maximum number of replacements to perform. -1 means replace all matches.

-1

no

Examples

Replace literal text

Source:

replace("Apples and Bananas", "and", "not")

Return:

"Apples not Bananas"
Replace using regular expression

Source:

replace("Apples and Bananas", r'(?i)bananas', "Pineapples")

Return:

"Apples and Pineapples"
Replace first instance

Source:

replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1)

Return:

"Pineapples and Bananas"
Replace with capture groups

Source:

replace("foo123bar", r'foo(?P<num>\d+)bar', "$num")

Return:

"123"

replace_with

Replaces all matching instances of pattern using a closure.

The pattern argument accepts a regular expression that can use capture groups.

The function uses the function closure syntax to compute the replacement values.

The closure takes a single parameter, which is an array, where the first item is always present and contains the entire string that matched pattern. The items from index one on contain the capture groups of the corresponding index. If a capture group is optional, the value may be null if it didn’t match.

The value returned by the closure must be a string and will replace the section of the input that was matched.

This returns a new string with the replacements, the original string is not mutated.

argument

Type

Description

default

required

value

string

The original string.

N/A

yes

pattern

regex

Replace all matches of this pattern. Must be a regular expression.

N/A

yes

count

integer

The maximum number of replacements to perform. -1 means replace all matches.

-1

no

Examples

Capitalize words

Source:

	replace_with("apples and bananas", r'\b(\w)(\w*)') -> |match| {
		upcase!(match.captures[0]) + string!(match.captures[1])
	}

Return:

"Apples And Bananas"
Replace with hash

Source:

	replace_with("email from test@example.com", r'\w+@example.com') -> |match| {
		sha2(match.string, variant: "SHA-512/224")
	}

Return:

"email from adf6e1bc4415d24912bd93072ad34ef825a7b6eb3bf53f68def1fc17"
Replace first instance

Source:

	replace_with("Apples and Apples", r'(?i)apples|cones', count: 1) -> |match| {
		"Pine" + downcase(match.string)
	}

Return:

"Pineapples and Apples"
Named capture group

Source:

	replace_with("level=error A message", r'level=(?P<level>\w+)') -> |match| {
		lvl = upcase!(match.level)
		"[{{lvl}}]"
	}

Return:

"[ERROR] A message"

sieve

Keeps only matches of pattern in value.

This can be used to define patterns that are allowed in the string and remove everything else.

argument

Type

Description

default

required

value

string

The original string.

N/A

yes

pattern

regex

Keep all matches of this pattern.

N/A

yes

replace_single

string

The string to use to replace single rejected characters.

N/A

no

replace_repeated

string

The string to use to replace multiple sequential instances of rejected characters.

N/A

no

Examples

Sieve with regex

Source:

sieve("test123%456.فوائد.net.", r'[a-z0-9.]')

Return:

"test123456..net."
Custom replacements

Source:

sieve("test123%456.فوائد.net.", r'[a-z.0-9]', replace_single: "X", replace_repeated: "<REMOVED>")

Return:

"test123X456.<REMOVED>.net."

slice

Returns a slice of value between the start and end positions.

If the start and end parameters are negative, they refer to positions counting from the right of the string or array. If end refers to a position that is greater than the length of the string or array, a slice up to the end of the string or array is returned.

argument

Type

Description

default

required

value

array, string

The string or array to slice.

N/A

yes

start

integer

The inclusive start position. A zero-based index that can be negative.

N/A

yes

end

integer

The exclusive end position. A zero-based index that can be negative.

String length

no

Examples

Slice a string (positive index)

Source:

slice!("Supercalifragilisticexpialidocious", start: 5, end: 13)

Return:

"califrag"
Slice a string (negative index)

Source:

slice!("Supercalifragilisticexpialidocious", start: 5, end: -14)

Return:

"califragilistic"

split

Splits the value string using pattern.

argument

Type

Description

default

required

value

string

The string to split.

N/A

yes

pattern

string, regex

The string is split whenever this pattern is matched.

N/A

yes

limit

integer

The maximum number of substrings to return.

N/A

no

Examples

Split a string (no limit)

Source:

split("apples and pears and bananas", " and ")

Return:

["apples","pears","bananas"]
Split a string (with a limit)

Source:

split("apples and pears and bananas", " and ", limit: 2)

Return:

["apples","pears and bananas"]

starts_with

Determines whether value begins with substring.

argument

Type

Description

default

required

value

string

The string to search.

N/A

yes

substring

string

The substring that the value must start with.

N/A

yes

case_sensitive

boolean

Whether the match should be case sensitive.

true

no

Examples

String starts with (case sensitive)

Source:

starts_with("The Needle In The Haystack", "The Needle")

Return:

true
String starts with (case insensitive)

Source:

starts_with("The Needle In The Haystack", "the needle", case_sensitive: false)

Return:

true

strip_ansi_escape_codes

Strips ANSI escape codes from value.

argument

Type

Description

default

required

value

string

The string to strip.

N/A

yes

Examples

Strip ANSI escape codes

Source:

strip_ansi_escape_codes("\e[46mfoo\e[0m bar")

Return:

"foo bar"

strip_whitespace

Strips whitespace from the start and end of value, where whitespace is defined by the Unicode White_Space property.

argument

Type

Description

default

required

value

string

The string to trim.

N/A

yes

Examples

Strip whitespace

Source:

strip_whitespace("  A sentence.  ")

Return:

"A sentence."

truncate

Truncates the value string up to the limit number of characters.

argument

Type

Description

default

required

value

string

The string to truncate.

N/A

yes

limit

integer, float

The number of characters to truncate the string after.

N/A

yes

ellipsis

boolean

This argument is deprecated. An ellipsis (...) is appended if the parameter is set to true and the value string is truncated because it exceeded the limit.

N/A

no

suffix

string

A custom suffix (...) is appended to truncated strings. If ellipsis is set to true, this parameter is ignored for backwards compatibility.

N/A

no

Examples

Truncate a string

Source:

truncate("A rather long sentence.", limit: 11, suffix: "...")

Return:

"A rather lo..."
Truncate a string

Source:

truncate("A rather long sentence.", limit: 11, suffix: "[TRUNCATED]")

Return:

"A rather lo[TRUNCATED]"

upcase

Upcases value, where upcase is defined according to the Unicode Derived Core Property Uppercase.

argument

Type

Description

default

required

value

string

The string to convert to uppercase.

N/A

yes

Examples

Upcase a string

Source:

upcase("Hello, World!")

Return:

"HELLO, WORLD!"

System Functions

get_env_var

Returns the value of the environment variable specified by name.

argument

Type

Description

default

required

name

string

The name of the environment variable.

N/A

yes

Errors

  • Environment variable name does not exist.
  • The value of environment variable name is not valid Unicode

Examples

Get an environment variable

Source:

get_env_var!("HOME")

Return:

"/root"

get_hostname

Returns the local system’s hostname.

Errors

  • Internal hostname resolution failed.

Examples

Get hostname

Source:

.hostname = get_hostname!()

get_timezone_name

Returns the name of the timezone in the Vector configuration (see global configuration options). If the configuration is set to local, then it attempts to determine the name of the timezone from the host OS. If this is not possible, then it returns the fixed offset of the local timezone for the current time in the format "[+-]HH:MM", for example, "+02:00".

Errors

  • Retrieval of local timezone information failed.

Examples

Get the IANA name of Vector’s timezone

Source:

.vector_timezone = get_timezone_name!()

Timestamp Functions

format_timestamp

Formats value into a string representation of the timestamp.

argument

Type

Description

default

required

value

timestamp

The timestamp to format as text.

N/A

yes

format

string

The format string as described by the Chrono library.

N/A

yes

timezone

string

The timezone to use when formatting the timestamp. The parameter uses the TZ identifier or local.

N/A

no

Examples

Format a timestamp (ISO8601/RFC 3339)

Source:

format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+")

Return:

"2020-10-21T16:00:00+00:00"
Format a timestamp (custom)

Source:

format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R")

Return:

"21-Oct-2020 16:00"

now

Returns the current timestamp in the UTC timezone with nanosecond precision.

Examples

Generate a current timestamp

Source:

now()

Return:

"2021-03-04T10:51:15.928937Z"

Type Functions

array

Returns value if it is an array, otherwise returns an error. This enables the type checker to guarantee that the returned value is an array and can be used in any function that expects an array.

argument

Type

Description

default

required

value

any

The value to check if it is an array.

N/A

yes

Errors

  • value is not an array.

Examples

Declare an array type

Source:

array!(.value)

Return:

[1,2,3]

bool

Returns value if it is a Boolean, otherwise returns an error. This enables the type checker to guarantee that the returned value is a Boolean and can be used in any function that expects a Boolean.

argument

Type

Description

default

requ