Observability Pipelines is not available on the US1-FED Datadog site.
Datadog Processing Language (DPL), or Vector Remap Language (VRL), contains built-in functions for transforming your data.
The functions are organized into the following categories:
Array Functions
append
Appends each item in the items array to the end of the value array.
Examples
Append to an array
Source:
Return:
chunks
Chunks value into slices of length chunk_size bytes.
The array of bytes to split.
The desired length of each chunk in bytes. This may be constrained by the host platform architecture.
Errors
chunk_size must be at least 1 byte.chunk_size is too large.
Examples
Split a string into chunks
Source:
Return:
Chunks do not respect unicode code point boundaries
Source:
Return:
push
Adds the item to the end of the value array.
Examples
Push an item onto an array
Source:
Return:
Codec Functions
decode_base16
Decodes the value (a Base16 string) into its original string.
Errors
value isn’t a valid encoded Base16 string.
Examples
Decode Base16 data
Source:
decode_base16!("796f752068617665207375636365737366756c6c79206465636f646564206d65")
Return:
"you have successfully decoded me"
decode_base64
Decodes the value (a Base64 string) into its original string.
The character set to use when decoding the data.
Errors
value isn’t a valid encoded Base64 string.
Examples
Decode Base64 data (default)
Source:
decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")
Return:
"you have successfully decoded me"
Decode Base64 data (URL safe)
Source:
decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")
Return:
"you can't make your heart feel something it won't"
decode_gzip
Decodes the value (a Gzip string) into its original string.
Errors
value isn’t a valid encoded Gzip string.
Examples
Decode Gzip data
Source:
encoded_text = decode_base64!("H4sIAHEAymMAA6vML1XISCxLVSguTU5OLS5OK83JqVRISU3OT0lNUchNBQD7BGDaIAAAAA==")
decode_gzip!(encoded_text)
Return:
"you have successfully decoded me"
decode_mime_q
Replaces q-encoded or base64-encoded encoded-word substrings in the value with their original string.
Errors
Examples
Decode single encoded-word
Source:
decode_mime_q!("=?utf-8?b?SGVsbG8sIFdvcmxkIQ==?=")
Return:
Embedded
Source:
decode_mime_q!("From: =?utf-8?b?SGVsbG8sIFdvcmxkIQ==?= <=?utf-8?q?hello=5Fworld=40example=2ecom?=>")
Return:
"From: Hello, World! <hello_world@example.com>"
Without charset
Source:
decode_mime_q!("?b?SGVsbG8sIFdvcmxkIQ==")
Return:
decode_percent
Decodes a percent-encoded value like a URL.
Examples
Percent decode a value
Source:
decode_percent("foo%20bar%3F")
Return:
decode_punycode
Decodes a punycode encoded value, like an internationalized domain name (IDN).
If enabled, checks if the input string is a valid domain name.
Errors
value is not valid punycode
Examples
Decode a punycode encoded internationalized domain name
Source:
decode_punycode!("www.xn--caf-dma.com")
Return:
Decode an ASCII only string
Source:
decode_punycode!("www.cafe.com")
Return:
Ignore validation
Source:
decode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)
Return:
decode_snappy
Decodes the value (a Snappy string) into its original string.
The Snappy data to decode.
Errors
value isn’t a valid encoded Snappy string.
Examples
Decode Snappy data
Source:
encoded_text = decode_base64!("LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==")
decode_snappy!(encoded_text)
Return:
"The quick brown fox jumps over 13 lazy dogs."
decode_zlib
Decodes the value (a Zlib string) into its original string.
Errors
value isn’t a valid encoded Zlib string.
Examples
Decode Zlib data
Source:
encoded_text = decode_base64!("eJwNy4ENwCAIBMCNXIlQ/KqplUSgCdvXAS41qPMHshCB2R1zJlWIVlR6UURX2+wx2YcuK3kAb9C1wd6dn7Fa+QH9gRxr")
decode_zlib!(encoded_text)
Return:
"you_have_successfully_decoded_me.congratulations.you_are_breathtaking."
decode_zstd
Decodes the value (a Zstandard string) into its original string.
Errors
value isn’t a valid encoded Zstd string.
Examples
Decode Zstd data
Source:
encoded_text = decode_base64!("KLUv/QBY/QEAYsQOFKClbQBedqXsb96EWDax/f/F/z+gNU4ZTInaUeAj82KqPFjUzKqhcfDqAIsLvAsnY1bI/N2mHzDixRQA")
decode_zstd!(encoded_text)
Return:
"you_have_successfully_decoded_me.congratulations.you_are_breathtaking."
encode_base16
Encodes the value to Base16.
Examples
Encode to Base16
Source:
encode_base16("please encode me")
Return:
"706c6561736520656e636f6465206d65"
encode_base64
Encodes the value to Base64.
Whether the Base64 output is padded.
The character set to use when encoding the data.
Examples
Encode to Base64 (default)
Source:
encode_base64("please encode me")
Return:
"cGxlYXNlIGVuY29kZSBtZQ=="
Encode to Base64 (without padding)
Source:
encode_base64("please encode me, no padding though", padding: false)
Return:
"cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g"
Encode to Base64 (URL safe)
Source:
encode_base64("please encode me, but safe for URLs", charset: "url_safe")
Return:
"cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM="
encode_gzip
Encodes the value to Gzip.
The default compression level.
Examples
Encode to Gzip
Source:
encoded_text = encode_gzip("please encode me")
encode_base64(encoded_text)
Return:
"H4sIAAAAAAAA/yvISU0sTlVIzUvOT0lVyE0FAI4R4vcQAAAA"
encode_json
Encodes the value to JSON.
The value to convert to a JSON string.
Whether to pretty print the JSON string or not.
Examples
Encode to JSON
Source:
.payload = encode_json({"hello": "world"})
Return:
encode_key_value
Encodes the value into key-value format with customizable delimiters. Default delimiters match
the logfmt format.
The value to convert to a string.
The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.
The string that separates the key from the value.
The string that separates each key-value pair.
Whether to encode key-value with a boolean value as a standalone key if true and nothing if false.
Errors
fields_ordering contains a non-string element.
Examples
Encode with default delimiters (no ordering)
Source:
encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return:
"lvl=info msg=\"This is a message\" ts=2021-06-05T17:20:00Z"
Encode with default delimiters (fields ordering)
Source:
encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return:
"ts=2021-06-05T17:20:00Z lvl=info msg=\"This is a message\" log_id=12345"
Encode with default delimiters (nested fields)
Source:
encode_key_value({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return:
"agent.name=foo event=log log.file.path=my.log"
Encode with default delimiters (nested fields ordering)
Source:
encode_key_value!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return:
"event=log log.file.path=my.log agent.name=foo"
Encode with custom delimiters (no ordering)
Source:
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
field_delimiter: ",",
key_value_delimiter: ":"
)
Return:
"lvl:info,msg:\"This is a message\",ts:2021-06-05T17:20:00Z"
Encode with custom delimiters and flatten boolean
Source:
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
field_delimiter: ",",
key_value_delimiter: ":",
flatten_boolean: true
)
Return:
"beta,lvl:info,msg:\"This is a message\",ts:2021-06-05T17:20:00Z"
encode_logfmt
Encodes the value to logfmt.
The value to convert to a logfmt string.
The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.
Errors
fields_ordering contains a non-string element.
Examples
Encode to logfmt (no ordering)
Source:
encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return:
"lvl=info msg=\"This is a message\" ts=2021-06-05T17:20:00Z"
Encode to logfmt (fields ordering)
Source:
encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return:
"ts=2021-06-05T17:20:00Z lvl=info msg=\"This is a message\" log_id=12345"
Encode to logfmt (nested fields)
Source:
encode_logfmt({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return:
"agent.name=foo event=log log.file.path=my.log"
Encode to logfmt (nested fields ordering)
Source:
encode_logfmt!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return:
"event=log log.file.path=my.log agent.name=foo"
encode_percent
Encodes a value with percent encoding to safely be used in URLs.
The ASCII set to use when encoding the data.
Examples
Percent encode all non-alphanumeric characters (default)
Source:
encode_percent("foo bar?")
Return:
Percent encode only control characters
Source:
encode_percent("foo bar", ascii_set: "CONTROLS")
Return:
encode_proto
Encodes the value into a protocol buffer payload.
The object to convert to a protocol buffer payload.
The path to the protobuf descriptor set file. Must be a literal string.
This file is the output of protoc -o …
The name of the message type to use for serializing.
Must be a literal string.
Errors
desc_file file does not exist.message_type message type does not exist in the descriptor file.
Examples
Encode to proto
Source:
.payload = encode_base64(encode_proto!({"name": "someone", "phones": [{"number": "123456"}]}, "resources/protobuf_descriptor_set.desc", "test_protobuf.Person"))
Return:
"Cgdzb21lb25lIggKBjEyMzQ1Ng=="
encode_punycode
Encodes a value to punycode. Useful for internationalized domain names (IDN).
Whether to validate the input string to check if it is a valid domain name.
Errors
value can not be encoded to punycode
Examples
Encode an internationalized domain name
Source:
encode_punycode!("www.café.com")
Return:
Encode an internationalized domain name with mixed case
Source:
encode_punycode!("www.CAFé.com")
Return:
Encode an ASCII only string
Source:
encode_punycode!("www.cafe.com")
Return:
Ignore validation
Source:
encode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)
Return:
"xn--8hbb.xn--fiba.xn--8hbf.xn--eib."
encode_snappy
Encodes the value to Snappy.
Errors
value cannot be encoded into a Snappy string.
Examples
Encode to Snappy
Source:
encoded_text = encode_snappy!("The quick brown fox jumps over 13 lazy dogs.")
encode_base64(encoded_text)
Return:
"LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg=="
encode_zlib
Encodes the value to Zlib.
The default compression level.
Examples
Encode to Zlib
Source:
encoded_text = encode_zlib("please encode me")
encode_base64(encoded_text)
Return:
"eJwryElNLE5VSM1Lzk9JVchNBQA0RQX7"
encode_zstd
Encodes the value to Zstandard.
The default compression level.
Examples
Encode to Zstd
Source:
encoded_text = encode_zstd("please encode me")
encode_base64(encoded_text)
Return:
"KLUv/QBYgQAAcGxlYXNlIGVuY29kZSBtZQ=="
Coerce Functions
to_bool
Coerces the value into a boolean.
boolean, integer, float, null, string
The value to convert to a Boolean.
Errors
value is not a supported boolean representation.
Examples
Coerce to a Boolean (string)
Source:
Return:
Coerce to a Boolean (float)
Source:
Return:
Coerce to a Boolean (int)
Source:
Return:
Coerce to a Boolean (null)
Source:
Return:
Coerce to a Boolean (Boolean)
Source:
Return:
to_float
Coerces the value into a float.
integer, float, boolean, string, timestamp
The value to convert to a float. Must be convertible to a float, otherwise an error is raised.
Errors
value is not a supported float representation.
Examples
Coerce to a float
Source:
Return:
Coerce to a float (timestamp)
Source:
to_float(t'2020-12-30T22:20:53.824727Z')
Return:
to_int
Coerces the value into an integer.
integer, float, boolean, string, timestamp, null
The value to convert to an integer.
Errors
value is a string but the text is not an integer.value is not a string, int, or timestamp.
Examples
Coerce to an int (string)
Source:
Return:
Coerce to an int (timestamp)
Source:
to_int(t'2020-12-30T22:20:53.824727Z')
Return:
to_regex
Coerces the value into a regex.
The value to convert to a regex.
Errors
Examples
Coerce to a regex
Source:
Return:
to_string
Coerces the value into a string.
integer, float, boolean, string, timestamp, null
The value to convert to a string.
Errors
value is not an integer, float, boolean, string, timestamp, or null.
Examples
Coerce to a string (Boolean)
Source:
Return:
Coerce to a string (int)
Source:
Return:
Coerce to a string (float)
Source:
Return:
Convert Functions
from_unix_timestamp
Converts the value integer from a Unix timestamp to a DPL timestamp.
Converts from the number of seconds since the Unix epoch by default. To convert from milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.
The Unix timestamp to convert.
Examples
Convert from a Unix timestamp (seconds)
Source:
Return:
Convert from a Unix timestamp (milliseconds)
Source:
from_unix_timestamp!(5000, unit: "milliseconds")
Return:
Convert from a Unix timestamp (nanoseconds)
Source:
from_unix_timestamp!(5000, unit: "nanoseconds")
Return:
"1970-01-01T00:00:00.000005Z"
to_syslog_facility
Converts the value, a Syslog facility code, into its corresponding
Syslog keyword. For example, 0 into "kern", 1 into "user", etc.
Errors
Examples
Coerce to a Syslog facility
Source:
Return:
to_syslog_level
Converts the value, a Syslog severity level, into its corresponding keyword,
i.e. 0 into "emerg", 1 into "alert", etc.
Errors
Examples
Coerce to a Syslog level
Source:
Return:
to_syslog_severity
Converts the value, a Syslog log level keyword, into a Syslog integer
severity level (0 to 7).
The Syslog level keyword to convert.
Errors
value is not a valid Syslog level keyword.
Examples
Coerce to Syslog severity
Source:
to_syslog_severity!("alert")
Return:
to_unix_timestamp
Converts the value timestamp into a Unix timestamp.
Returns the number of seconds since the Unix epoch by default. To return the number in milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.
The timestamp to convert into a Unix timestamp.
Examples
Convert to a Unix timestamp (seconds)
Source:
to_unix_timestamp(t'2021-01-01T00:00:00+00:00')
Return:
Convert to a Unix timestamp (milliseconds)
Source:
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")
Return:
Convert to a Unix timestamp (nanoseconds)
Source:
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")
Return:
Cryptography Functions
decrypt
Decrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
- AES-256-CFB (key = 32 bytes, iv = 16 bytes)
- AES-192-CFB (key = 24 bytes, iv = 16 bytes)
- AES-128-CFB (key = 16 bytes, iv = 16 bytes)
- AES-256-OFB (key = 32 bytes, iv = 16 bytes)
- AES-192-OFB (key = 24 bytes, iv = 16 bytes)
- AES-128-OFB (key = 16 bytes, iv = 16 bytes)
- AES-128-SIV (key = 32 bytes, iv = 16 bytes)
- AES-256-SIV (key = 64 bytes, iv = 16 bytes)
- Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
- Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
- Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
- CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
- XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
- XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
The string in raw bytes (not encoded) to decrypt.
The key in raw bytes (not encoded) for decryption. The length must match the algorithm requested.
The IV in raw bytes (not encoded) for decryption. The length must match the algorithm requested.
A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value.
The value should match the one used during encryption.
Errors
algorithm is not a supported algorithm.key length does not match the key size required for the algorithm specified.iv length does not match the iv size required for the algorithm specified.
Examples
Decrypt value
Source:
ciphertext = decode_base64!("5fLGcu1VHdzsPcGNDio7asLqE1P43QrVfPfmP4i4zOU=")
iv = decode_base64!("fVEIRkIiczCRWNxaarsyxA==")
key = "16_byte_keyxxxxx"
decrypt!(ciphertext, "AES-128-CBC-PKCS7", key, iv: iv)
Return:
encrypt
Encrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
- AES-256-CFB (key = 32 bytes, iv = 16 bytes)
- AES-192-CFB (key = 24 bytes, iv = 16 bytes)
- AES-128-CFB (key = 16 bytes, iv = 16 bytes)
- AES-256-OFB (key = 32 bytes, iv = 16 bytes)
- AES-192-OFB (key = 24 bytes, iv = 16 bytes)
- AES-128-OFB (key = 16 bytes, iv = 16 bytes)
- AES-128-SIV (key = 32 bytes, iv = 16 bytes)
- AES-256-SIV (key = 64 bytes, iv = 16 bytes)
- Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
- Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
- Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
- CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
- XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
- XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
The key in raw bytes (not encoded) for encryption. The length must match the algorithm requested.
The IV in raw bytes (not encoded) for encryption. The length must match the algorithm requested.
A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value.
Errors
algorithm is not a supported algorithm.key length does not match the key size required for the algorithm specified.iv length does not match the iv size required for the algorithm specified.
Examples
Encrypt value
Source:
plaintext = "super secret message"
iv = "1234567890123456" # typically you would call random_bytes(16)
key = "16_byte_keyxxxxx"
encrypted_message = encrypt!(plaintext, "AES-128-CBC-PKCS7", key, iv: iv)
encode_base64(encrypted_message)
Return:
"GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE="
hmac
Calculates a HMAC of the value using the given key.
The hashing algorithm used can be optionally specified.
For most use cases, the resulting bytestream should be encoded into a hex or base64
string using either encode_base16 or
encode_base64.
This function is infallible if either the default algorithm value or a recognized-valid compile-time
algorithm string literal is used. Otherwise, it is fallible.
The string to calculate the HMAC for.
The string to use as the cryptographic key.
The hashing algorithm to use.
Examples
Calculate message HMAC (defaults: SHA-256), encoding to a base64 string
Source:
encode_base64(hmac("Hello there", "super-secret-key"))
Return:
"eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI="
Calculate message HMAC using SHA-224, encoding to a hex-encoded string
Source:
encode_base16(hmac("Hello there", "super-secret-key", algorithm: "SHA-224"))
Return:
"42fccbc2b7d22a143b92f265a8046187558a94d11ddbb30622207e90"
Calculate message HMAC using a variable hash algorithm
Source:
.hash_algo = "SHA-256"
hmac_bytes, err = hmac("Hello there", "super-secret-key", algorithm: .hash_algo)
if err == null {
.hmac = encode_base16(hmac_bytes)
}
Return:
"78b184f1832f8aff3934f5e0212454671b2d04d494e3b25075c5e45167029662"
md5
Calculates an md5 hash of the value.
The string to calculate the hash for.
Examples
Create md5 hash
Source:
Return:
"acbd18db4cc2f85cedef654fccc4a4d8"
seahash
Calculates a Seahash hash of the value.
Note: Due to limitations in the underlying DPL data types, this function converts the unsigned 64-bit integer SeaHash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values.
The string to calculate the hash for.
Examples
Calculate seahash
Source:
Return:
Calculate negative seahash
Source:
Return:
sha1
Calculates a SHA-1 hash of the value.
The string to calculate the hash for.
Examples
Calculate sha1 hash
Source:
Return:
"0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33"
sha2
Calculates a SHA-2 hash of the value.
The string to calculate the hash for.
The variant of the algorithm to use.
Examples
Calculate sha2 hash
Source:
sha2("foo", variant: "SHA-512/224")
Return:
"d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be"
sha3
Calculates a SHA-3 hash of the value.
The string to calculate the hash for.
The variant of the algorithm to use.
Examples
Calculate sha3 hash
Source:
sha3("foo", variant: "SHA3-224")
Return:
"f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a"
Debug Functions
assert
Asserts the condition, which must be a Boolean expression. The program is aborted with
message if the condition evaluates to false.
An optional custom error message. If the equality assertion fails, message is
appended to the default message prefix. See the examples below
for a fully formed log message sample.
Errors
condition evaluates to false.
Examples
Assertion (true)
Source:
assert!("foo" == "foo", message: "\"foo\" must be \"foo\"!")
Return:
Assertion (false)
Source:
assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!")
assert_eq
Asserts that two expressions, left and right, have the same value. The program is
aborted with message if they do not have the same value.
The value to check for equality against right.
The value to check for equality against left.
An optional custom error message. If the equality assertion fails, message is
appended to the default message prefix. See the examples
below for a fully formed log message sample.
Examples
Successful assertion
Source:
Return:
Unsuccessful assertion
Source:
assert_eq!(127, [1, 2, 3])
Unsuccessful assertion with custom log message
Source:
assert_eq!(1, 0, message: "Unequal integers")
log
Logs the value to stdout at the specified level.
Specifies that the log message is output no more than once per the given number of seconds.
Use a value of 0 to turn rate limiting off.
Examples
Log a message
Source:
log("Hello, World!", level: "info", rate_limit_secs: 60)
Return:
Log an error
Source:
_, err = to_int(.field)
if err != null {
log(err, level: "error")
}
Return:
Enrichment Functions
find_enrichment_table_records
Searches an enrichment table for rows that match the
provided condition.
For file enrichment tables, this condition needs to be a DPL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields do not match, then no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity
can be specified using the case_sensitive argument. An exact match search can use an
index directly into the dataset, which should make this search fairly “cheap” from a
performance perspective.
Date range search. The given field must be greater than or equal to the from date
and less than or equal to the to date. A date range search involves
sequentially scanning through the rows that have been located using any exact match
criteria. This can be an expensive operation if there are many rows returned by any exact
match criteria. Therefore, use date ranges as the only criteria when the enrichment
data set is very small.
For geoip and mmdb enrichment tables, this condition needs to be a DPL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected
and without a value, null is used. This table can return the following fields:
To use this function, you need to update your configuration to
include an
enrichment_tables
parameter.
The enrichment table to search.
The condition to search on. Since the condition is used at boot time to create
indices into the data, these conditions must be statically defined.
A subset of fields from the enrichment table to return. If not specified,
all fields are returned.
Whether text fields need to match cases exactly.
Examples
Exact match
Source:
find_enrichment_table_records!("test",
{
"surname": "smith",
},
case_sensitive: false)
Return:
[{"id":1,"firstname":"Bob","surname":"Smith"},{"id":2,"firstname":"Fred","surname":"Smith"}]
Date range search
Source:
find_enrichment_table_records!("test",
{
"surname": "Smith",
"date_of_birth": {
"from": t'1985-01-01T00:00:00Z',
"to": t'1985-12-31T00:00:00Z'
}
})
Return:
[{"id":1,"firstname":"Bob","surname":"Smith"},{"id":2,"firstname":"Fred","surname":"Smith"}]
get_enrichment_table_record
Searches an enrichment table for a row that matches the
provided condition. A single row must be matched. If no rows are found or more than one row is
found, an error is returned.
For file enrichment tables, this condition needs to be a DPL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields do not match, then no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity
can be specified using the case_sensitive argument. An exact match search can use an
index directly into the dataset, which should make this search fairly “cheap” from a
performance perspective.
Date range search. The given field must be greater than or equal to the from date
and less than or equal to the to date. A date range search involves
sequentially scanning through the rows that have been located using any exact match
criteria. This can be an expensive operation if there are many rows returned by any exact
match criteria. Therefore, use date ranges as the only criteria when the enrichment
data set is very small.
For geoip and mmdb enrichment tables, this condition needs to be a DPL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected
and without a value, null is used. This table can return the following fields:
To use this function, you need to update your configuration to
include an
enrichment_tables
parameter.
The enrichment table to search.
The condition to search on. Since the condition is used at boot time to create
indices into the data, these conditions must be statically defined.
A subset of fields from the enrichment table to return. If not specified,
all fields are returned.
Whether the text fields match the case exactly.
Errors
- The row is not found.
- Multiple rows are found that match the condition.
Examples
Exact match
Source:
get_enrichment_table_record!("test",
{
"surname": "bob",
"firstname": "John"
},
case_sensitive: false)
Return:
{"id":1,"firstname":"Bob","surname":"Smith"}
Date range search
Source:
get_enrichment_table_record!("test",
{
"surname": "Smith",
"date_of_birth": {
"from": t'1985-01-01T00:00:00Z',
"to": t'1985-12-31T00:00:00Z'
}
})
Return:
{"id":1,"firstname":"Bob","surname":"Smith"}
Enumerate Functions
compact
Compacts the value by removing empty values, where empty values are defined using the
available parameters.
The object or array to compact.
Whether the compaction be recursive.
Whether null should be treated as an empty value.
Whether an empty string should be treated as an empty value.
Whether an empty object should be treated as an empty value.
Whether an empty array should be treated as an empty value.
Tests whether the value is “nullish” as defined by the is_nullish function.
Examples
Compact an array
Source:
compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true)
Return:
Compact an object
Source:
compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true)
Return:
filter
Filter elements from a collection.
This function currently does not support recursive iteration.
The function uses the function closure syntax to allow reading
the key-value or index-value combination for each item in the
collection.
The same scoping rules apply to closure blocks as they do for
regular blocks. This means that any variable defined in parent scopes
is accessible, and mutations to those variables are preserved,
but any new variables instantiated in the closure block are
unavailable outside of the block.
See the examples below to learn about the closure syntax.
The array or object to filter.
Examples
Filter elements
Source:
filter(array!(.tags)) -> |_index, value| {
# keep any elements that aren't equal to "foo"
value != "foo"
}
Return:
flatten
Flattens the value into a single-level representation.
The array or object to flatten.
The separator to join nested keys
Examples
Flatten array
Source:
flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])
Return:
Flatten object
Source:
flatten({
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
})
Return:
{"parent1.child1":1,"parent1.child2":2,"parent2.child3":3}
for_each
Iterate over a collection.
This function currently does not support recursive iteration.
The function uses the “function closure syntax” to allow reading
the key/value or index/value combination for each item in the
collection.
The same scoping rules apply to closure blocks as they do for
regular blocks. This means that any variable defined in parent scopes
is accessible, and mutations to those variables are preserved,
but any new variables instantiated in the closure block are
unavailable outside of the block.
See the examples below to learn about the closure syntax.
The array or object to iterate.
Examples
Tally elements
Source:
tally = {}
for_each(array!(.tags)) -> |_index, value| {
# Get the current tally for the `value`, or
# set to `0`.
count = int(get!(tally, [value])) ?? 0
# Increment the tally for the value by `1`.
tally = set!(tally, [value], count + 1)
}
tally
Return:
{"foo":2,"bar":1,"baz":1}
includes
Determines whether the value array includes the specified item.
Examples
Array includes
Source:
includes(["apple", "orange", "banana"], "banana")
Return:
keys
Returns the keys from the object passed into the function.
The object to extract keys from.
Examples
Get keys from the object
Source:
keys({"key1": "val1", "key2": "val2"})
Return:
length
Returns the length of the value.
- If
value is an array, returns the number of elements. - If
value is an object, returns the number of top-level keys. - If
value is a string, returns the number of bytes in the string. If
you want the number of characters, see strlen.
Examples
Length (object)
Source:
length({
"portland": "Trail Blazers",
"seattle": "Supersonics"
})
Return:
Length (nested object)
Source:
length({
"home": {
"city": "Portland",
"state": "Oregon"
},
"name": "Trail Blazers",
"mascot": {
"name": "Blaze the Trail Cat"
}
})
Return:
Length (array)
Source:
length(["Trail Blazers", "Supersonics", "Grizzlies"])
Return:
Length (string)
Source:
length("The Planet of the Apes Musical")
Return:
map_keys
Map the keys within an object.
If recursive is enabled, the function iterates into nested
objects, using the following rules:
- Iteration starts at the root.
- For every nested object type:
- First return the key of the object type itself.
- Then recurse into the object, and loop back to item (1)
in this list.
- Any mutation done on a nested object before recursing into
it, are preserved.
- For every nested array type:
- First return the key of the array type itself.
- Then find all objects within the array, and apply item (2)
to each individual object.
The above rules mean that map_keys with
recursive enabled finds all keys in the target,
regardless of whether nested objects are nested inside arrays.
The function uses the function closure syntax to allow reading
the key for each item in the object.
The same scoping rules apply to closure blocks as they do for
regular blocks. This means that any variable defined in parent scopes
is accessible, and mutations to those variables are preserved,
but any new variables instantiated in the closure block are
unavailable outside of the block.
See the examples below to learn about the closure syntax.
Whether to recursively iterate the collection.
Examples
Upcase keys
Source:
map_keys(.) -> |key| { upcase(key) }
Return:
{"FOO":"foo","BAR":"bar"}
De-dot keys
Source:
map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }
Return:
{"labels":{"app_kubernetes_io/name":"mysql"}}
map_values
Map the values within a collection.
If recursive is enabled, the function iterates into nested
collections, using the following rules:
- Iteration starts at the root.
- For every nested collection type:
- First return the collection type itself.
- Then recurse into the collection, and loop back to item (1)
in the list
- Any mutation done on a collection before recursing into it,
are preserved.
The function uses the function closure syntax to allow mutating
the value for each item in the collection.
The same scoping rules apply to closure blocks as they do for
regular blocks, meaning, any variable defined in parent scopes
are accessible, and mutations to those variables are preserved,
but any new variables instantiated in the closure block are
unavailable outside of the block.
Check out the examples below to learn about the closure syntax.
The object or array to iterate.
Whether to recursively iterate the collection.
Examples
Upcase values
Source:
map_values(.) -> |value| { upcase!(value) }
Return:
{"foo":"FOO","bar":"BAR"}
match_array
Determines whether the elements in the value array matches the pattern. By default, it checks that at least one element matches, but can be set to determine if all the elements match.
The regular expression pattern to match against.
Whether to match on all elements of value.
Examples
Match at least one element
Source:
match_array(["foobar", "bazqux"], r'foo')
Return:
Match all elements
Source:
match_array(["foo", "foobar", "barfoo"], r'foo', all: true)
Return:
No matches
Source:
match_array(["bazqux", "xyz"], r'foo')
Return:
Not all elements match
Source:
match_array(["foo", "foobar", "baz"], r'foo', all: true)
Return:
strlen
Returns the number of UTF-8 characters in value. This differs from
length which counts the number of bytes of a string.
Note: This is the count of Unicode scalar values
which can sometimes differ from Unicode code points.
Examples
unflatten
Unflattens the value into a nested representation.
The array or object to unflatten.
The separator to split flattened keys.
Whether to recursively unflatten the object values.
Examples
Unflatten
Source:
unflatten({
"foo.bar.baz": true,
"foo.bar.qux": false,
"foo.quux": 42
})
Return:
{"foo":{"bar":{"baz":true,"qux":false},"quux":42}}
Unflatten recursively
Source:
unflatten({
"flattened.parent": {
"foo.bar": true,
"foo.baz": false
}
})
Return:
{"flattened":{"parent":{"foo":{"bar":true,"baz":false}}}}
Unflatten non-recursively
Source:
unflatten({
"flattened.parent": {
"foo.bar": true,
"foo.baz": false
}
}, recursive: false)
Return:
{"flattened":{"parent":{"foo.bar":true,"foo.baz":false}}}
Ignore inconsistent keys values
Source:
unflatten({
"a": 3,
"a.b": 2,
"a.c": 4
})
Return:
unique
Returns the unique values for an array.
The first occurrence of each element is kept.
The array to return unique elements from.
Examples
Unique
Source:
unique(["foo", "bar", "foo", "baz"])
Return:
values
Returns the values from the object passed into the function.
The object to extract values from.
Examples
Get values from the object
Source:
values({"key1": "val1", "key2": "val2"})
Return:
Event Functions
get_secret
Returns the value of the given secret from an event.
Examples
Source:
get_secret("datadog_api_key")
Return:
remove_secret
Removes a secret from an event.
The name of the secret to remove.
Examples
Removes the Datadog API key from the event
Source:
remove_secret("datadog_api_key")
Return:
set_secret
Sets the given secret in the event.
Examples
Set the Datadog API key to the given value
Source:
set_secret("datadog_api_key", "abc122")
Return:
set_semantic_meaning
Sets a semantic meaning for an event. Note: This function assigns
meaning at startup, and has no runtime behavior. It is suggested
to put all calls to this function at the beginning of a DPL function. The function
cannot be conditionally called. For example, using an if statement cannot stop the meaning
from being assigned.
The path of the value that is assigned a meaning.
The name of the meaning to assign.
Examples
Sets custom field semantic meaning
Source:
set_semantic_meaning(.foo, "bar")
Return:
IP Functions
ip_aton
Converts IPv4 address in numbers-and-dots notation into network-order
bytes represented as an integer.
This behavior mimics inet_aton.
The IP address to convert to binary.
Errors
value is not a valid IPv4 address.
Examples
IPv4 to integer
Source:
Return:
ip_cidr_contains
Determines whether the ip is contained in the block referenced by the cidr.
The CIDR mask (v4 or v6).
The IP address (v4 or v6).
Errors
cidr is not a valid CIDR.ip is not a valid IP address.
Examples
IPv4 contains CIDR
Source:
ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")
Return:
IPv6 contains CIDR
Source:
ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")
Return:
ip_ntoa
Converts numeric representation of IPv4 address in network-order bytes
to numbers-and-dots notation.
This behavior mimics inet_ntoa.
The integer representation of an IPv4 address.
Errors
value cannot fit in an unsigned 32-bit integer.
Examples
Integer to IPv4
Source:
Return:
ip_ntop
Converts IPv4 and IPv6 addresses from binary to text form.
This behavior mimics inet_ntop.
The binary data to convert from.
For IPv4 addresses, it must be 4 bytes (32 bits) long.
For IPv6 addresses, it must be 16 bytes (128 bits) long.
Errors
value must be of length 4 or 16 bytes.
Examples
Convert IPv4 address from bytes after decoding from Base64
Source:
ip_ntop!(decode_base64!("wKgAAQ=="))
Return:
Convert IPv6 address from bytes after decoding from Base64
Source:
ip_ntop!(decode_base64!("IAENuIWjAAAAAIouA3BzNA=="))
Return:
"2001:db8:85a3::8a2e:370:7334"
ip_pton
Converts IPv4 and IPv6 addresses from text to binary form.
- The binary form of IPv4 addresses is 4 bytes (32 bits) long.
- The binary form of IPv6 addresses is 16 bytes (128 bits) long.
This behavior mimics inet_pton.
The IP address (v4 or v6) to convert to binary form.
Errors
value is not a valid IP (v4 or v6) address in text form.
Examples
Convert IPv4 address to bytes and encode to Base64
Source:
encode_base64(ip_pton!("192.168.0.1"))
Return:
Convert IPv6 address to bytes and encode to Base64
Source:
encode_base64(ip_pton!("2001:db8:85a3::8a2e:370:7334"))
Return:
"IAENuIWjAAAAAIouA3BzNA=="
ip_subnet
Extracts the subnet address from the ip using the supplied subnet.
The IP address (v4 or v6).
The subnet to extract from the IP address. This can be either a prefix length like /8 or a net mask
like 255.255.0.0. The net mask can be either an IPv4 or IPv6 address.
Errors
ip is not a valid IP address.subnet is not a valid subnet.
Examples
IPv4 subnet
Source:
ip_subnet!("192.168.10.32", "255.255.255.0")
Return:
IPv6 subnet
Source:
ip_subnet!("2404:6800:4003:c02::64", "/32")
Return:
ip_to_ipv6
Converts the ip to an IPv6 address.
The IP address to convert to IPv6.
Errors
ip is not a valid IP address.
Examples
IPv4 to IPv6
Source:
ip_to_ipv6!("192.168.10.32")
Return:
ipv6_to_ipv4
Converts the ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is
currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.
The IPv4-mapped IPv6 address to convert.
Errors
ip is not a valid IP address.ip is an IPv6 address that is not compatible with IPv4.
Examples
IPv6 to IPv4
Source:
ipv6_to_ipv4!("::ffff:192.168.0.1")
Return:
is_ipv4
Check if the string is a valid IPv4 address or not.
An IPv4-mapped or
IPv4-compatible IPv6 address is not considered
valid for the purpose of this function.
Examples
Valid IPv4 address
Source:
Return:
Valid IPv6 address
Source:
is_ipv4("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Return:
Arbitrary string
Source:
Return:
is_ipv6
Check if the string is a valid IPv6 address or not.
Examples
Valid IPv6 address
Source:
is_ipv6("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Return:
Valid IPv4 address
Source:
Return:
Arbitrary string
Source:
Return:
Number Functions
abs
Computes the absolute value of value.
The number to calculate the absolute value.
Examples
Computes the absolute value of the integer
Source:
Return:
Computes the absolute value of the float
Source:
Return:
ceil
Rounds the value up to the specified precision.
The number of decimal places to round to.
Examples
Round a number up (without precision)
Source:
Return:
Round a number up (with precision)
Source:
ceil(4.345, precision: 2)
Return:
floor
Rounds the value down to the specified precision.
The number to round down.
The number of decimal places to round to.
Examples
Round a number down (without precision)
Source:
Return:
Round a number down (with precision)
Source:
floor(4.345, precision: 2)
Return:
Formats the integer value into a string representation using the given base/radix.
The base to format the number in. Must be between 2 and 36 (inclusive).
- The base is not between 2 and 36.
Source:
Return:
Source:
Return:
Formats the value into a string representation of the number.
The number to format as a string.
The number of decimal places to display.
The character to use between the whole and decimal parts of the number.
The character to use between each thousands part of the number.
Source:
format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")
Return:
mod
Calculates the remainder of value divided by modulus.
The value the modulus is applied to.
Errors
value is not an integer or float.modulus is not an integer or float.modulus is equal to 0.
Examples
Calculate the remainder of two integers
Source:
Return:
round
Rounds the value to the specified precision.
The number of decimal places to round to.
Examples
Round a number (without precision)
Source:
Return:
Round a number (with precision)
Source:
round(4.345, precision: 2)
Return:
Object Functions
match_datadog_query
Matches an object against a Datadog Search Syntax query.
The Datadog Search Syntax query.
Examples
OR query
Source:
match_datadog_query({"message": "contains this and that"}, "this OR that")
Return:
AND query
Source:
match_datadog_query({"message": "contains only this"}, "this AND that")
Return:
Attribute wildcard
Source:
match_datadog_query({"name": "foobar"}, "@name:foo*")
Return:
Tag range
Source:
match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]')
Return:
merge
Merges the from object into the to object.
The object to merge into.
The object to merge from.
A deep merge is performed if true, otherwise only top-level fields are merged.
Examples
Object merge (shallow)
Source:
merge(
{
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
},
{
"parent1": {
"child2": 4,
"child5": 5
}
}
)
Return:
{"parent1":{"child2":4,"child5":5},"parent2":{"child3":3}}
Object merge (deep)
Source:
merge(
{
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
},
{
"parent1": {
"child2": 4,
"child5": 5
}
},
deep: true
)
Return:
{"parent1":{"child1":1,"child2":4,"child5":5},"parent2":{"child3":3}}
unnest
Unnest an array field from an object to create an array of objects using that field; keeping all other fields.
Assigning the array result of this to . results in multiple events being emitted from remap. See the
remap transform docs for more details.
This is also referred to as explode in some languages.
The path of the field to unnest.
Errors
- The field path referred to is not an array.
Examples
Unnest an array field
Source:
Unnest nested an array field
Source:
. = unnest!(.event.messages)
Parse Functions
parse_apache_log
Parses Apache access and error log lines. Lines can be in common,
combined, or the default error format.
The date/time format to use for
encoding the timestamp. The time is parsed in local time if the timestamp does not specify a timezone.
The format to use for parsing the log.
Errors
value does not match the specified format.timestamp_format is not a valid format string.- The timestamp in
value fails to parse using the provided timestamp_format.
Examples
Source:
parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}
Source:
parse_apache_log!(
s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
"combined",
)
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326,"referrer":"http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth","agent":"Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"}
Source:
parse_apache_log!(
s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
"error"
)
Return:
{"client":"147.159.108.175","message":"I will bypass the haptic COM bandwidth, that should matrix the CSS driver!","module":"ab","pid":4803,"port":24259,"severity":"alert","thread":"3814","timestamp":"2021-03-01T12:00:19Z"}
parse_aws_alb_log
Parses value in the Elastic Load Balancer Access format.
Access log of the Application Load Balancer.
Errors
value is not a properly formatted AWS ALB log.
Examples
Parse AWS ALB log
Source:
parse_aws_alb_log!(
"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)
Return:
{"type":"http","timestamp":"2018-11-30T22:23:00.186641Z","elb":"app/my-loadbalancer/50dc6c495c0c9188","client_host":"192.168.131.39:2817","target_host":null,"request_processing_time":0,"target_processing_time":0.001,"response_processing_time":0,"elb_status_code":"200","target_status_code":"200","received_bytes":34,"sent_bytes":366,"request_method":"GET","request_url":"http://www.example.com:80/","request_protocol":"HTTP/1.1","user_agent":"curl/7.46.0","ssl_cipher":null,"ssl_protocol":null,"target_group_arn":"arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067","trace_id":"Root=1-58337364-23a8c76965a2ef7629b185e3","traceability_id":null,"domain_name":null,"chosen_cert_arn":null,"matched_rule_priority":"0","request_creation_time":"2018-11-30T22:22:48.364000Z","actions_executed":"forward","redirect_url":null,"error_reason":null,"target_port_list":[],"target_status_code_list":[],"classification":null,"classification_reason":null}
parse_aws_cloudwatch_log_subscription_message
Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the
aws_kinesis_firehose source.
The string representation of the message to parse.
Errors
value is not a properly formatted AWS CloudWatch Log subscription message.
Examples
Parse AWS Cloudwatch Log subscription message
Source:
parse_aws_cloudwatch_log_subscription_message!(.message)
Return:
{"owner":"111111111111","message_type":"DATA_MESSAGE","log_group":"test","log_stream":"test","subscription_filters":["Destination"],"log_events":[{"id":"35683658089614582423604394983260738922885519999578275840","message":"{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}","timestamp":"2020-09-14T19:09:29.039Z"}]}
parse_aws_vpc_flow_log
Parses value in the VPC Flow Logs format.
Errors
value is not a properly formatted AWS VPC Flow log.
Examples
Source:
parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")
Return:
{"version":2,"account_id":"123456789010","interface_id":"eni-1235b8ca123456789","srcaddr":null,"dstaddr":null,"srcport":null,"dstport":null,"protocol":null,"packets":null,"bytes":null,"start":1431280876,"end":1431280934,"action":null,"log_status":"NODATA"}
Source:
parse_aws_vpc_flow_log!(
"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)
Return:
{"instance_id":null,"interface_id":"eni-1235b8ca123456789","srcaddr":"10.0.1.5","dstaddr":"10.0.0.220","pkt_srcaddr":"10.0.1.5","pkt_dstaddr":"203.0.113.5"}
Parse AWS VPC Flow log including v5 fields
Source:
parse_aws_vpc_flow_log!("5 52.95.128.179 10.0.0.71 80 34210 6 1616729292 1616729349 IPv4 14 15044 123456789012 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-0c50d5961bcb2d47b eni-1235b8ca123456789 ap-southeast-2 apse2-az3 - - ACCEPT 19 52.95.128.179 10.0.0.71 S3 - - ingress OK",
format: "version srcaddr dstaddr srcport dstport protocol start end type packets bytes account_id vpc_id subnet_id instance_id interface_id region az_id sublocation_type sublocation_id action tcp_flags pkt_srcaddr pkt_dstaddr pkt_src_aws_service pkt_dst_aws_service traffic_path flow_direction log_status")
Return:
{"account_id":"123456789012","action":"ACCEPT","az_id":"apse2-az3","bytes":15044,"dstaddr":"10.0.0.71","dstport":34210,"end":1616729349,"flow_direction":"ingress","instance_id":"i-0c50d5961bcb2d47b","interface_id":"eni-1235b8ca123456789","log_status":"OK","packets":14,"pkt_dst_aws_service":null,"pkt_dstaddr":"10.0.0.71","pkt_src_aws_service":"S3","pkt_srcaddr":"52.95.128.179","protocol":6,"region":"ap-southeast-2","srcaddr":"52.95.128.179","srcport":80,"start":1616729292,"sublocation_id":null,"sublocation_type":null,"subnet_id":"subnet-aaaaaaaa012345678","tcp_flags":19,"traffic_path":null,"type":"IPv4","version":5,"vpc_id":"vpc-abcdefab012345678"}
parse_cef
Parses the value in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.
Toggles translation of custom field pairs to key:value.
Errors
value is not a properly formatted CEF string.
Examples
Parse output generated by PTA
Source:
parse_cef!(
"CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None"
)
Return:
{"cefVersion":"0","deviceVendor":"CyberArk","deviceProduct":"PTA","deviceVersion":"12.6","deviceEventClassId":"1","name":"Suspected credentials theft","severity":"8","suser":"mike2@prod1.domain.com","shost":"prod1.domain.com","src":"1.1.1.1","duser":"andy@dev1.domain.com","dhost":"dev1.domain.com","dst":"2.2.2.2","cs1Label":"ExtraData","cs1":"None","cs2Label":"EventID","cs2":"52b06812ec3500ed864c461e","deviceCustomDate1Label":"detectionDate","deviceCustomDate1":"1388577900000","cs3Label":"PTAlink","cs3":"https://1.1.1.1/incidents/52b06812ec3500ed864c461e","cs4Label":"ExternalLink","cs4":"None"}
Source:
parse_cef!(
"Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232"
)
Return:
{"cefVersion":"1","deviceVendor":"Security","deviceProduct":"threatmanager","deviceVersion":"1.0","deviceEventClassId":"100","name":"worm successfully stopped","severity":"10","src":"10.0.0.1","dst":"2.1.2.2","spt":"1232"}
Translate custom fields
Source:
parse_cef!(
"CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address",
translate_custom_fields: true
)
Return:
{"cefVersion":"0","deviceVendor":"Dev","deviceProduct":"firewall","deviceVersion":"2.2","deviceEventClassId":"1","name":"Connection denied","severity":"5","Device IPv6 Address":"2345:0425:2CA1:0000:0000:0567:5673:23b5"}
parse_common_log
Parses the value using the Common Log Format (CLF).
Errors
value does not match the Common Log Format.timestamp_format is not a valid format string.- The timestamp in
value fails to parse using the provided timestamp_format.
Examples
Source:
parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}
Source:
parse_common_log!(
"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
"%+"
)
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}
parse_csv
Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.
The field delimiter to use when parsing. Must be a single-byte utf8 character.
Errors
- The delimiter must be a single-byte UTF-8 character.
value is not a valid CSV string.
Examples
Source:
parse_csv!("foo,bar,\"foo \"\", bar\"")
Return:
["foo","bar","foo \", bar"]
Source:
parse_csv!("foo bar", delimiter: " ")
Return:
parse_duration
Parses the value into a human-readable duration format specified by unit.
The string of the duration.
The output units for the duration.
Errors
value is not a properly formatted duration.
Examples
Parse duration (milliseconds)
Source:
parse_duration!("1005ms", unit: "s")
Return:
parse_etld
Parses the eTLD from value representing domain name.
Can be provided to get additional parts of the domain name. When 1 is passed,
eTLD+1 will be returned, which represents a domain registrable by a single
organization. Higher numbers will return subdomains.
Errors
- unable to determine eTLD for
value
Examples
Parse eTLD
Source:
parse_etld!("sub.sussex.ac.uk")
Return:
{"etld":"ac.uk","etld_plus":"ac.uk","known_suffix":true}
Parse eTLD+1
Source:
parse_etld!("sub.sussex.ac.uk", plus_parts: 1)
Return:
{"etld":"ac.uk","etld_plus":"sussex.ac.uk","known_suffix":true}
Parse eTLD with unknown suffix
Source:
parse_etld!("vector.acmecorp")
Return:
{"etld":"acmecorp","etld_plus":"acmecorp","known_suffix":false}
Parse eTLD with custom PSL
Source:
parse_etld!("vector.acmecorp", psl: "resources/public_suffix_list.dat")
Return:
{"etld":"acmecorp","etld_plus":"acmecorp","known_suffix":false}
parse_glog
Parses the value using the glog (Google Logging Library) format.
Errors
value does not match the glog format.
Examples
Parse using glog
Source:
parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")
Return:
{"level":"info","timestamp":"2021-01-31T14:48:54.411655Z","id":15520,"file":"main.c++","line":9,"message":"Hello world!"}
parse_grok
Parses the value using the grok format. All patterns listed here
are supported.
Errors
value fails to parse using the provided pattern.
Examples
Parse using Grok
Source:
parse_grok!(
"2020-10-02T23:22:12.223222Z info Hello world",
"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
)
Return:
{"timestamp":"2020-10-02T23:22:12.223222Z","level":"info","message":"Hello world"}
parse_groks
Parses the value using multiple grok patterns. All patterns listed here
are supported.
The Grok patterns, which are tried in order until the first match.
The shared set of grok aliases that can be referenced in the patterns to simplify them.
Errors
value fails to parse using the provided pattern.
Examples
Parse using multiple Grok patterns
Source:
parse_groks!(
"2020-10-02T23:22:12.223222Z info Hello world",
patterns: [
"%{common_prefix} %{_status} %{_message}",
"%{common_prefix} %{_message}",
],
aliases: {
"common_prefix": "%{_timestamp} %{_loglevel}",
"_timestamp": "%{TIMESTAMP_ISO8601:timestamp}",
"_loglevel": "%{LOGLEVEL:level}",
"_status": "%{POSINT:status}",
"_message": "%{GREEDYDATA:message}"
}
)
Return:
{"timestamp":"2020-10-02T23:22:12.223222Z","level":"info","message":"Hello world"}
parse_influxdb
Parses the value as an InfluxDB line protocol
string, producing a list of Vector-compatible metrics.
The string representation of the InfluxDB line protocol to parse.
Errors
value is not a valid InfluxDB line protocol string.- field set contains a field value of type
string. - field set contains a
NaN field value.
Examples
Parse InfluxDB line protocol
Source:
parse_influxdb!("cpu,host=A,region=us-west usage_system=64i,usage_user=10u,temperature=50.5,on=true,sleep=false 1590488773254420000")
Return:
[{"name":"cpu_usage_system","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":64}},{"name":"cpu_usage_user","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":10}},{"name":"cpu_temperature","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":50.5}},{"name":"cpu_on","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":1}},{"name":"cpu_sleep","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":0}}]
parse_int
Parses the string value representing a number in an optional base/radix to an integer.
The base the number is in. Must be between 2 and 36 (inclusive).
If unspecified, the string prefix is used to
determine the base: “0b”, 8 for “0” or “0o”, 16 for “0x”,
and 10 otherwise.
Errors
- The base is not between 2 and 36.
- The number cannot be parsed in the base.
Examples
Parse decimal
Source:
Return:
Parse binary
Source:
Return:
Parse octal
Source:
Return:
Parse hexadecimal
Source:
Return:
Parse explicit base
Source:
Return:
parse_json
Parses the value as JSON.
The string representation of the JSON to parse.
Number of layers to parse for nested JSON-formatted documents.
The value must be in the range of 1 to 128.
Whether to parse the JSON in a lossy manner. Replaces invalid UTF-8 characters
with the Unicode character � (U+FFFD) if set to true, otherwise returns an error
if there are any invalid UTF-8 characters present.
Errors
value is not a valid JSON-formatted payload.
Examples
Parse JSON
Source:
parse_json!("{\"key\": \"val\"}")
Return:
Parse JSON with max_depth
Source:
parse_json!("{\"top_level\":{\"key\": \"val\"}}", max_depth: 1)
Return:
{"top_level":"{\"key\": \"val\"}"}
parse_key_value
Parses the value in key-value format. Also known as logfmt.
- Keys and values can be wrapped with
". " characters can be escaped using \.
The string that separates the key from the value.
The string that separates each key-value pair.
Defines the acceptance of unnecessary whitespace surrounding the configured key_value_delimiter.
Whether a standalone key should be accepted, the resulting object associates such keys with the boolean value true.
Errors
value is not a properly formatted key-value string.
Examples
Parse logfmt log
Source:
parse_key_value!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return:
{"@timestamp":"Sun Jan 10 16:47:39 EST 2021","level":"info","msg":"Stopping all fetchers","tag#production":"stopping_fetchers","id":"ConsumerFetcherManager-1382721708341","module":"kafka.consumer.ConsumerFetcherManager"}
Parse comma delimited log
Source:
parse_key_value!(
"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
field_delimiter: ",",
key_value_delimiter: ":"
)
Return:
{"path":"/cart_link","host":"store.app.com","fwd":"102.30.171.16","dyno":"web.1","connect":"0ms","service":"87ms","status":"304","bytes":"632","protocol":"https"}
Parse comma delimited log with standalone keys
Source:
parse_key_value!(
"env:prod,service:backend,region:eu-east1,beta",
field_delimiter: ",",
key_value_delimiter: ":",
)
Return:
{"env":"prod","service":"backend","region":"eu-east1","beta":true}
Parse duplicate keys
Source:
parse_key_value!(
"at=info,method=GET,path=\"/index\",status=200,tags=dev,tags=dummy",
field_delimiter: ",",
key_value_delimiter: "=",
)
Return:
{"at":"info","method":"GET","path":"/index","status":"200","tags":["dev","dummy"]}
parse_klog
Parses the value using the klog format used by Kubernetes components.
Errors
value does not match the klog format.
Examples
Parse using klog
Source:
parse_klog!("I0505 17:59:40.692994 28133 klog.go:70] hello from klog")
Return:
{"file":"klog.go","id":28133,"level":"info","line":70,"message":"hello from klog","timestamp":"2024-05-05T17:59:40.692994Z"}
parse_linux_authorization
Parses Linux authorization logs usually found under either /var/log/auth.log (for Debian-based systems) or
/var/log/secure (for RedHat-based systems) according to Syslog format.
The text containing the message to parse.
Errors
value is not a properly formatted Syslog message.
Examples
Parse Linux authorization event
Source:
parse_linux_authorization!(
s'Mar 23 2023 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)
Return:
{"appname":"sshd","hostname":"localhost","message":"Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar","procid":1111,"timestamp":"2023-03-23T01:49:58Z"}
parse_logfmt
Parses the value in logfmt.
- Keys and values can be wrapped using the
" character. " characters can be escaped by the \ character.- As per this logfmt specification, the
parse_logfmt function
accepts standalone keys and assigns them a Boolean value of true.
Errors
value is not a properly formatted key-value string
Examples
Parse logfmt log
Source:
parse_logfmt!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return:
{"@timestamp":"Sun Jan 10 16:47:39 EST 2021","level":"info","msg":"Stopping all fetchers","tag#production":"stopping_fetchers","id":"ConsumerFetcherManager-1382721708341","module":"kafka.consumer.ConsumerFetcherManager"}
parse_nginx_log
Parses Nginx access and error log lines. Lines can be in combined,
ingress_upstreaminfo, or error format.
The date/time format to use for encoding the timestamp. The time is parsed
in local time if the timestamp doesn’t specify a timezone. The default format is %d/%b/%Y:%T %z for
combined logs and %Y/%m/%d %H:%M:%S for error logs.
The format to use for parsing the log.
Errors
value does not match the specified format.timestamp_format is not a valid format string.- The timestamp in
value fails to parse using the provided timestamp_format.
Examples
Source:
parse_nginx_log!(
s'172.17.0.1 - alice [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
"combined",
)
Return:
{"agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36","client":"172.17.0.1","compression":"2.75","referer":"http://localhost/somewhere","request":"POST /not-found HTTP/1.1","size":153,"status":404,"timestamp":"2021-04-01T12:02:31Z","user":"alice"}
Source:
parse_nginx_log!(
s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
"error"
)
Return:
{"timestamp":"2021-04-01T13:02:31Z","severity":"error","pid":31,"tid":31,"cid":1,"message":"open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)","client":"172.17.0.1","server":"localhost","request":"POST /not-found HTTP/1.1","host":"localhost:8081"}
Source:
parse_nginx_log!(
s'0.0.0.0 - bob [18/Mar/2023:15:00:00 +0000] "GET /some/path HTTP/2.0" 200 12312 "https://10.0.0.1/some/referer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 462 0.050 [some-upstream-service-9000] [some-other-upstream-5000] 10.0.50.80:9000 19437 0.049 200 752178adb17130b291aefd8c386279e7',
"ingress_upstreaminfo"
)
Return:
{"body_bytes_size":12312,"http_referer":"https://10.0.0.1/some/referer","http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36","proxy_alternative_upstream_name":"some-other-upstream-5000","proxy_upstream_name":"some-upstream-service-9000","remote_addr":"0.0.0.0","remote_user":"bob","req_id":"752178adb17130b291aefd8c386279e7","request":"GET /some/path HTTP/2.0","request_length":462,"request_time":0.05,"status":200,"timestamp":"2023-03-18T15:00:00Z","upstream_addr":"10.0.50.80:9000","upstream_response_length":19437,"upstream_response_time":0.049,"upstream_status":200}
parse_proto
Parses the value as a protocol buffer payload.
The protocol buffer payload to parse.
The path to the protobuf descriptor set file. Must be a literal string.
This file is the output of protoc -o …
The name of the message type to use for serializing.
Must be a literal string.
Errors
value is not a valid proto payload.desc_file file does not exist.message_type message type does not exist in the descriptor file.
Examples
Parse proto
Source:
parse_proto!(decode_base64!("Cgdzb21lb25lIggKBjEyMzQ1Ng=="), "resources/protobuf_descriptor_set.desc", "test_protobuf.Person")
Return:
{"name":"someone","phones":[{"number":"123456"}]}
parse_query_string
Parses the value as a query string.
Examples
Parse query string
Source:
parse_query_string("foo=%2B1&bar=2&bar=3&xyz")
Return:
{"foo":"+1","bar":["2","3"],"xyz":""}
Parse Ruby on Rails’ query string
Source:
parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")
Return:
parse_regex
Parses the value using the provided Regex pattern.
This function differs from the parse_regex_all function in that it returns only the first match.
The regular expression pattern to search against.
If true, the index of each group in the regular expression is also captured. Index 0
contains the whole match.
Errors
value fails to parse using the provided pattern.
Examples
Parse using Regex (with capture groups)
Source:
parse_regex!("first group and second group.", r'(?P<number>.*?) group')
Return:
Parse using Regex (without capture groups)
Source:
parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)
Return:
{"0":"first group","1":"first"}
parse_regex_all
Parses the value using the provided Regex pattern.
This function differs from the parse_regex function in that it returns all matches, not just the first.
The regular expression pattern to search against.
If true, the index of each group in the regular expression is also captured. Index 0
contains the whole match.
Errors
value fails to parse using the provided pattern.
Examples
Parse using Regex (all matches)
Source:
parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)
Return:
[{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}]
parse_ruby_hash
Parses the value as ruby hash.
The string representation of the ruby hash to parse.
Errors
value is not a valid ruby hash formatted payload.
Examples
Parse ruby hash
Source:
parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }')
Return:
{"test":"value","testNum":0.2,"testObj":{"testBool":true,"testNull":null}}
parse_syslog
Parses the value in Syslog format.
The text containing the Syslog message to parse.
Errors
value is not a properly formatted Syslog message.
Examples
Parse Syslog log (5424)
Source:
parse_syslog!(
s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!'
)
Return:
{"severity":"notice","facility":"user","timestamp":"2020-03-13T20:45:38.119Z","hostname":"dynamicwireless.name","appname":"non","procid":2426,"msgid":"ID931","message":"Try to override the THX port, maybe it will reboot the neural interface!","exampleSDID@32473":{"eventID":"1011","eventSource":"Application","iut":"3"},"version":1}
parse_timestamp
Parses the value in strptime format.
The text of the timestamp.
Errors
value fails to parse using the provided format.
Examples
Parse timestamp
Source:
parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z")
Return:
parse_tokens
Parses the value in token format. A token is considered to be one of the following:
- A word surrounded by whitespace.
- Text delimited by double quotes:
"..". Quotes can be included in the token if they are escaped by a backslash (\). - Text delimited by square brackets:
[..]. Closing square brackets can be included in the token if they are escaped by a backslash (\).
Errors
value is not a properly formatted tokenized string.
Examples
Parse tokens
Source:
parse_tokens(
"A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]"
)
Return:
["A","sentence","with \\\"a\\\" sentence inside","and","some brackets"]
parse_url
Parses the value in URL format.
If true and the port number is not specified in the input URL
string (or matches the default port for the scheme), it is
populated from well-known ports for the following schemes:
http, https, ws, wss, and ftp.
Errors
value is not a properly formatted URL.
Examples
Parse URL
Source:
parse_url!("ftp://foo:bar@example.com:4343/foobar?hello=world#123")
Return:
{"scheme":"ftp","username":"foo","password":"bar","host":"example.com","port":4343,"path":"/foobar","query":{"hello":"world"},"fragment":"123"}
Parse URL with default port
Source:
parse_url!("https://example.com", default_known_ports: true)
Return:
{"scheme":"https","username":"","password":"","host":"example.com","port":443,"path":"/","query":{},"fragment":null}
Parse URL with internationalized domain name
Source:
parse_url!("https://www.café.com")
Return:
{"scheme":"https","username":"","password":"","host":"www.xn--caf-dma.com","port":null,"path":"/","query":{},"fragment":null}
Parse URL with mixed case internationalized domain name
Source:
parse_url!("https://www.CAFé.com")
Return:
{"scheme":"https","username":"","password":"","host":"www.xn--caf-dma.com","port":null,"path":"/","query":{},"fragment":null}
parse_user_agent
Parses the value as a user agent string, which has a loosely defined format
so this parser only provides best effort guarantee.
Determines performance and reliability characteristics.
Examples
Fast mode
Source:
parse_user_agent(
"Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1"
)
Return:
{"browser":{"family":"Firefox","version":"1.0.1"},"device":{"category":"pc"},"os":{"family":"Linux","version":null}}
Reliable mode
Source:
parse_user_agent(
"Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
mode: "reliable"
)
Return:
{"browser":{"family":"Internet Explorer","version":"7.66"},"device":{"category":"pc"},"os":{"family":"Windows XP","version":"NT 5.1"}}
Enriched mode
Source:
parse_user_agent(
"Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54",
mode: "enriched"
)
Return:
{"browser":{"family":"Opera Mini","major":"4","minor":"3","patch":"24214","version":"10.54"},"device":{"brand":"Apple","category":"smartphone","family":"iPhone","model":"iPhone"},"os":{"family":"iOS","major":"4","minor":"2","patch":"1","patch_minor":null,"version":"4.2.1"}}
parse_xml
Parses the value as XML.
The string representation of the XML document to parse.
Include XML tag attributes in the returned object.
String prefix to use for XML tag attribute keys.
Key name to use for expanded text nodes.
Always return text nodes as {"<text_key>": "value"}.
Parse “true” and “false” as boolean.
Parse numbers as integers/floats.
Errors
value is not a valid XML document.
Examples
Parse XML
Source:
value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';
parse_xml!(value, text_key: "value", parse_number: false)
Return:
{"book":{"@category":"CHILDREN","author":"J K. Rowling","title":{"@lang":"en","value":"Harry Potter"},"year":"2005"}}
Path Functions
del
Removes the field specified by the static path from the target.
For dynamic path deletion, see the remove function.
The path of the field to delete.
After deletion, if compact is true and there is an empty object or array left,
the empty object or array is also removed, cascading up to the root. This only
applies to the path being deleted, and any parent paths.
Examples
Delete a field
Source:
Rename a field
Source:
.new_field = del(.old_field)
exists
Checks whether the path exists for the target.
This function distinguishes between a missing path
and a path with a null value. A regular path lookup,
such as .foo, cannot distinguish between the two cases
since it always returns null if the path doesn’t exist.
The path of the field to check.
Examples
Exists (field)
Source:
Return:
Exists (array element)
Source:
Return:
get
Dynamically get the value of a given path.
If you know the path you want to look up, use
static paths such as .foo.bar[1] to get the value of that
path. However, if you do not know the path names,
use the dynamic get function to get the requested
value.
The object or array to query.
An array of path segments to look for the value.
Errors
- The
path segment must be a string or an integer.
Examples
single-segment top-level field
Source:
get!(value: { "foo": "bar" }, path: ["foo"])
Return:
multi-segment nested field
Source:
get!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return:
array indexing
Source:
get!(value: ["foo", "bar", "baz"], path: [-2])
Return:
remove
Dynamically remove the value for a given path.
If you know the path you want to remove, use
the del function and static paths such as del(.foo.bar[1])
to remove the value at that path. The del function returns the
deleted value, and is more performant than remove.
However, if you do not know the path names, use the dynamic
remove function to remove the value at the provided path.
The object or array to remove data from.
An array of path segments to remove the value from.
After deletion, if compact is true, any empty objects or
arrays left are also removed.
Errors
- The
path segment must be a string or an integer.
Examples
single-segment top-level field
Source:
remove!(value: { "foo": "bar" }, path: ["foo"])
Return:
multi-segment nested field
Source:
remove!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return:
array indexing
Source:
remove!(value: ["foo", "bar", "baz"], path: [-2])
Return:
compaction
Source:
remove!(value: { "foo": { "bar": [42], "baz": true } }, path: ["foo", "bar", 0], compact: true)
Return:
set
Dynamically insert data into the path of a given object or array.
If you know the path you want to assign a value to,
use static path assignments such as .foo.bar[1] = true for
improved performance and readability. However, if you do not
know the path names, use the dynamic set function to
insert the data into the object or array.
The object or array to insert data into.
An array of path segments to insert the value into.
Errors
- The
path segment must be a string or an integer.
Examples
single-segment top-level field
Source:
set!(value: { "foo": "bar" }, path: ["foo"], data: "baz")
Return:
multi-segment nested field
Source:
set!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"], data: "qux")
Return:
array
Source:
set!(value: ["foo", "bar", "baz"], path: [-2], data: 42)
Return:
Random Functions
random_bool
Returns a random boolean.
Examples
Random boolean
Source:
is_boolean(random_bool())
Return:
random_bytes
A cryptographically secure random number generator. Returns a string value containing the number of
random bytes requested.
The number of bytes to generate. Must not be larger than 64k.
Errors
length is negative.length is larger than the maximum value (64k).
Examples
Generate random base 64 encoded bytes
Source:
encode_base64(random_bytes(16))
Return:
"LNu0BBgUbh7XAlXbjSOomQ=="
random_float
Returns a random float between [min, max).
Minimum value (inclusive).
Maximum value (exclusive).
Errors
max is not greater than min.
Examples
Random float from 0.0 to 10.0, not including 10.0
Source:
f = random_float(0.0, 10.0)
f >= 0 && f < 10
Return:
random_int
Returns a random integer between [min, max).
Minimum value (inclusive).
Maximum value (exclusive).
Errors
max is not greater than min.
Examples
Random integer from 0 to 10, not including 10
Source:
i = random_int(0, 10)
i >= 0 && i < 10
Return:
uuid_from_friendly_id
Convert a Friendly ID (base62 encoding a 128-bit word) to a UUID.
A string that is a Friendly ID
Errors
value is a string but the text uses characters outside of class [0-9A-Za-z].value is a base62 encoding of an integer, but the integer is greater than or equal to 2^128.
Examples
Convert a Friendly ID to a UUID
Source:
uuid_from_friendly_id!("3s87yEvnmkiPBMHsj8bwwc")
Return:
"7f41deed-d5e2-8b5e-7a13-ab4ff93cfad2"
uuid_v4
Generates a random UUIDv4 string.
Examples
Create a UUIDv4
Source:
Return:
"1d262f4f-199b-458d-879f-05fd0a5f0683"
uuid_v7
Generates a random UUIDv7 string.
The timestamp used to generate the UUIDv7.
Examples
Create a UUIDv7 with implicit now()
Source:
Return:
"06338364-8305-7b74-8000-de4963503139"
Create a UUIDv7 with explicit now()
Source:
Return:
"018e29b3-0bea-7f78-8af3-d32ccb1b93c1"
Create a UUIDv7 with custom timestamp
Source:
uuid_v7(t'2020-12-30T22:20:53.824727Z')
Return:
"0176b5bd-5d19-7394-bb60-c21028c6152b"
String Functions
camelcase
Takes the value string, and turns it into camelCase. Optionally, you can
pass in the existing case of the function, or else an attempt is made to determine the case automatically.
The string to convert to camelCase.
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
Examples
camelCase a string
Source:
camelcase("input-string")
Return:
camelCase a string
Source:
camelcase("input-string", "kebab-case")
Return:
Generates an ID based on the Community ID Spec.
The destination IP address.
The source port or ICMP type.
The destination port or ICMP code.
TCP
Source:
community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", source_port: 1122, destination_port: 3344, protocol: 6)
Return:
"1:wCb3OG7yAFWelaUydu0D+125CLM="
contains
Determines whether the value string contains the specified substring.
The substring to search for in value.
Whether the match should be case sensitive.
Examples
String contains (case sensitive)
Source:
contains("The Needle In The Haystack", "Needle")
Return:
String contains (case insensitive)
Source:
contains("The Needle In The Haystack", "needle", case_sensitive: false)
Return:
contains_all
Determines whether the value string contains all the specified substrings.
An array of substrings to search for in value.
Whether the match should be case sensitive.
Examples
String contains all
Source:
contains_all("The Needle In The Haystack", ["Needle", "Haystack"])
Return:
String contains all (case sensitive)
Source:
contains_all("the NEEDLE in the haystack", ["needle", "haystack"])
Return:
downcase
Downcases the value string, where downcase is defined according to the
Unicode Derived Core Property Lowercase.
The string to convert to lowercase.
Examples
Downcase a string
Source:
downcase("Hello, World!")
Return:
ends_with
Determines whether the value string ends with the specified substring.
The substring with which value must end.
Whether the match should be case sensitive.
Examples
String ends with (case sensitive)
Source:
ends_with("The Needle In The Haystack", "The Haystack")
Return:
String ends with (case insensitive)
Source:
ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false)
Return:
find
Determines from left to right the start position of the first found element in value
that matches pattern. Returns -1 if not found.
The string to find the pattern in.
The regular expression or string pattern to match against.
Offset to start searching.
Examples
Match text
Source:
Return:
Match regex
Source:
Return:
No matches
Source:
Return:
With an offset
Source:
find("foobarfoobarfoo", "bar", 4)
Return:
join
Joins each string in the value array into a single string, with items optionally separated from one another
by a separator.
The array of strings to join together.
The string separating each original element when joined.
Examples
Join array (no separator)
Source:
join!(["bring", "us", "together"])
Return:
Join array (comma separator)
Source:
join!(["sources", "transforms", "sinks"], separator: ", ")
Return:
"sources, transforms, sinks"
kebabcase
Takes the value string, and turns it into kebab-case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
The string to convert to kebab-case.
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
Examples
kebab-case a string
Source:
Return:
kebab-case a string
Source:
kebabcase("InputString", "PascalCase")
Return:
match
Determines whether the value matches the pattern.
The regular expression pattern to match against.
Examples
Regex match on a string
Source:
match("I'm a little teapot", r'teapot')
Return:
String does not match the regular expression
Source:
match("I'm a little teapot", r'.*balloon')
Return:
match_any
Determines whether value matches any of the given patterns. All
patterns are checked in a single pass over the target string, giving this
function a potential performance advantage over the multiple calls
in the match function.
The array of regular expression patterns to match against.
Examples
Regex match on a string
Source:
match_any("I'm a little teapot", [r'frying pan', r'teapot'])
Return:
parse_float
Parses the string value representing a floating point number in base 10 to a float.
Examples
Parse negative integer
Source:
Return:
Parse negative integer
Source:
Return:
Scientific notation
Source:
Return:
pascalcase
Takes the value string, and turns it into PascalCase. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
The string to convert to PascalCase.
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
Examples
PascalCase a string
Source:
pascalcase("input-string")
Return:
PascalCase a string
Source:
pascalcase("input-string", "kebab-case")
Return:
redact
Redact sensitive data in value such as:
This can help achieve compliance by ensuring sensitive data does not leave your network.
The value to redact sensitive data from.
The function’s behavior depends on value’s type:
- For strings, the sensitive data is redacted and a new string is returned.
- For arrays, the sensitive data is redacted in each string element.
- For objects, the sensitive data in each string value is masked, but the keys are not masked.
For arrays and objects, the function recurses into any nested arrays or objects. Any non-string elements are
skipped.
Redacted text is replaced with [REDACTED].
List of filters applied to value.
Each filter can be specified in the following ways:
- As a regular expression, which is used to redact text that match it.
- As an object with a
type key that corresponds to a named filter and additional keys for customizing that filter. - As a named filter, if it has no required parameters.
Named filters can be a:
pattern: Redacts text matching any regular expressions specified in the patterns
key, which is required. This is the expanded version of just passing a regular expression as a filter.us_social_security_number: Redacts US social security card numbers.
See examples for more details.
This parameter must be a static expression so that the argument can be validated at compile-time
to avoid runtime errors. You cannot use variables or other dynamic expressions with it.
Specifies what to replace the redacted strings with.
It is given as an object with a “type” key specifying the type of redactor to use
and additional keys depending on the type. The following types are supported:
full: The default. Replace with the string “[REDACTED]”.text: Replace with a custom string. The replacement key is required, and must
contain the string that is used as a replacement.sha2: Hash the redacted text with SHA-2 as with sha2. Supports two optional parameters:variant: The variant of the algorithm to use. Defaults to SHA-512/256.encoding: How to encode the hash as text. Can be base16 or base64.
Defaults to base64.
sha3: Hash the redacted text with SHA-3 as with sha3. Supports two optional parameters:variant: The variant of the algorithm to use. Defaults to SHA3-512.encoding: How to encode the hash as text. Can be base16 or base64.
Defaults to base64.
As a convenience you can use a string as a shorthand for common redactor patterns:
"full" is equivalent to {"type": "full"}"sha2" is equivalent to {"type": "sha2", "variant": "SHA-512/256", "encoding": "base64"}"sha3" is equivalent to {"type": "sha3", "variant": "SHA3-512", "encoding": "base64"}
This parameter must be a static expression so that the argument can be validated at compile-time
to avoid runtime errors. You cannot use variables or other dynamic expressions with it.
Examples
Replace text using a regex
Source:
redact("my id is 123456", filters: [r'\d+'])
Return:
Replace us social security numbers in any field
Source:
redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"])
Return:
{"name":"John Doe","ssn":"[REDACTED]"}
Replace with custom text
Source:
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "text", "replacement": "***"})
Return:
Replace with SHA-2 hash
Source:
redact("my id is 123456", filters: [r'\d+'], redactor: "sha2")
Return:
"my id is GEtTedW1p6tC094dDKH+3B8P+xSnZz69AmpjaXRd63I="
Replace with SHA-3 hash
Source:
redact("my id is 123456", filters: [r'\d+'], redactor: "sha3")
Return:
"my id is ZNCdmTDI7PeeUTFnpYjLdUObdizo+bIupZdl8yqnTKGdLx6X3JIqPUlUWUoFBikX+yTR+OcvLtAqWO11NPlNJw=="
Replace with SHA-256 hash using hex encoding
Source:
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})
Return:
"my id is 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92"
replace
Replaces all matching instances of pattern in value.
The pattern argument accepts regular expression capture groups.
Note when using capture groups:
- You will need to escape the
$ by using $$ to avoid Vector interpreting it as an
environment variable when loading configuration - If you want a literal
$ in the replacement pattern, you will also need to escape this
with $$. When combined with environment variable interpolation in config files this
means you will need to use $$$$ to have a literal $ in the replacement pattern.
Replace all matches of this pattern. Can be a static string or a regular expression.
The string that the matches are replaced with.
The maximum number of replacements to perform. -1 means replace all matches.
Examples
Replace literal text
Source:
replace("Apples and Bananas", "and", "not")
Return:
Replace using regular expression
Source:
replace("Apples and Bananas", r'(?i)bananas', "Pineapples")
Return:
Replace first instance
Source:
replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1)
Return:
Replace with capture groups (Note: Use $$num in config files)
Source:
replace("foo123bar", r'foo(?P<num>\d+)bar', "$num")
Return:
replace_with
Replaces all matching instances of pattern using a closure.
The pattern argument accepts a regular expression that can use capture groups.
The function uses the function closure syntax to compute the replacement values.
The closure takes a single parameter, which is an array, where the first item is always
present and contains the entire string that matched pattern. The items from index one on
contain the capture groups of the corresponding index. If a capture group is optional, the
value may be null if it didn’t match.
The value returned by the closure must be a string and will replace the section of
the input that was matched.
This returns a new string with the replacements, the original string is not mutated.
Replace all matches of this pattern. Must be a regular expression.
The maximum number of replacements to perform. -1 means replace all matches.
Examples
Capitalize words
Source:
replace_with("apples and bananas", r'\b(\w)(\w*)') -> |match| {
upcase!(match.captures[0]) + string!(match.captures[1])
}
Return:
Replace with hash
Source:
replace_with("email from test@example.com", r'\w+@example.com') -> |match| {
sha2(match.string, variant: "SHA-512/224")
}
Return:
"email from adf6e1bc4415d24912bd93072ad34ef825a7b6eb3bf53f68def1fc17"
Replace first instance
Source:
replace_with("Apples and Apples", r'(?i)apples|cones', count: 1) -> |match| {
"Pine" + downcase(match.string)
}
Return:
Named capture group
Source:
replace_with("level=error A message", r'level=(?P<level>\w+)') -> |match| {
lvl = upcase!(match.level)
"[{{lvl}}]"
}
Return:
screamingsnakecase
Takes the value string, and turns it into SCREAMING_SNAKE case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
The string to convert to SCREAMING_SNAKE case.
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
Examples
SCREAMING_SNAKE a string
Source:
screamingsnakecase("input-string")
Return:
SCREAMING_SNAKE a string
Source:
screamingsnakecase("input-string", "kebab-case")
Return:
sieve
Keeps only matches of pattern in value.
This can be used to define patterns that are allowed in the string and
remove everything else.
Keep all matches of this pattern.
The string to use to replace single rejected characters.
The string to use to replace multiple sequential instances of rejected characters.
Examples
Sieve with regex
Source:
sieve("test123%456.فوائد.net.", r'[a-z0-9.]')
Return:
Custom replacements
Source:
sieve("test123%456.فوائد.net.", r'[a-z.0-9]', replace_single: "X", replace_repeated: "<REMOVED>")
Return:
"test123X456.<REMOVED>.net."
slice
Returns a slice of value between the start and end positions.
If the start and end parameters are negative, they refer to positions counting from the right of the
string or array. If end refers to a position that is greater than the length of the string or array,
a slice up to the end of the string or array is returned.
The string or array to slice.
The inclusive start position. A zero-based index that can be negative.
The exclusive end position. A zero-based index that can be negative.
Examples
Slice a string (positive index)
Source:
slice!("Supercalifragilisticexpialidocious", start: 5, end: 13)
Return:
Slice a string (negative index)
Source:
slice!("Supercalifragilisticexpialidocious", start: 5, end: -14)
Return:
snakecase
Takes the value string, and turns it into snake-case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
The string to convert to snake-case.
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
Examples
snake-case a string
Source:
snakecase("input-string")
Return:
snake-case a string
Source:
snakecase("input-string", "kebab-case")
Return:
split
Splits the value string using pattern.
The string is split whenever this pattern is matched.
The maximum number of substrings to return.
Examples
Split a string (no limit)
Source:
split("apples and pears and bananas", " and ")
Return:
["apples","pears","bananas"]
Split a string (with a limit)
Source:
split("apples and pears and bananas", " and ", limit: 2)
Return:
["apples","pears and bananas"]
starts_with
Determines whether value begins with substring.
The substring that the value must start with.
Whether the match should be case sensitive.
Examples
String starts with (case sensitive)
Source:
starts_with("The Needle In The Haystack", "The Needle")
Return:
String starts with (case insensitive)
Source:
starts_with("The Needle In The Haystack", "the needle", case_sensitive: false)
Return:
strip_ansi_escape_codes
Strips ANSI escape codes from value.
Examples
Strip ANSI escape codes
Source:
strip_ansi_escape_codes("\e[46mfoo\e[0m bar")
Return:
strip_whitespace
Strips whitespace from the start and end of value, where whitespace is defined by the Unicode
White_Space property.
Examples
Strip whitespace
Source:
strip_whitespace(" A sentence. ")
Return:
truncate
Truncates the value string up to the limit number of characters.
The number of characters to truncate the string after.
This argument is deprecated. An ellipsis (...) is appended if the parameter is set to true and the value string
is truncated because it exceeded the limit.
A custom suffix (...) is appended to truncated strings.
If ellipsis is set to true, this parameter is ignored for backwards compatibility.
Examples
Truncate a string
Source:
truncate("A rather long sentence.", limit: 11, suffix: "...")
Return:
Truncate a string
Source:
truncate("A rather long sentence.", limit: 11, suffix: "[TRUNCATED]")
Return:
upcase
Upcases value, where upcase is defined according to the Unicode Derived Core Property
Uppercase.
The string to convert to uppercase.
Examples
Upcase a string
Source:
Return:
System Functions
get_env_var
Returns the value of the environment variable specified by name.
The name of the environment variable.
Errors
- Environment variable
name does not exist. - The value of environment variable
name is not valid Unicode
Examples
Get an environment variable
Source:
Return:
get_hostname
Returns the local system’s hostname.
Errors
- Internal hostname resolution failed.
Examples
Get hostname
Source:
.hostname = get_hostname!()
get_timezone_name
Returns the name of the timezone in the Vector configuration (see
global configuration options).
If the configuration is set to local, then it attempts to
determine the name of the timezone from the host OS. If this
is not possible, then it returns the fixed offset of the
local timezone for the current time in the format "[+-]HH:MM",
for example, "+02:00".
Errors
- Retrieval of local timezone information failed.
Examples
Get the IANA name of Vector’s timezone
Source:
.vector_timezone = get_timezone_name!()
Timestamp Functions
Formats value into a string representation of the timestamp.
The timestamp to format as text.
The timezone to use when formatting the timestamp. The parameter uses the TZ identifier or local.
Source:
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+")
Return:
"2020-10-21T16:00:00+00:00"
Source:
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R")
Return:
now
Returns the current timestamp in the UTC timezone with nanosecond precision.
Examples
Generate a current timestamp
Source:
Return:
"2021-03-04T10:51:15.928937Z"
Type Functions
array
Returns value if it is an array, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an array and can be used in any function that expects an array.
The value to check if it is an array.
Errors
Examples
Declare an array type
Source:
Return:
bool
Returns value if it is a Boolean, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a Boolean and can be used in any function that expects a Boolean.
The value to check if it is a Boolean.
Errors
Examples
Declare a Boolean type
Source:
Return:
float
Returns value if it is a float, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a float and can be used in any function that expects a float.
The value to check if it is a float.
Errors
Examples
Declare a float type
Source:
Return:
int
Returns value if it is an integer, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an integer and can be used in any function that expects an integer.
The value to check if it is an integer.
Errors
Examples
Declare an integer type
Source:
Return:
is_array
Check if the value’s type is an array.
The value to check if it is an array.
Examples
Valid array
Source:
Return:
Non-matching type
Source:
Return:
is_boolean
Check if the value’s type is a boolean.
The value to check if it is a Boolean.
Examples
Valid boolean
Source:
Return:
Non-matching type
Source:
Return:
is_empty
Check if the object, array, or string has a length of 0.
Examples
Empty array
Source:
Return:
Non-empty string
Source:
Return:
Non-empty object
Source:
Return:
is_float
Check if the value’s type is a float.
The value to check if it is a float.
Examples
Valid float
Source:
Return:
Non-matching type
Source:
Return:
is_integer
Check if the value`’s type is an integer.
The value to check if it is an integer.
Examples
Valid integer
Source:
Return:
Non-matching type
Source:
Return:
is_json
Check if the string is a valid JSON document.
The value to check if it is a valid JSON document.
The variant of the JSON type to explicitly check for.
Examples
Valid JSON object
Source:
Return:
Non-valid value
Source:
Return:
Exact variant
Source:
is_json("{}", variant: "object")
Return:
Non-valid exact variant
Source:
is_json("{}", variant: "array")
Return:
is_null
Check if value’s type is null. For a more relaxed function,
see is_nullish.
The value to check if it is null.
Examples
Null value
Source:
Return:
Non-matching type
Source:
Return:
is_nullish
Determines whether value is nullish, where nullish denotes the absence of a
meaningful value.
The value to check for nullishness, for example, a useless value.
Examples
Null detection (blank string)
Source:
Return:
Null detection (dash string)
Source:
Return:
Null detection (whitespace)
Source:
Return:
is_object
Check if value’s type is an object.
The value to check if it is an object.
Examples
Valid object
Source:
is_object({"foo": "bar"})
Return:
Non-matching type
Source:
Return:
is_regex
Check if value’s type is a regex.
The value to check if it is a regex.
Examples
Valid regex
Source:
Return:
Non-matching type
Source:
Return:
is_string
Check if value’s type is a string.
The value to check if it is a string.
Examples
Valid string
Source:
Return:
Non-matching type
Source:
Return:
is_timestamp
Check if value’s type is a timestamp.
The value to check if it is a timestamp.
Examples
Valid timestamp
Source:
is_timestamp(t'2021-03-26T16:00:00Z')
Return:
Non-matching type
Source:
Return:
object
Returns value if it is an object, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an object and can be used in any function that expects an object.
The value to check if it is an object.
Errors
Examples
Declare an object type
Source:
Return:
{"field1":"value1","field2":"value2"}
string
Returns value if it is a string, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a string and can be used in any function that expects a string.
The value to check if it is a string.
Errors
Examples
Declare a string type
Source:
Return:
tag_types_externally
Adds type information to all (nested) scalar values in the provided value.
The type information is added externally, meaning that value has the form of "type": value after this
transformation.
The value to tag with types.
Examples
Tag types externally (scalar)
Source:
tag_types_externally(123)
Return:
Tag types externally (object)
Source:
tag_types_externally({
"message": "Hello world",
"request": {
"duration_ms": 67.9
}
})
Return:
{"message":{"string":"Hello world"},"request":{"duration_ms":{"float":67.9}}}
Tag types externally (array)
Source:
tag_types_externally(["foo", "bar"])
Return:
[{"string":"foo"},{"string":"bar"}]
Tag types externally (null)
Source:
tag_types_externally(null)
Return:
timestamp
Returns value if it is a timestamp, otherwise returns an error. This enables the type checker to guarantee that
the returned value is a timestamp and can be used in any function that expects a timestamp.
The value to check if it is a timestamp.
Errors
value is not a timestamp.
Examples
Declare a timestamp type
Source:
timestamp(t'2020-10-10T16:00:00Z')
Return:
