Observability Pipelines is not available on the US1-FED Datadog site.
Datadog Processing Language (DPL), or Vector Remap Language (VRL), contains built-in functions for transforming your data.
The functions are organized into the following categories:
Appends each item in the items array to the end of the value array.
Chunks value into slices of length chunk_size bytes.
The array of bytes to split.
The desired length of each chunk in bytes. This may be constrained by the host platform architecture.
chunk_size must be at least 1 byte.chunk_size is too large.
Source:
Return:
Source:
Return:
Adds the item to the end of the value array.
Iterate over several arrays in parallel, producing a new array containing arrays of items from each source.
The resulting array will be as long as the shortest input array, with all the remaining elements dropped.
This function is modeled from the zip function in Python,
but similar methods can be found in Ruby
and Rust.
If a single parameter is given, it must contain an array of all the input arrays.
The first array of elements, or the array of input arrays if no other parameter is present.
The second array of elements. If not present, the first parameter contains all the arrays.
array_0 and array_1 must be arrays.
Source:
zip([1, 2, 3], [4, 5, 6, 7])
Return:
Source:
zip([[1, 2], [3, 4], [5, 6]])
Return:
Calculates a CRC of the value.
The CRC algorithm used can be optionally specified.
This function is infallible if either the default algorithm value or a recognized-valid compile-time
algorithm string literal is used. Otherwise, it is fallible.
The string to calculate the checksum for.
The CRC algorithm to use.
value is not a string.algorithm is not a supported algorithm.
Source:
Return:
Source:
crc("foo", algorithm: "CRC_32_CKSUM")
Return:
Decodes the value (a Base16 string) into its original string.
value isn’t a valid encoded Base16 string.
Source:
decode_base16!("796f752068617665207375636365737366756c6c79206465636f646564206d65")
Return:
"you have successfully decoded me"
Decodes the value (a Base64 string) into its original string.
The character set to use when decoding the data.
value isn’t a valid encoded Base64 string.
Source:
decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=")
Return:
"you have successfully decoded me"
Source:
decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe")
Return:
"you can't make your heart feel something it won't"
Decodes the value (a non-UTF8 string) to a UTF8 string using the specified character set.
The non-UTF8 string to decode.
Source:
decode_charset!(decode_base64!("vsiz58fPvLy/5A=="), "euc-kr")
Return:
Source:
decode_charset!(decode_base64!("pLOk86TLpMGkzw=="), "euc-jp")
Return:
Source:
decode_charset!(decode_base64!("xOO6ww=="), "gb2312")
Return:
Decodes the value (a Gzip string) into its original string.
value isn’t a valid encoded Gzip string.
Source:
encoded_text = decode_base64!("H4sIAHEAymMAA6vML1XISCxLVSguTU5OLS5OK83JqVRISU3OT0lNUchNBQD7BGDaIAAAAA==")
decode_gzip!(encoded_text)
Return:
"you have successfully decoded me"
Replaces q-encoded or base64-encoded encoded-word substrings in the value with their original string.
Source:
decode_mime_q!("=?utf-8?b?SGVsbG8sIFdvcmxkIQ==?=")
Return:
Source:
decode_mime_q!("From: =?utf-8?b?SGVsbG8sIFdvcmxkIQ==?= <=?utf-8?q?hello=5Fworld=40example=2ecom?=>")
Return:
"From: Hello, World! <hello_world@example.com>"
Source:
decode_mime_q!("?b?SGVsbG8sIFdvcmxkIQ==")
Return:
Decodes a percent-encoded value like a URL.
Source:
decode_percent("foo%20bar%3F")
Return:
Decodes a punycode encoded value, such as an internationalized domain name (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.
If enabled, checks if the input string is a valid domain name.
value is not valid punycode
Source:
decode_punycode!("www.xn--caf-dma.com")
Return:
Source:
decode_punycode!("www.cafe.com")
Return:
Source:
decode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)
Return:
Decodes the value (a Snappy string) into its original string.
The Snappy data to decode.
value isn’t a valid encoded Snappy string.
Source:
encoded_text = decode_base64!("LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==")
decode_snappy!(encoded_text)
Return:
"The quick brown fox jumps over 13 lazy dogs."
Decodes the value (a Zlib string) into its original string.
value isn’t a valid encoded Zlib string.
Source:
encoded_text = decode_base64!("eJwNy4ENwCAIBMCNXIlQ/KqplUSgCdvXAS41qPMHshCB2R1zJlWIVlR6UURX2+wx2YcuK3kAb9C1wd6dn7Fa+QH9gRxr")
decode_zlib!(encoded_text)
Return:
"you_have_successfully_decoded_me.congratulations.you_are_breathtaking."
Decodes the value (a Zstandard string) into its original string.
value isn’t a valid encoded Zstd string.
Source:
encoded_text = decode_base64!("KLUv/QBY/QEAYsQOFKClbQBedqXsb96EWDax/f/F/z+gNU4ZTInaUeAj82KqPFjUzKqhcfDqAIsLvAsnY1bI/N2mHzDixRQA")
decode_zstd!(encoded_text)
Return:
"you_have_successfully_decoded_me.congratulations.you_are_breathtaking."
Encodes the value to Base16.
Source:
encode_base16("please encode me")
Return:
"706c6561736520656e636f6465206d65"
Encodes the value to Base64.
Whether the Base64 output is padded.
The character set to use when encoding the data.
Source:
encode_base64("please encode me")
Return:
"cGxlYXNlIGVuY29kZSBtZQ=="
Source:
encode_base64("please encode me, no padding though", padding: false)
Return:
"cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g"
Source:
encode_base64("please encode me, but safe for URLs", charset: "url_safe")
Return:
"cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM="
Encodes the value (a UTF8 string) to a non-UTF8 string using the specified character set.
The UTF8 string to encode.
Source:
encode_base64(encode_charset!("안녕하세요", "euc-kr"))
Return:
Source:
encode_base64(encode_charset!("こんにちは", "euc-jp"))
Return:
Source:
encode_base64(encode_charset!("你好", "gb2312"))
Return:
Encodes the value to Gzip.
The default compression level.
Source:
encoded_text = encode_gzip("please encode me")
encode_base64(encoded_text)
Return:
"H4sIAAAAAAAA/yvISU0sTlVIzUvOT0lVyE0FAI4R4vcQAAAA"
Encodes the value to JSON.
The value to convert to a JSON string.
Whether to pretty print the JSON string or not.
Source:
.payload = encode_json({"hello": "world"})
Return:
Encodes the value into key-value format with customizable delimiters. Default delimiters match
the logfmt format.
The value to convert to a string.
The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.
The string that separates the key from the value.
The string that separates each key-value pair.
Whether to encode key-value with a boolean value as a standalone key if true and nothing if false.
fields_ordering contains a non-string element.
Source:
encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return:
"lvl=info msg=\"This is a message\" ts=2021-06-05T17:20:00Z"
Source:
encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return:
"ts=2021-06-05T17:20:00Z lvl=info msg=\"This is a message\" log_id=12345"
Source:
encode_key_value({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return:
"agent.name=foo event=log log.file.path=my.log"
Source:
encode_key_value!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return:
"event=log log.file.path=my.log agent.name=foo"
Source:
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"},
field_delimiter: ",",
key_value_delimiter: ":"
)
Return:
"lvl:info,msg:\"This is a message\",ts:2021-06-05T17:20:00Z"
Source:
encode_key_value(
{"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false},
field_delimiter: ",",
key_value_delimiter: ":",
flatten_boolean: true
)
Return:
"beta,lvl:info,msg:\"This is a message\",ts:2021-06-05T17:20:00Z"
Encodes the value to logfmt.
The value to convert to a logfmt string.
The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields.
fields_ordering contains a non-string element.
Source:
encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"})
Return:
"lvl=info msg=\"This is a message\" ts=2021-06-05T17:20:00Z"
Source:
encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"])
Return:
"ts=2021-06-05T17:20:00Z lvl=info msg=\"This is a message\" log_id=12345"
Source:
encode_logfmt({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"})
Return:
"agent.name=foo event=log log.file.path=my.log"
Source:
encode_logfmt!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"])
Return:
"event=log log.file.path=my.log agent.name=foo"
Encodes a value with percent encoding to safely be used in URLs.
The ASCII set to use when encoding the data.
Source:
encode_percent("foo bar?")
Return:
Source:
encode_percent("foo bar", ascii_set: "CONTROLS")
Return:
Encodes the value into a protocol buffer payload.
The object to convert to a protocol buffer payload.
The path to the protobuf descriptor set file. Must be a literal string.
This file is the output of protoc -o …
The name of the message type to use for serializing.
Must be a literal string.
desc_file file does not exist.message_type message type does not exist in the descriptor file.
Source:
.payload = encode_base64(encode_proto!({"name": "someone", "phones": [{"number": "123456"}]}, "resources/protobuf_descriptor_set.desc", "test_protobuf.Person"))
Return:
"Cgdzb21lb25lIggKBjEyMzQ1Ng=="
Encodes a value to punycode. Useful for internationalized domain names (IDN). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it.
Whether to validate the input string to check if it is a valid domain name.
value can not be encoded to punycode
Source:
encode_punycode!("www.café.com")
Return:
Source:
encode_punycode!("www.CAFé.com")
Return:
Source:
encode_punycode!("www.cafe.com")
Return:
Source:
encode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false)
Return:
"xn--8hbb.xn--fiba.xn--8hbf.xn--eib."
Encodes the value to Snappy.
value cannot be encoded into a Snappy string.
Source:
encoded_text = encode_snappy!("The quick brown fox jumps over 13 lazy dogs.")
encode_base64(encoded_text)
Return:
"LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg=="
Encodes the value to Zlib.
The default compression level.
Source:
encoded_text = encode_zlib("please encode me")
encode_base64(encoded_text)
Return:
"eJwryElNLE5VSM1Lzk9JVchNBQA0RQX7"
Encodes the value to Zstandard.
The default compression level.
Source:
encoded_text = encode_zstd("please encode me")
encode_base64(encoded_text)
Return:
"KLUv/QBYgQAAcGxlYXNlIGVuY29kZSBtZQ=="
Coerces the value into a boolean.
boolean, integer, float, null, string
The value to convert to a Boolean.
value is not a supported boolean representation.
Source:
Return:
Source:
Return:
Source:
Return:
Source:
Return:
Source:
Return:
Coerces the value into a float.
integer, float, boolean, string, timestamp
The value to convert to a float. Must be convertible to a float, otherwise an error is raised.
value is not a supported float representation.
Source:
Return:
Source:
to_float(t'2020-12-30T22:20:53.824727Z')
Return:
Coerces the value into an integer.
integer, float, boolean, string, timestamp, null
The value to convert to an integer.
value is a string but the text is not an integer.value is not a string, int, or timestamp.
Source:
Return:
Source:
to_int(t'2020-12-30T22:20:53.824727Z')
Return:
Coerces the value into a regex.
The value to convert to a regex.
Coerces the value into a string.
integer, float, boolean, string, timestamp, null
The value to convert to a string.
value is not an integer, float, boolean, string, timestamp, or null.
Source:
Return:
Source:
Return:
Source:
Return:
Converts the value integer from a Unix timestamp to a DPL timestamp.
Converts from the number of seconds since the Unix epoch by default. To convert from milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.
The Unix timestamp to convert.
Source:
Return:
Source:
from_unix_timestamp!(5000, unit: "milliseconds")
Return:
Source:
from_unix_timestamp!(5000, unit: "nanoseconds")
Return:
"1970-01-01T00:00:00.000005Z"
Converts the value, a Syslog facility code, into its corresponding
Syslog keyword. For example, 0 into "kern", 1 into "user", etc.
Converts the value, a Syslog facility keyword, into a Syslog integer
facility code (0 to 23).
The Syslog facility keyword to convert.
value is not a valid Syslog facility keyword.
Source:
to_syslog_facility_code!("authpriv")
Return:
Converts the value, a Syslog severity level, into its corresponding keyword,
i.e. 0 into "emerg", 1 into "alert", etc.
Converts the value, a Syslog log level keyword, into a Syslog integer
severity level (0 to 7).
The Syslog level keyword to convert.
value is not a valid Syslog level keyword.
Source:
to_syslog_severity!("alert")
Return:
Converts the value timestamp into a Unix timestamp.
Returns the number of seconds since the Unix epoch by default. To return the number in milliseconds or nanoseconds, set the unit argument to milliseconds or nanoseconds.
The timestamp to convert into a Unix timestamp.
Source:
to_unix_timestamp(t'2021-01-01T00:00:00+00:00')
Return:
Source:
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds")
Return:
Source:
to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds")
Return:
Decrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
- AES-256-CFB (key = 32 bytes, iv = 16 bytes)
- AES-192-CFB (key = 24 bytes, iv = 16 bytes)
- AES-128-CFB (key = 16 bytes, iv = 16 bytes)
- AES-256-OFB (key = 32 bytes, iv = 16 bytes)
- AES-192-OFB (key = 24 bytes, iv = 16 bytes)
- AES-128-OFB (key = 16 bytes, iv = 16 bytes)
- AES-128-SIV (key = 32 bytes, iv = 16 bytes)
- AES-256-SIV (key = 64 bytes, iv = 16 bytes)
- Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
- Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
- Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
- CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
- XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
- XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
The string in raw bytes (not encoded) to decrypt.
The key in raw bytes (not encoded) for decryption. The length must match the algorithm requested.
The IV in raw bytes (not encoded) for decryption. The length must match the algorithm requested.
A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value.
The value should match the one used during encryption.
algorithm is not a supported algorithm.key length does not match the key size required for the algorithm specified.iv length does not match the iv size required for the algorithm specified.
Source:
ciphertext = decode_base64!("5fLGcu1VHdzsPcGNDio7asLqE1P43QrVfPfmP4i4zOU=")
iv = decode_base64!("fVEIRkIiczCRWNxaarsyxA==")
key = "16_byte_keyxxxxx"
decrypt!(ciphertext, "AES-128-CBC-PKCS7", key, iv: iv)
Return:
Encrypts a string with a symmetric encryption algorithm.
Supported Algorithms:
- AES-256-CFB (key = 32 bytes, iv = 16 bytes)
- AES-192-CFB (key = 24 bytes, iv = 16 bytes)
- AES-128-CFB (key = 16 bytes, iv = 16 bytes)
- AES-256-OFB (key = 32 bytes, iv = 16 bytes)
- AES-192-OFB (key = 24 bytes, iv = 16 bytes)
- AES-128-OFB (key = 16 bytes, iv = 16 bytes)
- AES-128-SIV (key = 32 bytes, iv = 16 bytes)
- AES-256-SIV (key = 64 bytes, iv = 16 bytes)
- Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes)
- Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes)
- Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes)
- AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes)
- AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes)
- AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes)
- AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes)
- AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes)
- AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes)
- CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes)
- XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
- XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes)
The key in raw bytes (not encoded) for encryption. The length must match the algorithm requested.
The IV in raw bytes (not encoded) for encryption. The length must match the algorithm requested.
A new IV should be generated for every message. You can use random_bytes to generate a cryptographically secure random value.
algorithm is not a supported algorithm.key length does not match the key size required for the algorithm specified.iv length does not match the iv size required for the algorithm specified.
Source:
plaintext = "super secret message"
iv = "1234567890123456" # typically you would call random_bytes(16)
key = "16_byte_keyxxxxx"
encrypted_message = encrypt!(plaintext, "AES-128-CBC-PKCS7", key, iv: iv)
encode_base64(encrypted_message)
Return:
"GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE="
Calculates a HMAC of the value using the given key.
The hashing algorithm used can be optionally specified.
For most use cases, the resulting bytestream should be encoded into a hex or base64
string using either encode_base16 or
encode_base64.
This function is infallible if either the default algorithm value or a recognized-valid compile-time
algorithm string literal is used. Otherwise, it is fallible.
The string to calculate the HMAC for.
The string to use as the cryptographic key.
The hashing algorithm to use.
Source:
encode_base64(hmac("Hello there", "super-secret-key"))
Return:
"eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI="
Source:
encode_base16(hmac("Hello there", "super-secret-key", algorithm: "SHA-224"))
Return:
"42fccbc2b7d22a143b92f265a8046187558a94d11ddbb30622207e90"
Source:
.hash_algo = "SHA-256"
hmac_bytes, err = hmac("Hello there", "super-secret-key", algorithm: .hash_algo)
if err == null {
.hmac = encode_base16(hmac_bytes)
}
Return:
"78b184f1832f8aff3934f5e0212454671b2d04d494e3b25075c5e45167029662"
Calculates an md5 hash of the value.
The string to calculate the hash for.
Source:
Return:
"acbd18db4cc2f85cedef654fccc4a4d8"
Calculates a Seahash hash of the value.
Note: Due to limitations in the underlying DPL data types, this function converts the unsigned 64-bit integer SeaHash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values.
The string to calculate the hash for.
Source:
Return:
Source:
Return:
Calculates a SHA-1 hash of the value.
The string to calculate the hash for.
Source:
Return:
"0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33"
Calculates a SHA-2 hash of the value.
The string to calculate the hash for.
The variant of the algorithm to use.
Source:
sha2("foo", variant: "SHA-512/224")
Return:
"d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be"
Calculates a SHA-3 hash of the value.
The string to calculate the hash for.
The variant of the algorithm to use.
Source:
sha3("foo", variant: "SHA3-224")
Return:
"f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a"
Asserts the condition, which must be a Boolean expression. The program is aborted with
message if the condition evaluates to false.
An optional custom error message. If the equality assertion fails, message is
appended to the default message prefix. See the examples below
for a fully formed log message sample.
condition evaluates to false.
Source:
assert!("foo" == "foo", message: "\"foo\" must be \"foo\"!")
Return:
Source:
assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!")
Asserts that two expressions, left and right, have the same value. The program is
aborted with message if they do not have the same value.
The value to check for equality against right.
The value to check for equality against left.
An optional custom error message. If the equality assertion fails, message is
appended to the default message prefix. See the examples
below for a fully formed log message sample.
Source:
Return:
Source:
assert_eq!(127, [1, 2, 3])
Source:
assert_eq!(1, 0, message: "Unequal integers")
Logs the value to stdout at the specified level.
Specifies that the log message is output no more than once per the given number of seconds.
Use a value of 0 to turn rate limiting off.
Source:
log("Hello, World!", level: "info", rate_limit_secs: 60)
Return:
Source:
_, err = to_int(.field)
if err != null {
log(err, level: "error")
}
Return:
Searches an enrichment table for rows that match the
provided condition.
For file enrichment tables, this condition needs to be a DPL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields do not match, then no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity
can be specified using the case_sensitive argument. An exact match search can use an
index directly into the dataset, which should make this search fairly “cheap” from a
performance perspective.
Date range search. The given field must be greater than or equal to the from date
and less than or equal to the to date. A date range search involves
sequentially scanning through the rows that have been located using any exact match
criteria. This can be an expensive operation if there are many rows returned by any exact
match criteria. Therefore, use date ranges as the only criteria when the enrichment
data set is very small.
For geoip and mmdb enrichment tables, this condition needs to be a DPL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected
and without a value, null is used. This table can return the following fields:
To use this function, you need to update your configuration to
include an
enrichment_tables
parameter.
The enrichment table to search.
The condition to search on. Since the condition is used at boot time to create
indices into the data, these conditions must be statically defined.
A subset of fields from the enrichment table to return. If not specified,
all fields are returned.
Whether text fields need to match cases exactly.
Source:
find_enrichment_table_records!("test",
{
"surname": "smith",
},
case_sensitive: false)
Return:
[{"id":1,"firstname":"Bob","surname":"Smith"},{"id":2,"firstname":"Fred","surname":"Smith"}]
Source:
find_enrichment_table_records!("test",
{
"surname": "Smith",
"date_of_birth": {
"from": t'1985-01-01T00:00:00Z',
"to": t'1985-12-31T00:00:00Z'
}
})
Return:
[{"id":1,"firstname":"Bob","surname":"Smith"},{"id":2,"firstname":"Fred","surname":"Smith"}]
Searches an enrichment table for a row that matches the
provided condition. A single row must be matched. If no rows are found or more than one row is
found, an error is returned.
For file enrichment tables, this condition needs to be a DPL object in which
the key-value pairs indicate a field to search mapped to a value to search in that field.
This function returns the rows that match the provided condition(s). All fields need to
match for rows to be returned; if any fields do not match, then no rows are returned.
There are currently two forms of search criteria:
Exact match search. The given field must match the value exactly. Case sensitivity
can be specified using the case_sensitive argument. An exact match search can use an
index directly into the dataset, which should make this search fairly “cheap” from a
performance perspective.
Date range search. The given field must be greater than or equal to the from date
and less than or equal to the to date. A date range search involves
sequentially scanning through the rows that have been located using any exact match
criteria. This can be an expensive operation if there are many rows returned by any exact
match criteria. Therefore, use date ranges as the only criteria when the enrichment
data set is very small.
For geoip and mmdb enrichment tables, this condition needs to be a DPL object with a single key-value pair
whose value needs to be a valid IP address. Example: {"ip": .ip }. If a return field is expected
and without a value, null is used. This table can return the following fields:
To use this function, you need to update your configuration to
include an
enrichment_tables
parameter.
The enrichment table to search.
The condition to search on. Since the condition is used at boot time to create
indices into the data, these conditions must be statically defined.
A subset of fields from the enrichment table to return. If not specified,
all fields are returned.
Whether the text fields match the case exactly.
- The row is not found.
- Multiple rows are found that match the condition.
Source:
get_enrichment_table_record!("test",
{
"surname": "bob",
"firstname": "John"
},
case_sensitive: false)
Return:
{"id":1,"firstname":"Bob","surname":"Smith"}
Source:
get_enrichment_table_record!("test",
{
"surname": "Smith",
"date_of_birth": {
"from": t'1985-01-01T00:00:00Z',
"to": t'1985-12-31T00:00:00Z'
}
})
Return:
{"id":1,"firstname":"Bob","surname":"Smith"}
Compacts the value by removing empty values, where empty values are defined using the
available parameters.
The object or array to compact.
Whether the compaction be recursive.
Whether null should be treated as an empty value.
Whether an empty string should be treated as an empty value.
Whether an empty object should be treated as an empty value.
Whether an empty array should be treated as an empty value.
Tests whether the value is “nullish” as defined by the is_nullish function.
Source:
compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true)
Return:
Source:
compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true)
Return:
Filter elements from a collection.
This function currently does not support recursive iteration.
The function uses the function closure syntax to allow reading
the key-value or index-value combination for each item in the
collection.
The same scoping rules apply to closure blocks as they do for
regular blocks. This means that any variable defined in parent scopes
is accessible, and mutations to those variables are preserved,
but any new variables instantiated in the closure block are
unavailable outside of the block.
See the examples below to learn about the closure syntax.
The array or object to filter.
Source:
filter(array!(.tags)) -> |_index, value| {
# keep any elements that aren't equal to "foo"
value != "foo"
}
Return:
Flattens the value into a single-level representation.
The array or object to flatten.
The separator to join nested keys
Source:
flatten([1, [2, 3, 4], [5, [6, 7], 8], 9])
Return:
Source:
flatten({
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
})
Return:
{"parent1.child1":1,"parent1.child2":2,"parent2.child3":3}
Iterate over a collection.
This function currently does not support recursive iteration.
The function uses the “function closure syntax” to allow reading
the key/value or index/value combination for each item in the
collection.
The same scoping rules apply to closure blocks as they do for
regular blocks. This means that any variable defined in parent scopes
is accessible, and mutations to those variables are preserved,
but any new variables instantiated in the closure block are
unavailable outside of the block.
See the examples below to learn about the closure syntax.
The array or object to iterate.
Source:
tally = {}
for_each(array!(.tags)) -> |_index, value| {
# Get the current tally for the `value`, or
# set to `0`.
count = int(get!(tally, [value])) ?? 0
# Increment the tally for the value by `1`.
tally = set!(tally, [value], count + 1)
}
tally
Return:
{"foo":2,"bar":1,"baz":1}
Determines whether the value array includes the specified item.
Source:
includes(["apple", "orange", "banana"], "banana")
Return:
Returns the keys from the object passed into the function.
The object to extract keys from.
Source:
keys({"key1": "val1", "key2": "val2"})
Return:
Returns the length of the value.
- If
value is an array, returns the number of elements. - If
value is an object, returns the number of top-level keys. - If
value is a string, returns the number of bytes in the string. If
you want the number of characters, see strlen.
Source:
length({
"portland": "Trail Blazers",
"seattle": "Supersonics"
})
Return:
Source:
length({
"home": {
"city": "Portland",
"state": "Oregon"
},
"name": "Trail Blazers",
"mascot": {
"name": "Blaze the Trail Cat"
}
})
Return:
Source:
length(["Trail Blazers", "Supersonics", "Grizzlies"])
Return:
Source:
length("The Planet of the Apes Musical")
Return:
Map the keys within an object.
If recursive is enabled, the function iterates into nested
objects, using the following rules:
- Iteration starts at the root.
- For every nested object type:
- First return the key of the object type itself.
- Then recurse into the object, and loop back to item (1)
in this list.
- Any mutation done on a nested object before recursing into
it, are preserved.
- For every nested array type:
- First return the key of the array type itself.
- Then find all objects within the array, and apply item (2)
to each individual object.
The above rules mean that map_keys with
recursive enabled finds all keys in the target,
regardless of whether nested objects are nested inside arrays.
The function uses the function closure syntax to allow reading
the key for each item in the object.
The same scoping rules apply to closure blocks as they do for
regular blocks. This means that any variable defined in parent scopes
is accessible, and mutations to those variables are preserved,
but any new variables instantiated in the closure block are
unavailable outside of the block.
See the examples below to learn about the closure syntax.
Whether to recursively iterate the collection.
Source:
map_keys(.) -> |key| { upcase(key) }
Return:
{"FOO":"foo","BAR":"bar"}
Source:
map_keys(., recursive: true) -> |key| { replace(key, ".", "_") }
Return:
{"labels":{"app_kubernetes_io/name":"mysql"}}
Map the values within a collection.
If recursive is enabled, the function iterates into nested
collections, using the following rules:
- Iteration starts at the root.
- For every nested collection type:
- First return the collection type itself.
- Then recurse into the collection, and loop back to item (1)
in the list
- Any mutation done on a collection before recursing into it,
are preserved.
The function uses the function closure syntax to allow mutating
the value for each item in the collection.
The same scoping rules apply to closure blocks as they do for
regular blocks, meaning, any variable defined in parent scopes
are accessible, and mutations to those variables are preserved,
but any new variables instantiated in the closure block are
unavailable outside of the block.
Check out the examples below to learn about the closure syntax.
The object or array to iterate.
Whether to recursively iterate the collection.
Source:
map_values(.) -> |value| { upcase!(value) }
Return:
{"foo":"FOO","bar":"BAR"}
Determines whether the elements in the value array matches the pattern. By default, it checks that at least one element matches, but can be set to determine if all the elements match.
The regular expression pattern to match against.
Whether to match on all elements of value.
Source:
match_array(["foobar", "bazqux"], r'foo')
Return:
Source:
match_array(["foo", "foobar", "barfoo"], r'foo', all: true)
Return:
Source:
match_array(["bazqux", "xyz"], r'foo')
Return:
Source:
match_array(["foo", "foobar", "baz"], r'foo', all: true)
Return:
Returns the number of UTF-8 characters in value. This differs from
length which counts the number of bytes of a string.
Note: This is the count of Unicode scalar values
which can sometimes differ from Unicode code points.
Unflattens the value into a nested representation.
The array or object to unflatten.
The separator to split flattened keys.
Whether to recursively unflatten the object values.
Source:
unflatten({
"foo.bar.baz": true,
"foo.bar.qux": false,
"foo.quux": 42
})
Return:
{"foo":{"bar":{"baz":true,"qux":false},"quux":42}}
Source:
unflatten({
"flattened.parent": {
"foo.bar": true,
"foo.baz": false
}
})
Return:
{"flattened":{"parent":{"foo":{"bar":true,"baz":false}}}}
Source:
unflatten({
"flattened.parent": {
"foo.bar": true,
"foo.baz": false
}
}, recursive: false)
Return:
{"flattened":{"parent":{"foo.bar":true,"foo.baz":false}}}
Source:
unflatten({
"a": 3,
"a.b": 2,
"a.c": 4
})
Return:
Returns the unique values for an array.
The first occurrence of each element is kept.
The array to return unique elements from.
Source:
unique(["foo", "bar", "foo", "baz"])
Return:
Returns the values from the object passed into the function.
The object to extract values from.
Source:
values({"key1": "val1", "key2": "val2"})
Return:
Returns the value of the given secret from an event.
Source:
get_secret("datadog_api_key")
Return:
Removes a secret from an event.
The name of the secret to remove.
Source:
remove_secret("datadog_api_key")
Return:
Sets the given secret in the event.
Source:
set_secret("datadog_api_key", "abc122")
Return:
Sets a semantic meaning for an event. Note: This function assigns
meaning at startup, and has no runtime behavior. It is suggested
to put all calls to this function at the beginning of a DPL function. The function
cannot be conditionally called. For example, using an if statement cannot stop the meaning
from being assigned.
The path of the value that is assigned a meaning.
The name of the meaning to assign.
Source:
set_semantic_meaning(.foo, "bar")
Return:
Converts IPv4 address in numbers-and-dots notation into network-order
bytes represented as an integer.
This behavior mimics inet_aton.
The IP address to convert to binary.
value is not a valid IPv4 address.
Determines whether the ip is contained in the block referenced by the cidr.
The CIDR mask (v4 or v6).
The IP address (v4 or v6).
cidr is not a valid CIDR.ip is not a valid IP address.
Source:
ip_cidr_contains!("192.168.0.0/16", "192.168.10.32")
Return:
Source:
ip_cidr_contains!(["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"], "192.168.10.32")
Return:
Source:
ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1")
Return:
Converts numeric representation of IPv4 address in network-order bytes
to numbers-and-dots notation.
This behavior mimics inet_ntoa.
The integer representation of an IPv4 address.
value cannot fit in an unsigned 32-bit integer.
Converts IPv4 and IPv6 addresses from binary to text form.
This behavior mimics inet_ntop.
The binary data to convert from.
For IPv4 addresses, it must be 4 bytes (32 bits) long.
For IPv6 addresses, it must be 16 bytes (128 bits) long.
value must be of length 4 or 16 bytes.
Source:
ip_ntop!(decode_base64!("wKgAAQ=="))
Return:
Source:
ip_ntop!(decode_base64!("IAENuIWjAAAAAIouA3BzNA=="))
Return:
"2001:db8:85a3::8a2e:370:7334"
Converts IPv4 and IPv6 addresses from text to binary form.
- The binary form of IPv4 addresses is 4 bytes (32 bits) long.
- The binary form of IPv6 addresses is 16 bytes (128 bits) long.
This behavior mimics inet_pton.
The IP address (v4 or v6) to convert to binary form.
value is not a valid IP (v4 or v6) address in text form.
Source:
encode_base64(ip_pton!("192.168.0.1"))
Return:
Source:
encode_base64(ip_pton!("2001:db8:85a3::8a2e:370:7334"))
Return:
"IAENuIWjAAAAAIouA3BzNA=="
Extracts the subnet address from the ip using the supplied subnet.
The IP address (v4 or v6).
The subnet to extract from the IP address. This can be either a prefix length like /8 or a net mask
like 255.255.0.0. The net mask can be either an IPv4 or IPv6 address.
ip is not a valid IP address.subnet is not a valid subnet.
Source:
ip_subnet!("192.168.10.32", "255.255.255.0")
Return:
Source:
ip_subnet!("2404:6800:4003:c02::64", "/32")
Return:
Converts the ip to an IPv6 address.
The IP address to convert to IPv6.
ip is not a valid IP address.
Source:
ip_to_ipv6!("192.168.10.32")
Return:
Converts the ip to an IPv4 address. ip is returned unchanged if it’s already an IPv4 address. If ip is
currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown.
The IPv4-mapped IPv6 address to convert.
ip is not a valid IP address.ip is an IPv6 address that is not compatible with IPv4.
Source:
ipv6_to_ipv4!("::ffff:192.168.0.1")
Return:
Check if the string is a valid IPv4 address or not.
An IPv4-mapped or
IPv4-compatible IPv6 address is not considered
valid for the purpose of this function.
Source:
Return:
Source:
is_ipv4("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Return:
Source:
Return:
Check if the string is a valid IPv6 address or not.
Source:
is_ipv6("2001:0db8:85a3:0000:0000:8a2e:0370:7334")
Return:
Source:
Return:
Source:
Return:
Computes the absolute value of value.
The number to calculate the absolute value.
Source:
Return:
Source:
Return:
Rounds the value up to the specified precision.
The number of decimal places to round to.
Source:
Return:
Source:
ceil(4.345, precision: 2)
Return:
Rounds the value down to the specified precision.
The number to round down.
The number of decimal places to round to.
Source:
Return:
Source:
floor(4.345, precision: 2)
Return:
Formats the integer value into a string representation using the given base/radix.
The base to format the number in. Must be between 2 and 36 (inclusive).
- The base is not between 2 and 36.
Source:
Return:
Source:
Return:
Formats the value into a string representation of the number.
The number to format as a string.
The number of decimal places to display.
The character to use between the whole and decimal parts of the number.
The character to use between each thousands part of the number.
Source:
format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",")
Return:
Calculates the remainder of value divided by modulus.
The value the modulus is applied to.
value is not an integer or float.modulus is not an integer or float.modulus is equal to 0.
Rounds the value to the specified precision.
The number of decimal places to round to.
Source:
Return:
Source:
round(4.345, precision: 2)
Return:
Matches an object against a Datadog Search Syntax query.
The Datadog Search Syntax query.
Source:
match_datadog_query({"message": "contains this and that"}, "this OR that")
Return:
Source:
match_datadog_query({"message": "contains only this"}, "this AND that")
Return:
Source:
match_datadog_query({"name": "foobar"}, "@name:foo*")
Return:
Source:
match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]')
Return:
Merges the from object into the to object.
The object to merge into.
The object to merge from.
A deep merge is performed if true, otherwise only top-level fields are merged.
Source:
merge(
{
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
},
{
"parent1": {
"child2": 4,
"child5": 5
}
}
)
Return:
{"parent1":{"child2":4,"child5":5},"parent2":{"child3":3}}
Source:
merge(
{
"parent1": {
"child1": 1,
"child2": 2
},
"parent2": {
"child3": 3
}
},
{
"parent1": {
"child2": 4,
"child5": 5
}
},
deep: true
)
Return:
{"parent1":{"child1":1,"child2":4,"child5":5},"parent2":{"child3":3}}
Iterate over either one array of arrays or a pair of arrays and create an object out of all the key-value pairs contained in them.
With one array of arrays, any entries with no value use null instead.
Any keys that are null skip the corresponding value.
If a single parameter is given, it must contain an array of all the input arrays.
The first array of elements, or the array of input arrays if no other parameter is present.
The second array of elements. If not present, the first parameter must contain all the arrays.
values and keys must be arrays.- If
keys is not present, values must contain only arrays.
Source:
object_from_array([["one", 1], [null, 2], ["two", 3]])
Return:
Source:
object_from_array([1, 2, 3], keys: ["one", null, "two"])
Return:
Unnest an array field from an object to create an array of objects using that field; keeping all other fields.
Assigning the array result of this to . results in multiple events being emitted from remap. See the
remap transform docs for more details.
This is also referred to as explode in some languages.
The path of the field to unnest.
- The field path referred to is not an array.
Source:
Source:
. = unnest!(.event.messages)
Parses Apache access and error log lines. Lines can be in common,
combined, or the default error format.
The date/time format to use for
encoding the timestamp. The time is parsed in local time if the timestamp does not specify a timezone.
The format to use for parsing the log.
value does not match the specified format.timestamp_format is not a valid format string.- The timestamp in
value fails to parse using the provided timestamp_format.
Source:
parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common")
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}
Source:
parse_apache_log!(
s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"',
"combined",
)
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326,"referrer":"http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth","agent":"Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"}
Source:
parse_apache_log!(
s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!',
"error"
)
Return:
{"client":"147.159.108.175","message":"I will bypass the haptic COM bandwidth, that should matrix the CSS driver!","module":"ab","pid":4803,"port":24259,"severity":"alert","thread":"3814","timestamp":"2021-03-01T12:00:19Z"}
Parses value in the Elastic Load Balancer Access format.
Access log of the Application Load Balancer.
value is not a properly formatted AWS ALB log.
Source:
parse_aws_alb_log!(
"http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\""
)
Return:
{"type":"http","timestamp":"2018-11-30T22:23:00.186641Z","elb":"app/my-loadbalancer/50dc6c495c0c9188","client_host":"192.168.131.39:2817","target_host":null,"request_processing_time":0,"target_processing_time":0.001,"response_processing_time":0,"elb_status_code":"200","target_status_code":"200","received_bytes":34,"sent_bytes":366,"request_method":"GET","request_url":"http://www.example.com:80/","request_protocol":"HTTP/1.1","user_agent":"curl/7.46.0","ssl_cipher":null,"ssl_protocol":null,"target_group_arn":"arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067","trace_id":"Root=1-58337364-23a8c76965a2ef7629b185e3","traceability_id":null,"domain_name":null,"chosen_cert_arn":null,"matched_rule_priority":"0","request_creation_time":"2018-11-30T22:22:48.364000Z","actions_executed":"forward","redirect_url":null,"error_reason":null,"target_port_list":[],"target_status_code_list":[],"classification":null,"classification_reason":null}
Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the
aws_kinesis_firehose source.
The string representation of the message to parse.
value is not a properly formatted AWS CloudWatch Log subscription message.
Source:
parse_aws_cloudwatch_log_subscription_message!(.message)
Return:
{"owner":"111111111111","message_type":"DATA_MESSAGE","log_group":"test","log_stream":"test","subscription_filters":["Destination"],"log_events":[{"id":"35683658089614582423604394983260738922885519999578275840","message":"{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}","timestamp":"2020-09-14T19:09:29.039Z"}]}
Parses value in the VPC Flow Logs format.
value is not a properly formatted AWS VPC Flow log.
Source:
parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA")
Return:
{"version":2,"account_id":"123456789010","interface_id":"eni-1235b8ca123456789","srcaddr":null,"dstaddr":null,"srcport":null,"dstport":null,"protocol":null,"packets":null,"bytes":null,"start":1431280876,"end":1431280934,"action":null,"log_status":"NODATA"}
Source:
parse_aws_vpc_flow_log!(
"- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5",
"instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr"
)
Return:
{"instance_id":null,"interface_id":"eni-1235b8ca123456789","srcaddr":"10.0.1.5","dstaddr":"10.0.0.220","pkt_srcaddr":"10.0.1.5","pkt_dstaddr":"203.0.113.5"}
Source:
parse_aws_vpc_flow_log!("5 52.95.128.179 10.0.0.71 80 34210 6 1616729292 1616729349 IPv4 14 15044 123456789012 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-0c50d5961bcb2d47b eni-1235b8ca123456789 ap-southeast-2 apse2-az3 - - ACCEPT 19 52.95.128.179 10.0.0.71 S3 - - ingress OK",
format: "version srcaddr dstaddr srcport dstport protocol start end type packets bytes account_id vpc_id subnet_id instance_id interface_id region az_id sublocation_type sublocation_id action tcp_flags pkt_srcaddr pkt_dstaddr pkt_src_aws_service pkt_dst_aws_service traffic_path flow_direction log_status")
Return:
{"account_id":"123456789012","action":"ACCEPT","az_id":"apse2-az3","bytes":15044,"dstaddr":"10.0.0.71","dstport":34210,"end":1616729349,"flow_direction":"ingress","instance_id":"i-0c50d5961bcb2d47b","interface_id":"eni-1235b8ca123456789","log_status":"OK","packets":14,"pkt_dst_aws_service":null,"pkt_dstaddr":"10.0.0.71","pkt_src_aws_service":"S3","pkt_srcaddr":"52.95.128.179","protocol":6,"region":"ap-southeast-2","srcaddr":"52.95.128.179","srcport":80,"start":1616729292,"sublocation_id":null,"sublocation_type":null,"subnet_id":"subnet-aaaaaaaa012345678","tcp_flags":19,"traffic_path":null,"type":"IPv4","version":5,"vpc_id":"vpc-abcdefab012345678"}
Parses the value into a human-readable bytes format specified by unit and base.
The string of the duration with either binary or SI unit.
The output units for the byte.
The base for the byte, either 2 or 10.
value is not a properly formatted bytes.
Source:
parse_bytes!("1024KiB", unit: "MiB")
Return:
Source:
parse_bytes!("4TB", unit: "MB", base: "10")
Return:
Source:
parse_bytes!("1GB", unit: "B", base: "2")
Return:
Parses the value as CBOR.
The CBOR payload to parse.
value is not a valid CBOR-formatted payload.
Source:
parse_cbor!(decode_base64!("oWVmaWVsZGV2YWx1ZQ=="))
Return:
Parses the value in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.
Toggles translation of custom field pairs to key:value.
value is not a properly formatted CEF string.
Source:
parse_cef!(
"CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None"
)
Return:
{"cefVersion":"0","deviceVendor":"CyberArk","deviceProduct":"PTA","deviceVersion":"12.6","deviceEventClassId":"1","name":"Suspected credentials theft","severity":"8","suser":"mike2@prod1.domain.com","shost":"prod1.domain.com","src":"1.1.1.1","duser":"andy@dev1.domain.com","dhost":"dev1.domain.com","dst":"2.2.2.2","cs1Label":"ExtraData","cs1":"None","cs2Label":"EventID","cs2":"52b06812ec3500ed864c461e","deviceCustomDate1Label":"detectionDate","deviceCustomDate1":"1388577900000","cs3Label":"PTAlink","cs3":"https://1.1.1.1/incidents/52b06812ec3500ed864c461e","cs4Label":"ExternalLink","cs4":"None"}
Source:
parse_cef!(
"Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232"
)
Return:
{"cefVersion":"1","deviceVendor":"Security","deviceProduct":"threatmanager","deviceVersion":"1.0","deviceEventClassId":"100","name":"worm successfully stopped","severity":"10","src":"10.0.0.1","dst":"2.1.2.2","spt":"1232"}
Source:
parse_cef!(
"CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address",
translate_custom_fields: true
)
Return:
{"cefVersion":"0","deviceVendor":"Dev","deviceProduct":"firewall","deviceVersion":"2.2","deviceEventClassId":"1","name":"Connection denied","severity":"5","Device IPv6 Address":"2345:0425:2CA1:0000:0000:0567:5673:23b5"}
Parses the value using the Common Log Format (CLF).
value does not match the Common Log Format.timestamp_format is not a valid format string.- The timestamp in
value fails to parse using the provided timestamp_format.
Source:
parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326")
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}
Source:
parse_common_log!(
"127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326",
"%+"
)
Return:
{"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326}
Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value.
The field delimiter to use when parsing. Must be a single-byte utf8 character.
- The delimiter must be a single-byte UTF-8 character.
value is not a valid CSV string.
Source:
parse_csv!("foo,bar,\"foo \"\", bar\"")
Return:
["foo","bar","foo \", bar"]
Source:
parse_csv!("foo bar", delimiter: " ")
Return:
Parses the value as base64 encoded DNSTAP data.
The base64 encoded representation of the DNSTAP data to parse.
Whether to turn all hostnames found in resulting data lowercase, for consistency.
value is not a valid base64 encoded string.- dnstap parsing failed for
value
Source:
parse_dnstap!("ChVqYW1lcy1WaXJ0dWFsLU1hY2hpbmUSC0JJTkQgOS4xNi4zGgBy5wEIAxACGAEiEAAAAAAAAAAAAAAAAAAAAAAqECABBQJwlAAAAAAAAAAAADAw8+0CODVA7+zq9wVNMU3WNlI2kwIAAAABAAAAAAABCWZhY2Vib29rMQNjb20AAAEAAQAAKQIAAACAAAAMAAoACOxjCAG9zVgzWgUDY29tAGAAbQAAAAByZLM4AAAAAQAAAAAAAQJoNQdleGFtcGxlA2NvbQAABgABAAApBNABAUAAADkADwA1AAlubyBTRVAgbWF0Y2hpbmcgdGhlIERTIGZvdW5kIGZvciBkbnNzZWMtZmFpbGVkLm9yZy54AQ==")
Return:
{"dataType":"Message","dataTypeId":1,"extraInfo":"","messageType":"ResolverQuery","messageTypeId":3,"queryZone":"com.","requestData":{"fullRcode":0,"header":{"aa":false,"ad":false,"anCount":0,"arCount":1,"cd":false,"id":37634,"nsCount":0,"opcode":0,"qdCount":1,"qr":0,"ra":false,"rcode":0,"rd":false,"tc":false},"opt":{"do":true,"ednsVersion":0,"extendedRcode":0,"options":[{"optCode":10,"optName":"Cookie","optValue":"7GMIAb3NWDM="}],"udpPayloadSize":512},"question":[{"class":"IN","domainName":"facebook1.com.","questionType":"A","questionTypeId":1}],"rcodeName":"NoError"},"responseData":{"fullRcode":16,"header":{"aa":false,"ad":false,"anCount":0,"arCount":1,"cd":false,"id":45880,"nsCount":0,"opcode":0,"qdCount":1,"qr":0,"ra":false,"rcode":16,"rd":false,"tc":false},"opt":{"do":false,"ednsVersion":1,"extendedRcode":1,"ede":[{"extraText":"no SEP matching the DS found for dnssec-failed.org.","infoCode":9,"purpose":"DNSKEY Missing"}],"udpPayloadSize":1232},"question":[{"class":"IN","domainName":"h5.example.com.","questionType":"SOA","questionTypeId":6}],"rcodeName":"BADSIG"},"responseAddress":"2001:502:7094::30","responsePort":53,"serverId":"james-Virtual-Machine","serverVersion":"BIND 9.16.3","socketFamily":"INET6","socketProtocol":"UDP","sourceAddress":"::","sourcePort":46835,"time":1593489007920014000,"timePrecision":"ns","timestamp":"2020-06-30T03:50:07.920014129Z"}
Parses the value into a human-readable duration format specified by unit.
The string of the duration.
The output units for the duration.
value is not a properly formatted duration.
Source:
parse_duration!("1005ms", unit: "s")
Return:
Source:
parse_duration!("1s 1ms", unit: "ms")
Return:
Parses the eTLD from value representing domain name.
Can be provided to get additional parts of the domain name. When 1 is passed,
eTLD+1 will be returned, which represents a domain registrable by a single
organization. Higher numbers will return subdomains.
- unable to determine eTLD for
value
Source:
parse_etld!("sub.sussex.ac.uk")
Return:
{"etld":"ac.uk","etld_plus":"ac.uk","known_suffix":true}
Source:
parse_etld!("sub.sussex.ac.uk", plus_parts: 1)
Return:
{"etld":"ac.uk","etld_plus":"sussex.ac.uk","known_suffix":true}
Source:
parse_etld!("vector.acmecorp")
Return:
{"etld":"acmecorp","etld_plus":"acmecorp","known_suffix":false}
Source:
parse_etld!("vector.acmecorp", psl: "resources/public_suffix_list.dat")
Return:
{"etld":"acmecorp","etld_plus":"acmecorp","known_suffix":false}
Parses the value using the glog (Google Logging Library) format.
value does not match the glog format.
Source:
parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!")
Return:
{"level":"info","timestamp":"2021-01-31T14:48:54.411655Z","id":15520,"file":"main.c++","line":9,"message":"Hello world!"}
Parses the value using the grok format. All patterns listed here
are supported.
value fails to parse using the provided pattern.
Source:
parse_grok!(
"2020-10-02T23:22:12.223222Z info Hello world",
"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}"
)
Return:
{"timestamp":"2020-10-02T23:22:12.223222Z","level":"info","message":"Hello world"}
Parses the value using multiple grok patterns. All patterns listed here
are supported.
The Grok patterns, which are tried in order until the first match.
The shared set of grok aliases that can be referenced in the patterns to simplify them.
Path to the file containing aliases in a JSON format.
value fails to parse using the provided pattern.patterns is not an array.aliases is not an object.alias_sources is not a string or doesn’t point to a valid file.
Source:
parse_groks!(
"2020-10-02T23:22:12.223222Z info Hello world",
patterns: [
"%{common_prefix} %{_status} %{_message}",
"%{common_prefix} %{_message}",
],
aliases: {
"common_prefix": "%{_timestamp} %{_loglevel}",
"_timestamp": "%{TIMESTAMP_ISO8601:timestamp}",
"_loglevel": "%{LOGLEVEL:level}",
"_status": "%{POSINT:status}",
"_message": "%{GREEDYDATA:message}"
}
)
Return:
{"timestamp":"2020-10-02T23:22:12.223222Z","level":"info","message":"Hello world"}
Source:
parse_groks!(
"username=foo",
patterns: [ "%{PATTERN_A}" ],
alias_sources: [ "path/to/aliases.json" ]
)
# aliases.json contents:
# {
# "PATTERN_A": "%{PATTERN_B}",
# "PATTERN_B": "username=%{USERNAME:username}"
# }
Parses the value as an InfluxDB line protocol
string, producing a list of Vector-compatible metrics.
The string representation of the InfluxDB line protocol to parse.
value is not a valid InfluxDB line protocol string.- field set contains a field value of type
string. - field set contains a
NaN field value.
Source:
parse_influxdb!("cpu,host=A,region=us-west usage_system=64i,usage_user=10u,temperature=50.5,on=true,sleep=false 1590488773254420000")
Return:
[{"name":"cpu_usage_system","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":64}},{"name":"cpu_usage_user","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":10}},{"name":"cpu_temperature","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":50.5}},{"name":"cpu_on","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":1}},{"name":"cpu_sleep","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":0}}]
Parses the string value representing a number in an optional base/radix to an integer.
The base the number is in. Must be between 2 and 36 (inclusive).
If unspecified, the string prefix is used to
determine the base: “0b”, 8 for “0” or “0o”, 16 for “0x”,
and 10 otherwise.
- The base is not between 2 and 36.
- The number cannot be parsed in the base.
Source:
Return:
Source:
Return:
Source:
Return:
Source:
Return:
Source:
Return:
Parses the value as JSON.
The string representation of the JSON to parse.
Number of layers to parse for nested JSON-formatted documents.
The value must be in the range of 1 to 128.
Whether to parse the JSON in a lossy manner. Replaces invalid UTF-8 characters
with the Unicode character � (U+FFFD) if set to true, otherwise returns an error
if there are any invalid UTF-8 characters present.
value is not a valid JSON-formatted payload.
Source:
parse_json!("{\"key\": \"val\"}")
Return:
Source:
parse_json!("{\"top_level\":{\"key\": \"val\"}}", max_depth: 1)
Return:
{"top_level":"{\"key\": \"val\"}"}
Parses the value in key-value format. Also known as logfmt.
- Keys and values can be wrapped with
". " characters can be escaped using \.
The string that separates the key from the value.
The string that separates each key-value pair.
Defines the acceptance of unnecessary whitespace surrounding the configured key_value_delimiter.
Whether a standalone key should be accepted, the resulting object associates such keys with the boolean value true.
value is not a properly formatted key-value string.
Source:
parse_key_value!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return:
{"@timestamp":"Sun Jan 10 16:47:39 EST 2021","level":"info","msg":"Stopping all fetchers","tag#production":"stopping_fetchers","id":"ConsumerFetcherManager-1382721708341","module":"kafka.consumer.ConsumerFetcherManager"}
Source:
parse_key_value!(
"path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https",
field_delimiter: ",",
key_value_delimiter: ":"
)
Return:
{"path":"/cart_link","host":"store.app.com","fwd":"102.30.171.16","dyno":"web.1","connect":"0ms","service":"87ms","status":"304","bytes":"632","protocol":"https"}
Source:
parse_key_value!(
"env:prod,service:backend,region:eu-east1,beta",
field_delimiter: ",",
key_value_delimiter: ":",
)
Return:
{"env":"prod","service":"backend","region":"eu-east1","beta":true}
Source:
parse_key_value!(
"at=info,method=GET,path=\"/index\",status=200,tags=dev,tags=dummy",
field_delimiter: ",",
key_value_delimiter: "=",
)
Return:
{"at":"info","method":"GET","path":"/index","status":"200","tags":["dev","dummy"]}
Parses the value using the klog format used by Kubernetes components.
value does not match the klog format.
Source:
parse_klog!("I0505 17:59:40.692994 28133 klog.go:70] hello from klog")
Return:
{"file":"klog.go","id":28133,"level":"info","line":70,"message":"hello from klog","timestamp":"2025-05-05T17:59:40.692994Z"}
Parses Linux authorization logs usually found under either /var/log/auth.log (for Debian-based systems) or
/var/log/secure (for RedHat-based systems) according to Syslog format.
The text containing the message to parse.
value is not a properly formatted Syslog message.
Source:
parse_linux_authorization!(
s'Mar 23 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar'
)
Return:
{"appname":"sshd","hostname":"localhost","message":"Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar","procid":1111,"timestamp":"2025-03-23T01:49:58Z"}
Parses the value in logfmt.
- Keys and values can be wrapped using the
" character. " characters can be escaped by the \ character.- As per this logfmt specification, the
parse_logfmt function
accepts standalone keys and assigns them a Boolean value of true.
value is not a properly formatted key-value string
Source:
parse_logfmt!(
"@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager"
)
Return:
{"@timestamp":"Sun Jan 10 16:47:39 EST 2021","level":"info","msg":"Stopping all fetchers","tag#production":"stopping_fetchers","id":"ConsumerFetcherManager-1382721708341","module":"kafka.consumer.ConsumerFetcherManager"}
Parses Nginx access and error log lines. Lines can be in [`combined`](https://nginx.org/en/docs/http/ngx_http_log_module.html),
[`ingress_upstreaminfo`](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/log-format/), [`main`](https://hg.nginx.org/pkg-oss/file/tip/debian/debian/nginx.conf) or [`error`](https://github.com/nginx/nginx/blob/branches/stable-1.18/src/core/ngx_log.c#L102) format.
The date/time format to use for encoding the timestamp. The time is parsed
in local time if the timestamp doesn’t specify a timezone. The default format is %d/%b/%Y:%T %z for
combined logs and %Y/%m/%d %H:%M:%S for error logs.
The format to use for parsing the log.
value does not match the specified format.timestamp_format is not a valid format string.- The timestamp in
value fails to parse using the provided timestamp_format.
Source:
parse_nginx_log!(
s'172.17.0.1 - alice [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"',
"combined",
)
Return:
{"agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36","client":"172.17.0.1","compression":"2.75","referer":"http://localhost/somewhere","request":"POST /not-found HTTP/1.1","size":153,"status":404,"timestamp":"2021-04-01T12:02:31Z","user":"alice"}
Source:
parse_nginx_log!(
s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"',
"error"
)
Return:
{"timestamp":"2021-04-01T13:02:31Z","severity":"error","pid":31,"tid":31,"cid":1,"message":"open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)","client":"172.17.0.1","server":"localhost","request":"POST /not-found HTTP/1.1","host":"localhost:8081"}
Source:
parse_nginx_log!(
s'0.0.0.0 - bob [18/Mar/2023:15:00:00 +0000] "GET /some/path HTTP/2.0" 200 12312 "https://10.0.0.1/some/referer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 462 0.050 [some-upstream-service-9000] [some-other-upstream-5000] 10.0.50.80:9000 19437 0.049 200 752178adb17130b291aefd8c386279e7',
"ingress_upstreaminfo"
)
Return:
{"body_bytes_size":12312,"http_referer":"https://10.0.0.1/some/referer","http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36","proxy_alternative_upstream_name":"some-other-upstream-5000","proxy_upstream_name":"some-upstream-service-9000","remote_addr":"0.0.0.0","remote_user":"bob","req_id":"752178adb17130b291aefd8c386279e7","request":"GET /some/path HTTP/2.0","request_length":462,"request_time":0.05,"status":200,"timestamp":"2023-03-18T15:00:00Z","upstream_addr":"10.0.50.80:9000","upstream_response_length":19437,"upstream_response_time":0.049,"upstream_status":200}
Source:
parse_nginx_log!(
s'172.24.0.3 - alice [31/Dec/2024:17:32:06 +0000] "GET / HTTP/1.1" 200 615 "https://domain.tld/path" "curl/8.11.1" "1.2.3.4, 10.10.1.1"',
"main"
)
Return:
{"body_bytes_size":615,"http_referer":"https://domain.tld/path","http_user_agent":"curl/8.11.1","http_x_forwarded_for":"1.2.3.4, 10.10.1.1","remote_addr":"172.24.0.3","remote_user":"alice","request":"GET / HTTP/1.1","status":200,"timestamp":"2024-12-31T17:32:06Z"}
Parses the value as a protocol buffer payload.
The protocol buffer payload to parse.
The path to the protobuf descriptor set file. Must be a literal string.
This file is the output of protoc -o …
The name of the message type to use for serializing.
Must be a literal string.
value is not a valid proto payload.desc_file file does not exist.message_type message type does not exist in the descriptor file.
Source:
parse_proto!(decode_base64!("Cgdzb21lb25lIggKBjEyMzQ1Ng=="), "resources/protobuf_descriptor_set.desc", "test_protobuf.Person")
Return:
{"name":"someone","phones":[{"number":"123456"}]}
Parses the value as a query string.
Source:
parse_query_string("foo=%2B1&bar=2&bar=3&xyz")
Return:
{"foo":"+1","bar":["2","3"],"xyz":""}
Source:
parse_query_string("?foo%5b%5d=1&foo%5b%5d=2")
Return:
Parses the value using the provided Regex pattern.
This function differs from the parse_regex_all function in that it returns only the first match.
The regular expression pattern to search against.
If true, the index of each group in the regular expression is also captured. Index 0
contains the whole match.
value fails to parse using the provided pattern.
Source:
parse_regex!("first group and second group.", r'(?P<number>.*?) group')
Return:
Source:
parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true)
Return:
{"0":"first group","1":"first"}
Parses the value using the provided Regex pattern.
This function differs from the parse_regex function in that it returns all matches, not just the first.
The regular expression pattern to search against.
If true, the index of each group in the regular expression is also captured. Index 0
contains the whole match.
value is not a string.pattern is not a regex.
Source:
parse_regex_all!("first group and second group.", r'(?P<number>\w+) group', numeric_groups: true)
Return:
[{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}]
Parses the value as ruby hash.
The string representation of the ruby hash to parse.
value is not a valid ruby hash formatted payload.
Source:
parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }')
Return:
{"test":"value","testNum":0.2,"testObj":{"testBool":true,"testNull":null}}
Parses the value in Syslog format.
The text containing the Syslog message to parse.
value is not a properly formatted Syslog message.
Source:
parse_syslog!(
s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!'
)
Return:
{"severity":"notice","facility":"user","timestamp":"2020-03-13T20:45:38.119Z","hostname":"dynamicwireless.name","appname":"non","procid":2426,"msgid":"ID931","message":"Try to override the THX port, maybe it will reboot the neural interface!","exampleSDID@32473":{"eventID":"1011","eventSource":"Application","iut":"3"},"version":1}
Parses the value in strptime format.
The text of the timestamp.
The TZ database format. By default, this function parses the timestamp by global timezone option.
This argument overwrites the setting and is useful for parsing timestamps without a specified timezone, such as 16/10/2019 12:00:00.
value fails to parse using the provided format.value fails to parse using the provided timezone.
Source:
parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z")
Return:
Source:
parse_timestamp!("16/10/2019 12:00:00", format: "%d/%m/%Y %H:%M:%S", timezone: "Asia/Taipei")
Return:
Parses the value in token format. A token is considered to be one of the following:
- A word surrounded by whitespace.
- Text delimited by double quotes:
"..". Quotes can be included in the token if they are escaped by a backslash (\). - Text delimited by square brackets:
[..]. Closing square brackets can be included in the token if they are escaped by a backslash (\).
value is not a properly formatted tokenized string.
Source:
parse_tokens(
"A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]"
)
Return:
["A","sentence","with \\\"a\\\" sentence inside","and","some brackets"]
Parses the value in URL format.
If true and the port number is not specified in the input URL
string (or matches the default port for the scheme), it is
populated from well-known ports for the following schemes:
http, https, ws, wss, and ftp.
value is not a properly formatted URL.
Source:
parse_url!("ftp://foo:bar@example.com:4343/foobar?hello=world#123")
Return:
{"scheme":"ftp","username":"foo","password":"bar","host":"example.com","port":4343,"path":"/foobar","query":{"hello":"world"},"fragment":"123"}
Source:
parse_url!("https://example.com", default_known_ports: true)
Return:
{"scheme":"https","username":"","password":"","host":"example.com","port":443,"path":"/","query":{},"fragment":null}
Source:
parse_url!("https://www.café.com")
Return:
{"scheme":"https","username":"","password":"","host":"www.xn--caf-dma.com","port":null,"path":"/","query":{},"fragment":null}
Source:
parse_url!("https://www.CAFé.com")
Return:
{"scheme":"https","username":"","password":"","host":"www.xn--caf-dma.com","port":null,"path":"/","query":{},"fragment":null}
Parses the value as a user agent string, which has a loosely defined format
so this parser only provides best effort guarantee.
Determines performance and reliability characteristics.
Source:
parse_user_agent(
"Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1"
)
Return:
{"browser":{"family":"Firefox","version":"1.0.1"},"device":{"category":"pc"},"os":{"family":"Linux","version":null}}
Source:
parse_user_agent(
"Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)",
mode: "reliable"
)
Return:
{"browser":{"family":"Internet Explorer","version":"7.66"},"device":{"category":"pc"},"os":{"family":"Windows XP","version":"NT 5.1"}}
Source:
parse_user_agent(
"Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54",
mode: "enriched"
)
Return:
{"browser":{"family":"Opera Mini","major":"4","minor":"3","patch":"24214","version":"10.54"},"device":{"brand":"Apple","category":"smartphone","family":"iPhone","model":"iPhone"},"os":{"family":"iOS","major":"4","minor":"2","patch":"1","patch_minor":null,"version":"4.2.1"}}
Parses the value as XML.
The string representation of the XML document to parse.
Include XML tag attributes in the returned object.
String prefix to use for XML tag attribute keys.
Key name to use for expanded text nodes.
Always return text nodes as {"<text_key>": "value"}.
Parse “true” and “false” as boolean.
Parse numbers as integers/floats.
value is not a valid XML document.
Source:
value = s'<book category="CHILDREN"><title lang="en">Harry Potter</title><author>J K. Rowling</author><year>2005</year></book>';
parse_xml!(value, text_key: "value", parse_number: false)
Return:
{"book":{"@category":"CHILDREN","author":"J K. Rowling","title":{"@lang":"en","value":"Harry Potter"},"year":"2005"}}
Removes the field specified by the static path from the target.
For dynamic path deletion, see the remove function.
The path of the field to delete.
After deletion, if compact is true and there is an empty object or array left,
the empty object or array is also removed, cascading up to the root. This only
applies to the path being deleted, and any parent paths.
Source:
Source:
.new_field = del(.old_field)
Checks whether the path exists for the target.
This function distinguishes between a missing path
and a path with a null value. A regular path lookup,
such as .foo, cannot distinguish between the two cases
since it always returns null if the path doesn’t exist.
The path of the field to check.
Source:
Return:
Source:
Return:
Dynamically get the value of a given path.
If you know the path you want to look up, use
static paths such as .foo.bar[1] to get the value of that
path. However, if you do not know the path names,
use the dynamic get function to get the requested
value.
The object or array to query.
An array of path segments to look for the value.
- The
path segment must be a string or an integer.
Source:
get!(value: { "foo": "bar" }, path: ["foo"])
Return:
Source:
get!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return:
Source:
get!(value: ["foo", "bar", "baz"], path: [-2])
Return:
Dynamically remove the value for a given path.
If you know the path you want to remove, use
the del function and static paths such as del(.foo.bar[1])
to remove the value at that path. The del function returns the
deleted value, and is more performant than remove.
However, if you do not know the path names, use the dynamic
remove function to remove the value at the provided path.
The object or array to remove data from.
An array of path segments to remove the value from.
After deletion, if compact is true, any empty objects or
arrays left are also removed.
- The
path segment must be a string or an integer.
Source:
remove!(value: { "foo": "bar" }, path: ["foo"])
Return:
Source:
remove!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"])
Return:
Source:
remove!(value: ["foo", "bar", "baz"], path: [-2])
Return:
Source:
remove!(value: { "foo": { "bar": [42], "baz": true } }, path: ["foo", "bar", 0], compact: true)
Return:
Dynamically insert data into the path of a given object or array.
If you know the path you want to assign a value to,
use static path assignments such as .foo.bar[1] = true for
improved performance and readability. However, if you do not
know the path names, use the dynamic set function to
insert the data into the object or array.
The object or array to insert data into.
An array of path segments to insert the value into.
- The
path segment must be a string or an integer.
Source:
set!(value: { "foo": "bar" }, path: ["foo"], data: "baz")
Return:
Source:
set!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"], data: "qux")
Return:
Source:
set!(value: ["foo", "bar", "baz"], path: [-2], data: 42)
Return:
Returns a random boolean.
Source:
is_boolean(random_bool())
Return:
A cryptographically secure random number generator. Returns a string value containing the number of
random bytes requested.
The number of bytes to generate. Must not be larger than 64k.
length is negative.length is larger than the maximum value (64k).
Source:
encode_base64(random_bytes(16))
Return:
"LNu0BBgUbh7XAlXbjSOomQ=="
Returns a random float between [min, max).
Minimum value (inclusive).
Maximum value (exclusive).
max is not greater than min.
Source:
f = random_float(0.0, 10.0)
f >= 0 && f < 10
Return:
Returns a random integer between [min, max).
Minimum value (inclusive).
Maximum value (exclusive).
max is not greater than min.
Source:
i = random_int(0, 10)
i >= 0 && i < 10
Return:
Convert a Friendly ID (base62 encoding a 128-bit word) to a UUID.
A string that is a Friendly ID
value is a string but the text uses characters outside of class [0-9A-Za-z].value is a base62 encoding of an integer, but the integer is greater than or equal to 2^128.
Source:
uuid_from_friendly_id!("3s87yEvnmkiPBMHsj8bwwc")
Return:
"7f41deed-d5e2-8b5e-7a13-ab4ff93cfad2"
Generates a random UUIDv4 string.
Source:
Return:
"1d262f4f-199b-458d-879f-05fd0a5f0683"
Generates a random UUIDv7 string.
The timestamp used to generate the UUIDv7.
Source:
Return:
"06338364-8305-7b74-8000-de4963503139"
Source:
Return:
"018e29b3-0bea-7f78-8af3-d32ccb1b93c1"
Source:
uuid_v7(t'2020-12-30T22:20:53.824727Z')
Return:
"0176b5bd-5d19-7394-bb60-c21028c6152b"
Takes the value string, and turns it into camelCase. Optionally, you can
pass in the existing case of the function, or else an attempt is made to determine the case automatically.
The string to convert to camelCase.
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
Source:
camelcase("input-string")
Return:
Source:
camelcase("input-string", "kebab-case")
Return:
Generates an ID based on the Community ID Spec.
The destination IP address.
The source port or ICMP type.
The destination port or ICMP code.
Source:
community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", source_port: 1122, destination_port: 3344, protocol: 6)
Return:
"1:wCb3OG7yAFWelaUydu0D+125CLM="
Determines whether the value string contains the specified substring.
The substring to search for in value.
Whether the match should be case sensitive.
Source:
contains("The Needle In The Haystack", "Needle")
Return:
Source:
contains("The Needle In The Haystack", "needle", case_sensitive: false)
Return:
Determines whether the value string contains all the specified substrings.
An array of substrings to search for in value.
Whether the match should be case sensitive.
Source:
contains_all("The Needle In The Haystack", ["Needle", "Haystack"])
Return:
Source:
contains_all("the NEEDLE in the haystack", ["needle", "haystack"])
Return:
Downcases the value string, where downcase is defined according to the
Unicode Derived Core Property Lowercase.
The string to convert to lowercase.
Source:
downcase("Hello, World!")
Return:
Determines whether the value string ends with the specified substring.
The substring with which value must end.
Whether the match should be case sensitive.
Source:
ends_with("The Needle In The Haystack", "The Haystack")
Return:
Source:
ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false)
Return:
Determines from left to right the start position of the first found element in value
that matches pattern. Returns -1 if not found.
The string to find the pattern in.
The regular expression or string pattern to match against.
Offset to start searching.
Source:
Return:
Source:
Return:
Source:
Return:
Source:
find("foobarfoobarfoo", "bar", 4)
Return:
Joins each string in the value array into a single string, with items optionally separated from one another
by a separator.
The array of strings to join together.
The string separating each original element when joined.
Source:
join!(["bring", "us", "together"])
Return:
Source:
join!(["sources", "transforms", "sinks"], separator: ", ")
Return:
"sources, transforms, sinks"
Takes the value string, and turns it into kebab-case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
The string to convert to kebab-case.
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
Source:
Return:
Source:
kebabcase("InputString", "PascalCase")
Return:
Determines whether the value matches the pattern.
The regular expression pattern to match against.
Source:
match("I'm a little teapot", r'teapot')
Return:
Source:
match("I'm a little teapot", r'.*balloon')
Return:
Determines whether value matches any of the given patterns. All
patterns are checked in a single pass over the target string, giving this
function a potential performance advantage over the multiple calls
in the match function.
The array of regular expression patterns to match against.
Source:
match_any("I'm a little teapot", [r'frying pan', r'teapot'])
Return:
Parses the string value representing a floating point number in base 10 to a float.
Source:
Return:
Source:
Return:
Source:
Return:
Takes the value string, and turns it into PascalCase. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
The string to convert to PascalCase.
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
Source:
pascalcase("input-string")
Return:
Source:
pascalcase("input-string", "kebab-case")
Return:
Redact sensitive data in value such as:
This can help achieve compliance by ensuring sensitive data does not leave your network.
The value to redact sensitive data from.
The function’s behavior depends on value’s type:
- For strings, the sensitive data is redacted and a new string is returned.
- For arrays, the sensitive data is redacted in each string element.
- For objects, the sensitive data in each string value is masked, but the keys are not masked.
For arrays and objects, the function recurses into any nested arrays or objects. Any non-string elements are
skipped.
Redacted text is replaced with [REDACTED].
List of filters applied to value.
Each filter can be specified in the following ways:
- As a regular expression, which is used to redact text that match it.
- As an object with a
type key that corresponds to a named filter and additional keys for customizing that filter. - As a named filter, if it has no required parameters.
Named filters can be a:
pattern: Redacts text matching any regular expressions specified in the patterns
key, which is required. This is the expanded version of just passing a regular expression as a filter.us_social_security_number: Redacts US social security card numbers.
See examples for more details.
This parameter must be a static expression so that the argument can be validated at compile-time
to avoid runtime errors. You cannot use variables or other dynamic expressions with it.
Specifies what to replace the redacted strings with.
It is given as an object with a “type” key specifying the type of redactor to use
and additional keys depending on the type. The following types are supported:
full: The default. Replace with the string “[REDACTED]”.text: Replace with a custom string. The replacement key is required, and must
contain the string that is used as a replacement.sha2: Hash the redacted text with SHA-2 as with sha2. Supports two optional parameters:variant: The variant of the algorithm to use. Defaults to SHA-512/256.encoding: How to encode the hash as text. Can be base16 or base64.
Defaults to base64.
sha3: Hash the redacted text with SHA-3 as with sha3. Supports two optional parameters:variant: The variant of the algorithm to use. Defaults to SHA3-512.encoding: How to encode the hash as text. Can be base16 or base64.
Defaults to base64.
As a convenience you can use a string as a shorthand for common redactor patterns:
"full" is equivalent to {"type": "full"}"sha2" is equivalent to {"type": "sha2", "variant": "SHA-512/256", "encoding": "base64"}"sha3" is equivalent to {"type": "sha3", "variant": "SHA3-512", "encoding": "base64"}
This parameter must be a static expression so that the argument can be validated at compile-time
to avoid runtime errors. You cannot use variables or other dynamic expressions with it.
Source:
redact("my id is 123456", filters: [r'\d+'])
Return:
Source:
redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"])
Return:
{"name":"John Doe","ssn":"[REDACTED]"}
Source:
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "text", "replacement": "***"})
Return:
Source:
redact("my id is 123456", filters: [r'\d+'], redactor: "sha2")
Return:
"my id is GEtTedW1p6tC094dDKH+3B8P+xSnZz69AmpjaXRd63I="
Source:
redact("my id is 123456", filters: [r'\d+'], redactor: "sha3")
Return:
"my id is ZNCdmTDI7PeeUTFnpYjLdUObdizo+bIupZdl8yqnTKGdLx6X3JIqPUlUWUoFBikX+yTR+OcvLtAqWO11NPlNJw=="
Source:
redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"})
Return:
"my id is 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92"
Replaces all matching instances of pattern in value.
The pattern argument accepts regular expression capture groups.
Note when using capture groups:
- You will need to escape the
$ by using $$ to avoid Vector interpreting it as an
environment variable when loading configuration - If you want a literal
$ in the replacement pattern, you will also need to escape this
with $$. When combined with environment variable interpolation in config files this
means you will need to use $$$$ to have a literal $ in the replacement pattern.
Replace all matches of this pattern. Can be a static string or a regular expression.
The string that the matches are replaced with.
The maximum number of replacements to perform. -1 means replace all matches.
Source:
replace("Apples and Bananas", "and", "not")
Return:
Source:
replace("Apples and Bananas", r'(?i)bananas', "Pineapples")
Return:
Source:
replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1)
Return:
Source:
replace("foo123bar", r'foo(?P<num>\d+)bar', "$num")
Return:
Replaces all matching instances of pattern using a closure.
The pattern argument accepts a regular expression that can use capture groups.
The function uses the function closure syntax to compute the replacement values.
The closure takes a single parameter, which is an array, where the first item is always
present and contains the entire string that matched pattern. The items from index one on
contain the capture groups of the corresponding index. If a capture group is optional, the
value may be null if it didn’t match.
The value returned by the closure must be a string and will replace the section of
the input that was matched.
This returns a new string with the replacements, the original string is not mutated.
Replace all matches of this pattern. Must be a regular expression.
The maximum number of replacements to perform. -1 means replace all matches.
Source:
replace_with("apples and bananas", r'\b(\w)(\w*)') -> |match| {
upcase!(match.captures[0]) + string!(match.captures[1])
}
Return:
Source:
replace_with("email from test@example.com", r'\w+@example.com') -> |match| {
sha2(match.string, variant: "SHA-512/224")
}
Return:
"email from adf6e1bc4415d24912bd93072ad34ef825a7b6eb3bf53f68def1fc17"
Source:
replace_with("Apples and Apples", r'(?i)apples|cones', count: 1) -> |match| {
"Pine" + downcase(match.string)
}
Return:
Source:
replace_with("level=error A message", r'level=(?P<level>\w+)') -> |match| {
lvl = upcase!(match.level)
"[{{lvl}}]"
}
Return:
Takes the value string, and turns it into SCREAMING_SNAKE case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
The string to convert to SCREAMING_SNAKE case.
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
Source:
screamingsnakecase("input-string")
Return:
Source:
screamingsnakecase("input-string", "kebab-case")
Return:
Generates Shannon entropy from given string. It can generate it
based on string bytes, codepoints, or graphemes.
Defines how to split the string to calculate entropy, based on occurrences of
segments.
Byte segmentation is the fastest, but it might give undesired results when handling
UTF-8 strings, while grapheme segmentation is the slowest, but most correct in these
cases.
Source:
floor(shannon_entropy("vector.dev"), precision: 4)
Return:
Source:
floor(shannon_entropy("test123%456.فوائد.net."), precision: 4)
Return:
Source:
floor(shannon_entropy("test123%456.فوائد.net.", segmentation: "grapheme"), precision: 4)
Return:
Keeps only matches of pattern in value.
This can be used to define patterns that are allowed in the string and
remove everything else.
Keep all matches of this pattern.
The string to use to replace single rejected characters.
The string to use to replace multiple sequential instances of rejected characters.
Source:
sieve("test123%456.فوائد.net.", r'[a-z0-9.]')
Return:
Source:
sieve("test123%456.فوائد.net.", r'[a-z.0-9]', replace_single: "X", replace_repeated: "<REMOVED>")
Return:
"test123X456.<REMOVED>.net."
Returns a slice of value between the start and end positions.
If the start and end parameters are negative, they refer to positions counting from the right of the
string or array. If end refers to a position that is greater than the length of the string or array,
a slice up to the end of the string or array is returned.
The string or array to slice.
The inclusive start position. A zero-based index that can be negative.
The exclusive end position. A zero-based index that can be negative.
Source:
slice!("Supercalifragilisticexpialidocious", start: 5, end: 13)
Return:
Source:
slice!("Supercalifragilisticexpialidocious", start: 5, end: -14)
Return:
Takes the value string, and turns it into snake-case. Optionally, you can
pass in the existing case of the function, or else we will try to figure out the case automatically.
The string to convert to snake-case.
Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case
Source:
snakecase("input-string")
Return:
Source:
snakecase("input-string", "kebab-case")
Return:
Splits the value string using pattern.
The string is split whenever this pattern is matched.
The maximum number of substrings to return.
Source:
split("apples and pears and bananas", " and ")
Return:
["apples","pears","bananas"]
Source:
split("apples and pears and bananas", " and ", limit: 2)
Return:
["apples","pears and bananas"]
Determines whether value begins with substring.
The substring that the value must start with.
Whether the match should be case sensitive.
Source:
starts_with("The Needle In The Haystack", "The Needle")
Return:
Source:
starts_with("The Needle In The Haystack", "the needle", case_sensitive: false)
Return:
Strips ANSI escape codes from value.
Source:
strip_ansi_escape_codes("\e[46mfoo\e[0m bar")
Return:
Strips whitespace from the start and end of value, where whitespace is defined by the Unicode
White_Space property.
Source:
strip_whitespace(" A sentence. ")
Return:
Truncates the value string up to the limit number of characters.
The number of characters to truncate the string after.
This argument is deprecated. An ellipsis (...) is appended if the parameter is set to true and the value string
is truncated because it exceeded the limit.
A custom suffix (...) is appended to truncated strings.
If ellipsis is set to true, this parameter is ignored for backwards compatibility.
Source:
truncate("A rather long sentence.", limit: 11, suffix: "...")
Return:
Source:
truncate("A rather long sentence.", limit: 11, suffix: "[TRUNCATED]")
Return:
Upcases value, where upcase is defined according to the Unicode Derived Core Property
Uppercase.
The string to convert to uppercase.
Returns the value of the environment variable specified by name.
The name of the environment variable.
- Environment variable
name does not exist. - The value of environment variable
name is not valid Unicode
Returns the local system’s hostname.
- Internal hostname resolution failed.
Source:
.hostname = get_hostname!()
Returns the name of the timezone in the Vector configuration (see
global configuration options).
If the configuration is set to local, then it attempts to
determine the name of the timezone from the host OS. If this
is not possible, then it returns the fixed offset of the
local timezone for the current time in the format "[+-]HH:MM",
for example, "+02:00".
- Retrieval of local timezone information failed.
Source:
.vector_timezone = get_timezone_name!()
Formats value into a string representation of the timestamp.
The timestamp to format as text.
The timezone to use when formatting the timestamp. The parameter uses the TZ identifier or local.
Source:
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+")
Return:
"2020-10-21T16:00:00+00:00"
Source:
format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R")
Return:
Returns the current timestamp in the UTC timezone with nanosecond precision.
Source:
Return:
"2021-03-04T10:51:15.928937Z"
Returns value if it is an array, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an array and can be used in any function that expects an array.
The value to check if it is an array.
Returns value if it is a Boolean, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a Boolean and can be used in any function that expects a Boolean.
The value to check if it is a Boolean.
Returns value if it is a float, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a float and can be used in any function that expects a float.
The value to check if it is a float.
Returns value if it is an integer, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an integer and can be used in any function that expects an integer.
The value to check if it is an integer.
Check if the value’s type is an array.
The value to check if it is an array.
Source:
Return:
Source:
Return:
Check if the value’s type is a boolean.
The value to check if it is a Boolean.
Source:
Return:
Source:
Return:
Check if the object, array, or string has a length of 0.
Source:
Return:
Source:
Return:
Source:
Return:
Check if the value’s type is a float.
The value to check if it is a float.
Source:
Return:
Source:
Return:
Check if the value`’s type is an integer.
The value to check if it is an integer.
Source:
Return:
Source:
Return:
Check if the string is a valid JSON document.
The value to check if it is a valid JSON document.
The variant of the JSON type to explicitly check for.
Source:
Return:
Source:
Return:
Source:
is_json("{}", variant: "object")
Return:
Source:
is_json("{}", variant: "array")
Return:
Check if value’s type is null. For a more relaxed function,
see is_nullish.
The value to check if it is null.
Source:
Return:
Source:
Return:
Determines whether value is nullish. Returns true if the specified value is null,
an empty string, a string containing only whitespace, or the string "-". Returns false otherwise.
The value to check for nullishness, for example, a useless value.
Source:
Return:
Source:
Return:
Source:
Return:
Check if value’s type is an object.
The value to check if it is an object.
Source:
is_object({"foo": "bar"})
Return:
Source:
Return:
Check if value’s type is a regex.
The value to check if it is a regex.
Source:
Return:
Source:
Return:
Check if value’s type is a string.
The value to check if it is a string.
Source:
Return:
Source:
Return:
Check if value’s type is a timestamp.
The value to check if it is a timestamp.
Source:
is_timestamp(t'2021-03-26T16:00:00Z')
Return:
Source:
Return:
Returns value if it is an object, otherwise returns an error. This enables the type checker to guarantee that the
returned value is an object and can be used in any function that expects an object.
The value to check if it is an object.
Source:
Return:
{"field1":"value1","field2":"value2"}
Returns value if it is a string, otherwise returns an error. This enables the type checker to guarantee that the
returned value is a string and can be used in any function that expects a string.
The value to check if it is a string.
Adds type information to all (nested) scalar values in the provided value.
The type information is added externally, meaning that value has the form of "type": value after this
transformation.
The value to tag with types.
Source:
tag_types_externally(123)
Return:
Source:
tag_types_externally({
"message": "Hello world",
"request": {
"duration_ms": 67.9
}
})
Return:
{"message":{"string":"Hello world"},"request":{"duration_ms":{"float":67.9}}}
Source:
tag_types_externally(["foo", "bar"])
Return:
[{"string":"foo"},{"string":"bar"}]
Source:
tag_types_externally(null)
Return:
Returns value if it is a timestamp, otherwise returns an error. This enables the type checker to guarantee that
the returned value is a timestamp and can be used in any function that expects a timestamp.
The value to check if it is a timestamp.
value is not a timestamp.
Source:
timestamp(t'2020-10-10T16:00:00Z')
Return:
