--- title: Custom Processor description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: Docs > Observability Pipelines > Processors > Custom Processor --- # Custom Processor {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site). (). {% /alert %} {% /callout %} Available for: {% icon name="icon-logs" /%} Logs | {% icon name="icon-metrics" /%} Metrics ## Overview{% #overview %} Use this processor with Vector Remap Language (VRL) to modify and enrich your logs. VRL is an expression-oriented, domain specific language designed for transforming logs. It features built-in functions for observability use cases. You can use custom functions in the following ways: - Manipulate arrays, strings, and other data types. - Encode and decode values using Codec. - Encrypt and decrypt values. - Coerce one datatype to another datatype (for example, from an integer to a string). - Convert syslog values to read-able values. - Enrich values by using enrichment tables. - Manipulate IP values. - Parse values with custom rules (for example, grok, regex, and so on) and out-of-the-box functions (for example, syslog, apache, VPC flow logs, and so on). See [Writing Effective Grok Parsing Rules with Regular Expressions](https://docs.datadoghq.com/logs/guide/regex_log_parsing/) for information. - Manipulate event paths. See Custom functions for the full list of available functions. See [Remap Reserved Attributes](https://docs.datadoghq.com/observability_pipelines/guide/remap_reserved_attributes) on how to use the Custom Processor to manually and dynamically remap attributes. ## Setup{% #setup %} To set up this processor: - If you have not created any functions yet, click **Add custom processor** and follow the instructions in Add a function to create a function. - If you have already added custom functions, click **Manage custom processors**. Click on a function in the list to edit or delete it. You can use the search bar to find a function by its name. Click **Add Custom Processor** to add a function. ### Add a function{% #add-a-function %} 1. Enter a name for your custom processor. 1. Add your script to modify your logs using [custom functions](https://docs.datadoghq.com/observability_pipelines/guide/remap_reserved_attributes). You can also click **Autofill with Example** and select one of the common use cases to get started. Click the copy icon for the example script and paste it into your script. See [Get Started with the Custom Processor](https://docs.datadoghq.com/observability_pipelines/guide/get_started_with_the_custom_processor) for more information. 1. Optionally, check **Drop events on error** if you want to drop events that encounter an error during processing. 1. Enter a sample log event. 1. Click **Run** to preview how the functions process the log. After the script has run, you can see the output for the log. 1. Click **Save**. ## Custom functions{% #custom-functions %} - [Array](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#array) - [Codec](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#codec) - [Convert](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#convert) - [Cryptography](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#cryptography) - [Debug](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#debug) - [Enrichment](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#enrichment) - [IP](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#ip) - [Number](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#number) - [Object](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#object) - [Parse](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#parse) - [Path](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#path) - [Random](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#random) - [String](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#string) - [System](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#system) - [Timestamp](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#timestamp) - [Type](https://docs.datadoghq.com/observability_pipelines/processors/custom_processor/#type) Array Functions ### append{% #append %} Appends each item in the `items` array to the end of the `value` array. argumentTypeDescriptiondefaultrequired value array The initial array. N/A yes items array The items to append. N/A yes #### Examples{% #append-examples %} ##### Append to an array{% #append-to-an-array %} Source: ```bash append([1, 2], [3, 4]) ``` Return: ```bash [1,2,3,4] ``` ### chunks{% #chunks %} Chunks `value` into slices of length `chunk_size` bytes. argumentTypeDescriptiondefaultrequired value array, string The array of bytes to split. N/A yes chunk_size integer The desired length of each chunk in bytes. This may be constrained by the host platform architecture. N/A yes #### Errors{% #chunks-errors %} - `chunk_size` must be at least 1 byte. - `chunk_size` is too large. #### Examples{% #chunks-examples %} ##### Split a string into chunks{% #split-a-string-into-chunks %} Source: ```bash chunks("abcdefgh", 4) ``` Return: ```bash ["abcd","efgh"] ``` ##### Chunks do not respect unicode code point boundaries{% #chunks-do-not-respect-unicode-code-point-boundaries %} Source: ```bash chunks("ab你好", 4) ``` Return: ```bash ["ab�","�好"] ``` ### push{% #push %} Adds the `item` to the end of the `value` array. argumentTypeDescriptiondefaultrequired value array The target array. N/A yes item any The item to push. N/A yes #### Examples{% #push-examples %} ##### Push an item onto an array{% #push-an-item-onto-an-array %} Source: ```bash push([1, 2], 3) ``` Return: ```bash [1,2,3] ``` ### zip{% #zip %} Iterate over several arrays in parallel, producing a new array containing arrays of items from each source. The resulting array will be as long as the shortest input array, with all the remaining elements dropped. This function is modeled from the `zip` function [in Python](https://docs.python.org/3/library/functions.html#zip), but similar methods can be found in [Ruby](https://docs.ruby-lang.org/en/master/Array.html#method-i-zip) and [Rust](https://doc.rust-lang.org/stable/std/iter/trait.Iterator.html#method.zip). If a single parameter is given, it must contain an array of all the input arrays. argumentTypeDescriptiondefaultrequired array_0 array The first array of elements, or the array of input arrays if no other parameter is present. N/A yes array_1 array The second array of elements. If not present, the first parameter contains all the arrays. N/A no #### Errors{% #zip-errors %} - `array_0` and `array_1` must be arrays. #### Examples{% #zip-examples %} ##### Merge two arrays{% #merge-two-arrays %} Source: ```bash zip([1, 2, 3], [4, 5, 6, 7]) ``` Return: ```bash [[1,4],[2,5],[3,6]] ``` ##### Merge three arrays{% #merge-three-arrays %} Source: ```bash zip([[1, 2], [3, 4], [5, 6]]) ``` Return: ```bash [[1,3,5],[2,4,6]] ``` Checksum Functions ### crc{% #crc %} Calculates a CRC of the `value`. The CRC `algorithm` used can be optionally specified. This function is infallible if either the default `algorithm` value or a recognized-valid compile-time `algorithm` string literal is used. Otherwise, it is fallible. argumentTypeDescriptiondefaultrequired value string The string to calculate the checksum for. N/A yes algorithm string The CRC algorithm to use. `CRC_32_ISO_HDLC` no #### Errors{% #crc-errors %} - `value` is not a string. - `algorithm` is not a supported algorithm. #### Examples{% #crc-examples %} ##### Create CRC checksum using the default algorithm{% #create-crc-checksum-using-the-default-algorithm %} Source: ```bash crc("foo") ``` Return: ```bash "2356372769" ``` ##### Create CRC checksum using the CRC_32_CKSUM algorithm{% #create-crc-checksum-using-the-crc_32_cksum-algorithm %} Source: ```bash crc("foo", algorithm: "CRC_32_CKSUM") ``` Return: ```bash "4271552933" ``` Codec Functions ### decode_base16{% #decode_base16 %} Decodes the `value` (a [Base16](https://en.wikipedia.org/wiki/Hexadecimal) string) into its original string. argumentTypeDescriptiondefaultrequired value string The [Base16](https://en.wikipedia.org/wiki/Hexadecimal) data to decode. N/A yes #### Errors{% #decode_base16-errors %} - `value` isn't a valid encoded Base16 string. #### Examples{% #decode_base16-examples %} ##### Decode Base16 data{% #decode-base16-data %} Source: ```bash decode_base16!("796f752068617665207375636365737366756c6c79206465636f646564206d65") ``` Return: ```bash "you have successfully decoded me" ``` ### decode_base64{% #decode_base64 %} Decodes the `value` (a [Base64](https://en.wikipedia.org/wiki/Base64) string) into its original string. argumentTypeDescriptiondefaultrequired value string The [Base64](https://en.wikipedia.org/wiki/Base64) data to decode. N/A yes charset string The character set to use when decoding the data. `standard` no #### Errors{% #decode_base64-errors %} - `value` isn't a valid encoded Base64 string. #### Examples{% #decode_base64-examples %} ##### Decode Base64 data (default){% #decode-base64-data-default %} Source: ```bash decode_base64!("eW91IGhhdmUgc3VjY2Vzc2Z1bGx5IGRlY29kZWQgbWU=") ``` Return: ```bash "you have successfully decoded me" ``` ##### Decode Base64 data (URL safe){% #decode-base64-data-url-safe %} Source: ```bash decode_base64!("eW91IGNhbid0IG1ha2UgeW91ciBoZWFydCBmZWVsIHNvbWV0aGluZyBpdCB3b24ndA==", charset: "url_safe") ``` Return: ```bash "you can't make your heart feel something it won't" ``` ### decode_charset{% #decode_charset %} Decodes the `value` (a non-UTF8 string) to a UTF8 string using the specified [character set](https://encoding.spec.whatwg.org/#names-and-labels). argumentTypeDescriptiondefaultrequired value string The non-UTF8 string to decode. N/A yes from_charset string The [character set](https://encoding.spec.whatwg.org/#names-and-labels) to use when decoding the data. N/A yes #### Errors{% #decode_charset-errors %} - `from_charset` isn't a valid [character set](https://encoding.spec.whatwg.org/#names-and-labels). #### Examples{% #decode_charset-examples %} ##### Decode EUC-KR string{% #decode-euc-kr-string %} Source: ```bash decode_charset!(decode_base64!("vsiz58fPvLy/5A=="), "euc-kr") ``` Return: ```bash "안녕하세요" ``` ##### Decode EUC-JP string{% #decode-euc-jp-string %} Source: ```bash decode_charset!(decode_base64!("pLOk86TLpMGkzw=="), "euc-jp") ``` Return: ```bash "こんにちは" ``` ##### Decode GB2312 string{% #decode-gb2312-string %} Source: ```bash decode_charset!(decode_base64!("xOO6ww=="), "gb2312") ``` Return: ```bash "你好" ``` ### decode_gzip{% #decode_gzip %} Decodes the `value` (a [Gzip](https://www.gzip.org/) string) into its original string. argumentTypeDescriptiondefaultrequired value string The [Gzip](https://www.gzip.org/) data to decode. N/A yes #### Errors{% #decode_gzip-errors %} - `value` isn't a valid encoded Gzip string. #### Examples{% #decode_gzip-examples %} ##### Decode Gzip data{% #decode-gzip-data %} Source: ```bash encoded_text = decode_base64!("H4sIAHEAymMAA6vML1XISCxLVSguTU5OLS5OK83JqVRISU3OT0lNUchNBQD7BGDaIAAAAA==") decode_gzip!(encoded_text) ``` Return: ```bash "you have successfully decoded me" ``` ### decode_lz4{% #decode_lz4 %} Decodes the `value` (an lz4 string) into its original string. argumentTypeDescriptiondefaultrequired value string The lz4 block data to decode. N/A yes #### Errors{% #decode_lz4-errors %} - `value` unable to decode value with lz4 decoder. #### Examples{% #decode_lz4-examples %} ##### Decode Lz4 data{% #decode-lz4-data %} Source: ```bash encoded_text = decode_base64!("LAAAAPAdVGhlIHF1aWNrIGJyb3duIGZveCBqdW1wcyBvdmVyIDEzIGxhenkgZG9ncy4=") decode_lz4!(encoded_text) ``` Return: ```bash "The quick brown fox jumps over 13 lazy dogs." ``` ### decode_mime_q{% #decode_mime_q %} Replaces q-encoded or base64-encoded [encoded-word](https://datatracker.ietf.org/doc/html/rfc2047#section-2) substrings in the `value` with their original string. argumentTypeDescriptiondefaultrequired value string The string with [encoded-words](https://datatracker.ietf.org/doc/html/rfc2047#section-2) to decode. N/A yes #### Errors{% #decode_mime_q-errors %} - `value` has invalid encoded [encoded-word](https://datatracker.ietf.org/doc/html/rfc2047#section-2) string. #### Examples{% #decode_mime_q-examples %} ##### Decode single encoded-word{% #decode-single-encoded-word %} Source: ```bash decode_mime_q!("=?utf-8?b?SGVsbG8sIFdvcmxkIQ==?=") ``` Return: ```bash "Hello, World!" ``` ##### Embedded{% #embedded %} Source: ```bash decode_mime_q!("From: =?utf-8?b?SGVsbG8sIFdvcmxkIQ==?= <=?utf-8?q?hello=5Fworld=40example=2ecom?=>") ``` Return: ```bash "From: Hello, World! " ``` ##### Without charset{% #without-charset %} Source: ```bash decode_mime_q!("?b?SGVsbG8sIFdvcmxkIQ==") ``` Return: ```bash "Hello, World!" ``` ### decode_percent{% #decode_percent %} Decodes a [percent-encoded](https://url.spec.whatwg.org/#percent-encoded-bytes) `value` like a URL. argumentTypeDescriptiondefaultrequired value string The string to decode. N/A yes #### Examples{% #decode_percent-examples %} ##### Percent decode a value{% #percent-decode-a-value %} Source: ```bash decode_percent("foo%20bar%3F") ``` Return: ```bash "foo bar?" ``` ### decode_punycode{% #decode_punycode %} Decodes a [punycode](https://en.wikipedia.org/wiki/Punycode) encoded `value`, such as an internationalized domain name ([IDN](https://en.wikipedia.org/wiki/Internationalized_domain_name)). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it. argumentTypeDescriptiondefaultrequired value string The string to decode. N/A yes validate boolean If enabled, checks if the input string is a valid domain name. `true` no #### Errors{% #decode_punycode-errors %} - `value` is not valid `punycode` #### Examples{% #decode_punycode-examples %} ##### Decode a punycode encoded internationalized domain name{% #decode-a-punycode-encoded-internationalized-domain-name %} Source: ```bash decode_punycode!("www.xn--caf-dma.com") ``` Return: ```bash "www.café.com" ``` ##### Decode an ASCII only string{% #decode-an-ascii-only-string %} Source: ```bash decode_punycode!("www.cafe.com") ``` Return: ```bash "www.cafe.com" ``` ##### Ignore validation{% #ignore-validation %} Source: ```bash decode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false) ``` Return: ```bash "١٠.٦٦.٣٠.٥." ``` ### decode_snappy{% #decode_snappy %} Decodes the `value` (a Snappy string) into its original string. argumentTypeDescriptiondefaultrequired value string The Snappy data to decode. N/A yes #### Errors{% #decode_snappy-errors %} - `value` isn't a valid encoded Snappy string. #### Examples{% #decode_snappy-examples %} ##### Decode Snappy data{% #decode-snappy-data %} Source: ```bash encoded_text = decode_base64!("LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==") decode_snappy!(encoded_text) ``` Return: ```bash "The quick brown fox jumps over 13 lazy dogs." ``` ### decode_zlib{% #decode_zlib %} Decodes the `value` (a [Zlib](https://www.zlib.net) string) into its original string. argumentTypeDescriptiondefaultrequired value string The [Zlib](https://www.zlib.net) data to decode. N/A yes #### Errors{% #decode_zlib-errors %} - `value` isn't a valid encoded Zlib string. #### Examples{% #decode_zlib-examples %} ##### Decode Zlib data{% #decode-zlib-data %} Source: ```bash encoded_text = decode_base64!("eJwNy4ENwCAIBMCNXIlQ/KqplUSgCdvXAS41qPMHshCB2R1zJlWIVlR6UURX2+wx2YcuK3kAb9C1wd6dn7Fa+QH9gRxr") decode_zlib!(encoded_text) ``` Return: ```bash "you_have_successfully_decoded_me.congratulations.you_are_breathtaking." ``` ### decode_zstd{% #decode_zstd %} Decodes the `value` (a [Zstandard](https://facebook.github.io/zstd) string) into its original string. argumentTypeDescriptiondefaultrequired value string The [Zstandard](https://facebook.github.io/zstd) data to decode. N/A yes #### Errors{% #decode_zstd-errors %} - `value` isn't a valid encoded Zstd string. #### Examples{% #decode_zstd-examples %} ##### Decode Zstd data{% #decode-zstd-data %} Source: ```bash encoded_text = decode_base64!("KLUv/QBY/QEAYsQOFKClbQBedqXsb96EWDax/f/F/z+gNU4ZTInaUeAj82KqPFjUzKqhcfDqAIsLvAsnY1bI/N2mHzDixRQA") decode_zstd!(encoded_text) ``` Return: ```bash "you_have_successfully_decoded_me.congratulations.you_are_breathtaking." ``` ### encode_base16{% #encode_base16 %} Encodes the `value` to [Base16](https://en.wikipedia.org/wiki/Hexadecimal). argumentTypeDescriptiondefaultrequired value string The string to encode. N/A yes #### Examples{% #encode_base16-examples %} ##### Encode to Base16{% #encode-to-base16 %} Source: ```bash encode_base16("please encode me") ``` Return: ```bash "706c6561736520656e636f6465206d65" ``` ### encode_base64{% #encode_base64 %} Encodes the `value` to [Base64](https://en.wikipedia.org/wiki/Base64). argumentTypeDescriptiondefaultrequired value string The string to encode. N/A yes padding boolean Whether the Base64 output is [padded](https://en.wikipedia.org/wiki/Base64#Output_padding). `true` no charset string The character set to use when encoding the data. `standard` no #### Examples{% #encode_base64-examples %} ##### Encode to Base64 (default){% #encode-to-base64-default %} Source: ```bash encode_base64("please encode me") ``` Return: ```bash "cGxlYXNlIGVuY29kZSBtZQ==" ``` ##### Encode to Base64 (without padding){% #encode-to-base64-without-padding %} Source: ```bash encode_base64("please encode me, no padding though", padding: false) ``` Return: ```bash "cGxlYXNlIGVuY29kZSBtZSwgbm8gcGFkZGluZyB0aG91Z2g" ``` ##### Encode to Base64 (URL safe){% #encode-to-base64-url-safe %} Source: ```bash encode_base64("please encode me, but safe for URLs", charset: "url_safe") ``` Return: ```bash "cGxlYXNlIGVuY29kZSBtZSwgYnV0IHNhZmUgZm9yIFVSTHM=" ``` ### encode_charset{% #encode_charset %} Encodes the `value` (a UTF8 string) to a non-UTF8 string using the specified [character set](https://encoding.spec.whatwg.org/#names-and-labels). argumentTypeDescriptiondefaultrequired value string The UTF8 string to encode. N/A yes to_charset string The [character set](https://encoding.spec.whatwg.org/#names-and-labels) to use when encoding the data. N/A yes #### Errors{% #encode_charset-errors %} - `to_charset` isn't a valid [character set](https://encoding.spec.whatwg.org/#names-and-labels). #### Examples{% #encode_charset-examples %} ##### Encode UTF8 string to EUC-KR{% #encode-utf8-string-to-euc-kr %} Source: ```bash encode_base64(encode_charset!("안녕하세요", "euc-kr")) ``` Return: ```bash "vsiz58fPvLy/5A==" ``` ##### Encode UTF8 string to EUC-JP{% #encode-utf8-string-to-euc-jp %} Source: ```bash encode_base64(encode_charset!("こんにちは", "euc-jp")) ``` Return: ```bash "pLOk86TLpMGkzw==" ``` ##### Encode UTF8 string to GB2312{% #encode-utf8-string-to-gb2312 %} Source: ```bash encode_base64(encode_charset!("你好", "gb2312")) ``` Return: ```bash "xOO6ww==" ``` ### encode_gzip{% #encode_gzip %} Encodes the `value` to [Gzip](https://www.gzip.org/). argumentTypeDescriptiondefaultrequired value string The string to encode. N/A yes compression_level integer The default compression level. `6` no #### Examples{% #encode_gzip-examples %} ##### Encode to Gzip{% #encode-to-gzip %} Source: ```bash encoded_text = encode_gzip("please encode me") encode_base64(encoded_text) ``` Return: ```bash "H4sIAAAAAAAA/yvISU0sTlVIzUvOT0lVyE0FAI4R4vcQAAAA" ``` ### encode_json{% #encode_json %} Encodes the `value` to JSON. argumentTypeDescriptiondefaultrequired value any The value to convert to a JSON string. N/A yes pretty boolean Whether to pretty print the JSON string or not. N/A no #### Examples{% #encode_json-examples %} ##### Encode to JSON{% #encode-to-json %} Source: ```bash .payload = encode_json({"hello": "world"}) ``` Return: ```bash "{\"hello\":\"world\"}" ``` ### encode_key_value{% #encode_key_value %} Encodes the `value` into key-value format with customizable delimiters. Default delimiters match the [logfmt](https://brandur.org/logfmt) format. argumentTypeDescriptiondefaultrequired value object The value to convert to a string. N/A yes fields_ordering array The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields. N/A no key_value_delimiter string The string that separates the key from the value. `=` no field_delimiter string The string that separates each key-value pair. N/A no flatten_boolean boolean Whether to encode key-value with a boolean value as a standalone key if `true` and nothing if `false`. `false` no #### Errors{% #encode_key_value-errors %} - `fields_ordering` contains a non-string element. #### Examples{% #encode_key_value-examples %} ##### Encode with default delimiters (no ordering){% #encode-with-default-delimiters-no-ordering %} Source: ```bash encode_key_value({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"}) ``` Return: ```bash "lvl=info msg=\"This is a message\" ts=2021-06-05T17:20:00Z" ``` ##### Encode with default delimiters (fields ordering){% #encode-with-default-delimiters-fields-ordering %} Source: ```bash encode_key_value!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"]) ``` Return: ```bash "ts=2021-06-05T17:20:00Z lvl=info msg=\"This is a message\" log_id=12345" ``` ##### Encode with default delimiters (nested fields){% #encode-with-default-delimiters-nested-fields %} Source: ```bash encode_key_value({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}) ``` Return: ```bash "agent.name=foo event=log log.file.path=my.log" ``` ##### Encode with default delimiters (nested fields ordering){% #encode-with-default-delimiters-nested-fields-ordering %} Source: ```bash encode_key_value!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"]) ``` Return: ```bash "event=log log.file.path=my.log agent.name=foo" ``` ##### Encode with custom delimiters (no ordering){% #encode-with-custom-delimiters-no-ordering %} Source: ```bash encode_key_value( {"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"}, field_delimiter: ",", key_value_delimiter: ":" ) ``` Return: ```bash "lvl:info,msg:\"This is a message\",ts:2021-06-05T17:20:00Z" ``` ##### Encode with custom delimiters and flatten boolean{% #encode-with-custom-delimiters-and-flatten-boolean %} Source: ```bash encode_key_value( {"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "beta": true, "dropped": false}, field_delimiter: ",", key_value_delimiter: ":", flatten_boolean: true ) ``` Return: ```bash "beta,lvl:info,msg:\"This is a message\",ts:2021-06-05T17:20:00Z" ``` ### encode_logfmt{% #encode_logfmt %} Encodes the `value` to [logfmt](https://brandur.org/logfmt). argumentTypeDescriptiondefaultrequired value object The value to convert to a logfmt string. N/A yes fields_ordering array The ordering of fields to preserve. Any fields not in this list are listed unordered, after all ordered fields. N/A no #### Errors{% #encode_logfmt-errors %} - `fields_ordering` contains a non-string element. #### Examples{% #encode_logfmt-examples %} ##### Encode to logfmt (no ordering){% #encode-to-logfmt-no-ordering %} Source: ```bash encode_logfmt({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info"}) ``` Return: ```bash "lvl=info msg=\"This is a message\" ts=2021-06-05T17:20:00Z" ``` ##### Encode to logfmt (fields ordering){% #encode-to-logfmt-fields-ordering %} Source: ```bash encode_logfmt!({"ts": "2021-06-05T17:20:00Z", "msg": "This is a message", "lvl": "info", "log_id": 12345}, ["ts", "lvl", "msg"]) ``` Return: ```bash "ts=2021-06-05T17:20:00Z lvl=info msg=\"This is a message\" log_id=12345" ``` ##### Encode to logfmt (nested fields){% #encode-to-logfmt-nested-fields %} Source: ```bash encode_logfmt({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}) ``` Return: ```bash "agent.name=foo event=log log.file.path=my.log" ``` ##### Encode to logfmt (nested fields ordering){% #encode-to-logfmt-nested-fields-ordering %} Source: ```bash encode_logfmt!({"agent": {"name": "foo"}, "log": {"file": {"path": "my.log"}}, "event": "log"}, ["event", "log.file.path", "agent.name"]) ``` Return: ```bash "event=log log.file.path=my.log agent.name=foo" ``` ### encode_lz4{% #encode_lz4 %} Encodes the `value` to [Lz4](https://lz4.github.io/lz4/). argumentTypeDescriptiondefaultrequired value string The string to encode. N/A yes #### Examples{% #encode_lz4-examples %} ##### Encode to Lz4{% #encode-to-lz4 %} Source: ```bash encoded_text = encode_lz4!("The quick brown fox jumps over 13 lazy dogs.") encode_base64(encoded_text) ``` Return: ```bash "LAAAAPAdVGhlIHF1aWNrIGJyb3duIGZveCBqdW1wcyBvdmVyIDEzIGxhenkgZG9ncy4=" ``` ### encode_percent{% #encode_percent %} Encodes a `value` with [percent encoding](https://url.spec.whatwg.org/#percent-encoded-bytes) to safely be used in URLs. argumentTypeDescriptiondefaultrequired value string The string to encode. N/A yes ascii_set string The ASCII set to use when encoding the data. `NON_ALPHANUMERIC` no #### Examples{% #encode_percent-examples %} ##### Percent encode all non-alphanumeric characters (default){% #percent-encode-all-non-alphanumeric-characters-default %} Source: ```bash encode_percent("foo bar?") ``` Return: ```bash "foo%20bar%3F" ``` ##### Percent encode only control characters{% #percent-encode-only-control-characters %} Source: ```bash encode_percent("foo bar", ascii_set: "CONTROLS") ``` Return: ```bash "foo %09bar" ``` ### encode_proto{% #encode_proto %} Encodes the `value` into a protocol buffer payload. argumentTypeDescriptiondefaultrequired value object The object to convert to a protocol buffer payload. N/A yes desc_file string The path to the protobuf descriptor set file. Must be a literal string. This file is the output of protoc -o… N/A yes message_type string The name of the message type to use for serializing. Must be a literal string. N/A yes #### Errors{% #encode_proto-errors %} - `desc_file` file does not exist. - `message_type` message type does not exist in the descriptor file. #### Examples{% #encode_proto-examples %} ##### Encode to proto{% #encode-to-proto %} Source: ```bash .payload = encode_base64(encode_proto!({"name": "someone", "phones": [{"number": "123456"}]}, "resources/protobuf_descriptor_set.desc", "test_protobuf.Person")) ``` Return: ```bash "Cgdzb21lb25lIggKBjEyMzQ1Ng==" ``` ### encode_punycode{% #encode_punycode %} Encodes a `value` to [punycode](https://en.wikipedia.org/wiki/Punycode). Useful for internationalized domain names ([IDN](https://en.wikipedia.org/wiki/Internationalized_domain_name)). This function assumes that the value passed is meant to be used in IDN context and that it is either a domain name or a part of it. argumentTypeDescriptiondefaultrequired value string The string to encode. N/A yes validate boolean Whether to validate the input string to check if it is a valid domain name. `true` no #### Errors{% #encode_punycode-errors %} - `value` can not be encoded to `punycode` #### Examples{% #encode_punycode-examples %} ##### Encode an internationalized domain name{% #encode-an-internationalized-domain-name %} Source: ```bash encode_punycode!("www.café.com") ``` Return: ```bash "www.xn--caf-dma.com" ``` ##### Encode an internationalized domain name with mixed case{% #encode-an-internationalized-domain-name-with-mixed-case %} Source: ```bash encode_punycode!("www.CAFé.com") ``` Return: ```bash "www.xn--caf-dma.com" ``` ##### Encode an ASCII only string{% #encode-an-ascii-only-string %} Source: ```bash encode_punycode!("www.cafe.com") ``` Return: ```bash "www.cafe.com" ``` ##### Ignore validation{% #ignore-validation %} Source: ```bash encode_punycode!("xn--8hbb.xn--fiba.xn--8hbf.xn--eib.", validate: false) ``` Return: ```bash "xn--8hbb.xn--fiba.xn--8hbf.xn--eib." ``` ### encode_snappy{% #encode_snappy %} Encodes the `value` to Snappy. argumentTypeDescriptiondefaultrequired value string The string to encode. N/A yes #### Errors{% #encode_snappy-errors %} - `value` cannot be encoded into a Snappy string. #### Examples{% #encode_snappy-examples %} ##### Encode to Snappy{% #encode-to-snappy %} Source: ```bash encoded_text = encode_snappy!("The quick brown fox jumps over 13 lazy dogs.") encode_base64(encoded_text) ``` Return: ```bash "LKxUaGUgcXVpY2sgYnJvd24gZm94IGp1bXBzIG92ZXIgMTMgbGF6eSBkb2dzLg==" ``` ### encode_zlib{% #encode_zlib %} Encodes the `value` to [Zlib](https://www.zlib.net). argumentTypeDescriptiondefaultrequired value string The string to encode. N/A yes compression_level integer The default compression level. `6` no #### Examples{% #encode_zlib-examples %} ##### Encode to Zlib{% #encode-to-zlib %} Source: ```bash encoded_text = encode_zlib("please encode me") encode_base64(encoded_text) ``` Return: ```bash "eJwryElNLE5VSM1Lzk9JVchNBQA0RQX7" ``` ### encode_zstd{% #encode_zstd %} Encodes the `value` to [Zstandard](https://facebook.github.io/zstd). argumentTypeDescriptiondefaultrequired value string The string to encode. N/A yes compression_level integer The default compression level. `3` no #### Examples{% #encode_zstd-examples %} ##### Encode to Zstd{% #encode-to-zstd %} Source: ```bash encoded_text = encode_zstd("please encode me") encode_base64(encoded_text) ``` Return: ```bash "KLUv/QBYgQAAcGxlYXNlIGVuY29kZSBtZQ==" ``` Coerce Functions ### to_bool{% #to_bool %} Coerces the `value` into a boolean. argumentTypeDescriptiondefaultrequired value boolean, integer, float, null, string The value to convert to a Boolean. N/A yes #### Errors{% #to_bool-errors %} - `value` is not a supported boolean representation. #### Examples{% #to_bool-examples %} ##### Coerce to a Boolean (string){% #coerce-to-a-boolean-string %} Source: ```bash to_bool!("yes") ``` Return: ```bash true ``` ##### Coerce to a Boolean (float){% #coerce-to-a-boolean-float %} Source: ```bash to_bool(0.0) ``` Return: ```bash false ``` ##### Coerce to a Boolean (int){% #coerce-to-a-boolean-int %} Source: ```bash to_bool(0) ``` Return: ```bash false ``` ##### Coerce to a Boolean (null){% #coerce-to-a-boolean-null %} Source: ```bash to_bool(null) ``` Return: ```bash false ``` ##### Coerce to a Boolean (Boolean){% #coerce-to-a-boolean-boolean %} Source: ```bash to_bool(true) ``` Return: ```bash true ``` ### to_float{% #to_float %} Coerces the `value` into a float. argumentTypeDescriptiondefaultrequired value integer, float, boolean, string, timestamp The value to convert to a float. Must be convertible to a float, otherwise an error is raised. N/A yes #### Errors{% #to_float-errors %} - `value` is not a supported float representation. #### Examples{% #to_float-examples %} ##### Coerce to a float{% #coerce-to-a-float %} Source: ```bash to_float!("3.145") ``` Return: ```bash 3.145 ``` ##### Coerce to a float (timestamp){% #coerce-to-a-float-timestamp %} Source: ```bash to_float(t'2020-12-30T22:20:53.824727Z') ``` Return: ```bash 1609366853.824727 ``` ### to_int{% #to_int %} Coerces the `value` into an integer. argumentTypeDescriptiondefaultrequired value integer, float, boolean, string, timestamp, null The value to convert to an integer. N/A yes #### Errors{% #to_int-errors %} - `value` is a string but the text is not an integer. - `value` is not a string, int, or timestamp. #### Examples{% #to_int-examples %} ##### Coerce to an int (string){% #coerce-to-an-int-string %} Source: ```bash to_int!("2") ``` Return: ```bash 2 ``` ##### Coerce to an int (timestamp){% #coerce-to-an-int-timestamp %} Source: ```bash to_int(t'2020-12-30T22:20:53.824727Z') ``` Return: ```bash 1609366853 ``` ### to_regex{% #to_regex %} Coerces the `value` into a regex. argumentTypeDescriptiondefaultrequired value string The value to convert to a regex. N/A yes #### Errors{% #to_regex-errors %} - `value` is not a string. #### Examples{% #to_regex-examples %} ##### Coerce to a regex{% #coerce-to-a-regex %} Source: ```bash to_regex("^foo$") ?? r'' ``` Return: ```bash "^foo$" ``` ### to_string{% #to_string %} Coerces the `value` into a string. argumentTypeDescriptiondefaultrequired value integer, float, boolean, string, timestamp, null The value to convert to a string. N/A yes #### Errors{% #to_string-errors %} - `value` is not an integer, float, boolean, string, timestamp, or null. #### Examples{% #to_string-examples %} ##### Coerce to a string (Boolean){% #coerce-to-a-string-boolean %} Source: ```bash to_string(true) ``` Return: ```bash "true" ``` ##### Coerce to a string (int){% #coerce-to-a-string-int %} Source: ```bash to_string(52) ``` Return: ```bash "52" ``` ##### Coerce to a string (float){% #coerce-to-a-string-float %} Source: ```bash to_string(52.2) ``` Return: ```bash "52.2" ``` Convert Functions ### from_unix_timestamp{% #from_unix_timestamp %} Converts the `value` integer from a [Unix timestamp](https://en.wikipedia.org/wiki/Unix_time) to a DPL `timestamp`. Converts from the number of seconds since the Unix epoch by default. To convert from milliseconds or nanoseconds, set the `unit` argument to `milliseconds` or `nanoseconds`. argumentTypeDescriptiondefaultrequired value integer The Unix timestamp to convert. N/A yes unit string The time unit. `seconds` no #### Examples{% #from_unix_timestamp-examples %} ##### Convert from a Unix timestamp (seconds){% #convert-from-a-unix-timestamp-seconds %} Source: ```bash from_unix_timestamp!(5) ``` Return: ```bash "1970-01-01T00:00:05Z" ``` ##### Convert from a Unix timestamp (milliseconds){% #convert-from-a-unix-timestamp-milliseconds %} Source: ```bash from_unix_timestamp!(5000, unit: "milliseconds") ``` Return: ```bash "1970-01-01T00:00:05Z" ``` ##### Convert from a Unix timestamp (nanoseconds){% #convert-from-a-unix-timestamp-nanoseconds %} Source: ```bash from_unix_timestamp!(5000, unit: "nanoseconds") ``` Return: ```bash "1970-01-01T00:00:00.000005Z" ``` ### to_syslog_facility{% #to_syslog_facility %} Converts the `value`, a Syslog [facility code](https://en.wikipedia.org/wiki/Syslog#Facility), into its corresponding Syslog keyword. For example, `0` into `"kern"`, `1` into `"user"`, etc. argumentTypeDescriptiondefaultrequired value integer The facility code. N/A yes #### Errors{% #to_syslog_facility-errors %} - `value` is not a valid Syslog [facility code](https://en.wikipedia.org/wiki/Syslog#Facility). #### Examples{% #to_syslog_facility-examples %} ##### Coerce to a Syslog facility{% #coerce-to-a-syslog-facility %} Source: ```bash to_syslog_facility!(4) ``` Return: ```bash "auth" ``` ### to_syslog_facility_code{% #to_syslog_facility_code %} Converts the `value`, a Syslog [facility keyword](https://en.wikipedia.org/wiki/Syslog#Facility), into a Syslog integer facility code (`0` to `23`). argumentTypeDescriptiondefaultrequired value string The Syslog facility keyword to convert. N/A yes #### Errors{% #to_syslog_facility_code-errors %} - `value` is not a valid Syslog facility keyword. #### Examples{% #to_syslog_facility_code-examples %} ##### Coerce to Syslog facility code{% #coerce-to-syslog-facility-code %} Source: ```bash to_syslog_facility_code!("authpriv") ``` Return: ```bash 10 ``` ### to_syslog_level{% #to_syslog_level %} Converts the `value`, a Syslog [severity level](https://en.wikipedia.org/wiki/Syslog#Severity_level), into its corresponding keyword, i.e. 0 into `"emerg"`, 1 into `"alert"`, etc. argumentTypeDescriptiondefaultrequired value integer The severity level. N/A yes #### Errors{% #to_syslog_level-errors %} - `value` isn't a valid Syslog [severity level](https://en.wikipedia.org/wiki/Syslog#Severity_level). #### Examples{% #to_syslog_level-examples %} ##### Coerce to a Syslog level{% #coerce-to-a-syslog-level %} Source: ```bash to_syslog_level!(5) ``` Return: ```bash "notice" ``` ### to_syslog_severity{% #to_syslog_severity %} Converts the `value`, a Syslog [log level keyword](https://en.wikipedia.org/wiki/Syslog#Severity_level), into a Syslog integer severity level (`0` to `7`). argumentTypeDescriptiondefaultrequired value string The Syslog level keyword to convert. N/A yes #### Errors{% #to_syslog_severity-errors %} - `value` is not a valid Syslog level keyword. #### Examples{% #to_syslog_severity-examples %} ##### Coerce to Syslog severity{% #coerce-to-syslog-severity %} Source: ```bash to_syslog_severity!("alert") ``` Return: ```bash 1 ``` ### to_unix_timestamp{% #to_unix_timestamp %} Converts the `value` timestamp into a [Unix timestamp](https://en.wikipedia.org/wiki/Unix_time). Returns the number of seconds since the Unix epoch by default. To return the number in milliseconds or nanoseconds, set the `unit` argument to `milliseconds` or `nanoseconds`. argumentTypeDescriptiondefaultrequired value timestamp The timestamp to convert into a Unix timestamp. N/A yes unit string The time unit. `seconds` no #### Examples{% #to_unix_timestamp-examples %} ##### Convert to a Unix timestamp (seconds){% #convert-to-a-unix-timestamp-seconds %} Source: ```bash to_unix_timestamp(t'2021-01-01T00:00:00+00:00') ``` Return: ```bash 1609459200 ``` ##### Convert to a Unix timestamp (milliseconds){% #convert-to-a-unix-timestamp-milliseconds %} Source: ```bash to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "milliseconds") ``` Return: ```bash 1609459200000 ``` ##### Convert to a Unix timestamp (nanoseconds){% #convert-to-a-unix-timestamp-nanoseconds %} Source: ```bash to_unix_timestamp(t'2021-01-01T00:00:00Z', unit: "nanoseconds") ``` Return: ```bash 1609459200000000000 ``` Cryptography Functions ### decrypt{% #decrypt %} Decrypts a string with a symmetric encryption algorithm. Supported Algorithms: - AES-256-CFB (key = 32 bytes, iv = 16 bytes) - AES-192-CFB (key = 24 bytes, iv = 16 bytes) - AES-128-CFB (key = 16 bytes, iv = 16 bytes) - AES-256-OFB (key = 32 bytes, iv = 16 bytes) - AES-192-OFB (key = 24 bytes, iv = 16 bytes) - AES-128-OFB (key = 16 bytes, iv = 16 bytes) - AES-128-SIV (key = 32 bytes, iv = 16 bytes) - AES-256-SIV (key = 64 bytes, iv = 16 bytes) - Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes) - Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes) - Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes) - AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes) - AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes) - AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes) - AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes) - AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes) - AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes) - AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes) - AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes) - AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes) - AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes) - AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes) - AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes) - AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes) - AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes) - AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes) - AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes) - AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes) - AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes) - CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes) - XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes) - XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes) argumentTypeDescriptiondefaultrequired ciphertext string The string in raw bytes (not encoded) to decrypt. N/A yes algorithm string The algorithm to use. N/A yes key string The key in raw bytes (not encoded) for decryption. The length must match the algorithm requested. N/A yes iv string The IV in raw bytes (not encoded) for decryption. The length must match the algorithm requested. A new IV should be generated for every message. You can use `random_bytes` to generate a cryptographically secure random value. The value should match the one used during encryption. N/A yes #### Errors{% #decrypt-errors %} - `algorithm` is not a supported algorithm. - `key` length does not match the key size required for the algorithm specified. - `iv` length does not match the `iv` size required for the algorithm specified. #### Examples{% #decrypt-examples %} ##### Decrypt value{% #decrypt-value %} Source: ```bash ciphertext = decode_base64!("5fLGcu1VHdzsPcGNDio7asLqE1P43QrVfPfmP4i4zOU=") iv = decode_base64!("fVEIRkIiczCRWNxaarsyxA==") key = "16_byte_keyxxxxx" decrypt!(ciphertext, "AES-128-CBC-PKCS7", key, iv: iv) ``` Return: ```bash "super_secret_message" ``` ### encrypt{% #encrypt %} Encrypts a string with a symmetric encryption algorithm. Supported Algorithms: - AES-256-CFB (key = 32 bytes, iv = 16 bytes) - AES-192-CFB (key = 24 bytes, iv = 16 bytes) - AES-128-CFB (key = 16 bytes, iv = 16 bytes) - AES-256-OFB (key = 32 bytes, iv = 16 bytes) - AES-192-OFB (key = 24 bytes, iv = 16 bytes) - AES-128-OFB (key = 16 bytes, iv = 16 bytes) - AES-128-SIV (key = 32 bytes, iv = 16 bytes) - AES-256-SIV (key = 64 bytes, iv = 16 bytes) - Deprecated - AES-256-CTR (key = 32 bytes, iv = 16 bytes) - Deprecated - AES-192-CTR (key = 24 bytes, iv = 16 bytes) - Deprecated - AES-128-CTR (key = 16 bytes, iv = 16 bytes) - AES-256-CTR-LE (key = 32 bytes, iv = 16 bytes) - AES-192-CTR-LE (key = 24 bytes, iv = 16 bytes) - AES-128-CTR-LE (key = 16 bytes, iv = 16 bytes) - AES-256-CTR-BE (key = 32 bytes, iv = 16 bytes) - AES-192-CTR-BE (key = 24 bytes, iv = 16 bytes) - AES-128-CTR-BE (key = 16 bytes, iv = 16 bytes) - AES-256-CBC-PKCS7 (key = 32 bytes, iv = 16 bytes) - AES-192-CBC-PKCS7 (key = 24 bytes, iv = 16 bytes) - AES-128-CBC-PKCS7 (key = 16 bytes, iv = 16 bytes) - AES-256-CBC-ANSIX923 (key = 32 bytes, iv = 16 bytes) - AES-192-CBC-ANSIX923 (key = 24 bytes, iv = 16 bytes) - AES-128-CBC-ANSIX923 (key = 16 bytes, iv = 16 bytes) - AES-256-CBC-ISO7816 (key = 32 bytes, iv = 16 bytes) - AES-192-CBC-ISO7816 (key = 24 bytes, iv = 16 bytes) - AES-128-CBC-ISO7816 (key = 16 bytes, iv = 16 bytes) - AES-256-CBC-ISO10126 (key = 32 bytes, iv = 16 bytes) - AES-192-CBC-ISO10126 (key = 24 bytes, iv = 16 bytes) - AES-128-CBC-ISO10126 (key = 16 bytes, iv = 16 bytes) - CHACHA20-POLY1305 (key = 32 bytes, iv = 12 bytes) - XCHACHA20-POLY1305 (key = 32 bytes, iv = 24 bytes) - XSALSA20-POLY1305 (key = 32 bytes, iv = 24 bytes) argumentTypeDescriptiondefaultrequired plaintext string The string to encrypt. N/A yes algorithm string The algorithm to use. N/A yes key string The key in raw bytes (not encoded) for encryption. The length must match the algorithm requested. N/A yes iv string The IV in raw bytes (not encoded) for encryption. The length must match the algorithm requested. A new IV should be generated for every message. You can use `random_bytes` to generate a cryptographically secure random value. N/A yes #### Errors{% #encrypt-errors %} - `algorithm` is not a supported algorithm. - `key` length does not match the key size required for the algorithm specified. - `iv` length does not match the `iv` size required for the algorithm specified. #### Examples{% #encrypt-examples %} ##### Encrypt value{% #encrypt-value %} Source: ```bash plaintext = "super secret message" iv = "1234567890123456" # typically you would call random_bytes(16) key = "16_byte_keyxxxxx" encrypted_message = encrypt!(plaintext, "AES-128-CBC-PKCS7", key, iv: iv) encode_base64(encrypted_message) ``` Return: ```bash "GBw8Mu00v0Kc38+/PvsVtGgWuUJ+ZNLgF8Opy8ohIYE=" ``` ### hmac{% #hmac %} Calculates a [HMAC](https://en.wikipedia.org/wiki/HMAC) of the `value` using the given `key`. The hashing `algorithm` used can be optionally specified. For most use cases, the resulting bytestream should be encoded into a hex or base64 string using either encode_base16 or encode_base64. This function is infallible if either the default `algorithm` value or a recognized-valid compile-time `algorithm` string literal is used. Otherwise, it is fallible. argumentTypeDescriptiondefaultrequired value string The string to calculate the HMAC for. N/A yes key string The string to use as the cryptographic key. N/A yes algorithm string The hashing algorithm to use. `SHA-256` no #### Examples{% #hmac-examples %} ##### Calculate message HMAC (defaults: SHA-256), encoding to a base64 string{% #calculate-message-hmac-defaults-sha-256-encoding-to-a-base64-string %} Source: ```bash encode_base64(hmac("Hello there", "super-secret-key")) ``` Return: ```bash "eLGE8YMviv85NPXgISRUZxstBNSU47JQdcXkUWcClmI=" ``` ##### Calculate message HMAC using SHA-224, encoding to a hex-encoded string{% #calculate-message-hmac-using-sha-224-encoding-to-a-hex-encoded-string %} Source: ```bash encode_base16(hmac("Hello there", "super-secret-key", algorithm: "SHA-224")) ``` Return: ```bash "42fccbc2b7d22a143b92f265a8046187558a94d11ddbb30622207e90" ``` ##### Calculate message HMAC using a variable hash algorithm{% #calculate-message-hmac-using-a-variable-hash-algorithm %} Source: ```bash .hash_algo = "SHA-256" hmac_bytes, err = hmac("Hello there", "super-secret-key", algorithm: .hash_algo) if err == null { .hmac = encode_base16(hmac_bytes) } ``` Return: ```bash "78b184f1832f8aff3934f5e0212454671b2d04d494e3b25075c5e45167029662" ``` ### md5{% #md5 %} Calculates an md5 hash of the `value`. argumentTypeDescriptiondefaultrequired value string The string to calculate the hash for. N/A yes #### Examples{% #md5-examples %} ##### Create md5 hash{% #create-md5-hash %} Source: ```bash md5("foo") ``` Return: ```bash "acbd18db4cc2f85cedef654fccc4a4d8" ``` ### seahash{% #seahash %} Calculates a [Seahash](https://docs.rs/seahash/latest/seahash/) hash of the `value`. **Note**: Due to limitations in the underlying DPL data types, this function converts the unsigned 64-bit integer SeaHash result to a signed 64-bit integer. Results higher than the signed 64-bit integer maximum value wrap around to negative values. argumentTypeDescriptiondefaultrequired value string The string to calculate the hash for. N/A yes #### Examples{% #seahash-examples %} ##### Calculate seahash{% #calculate-seahash %} Source: ```bash seahash("foobar") ``` Return: ```bash 5348458858952426000 ``` ##### Calculate negative seahash{% #calculate-negative-seahash %} Source: ```bash seahash("bar") ``` Return: ```bash -2796170501982571500 ``` ### sha1{% #sha1 %} Calculates a [SHA-1](https://en.wikipedia.org/wiki/SHA-1) hash of the `value`. argumentTypeDescriptiondefaultrequired value string The string to calculate the hash for. N/A yes #### Examples{% #sha1-examples %} ##### Calculate sha1 hash{% #calculate-sha1-hash %} Source: ```bash sha1("foo") ``` Return: ```bash "0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33" ``` ### sha2{% #sha2 %} Calculates a [SHA-2](https://en.wikipedia.org/wiki/SHA-2) hash of the `value`. argumentTypeDescriptiondefaultrequired value string The string to calculate the hash for. N/A yes variant string The variant of the algorithm to use. `SHA-512/256` no #### Examples{% #sha2-examples %} ##### Calculate sha2 hash{% #calculate-sha2-hash %} Source: ```bash sha2("foo", variant: "SHA-512/224") ``` Return: ```bash "d68f258d37d670cfc1ec1001a0394784233f88f056994f9a7e5e99be" ``` ### sha3{% #sha3 %} Calculates a [SHA-3](https://en.wikipedia.org/wiki/SHA-3) hash of the `value`. argumentTypeDescriptiondefaultrequired value string The string to calculate the hash for. N/A yes variant string The variant of the algorithm to use. `SHA3-512` no #### Examples{% #sha3-examples %} ##### Calculate sha3 hash{% #calculate-sha3-hash %} Source: ```bash sha3("foo", variant: "SHA3-224") ``` Return: ```bash "f4f6779e153c391bbd29c95e72b0708e39d9166c7cea51d1f10ef58a" ``` Debug Functions ### assert{% #assert %} Asserts the `condition`, which must be a Boolean expression. The program is aborted with `message` if the condition evaluates to `false`. argumentTypeDescriptiondefaultrequired condition boolean The condition to check. N/A yes message string An optional custom error message. If the equality assertion fails, `message` is appended to the default message prefix. See the examples below for a fully formed log message sample. N/A no #### Errors{% #assert-errors %} - `condition` evaluates to `false`. #### Examples{% #assert-examples %} ##### Assertion (true){% #assertion-true %} Source: ```bash assert!("foo" == "foo", message: "\"foo\" must be \"foo\"!") ``` Return: ```bash true ``` ##### Assertion (false){% #assertion-false %} Source: ```bash assert!("foo" == "bar", message: "\"foo\" must be \"foo\"!") ``` ### assert_eq{% #assert_eq %} Asserts that two expressions, `left` and `right`, have the same value. The program is aborted with `message` if they do not have the same value. argumentTypeDescriptiondefaultrequired left any The value to check for equality against `right`. N/A yes right any The value to check for equality against `left`. N/A yes message string An optional custom error message. If the equality assertion fails, `message` is appended to the default message prefix. See the examples below for a fully formed log message sample. N/A no #### Examples{% #assert_eq-examples %} ##### Successful assertion{% #successful-assertion %} Source: ```bash assert_eq!(1, 1) ``` Return: ```bash true ``` ##### Unsuccessful assertion{% #unsuccessful-assertion %} Source: ```bash assert_eq!(127, [1, 2, 3]) ``` ##### Unsuccessful assertion with custom log message{% #unsuccessful-assertion-with-custom-log-message %} Source: ```bash assert_eq!(1, 0, message: "Unequal integers") ``` ### log{% #log %} Logs the `value` to [stdout](https://en.wikipedia.org/wiki/Standard_streams#Standard_output_%28stdout%29) at the specified `level`. argumentTypeDescriptiondefaultrequired value any The value to log. N/A yes level string The log level. `info` no rate_limit_secs integer Specifies that the log message is output no more than once per the given number of seconds. Use a value of `0` to turn rate limiting off. `1` no #### Examples{% #log-examples %} ##### Log a message{% #log-a-message %} Source: ```bash log("Hello, World!", level: "info", rate_limit_secs: 60) ``` Return: ```bash null ``` ##### Log an error{% #log-an-error %} Source: ```bash _, err = to_int(.field) if err != null { log(err, level: "error") } ``` Return: ```bash null ``` Enrichment Functions ### find_enrichment_table_records{% #find_enrichment_table_records %} Searches an enrichment table for rows that match the provided condition. For `file` enrichment tables, this condition needs to be a DPL object in which the key-value pairs indicate a field to search mapped to a value to search in that field. This function returns the rows that match the provided condition(s). *All* fields need to match for rows to be returned; if any fields do not match, then no rows are returned. There are currently two forms of search criteria: 1. **Exact match search**. The given field must match the value exactly. Case sensitivity can be specified using the `case_sensitive` argument. An exact match search can use an index directly into the dataset, which should make this search fairly "cheap" from a performance perspective. 1. **Date range search**. The given field must be greater than or equal to the `from` date and/or less than or equal to the `to` date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the *only* criteria when the enrichment data set is very small. For `geoip` and `mmdb` enrichment tables, this condition needs to be a DPL object with a single key-value pair whose value needs to be a valid IP address. Example: `{"ip": .ip }`. If a return field is expected and without a value, `null` is used. This table can return the following fields: - ISP databases: - `autonomous_system_number` - `autonomous_system_organization` - `isp` - `organization` - City databases: - `city_name` - `continent_code` - `country_code` - `country_name` - `region_code` - `region_name` - `metro_code` - `latitude` - `longitude` - `postal_code` - `timezone` - Connection-Type databases: - `connection_type` To use this function, you need to update your configuration to include an `enrichment_tables` parameter. argumentTypeDescriptiondefaultrequired table string The enrichment table to search. N/A yes condition object The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined. N/A yes select array A subset of fields from the enrichment table to return. If not specified, all fields are returned. N/A no case_sensitive boolean Whether text fields need to match cases exactly. `true` no #### Examples{% #find_enrichment_table_records-examples %} ##### Exact match{% #exact-match %} Source: ```bash find_enrichment_table_records!("test", { "surname": "smith", }, case_sensitive: false) ``` Return: ```bash [{"id":1,"firstname":"Bob","surname":"Smith"},{"id":2,"firstname":"Fred","surname":"Smith"}] ``` ##### Date range search{% #date-range-search %} Source: ```bash find_enrichment_table_records!("test", { "surname": "Smith", "date_of_birth": { "from": t'1985-01-01T00:00:00Z', "to": t'1985-12-31T00:00:00Z' } }) ``` Return: ```bash [{"id":1,"firstname":"Bob","surname":"Smith"},{"id":2,"firstname":"Fred","surname":"Smith"}] ``` ### get_enrichment_table_record{% #get_enrichment_table_record %} Searches an enrichment table for a row that matches the provided condition. A single row must be matched. If no rows are found or more than one row is found, an error is returned. For `file` enrichment tables, this condition needs to be a DPL object in which the key-value pairs indicate a field to search mapped to a value to search in that field. This function returns the rows that match the provided condition(s). *All* fields need to match for rows to be returned; if any fields do not match, then no rows are returned. There are currently two forms of search criteria: 1. **Exact match search**. The given field must match the value exactly. Case sensitivity can be specified using the `case_sensitive` argument. An exact match search can use an index directly into the dataset, which should make this search fairly "cheap" from a performance perspective. 1. **Date range search**. The given field must be greater than or equal to the `from` date and/or less than or equal to the `to` date. A date range search involves sequentially scanning through the rows that have been located using any exact match criteria. This can be an expensive operation if there are many rows returned by any exact match criteria. Therefore, use date ranges as the *only* criteria when the enrichment data set is very small. For `geoip` and `mmdb` enrichment tables, this condition needs to be a DPL object with a single key-value pair whose value needs to be a valid IP address. Example: `{"ip": .ip }`. If a return field is expected and without a value, `null` is used. This table can return the following fields: - ISP databases: - `autonomous_system_number` - `autonomous_system_organization` - `isp` - `organization` - City databases: - `city_name` - `continent_code` - `country_code` - `country_name` - `region_code` - `region_name` - `metro_code` - `latitude` - `longitude` - `postal_code` - `timezone` - Connection-Type databases: - `connection_type` To use this function, you need to update your configuration to include an `enrichment_tables` parameter. argumentTypeDescriptiondefaultrequired table string The enrichment table to search. N/A yes condition object The condition to search on. Since the condition is used at boot time to create indices into the data, these conditions must be statically defined. N/A yes select array A subset of fields from the enrichment table to return. If not specified, all fields are returned. N/A no case_sensitive boolean Whether the text fields match the case exactly. `true` no #### Errors{% #get_enrichment_table_record-errors %} - The row is not found. - Multiple rows are found that match the condition. #### Examples{% #get_enrichment_table_record-examples %} ##### Exact match{% #exact-match %} Source: ```bash get_enrichment_table_record!("test", { "surname": "bob", "firstname": "John" }, case_sensitive: false) ``` Return: ```bash {"id":1,"firstname":"Bob","surname":"Smith"} ``` ##### Date range search{% #date-range-search %} Source: ```bash get_enrichment_table_record!("test", { "surname": "Smith", "date_of_birth": { "from": t'1985-01-01T00:00:00Z', "to": t'1985-12-31T00:00:00Z' } }) ``` Return: ```bash {"id":1,"firstname":"Bob","surname":"Smith"} ``` Enumerate Functions ### compact{% #compact %} Compacts the `value` by removing empty values, where empty values are defined using the available parameters. argumentTypeDescriptiondefaultrequired value array, object The object or array to compact. N/A yes recursive boolean Whether the compaction be recursive. `true` no null boolean Whether null should be treated as an empty value. `true` no string boolean Whether an empty string should be treated as an empty value. `true` no object boolean Whether an empty object should be treated as an empty value. `true` no array boolean Whether an empty array should be treated as an empty value. `true` no nullish boolean Tests whether the value is "nullish" as defined by the `is_nullish` function. `false` no #### Examples{% #compact-examples %} ##### Compact an array{% #compact-an-array %} Source: ```bash compact(["foo", "bar", "", null, [], "buzz"], string: true, array: true, null: true) ``` Return: ```bash ["foo","bar","buzz"] ``` ##### Compact an object{% #compact-an-object %} Source: ```bash compact({"field1": 1, "field2": "", "field3": [], "field4": null}, string: true, array: true, null: true) ``` Return: ```bash {"field1":1} ``` ### filter{% #filter %} Filter elements from a collection. This function currently *does not* support recursive iteration. The function uses the function closure syntax to allow reading the key-value or index-value combination for each item in the collection. The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block. See the examples below to learn about the closure syntax. argumentTypeDescriptiondefaultrequired value array, object The array or object to filter. N/A yes #### Examples{% #filter-examples %} ##### Filter elements{% #filter-elements %} Source: ```bash filter(array!(.tags)) -> |_index, value| { # keep any elements that aren't equal to "foo" value != "foo" } ``` Return: ```bash ["bar","baz"] ``` ### flatten{% #flatten %} Flattens the `value` into a single-level representation. argumentTypeDescriptiondefaultrequired value array, object The array or object to flatten. N/A yes separator string The separator to join nested keys `.` no #### Examples{% #flatten-examples %} ##### Flatten array{% #flatten-array %} Source: ```bash flatten([1, [2, 3, 4], [5, [6, 7], 8], 9]) ``` Return: ```bash [1,2,3,4,5,6,7,8,9] ``` ##### Flatten object{% #flatten-object %} Source: ```bash flatten({ "parent1": { "child1": 1, "child2": 2 }, "parent2": { "child3": 3 } }) ``` Return: ```bash {"parent1.child1":1,"parent1.child2":2,"parent2.child3":3} ``` ### for_each{% #for_each %} Iterate over a collection. This function currently *does not* support recursive iteration. The function uses the "function closure syntax" to allow reading the key/value or index/value combination for each item in the collection. The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block. See the examples below to learn about the closure syntax. argumentTypeDescriptiondefaultrequired value array, object The array or object to iterate. N/A yes #### Examples{% #for_each-examples %} ##### Tally elements{% #tally-elements %} Source: ```bash tally = {} for_each(array!(.tags)) -> |_index, value| { # Get the current tally for the `value`, or # set to `0`. count = int(get!(tally, [value])) ?? 0 # Increment the tally for the value by `1`. tally = set!(tally, [value], count + 1) } tally ``` Return: ```bash {"foo":2,"bar":1,"baz":1} ``` ### includes{% #includes %} Determines whether the `value` array includes the specified `item`. argumentTypeDescriptiondefaultrequired value array The array. N/A yes item any The item to check. N/A yes #### Examples{% #includes-examples %} ##### Array includes{% #array-includes %} Source: ```bash includes(["apple", "orange", "banana"], "banana") ``` Return: ```bash true ``` ### keys{% #keys %} Returns the keys from the object passed into the function. argumentTypeDescriptiondefaultrequired value object The object to extract keys from. N/A yes #### Examples{% #keys-examples %} ##### Get keys from the object{% #get-keys-from-the-object %} Source: ```bash keys({"key1": "val1", "key2": "val2"}) ``` Return: ```bash ["key1","key2"] ``` ### length{% #length %} Returns the length of the `value`. - If `value` is an array, returns the number of elements. - If `value` is an object, returns the number of top-level keys. - If `value` is a string, returns the number of bytes in the string. If you want the number of characters, see `strlen`. argumentTypeDescriptiondefaultrequired value array, object, string The array or object. N/A yes #### Examples{% #length-examples %} ##### Length (object){% #length-object %} Source: ```bash length({ "portland": "Trail Blazers", "seattle": "Supersonics" }) ``` Return: ```bash 2 ``` ##### Length (nested object){% #length-nested-object %} Source: ```bash length({ "home": { "city": "Portland", "state": "Oregon" }, "name": "Trail Blazers", "mascot": { "name": "Blaze the Trail Cat" } }) ``` Return: ```bash 3 ``` ##### Length (array){% #length-array %} Source: ```bash length(["Trail Blazers", "Supersonics", "Grizzlies"]) ``` Return: ```bash 3 ``` ##### Length (string){% #length-string %} Source: ```bash length("The Planet of the Apes Musical") ``` Return: ```bash 30 ``` ### map_keys{% #map_keys %} Map the keys within an object. If `recursive` is enabled, the function iterates into nested objects, using the following rules: 1. Iteration starts at the root. 1. For every nested object type: - First return the key of the object type itself. - Then recurse into the object, and loop back to item (1) in this list. - Any mutation done on a nested object *before* recursing into it, are preserved. 1. For every nested array type: - First return the key of the array type itself. - Then find all objects within the array, and apply item (2) to each individual object. The above rules mean that `map_keys` with `recursive` enabled finds *all* keys in the target, regardless of whether nested objects are nested inside arrays. The function uses the function closure syntax to allow reading the key for each item in the object. The same scoping rules apply to closure blocks as they do for regular blocks. This means that any variable defined in parent scopes is accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block. See the examples below to learn about the closure syntax. argumentTypeDescriptiondefaultrequired value object The object to iterate. N/A yes recursive boolean Whether to recursively iterate the collection. `false` no #### Examples{% #map_keys-examples %} ##### Upcase keys{% #upcase-keys %} Source: ```bash map_keys(.) -> |key| { upcase(key) } ``` Return: ```bash {"FOO":"foo","BAR":"bar"} ``` ##### De-dot keys{% #de-dot-keys %} Source: ```bash map_keys(., recursive: true) -> |key| { replace(key, ".", "_") } ``` Return: ```bash {"labels":{"app_kubernetes_io/name":"mysql"}} ``` ### map_values{% #map_values %} Map the values within a collection. If `recursive` is enabled, the function iterates into nested collections, using the following rules: 1. Iteration starts at the root. 1. For every nested collection type: - First return the collection type itself. - Then recurse into the collection, and loop back to item (1) in the list - Any mutation done on a collection *before* recursing into it, are preserved. The function uses the function closure syntax to allow mutating the value for each item in the collection. The same scoping rules apply to closure blocks as they do for regular blocks, meaning, any variable defined in parent scopes are accessible, and mutations to those variables are preserved, but any new variables instantiated in the closure block are unavailable outside of the block. Check out the examples below to learn about the closure syntax. argumentTypeDescriptiondefaultrequired value array, object The object or array to iterate. N/A yes recursive boolean Whether to recursively iterate the collection. `false` no #### Examples{% #map_values-examples %} ##### Upcase values{% #upcase-values %} Source: ```bash map_values(.) -> |value| { upcase!(value) } ``` Return: ```bash {"foo":"FOO","bar":"BAR"} ``` ### match_array{% #match_array %} Determines whether the elements in the `value` array matches the `pattern`. By default, it checks that at least one element matches, but can be set to determine if all the elements match. argumentTypeDescriptiondefaultrequired value array The array. N/A yes pattern regex The regular expression pattern to match against. N/A yes all boolean Whether to match on all elements of `value`. `false` no #### Examples{% #match_array-examples %} ##### Match at least one element{% #match-at-least-one-element %} Source: ```bash match_array(["foobar", "bazqux"], r'foo') ``` Return: ```bash true ``` ##### Match all elements{% #match-all-elements %} Source: ```bash match_array(["foo", "foobar", "barfoo"], r'foo', all: true) ``` Return: ```bash true ``` ##### No matches{% #no-matches %} Source: ```bash match_array(["bazqux", "xyz"], r'foo') ``` Return: ```bash false ``` ##### Not all elements match{% #not-all-elements-match %} Source: ```bash match_array(["foo", "foobar", "baz"], r'foo', all: true) ``` Return: ```bash false ``` ### strlen{% #strlen %} Returns the number of UTF-8 characters in `value`. This differs from `length` which counts the number of bytes of a string. **Note**: This is the count of [Unicode scalar values](https://www.unicode.org/glossary/#unicode_scalar_value) which can sometimes differ from [Unicode code points](https://www.unicode.org/glossary/#code_point). argumentTypeDescriptiondefaultrequired value string The string. N/A yes #### Examples{% #strlen-examples %} ##### strlen{% #strlen %} Source: ```bash strlen("ñandú") ``` Return: ```bash 5 ``` ### unflatten{% #unflatten %} Unflattens the `value` into a nested representation. argumentTypeDescriptiondefaultrequired value object The array or object to unflatten. N/A yes separator string The separator to split flattened keys. `.` no recursive boolean Whether to recursively unflatten the object values. `true` no #### Examples{% #unflatten-examples %} ##### Unflatten{% #unflatten %} Source: ```bash unflatten({ "foo.bar.baz": true, "foo.bar.qux": false, "foo.quux": 42 }) ``` Return: ```bash {"foo":{"bar":{"baz":true,"qux":false},"quux":42}} ``` ##### Unflatten recursively{% #unflatten-recursively %} Source: ```bash unflatten({ "flattened.parent": { "foo.bar": true, "foo.baz": false } }) ``` Return: ```bash {"flattened":{"parent":{"foo":{"bar":true,"baz":false}}}} ``` ##### Unflatten non-recursively{% #unflatten-non-recursively %} Source: ```bash unflatten({ "flattened.parent": { "foo.bar": true, "foo.baz": false } }, recursive: false) ``` Return: ```bash {"flattened":{"parent":{"foo.bar":true,"foo.baz":false}}} ``` ##### Ignore inconsistent keys values{% #ignore-inconsistent-keys-values %} Source: ```bash unflatten({ "a": 3, "a.b": 2, "a.c": 4 }) ``` Return: ```bash {"a":{"b":2,"c":4}} ``` ### unique{% #unique %} Returns the unique values for an array. The first occurrence of each element is kept. argumentTypeDescriptiondefaultrequired value array The array to return unique elements from. N/A yes #### Examples{% #unique-examples %} ##### Unique{% #unique %} Source: ```bash unique(["foo", "bar", "foo", "baz"]) ``` Return: ```bash ["foo","bar","baz"] ``` ### values{% #values %} Returns the values from the object passed into the function. argumentTypeDescriptiondefaultrequired value object The object to extract values from. N/A yes #### Examples{% #values-examples %} ##### Get values from the object{% #get-values-from-the-object %} Source: ```bash values({"key1": "val1", "key2": "val2"}) ``` Return: ```bash ["val1","val2"] ``` IP Functions ### ip_aton{% #ip_aton %} Converts IPv4 address in numbers-and-dots notation into network-order bytes represented as an integer. This behavior mimics [inet_aton](https://linux.die.net/man/3/inet_aton). argumentTypeDescriptiondefaultrequired value string The IP address to convert to binary. N/A yes #### Errors{% #ip_aton-errors %} - `value` is not a valid IPv4 address. #### Examples{% #ip_aton-examples %} ##### IPv4 to integer{% #ipv4-to-integer %} Source: ```bash ip_aton!("1.2.3.4") ``` Return: ```bash 16909060 ``` ### ip_cidr_contains{% #ip_cidr_contains %} Determines whether the `ip` is contained in the block referenced by the `cidr`. argumentTypeDescriptiondefaultrequired cidr string, array The CIDR mask (v4 or v6). N/A yes ip string The IP address (v4 or v6). N/A yes #### Errors{% #ip_cidr_contains-errors %} - `cidr` is not a valid CIDR. - `ip` is not a valid IP address. #### Examples{% #ip_cidr_contains-examples %} ##### IPv4 contains CIDR{% #ipv4-contains-cidr %} Source: ```bash ip_cidr_contains!("192.168.0.0/16", "192.168.10.32") ``` Return: ```bash true ``` ##### IPv4 is private{% #ipv4-is-private %} Source: ```bash ip_cidr_contains!(["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"], "192.168.10.32") ``` Return: ```bash true ``` ##### IPv6 contains CIDR{% #ipv6-contains-cidr %} Source: ```bash ip_cidr_contains!("2001:4f8:4:ba::/64", "2001:4f8:4:ba:2e0:81ff:fe22:d1f1") ``` Return: ```bash true ``` ### ip_ntoa{% #ip_ntoa %} Converts numeric representation of IPv4 address in network-order bytes to numbers-and-dots notation. This behavior mimics [inet_ntoa](https://linux.die.net/man/3/inet_ntoa). argumentTypeDescriptiondefaultrequired value string The integer representation of an IPv4 address. N/A yes #### Errors{% #ip_ntoa-errors %} - `value` cannot fit in an unsigned 32-bit integer. #### Examples{% #ip_ntoa-examples %} ##### Integer to IPv4{% #integer-to-ipv4 %} Source: ```bash ip_ntoa!(16909060) ``` Return: ```bash "1.2.3.4" ``` ### ip_ntop{% #ip_ntop %} Converts IPv4 and IPv6 addresses from binary to text form. This behavior mimics [inet_ntop](https://linux.die.net/man/3/inet_ntop). argumentTypeDescriptiondefaultrequired value string The binary data to convert from. For IPv4 addresses, it must be 4 bytes (32 bits) long. For IPv6 addresses, it must be 16 bytes (128 bits) long. N/A yes #### Errors{% #ip_ntop-errors %} - `value` must be of length 4 or 16 bytes. #### Examples{% #ip_ntop-examples %} ##### Convert IPv4 address from bytes after decoding from Base64{% #convert-ipv4-address-from-bytes-after-decoding-from-base64 %} Source: ```bash ip_ntop!(decode_base64!("wKgAAQ==")) ``` Return: ```bash "192.168.0.1" ``` ##### Convert IPv6 address from bytes after decoding from Base64{% #convert-ipv6-address-from-bytes-after-decoding-from-base64 %} Source: ```bash ip_ntop!(decode_base64!("IAENuIWjAAAAAIouA3BzNA==")) ``` Return: ```bash "2001:db8:85a3::8a2e:370:7334" ``` ### ip_pton{% #ip_pton %} Converts IPv4 and IPv6 addresses from text to binary form. - The binary form of IPv4 addresses is 4 bytes (32 bits) long. - The binary form of IPv6 addresses is 16 bytes (128 bits) long. This behavior mimics [inet_pton](https://linux.die.net/man/3/inet_pton). argumentTypeDescriptiondefaultrequired value string The IP address (v4 or v6) to convert to binary form. N/A yes #### Errors{% #ip_pton-errors %} - `value` is not a valid IP (v4 or v6) address in text form. #### Examples{% #ip_pton-examples %} ##### Convert IPv4 address to bytes and encode to Base64{% #convert-ipv4-address-to-bytes-and-encode-to-base64 %} Source: ```bash encode_base64(ip_pton!("192.168.0.1")) ``` Return: ```bash "wKgAAQ==" ``` ##### Convert IPv6 address to bytes and encode to Base64{% #convert-ipv6-address-to-bytes-and-encode-to-base64 %} Source: ```bash encode_base64(ip_pton!("2001:db8:85a3::8a2e:370:7334")) ``` Return: ```bash "IAENuIWjAAAAAIouA3BzNA==" ``` ### ip_subnet{% #ip_subnet %} Extracts the subnet address from the `ip` using the supplied `subnet`. argumentTypeDescriptiondefaultrequired ip string The IP address (v4 or v6). N/A yes subnet string The subnet to extract from the IP address. This can be either a prefix length like `/8` or a net mask like `255.255.0.0`. The net mask can be either an IPv4 or IPv6 address. N/A yes #### Errors{% #ip_subnet-errors %} - `ip` is not a valid IP address. - `subnet` is not a valid subnet. #### Examples{% #ip_subnet-examples %} ##### IPv4 subnet{% #ipv4-subnet %} Source: ```bash ip_subnet!("192.168.10.32", "255.255.255.0") ``` Return: ```bash "192.168.10.0" ``` ##### IPv6 subnet{% #ipv6-subnet %} Source: ```bash ip_subnet!("2404:6800:4003:c02::64", "/32") ``` Return: ```bash "2404:6800::" ``` ### ip_to_ipv6{% #ip_to_ipv6 %} Converts the `ip` to an IPv6 address. argumentTypeDescriptiondefaultrequired ip string The IP address to convert to IPv6. N/A yes #### Errors{% #ip_to_ipv6-errors %} - `ip` is not a valid IP address. #### Examples{% #ip_to_ipv6-examples %} ##### IPv4 to IPv6{% #ipv4-to-ipv6 %} Source: ```bash ip_to_ipv6!("192.168.10.32") ``` Return: ```bash "::ffff:192.168.10.32" ``` ### ipv6_to_ipv4{% #ipv6_to_ipv4 %} Converts the `ip` to an IPv4 address. `ip` is returned unchanged if it's already an IPv4 address. If `ip` is currently an IPv6 address then it needs to be IPv4 compatible, otherwise an error is thrown. argumentTypeDescriptiondefaultrequired ip string The IPv4-mapped IPv6 address to convert. N/A yes #### Errors{% #ipv6_to_ipv4-errors %} - `ip` is not a valid IP address. - `ip` is an IPv6 address that is not compatible with IPv4. #### Examples{% #ipv6_to_ipv4-examples %} ##### IPv6 to IPv4{% #ipv6-to-ipv4 %} Source: ```bash ipv6_to_ipv4!("::ffff:192.168.0.1") ``` Return: ```bash "192.168.0.1" ``` ### is_ipv4{% #is_ipv4 %} Check if the string is a valid IPv4 address or not. An [IPv4-mapped](https://datatracker.ietf.org/doc/html/rfc6890) or [IPv4-compatible](https://datatracker.ietf.org/doc/html/rfc6890) IPv6 address is not considered valid for the purpose of this function. argumentTypeDescriptiondefaultrequired value string The IP address to check N/A yes #### Examples{% #is_ipv4-examples %} ##### Valid IPv4 address{% #valid-ipv4-address %} Source: ```bash is_ipv4("10.0.102.37") ``` Return: ```bash true ``` ##### Valid IPv6 address{% #valid-ipv6-address %} Source: ```bash is_ipv4("2001:0db8:85a3:0000:0000:8a2e:0370:7334") ``` Return: ```bash false ``` ##### Arbitrary string{% #arbitrary-string %} Source: ```bash is_ipv4("foobar") ``` Return: ```bash false ``` ### is_ipv6{% #is_ipv6 %} Check if the string is a valid IPv6 address or not. argumentTypeDescriptiondefaultrequired value string The IP address to check N/A yes #### Examples{% #is_ipv6-examples %} ##### Valid IPv6 address{% #valid-ipv6-address %} Source: ```bash is_ipv6("2001:0db8:85a3:0000:0000:8a2e:0370:7334") ``` Return: ```bash true ``` ##### Valid IPv4 address{% #valid-ipv4-address %} Source: ```bash is_ipv6("10.0.102.37") ``` Return: ```bash false ``` ##### Arbitrary string{% #arbitrary-string %} Source: ```bash is_ipv6("foobar") ``` Return: ```bash false ``` Number Functions ### abs{% #abs %} Computes the absolute value of `value`. argumentTypeDescriptiondefaultrequired value integer, float The number to calculate the absolute value. N/A yes #### Examples{% #abs-examples %} ##### Computes the absolute value of the integer{% #computes-the-absolute-value-of-the-integer %} Source: ```bash abs(-42) ``` Return: ```bash 42 ``` ##### Computes the absolute value of the float{% #computes-the-absolute-value-of-the-float %} Source: ```bash abs(-42.2) ``` Return: ```bash 42.2 ``` ### ceil{% #ceil %} Rounds the `value` up to the specified `precision`. argumentTypeDescriptiondefaultrequired value integer, float The number to round up. N/A yes precision integer The number of decimal places to round to. `0` no #### Examples{% #ceil-examples %} ##### Round a number up (without precision){% #round-a-number-up-without-precision %} Source: ```bash ceil(4.345) ``` Return: ```bash 5 ``` ##### Round a number up (with precision){% #round-a-number-up-with-precision %} Source: ```bash ceil(4.345, precision: 2) ``` Return: ```bash 4.35 ``` ### floor{% #floor %} Rounds the `value` down to the specified `precision`. argumentTypeDescriptiondefaultrequired value integer, float The number to round down. N/A yes precision integer The number of decimal places to round to. `0` no #### Examples{% #floor-examples %} ##### Round a number down (without precision){% #round-a-number-down-without-precision %} Source: ```bash floor(4.345) ``` Return: ```bash 4 ``` ##### Round a number down (with precision){% #round-a-number-down-with-precision %} Source: ```bash floor(4.345, precision: 2) ``` Return: ```bash 4.34 ``` ### format_int{% #format_int %} Formats the integer `value` into a string representation using the given base/radix. argumentTypeDescriptiondefaultrequired value integer The number to format. N/A yes base integer The base to format the number in. Must be between 2 and 36 (inclusive). `10` no #### Errors{% #format_int-errors %} - The base is not between 2 and 36. #### Examples{% #format_int-examples %} ##### Format as a hexadecimal integer{% #format-as-a-hexadecimal-integer %} Source: ```bash format_int!(42, 16) ``` Return: ```bash "2a" ``` ##### Format as a negative hexadecimal integer{% #format-as-a-negative-hexadecimal-integer %} Source: ```bash format_int!(-42, 16) ``` Return: ```bash "-2a" ``` ### format_number{% #format_number %} Formats the `value` into a string representation of the number. argumentTypeDescriptiondefaultrequired value integer, float The number to format as a string. N/A yes scale integer The number of decimal places to display. N/A no decimal_separator string The character to use between the whole and decimal parts of the number. `.` no grouping_separator string The character to use between each thousands part of the number. `,` no #### Examples{% #format_number-examples %} ##### Format a number (3 decimals){% #format-a-number-3-decimals %} Source: ```bash format_number(1234567.89, 3, decimal_separator: ".", grouping_separator: ",") ``` Return: ```bash "1,234,567.890" ``` ### mod{% #mod %} Calculates the remainder of `value` divided by `modulus`. argumentTypeDescriptiondefaultrequired value integer, float The value the `modulus` is applied to. N/A yes modulus integer, float The `modulus` value. N/A yes #### Errors{% #mod-errors %} - `value` is not an integer or float. - `modulus` is not an integer or float. - `modulus` is equal to 0. #### Examples{% #mod-examples %} ##### Calculate the remainder of two integers{% #calculate-the-remainder-of-two-integers %} Source: ```bash mod(5, 2) ``` Return: ```bash 1 ``` ### round{% #round %} Rounds the `value` to the specified `precision`. argumentTypeDescriptiondefaultrequired value integer, float The number to round. N/A yes precision integer The number of decimal places to round to. `0` no #### Examples{% #round-examples %} ##### Round a number (without precision){% #round-a-number-without-precision %} Source: ```bash round(4.345) ``` Return: ```bash 4 ``` ##### Round a number (with precision){% #round-a-number-with-precision %} Source: ```bash round(4.345, precision: 2) ``` Return: ```bash 4.35 ``` Object Functions ### match_datadog_query{% #match_datadog_query %} Matches an object against a [Datadog Search Syntax](https://docs.datadoghq.com/logs/explorer/search_syntax/) query. argumentTypeDescriptiondefaultrequired value object The object. N/A yes query string The Datadog Search Syntax query. N/A yes #### Examples{% #match_datadog_query-examples %} ##### OR query{% #or-query %} Source: ```bash match_datadog_query({"message": "contains this and that"}, "this OR that") ``` Return: ```bash true ``` ##### AND query{% #and-query %} Source: ```bash match_datadog_query({"message": "contains only this"}, "this AND that") ``` Return: ```bash false ``` ##### Attribute wildcard{% #attribute-wildcard %} Source: ```bash match_datadog_query({"name": "foobar"}, "@name:foo*") ``` Return: ```bash true ``` ##### Tag range{% #tag-range %} Source: ```bash match_datadog_query({"tags": ["a:x", "b:y", "c:z"]}, s'b:["x" TO "z"]') ``` Return: ```bash true ``` ### merge{% #merge %} Merges the `from` object into the `to` object. argumentTypeDescriptiondefaultrequired to object The object to merge into. N/A yes from object The object to merge from. N/A yes deep boolean A deep merge is performed if `true`, otherwise only top-level fields are merged. `false` no #### Examples{% #merge-examples %} ##### Object merge (shallow){% #object-merge-shallow %} Source: ```bash merge( { "parent1": { "child1": 1, "child2": 2 }, "parent2": { "child3": 3 } }, { "parent1": { "child2": 4, "child5": 5 } } ) ``` Return: ```bash {"parent1":{"child2":4,"child5":5},"parent2":{"child3":3}} ``` ##### Object merge (deep){% #object-merge-deep %} Source: ```bash merge( { "parent1": { "child1": 1, "child2": 2 }, "parent2": { "child3": 3 } }, { "parent1": { "child2": 4, "child5": 5 } }, deep: true ) ``` Return: ```bash {"parent1":{"child1":1,"child2":4,"child5":5},"parent2":{"child3":3}} ``` ### object_from_array{% #object_from_array %} Iterate over either one array of arrays or a pair of arrays and create an object out of all the key-value pairs contained in them. With one array of arrays, any entries with no value use `null` instead. Any keys that are `null` skip the corresponding value. If a single parameter is given, it must contain an array of all the input arrays. argumentTypeDescriptiondefaultrequired values array The first array of elements, or the array of input arrays if no other parameter is present. N/A yes keys array The second array of elements. If not present, the first parameter must contain all the arrays. N/A no #### Errors{% #object_from_array-errors %} - `values` and `keys` must be arrays. - If `keys` is not present, `values` must contain only arrays. #### Examples{% #object_from_array-examples %} ##### Create an object from one array{% #create-an-object-from-one-array %} Source: ```bash object_from_array([["one", 1], [null, 2], ["two", 3]]) ``` Return: ```bash {"one":1,"two":3} ``` ##### Create an object from separate key and value arrays{% #create-an-object-from-separate-key-and-value-arrays %} Source: ```bash object_from_array([1, 2, 3], keys: ["one", null, "two"]) ``` Return: ```bash {"one":1,"two":3} ``` ### unnest{% #unnest %} Unnest an array field from an object to create an array of objects using that field; keeping all other fields. Assigning the array result of this to `.` results in multiple events being emitted from `remap`. See the `remap` transform docs for more details. This is also referred to as `explode` in some languages. argumentTypeDescriptiondefaultrequired path path The path of the field to unnest. N/A yes #### Errors{% #unnest-errors %} - The field path referred to is not an array. #### Examples{% #unnest-examples %} ##### Unnest an array field{% #unnest-an-array-field %} Source: ```bash . = unnest!(.messages) ``` ##### Unnest nested an array field{% #unnest-nested-an-array-field %} Source: ```bash . = unnest!(.event.messages) ``` Parse Functions ### parse_apache_log{% #parse_apache_log %} Parses Apache access and error log lines. Lines can be in [`common`](https://httpd.apache.org/docs/current/logs.html#common), [`combined`](https://httpd.apache.org/docs/current/logs.html#combined), or the default [`error`](https://httpd.apache.org/docs/current/logs.html#errorlog) format. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes timestamp_format string The [date/time format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) to use for encoding the timestamp. The time is parsed in local time if the timestamp does not specify a timezone. `%d/%b/%Y:%T %z` no format string The format to use for parsing the log. N/A yes #### Errors{% #parse_apache_log-errors %} - `value` does not match the specified format. - `timestamp_format` is not a valid format string. - The timestamp in `value` fails to parse using the provided `timestamp_format`. #### Examples{% #parse_apache_log-examples %} ##### Parse using Apache log format (common){% #parse-using-apache-log-format-common %} Source: ```bash parse_apache_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", format: "common") ``` Return: ```bash {"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326} ``` ##### Parse using Apache log format (combined){% #parse-using-apache-log-format-combined %} Source: ```bash parse_apache_log!( s'127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth" "Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"', "combined", ) ``` Return: ```bash {"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326,"referrer":"http://www.seniorinfomediaries.com/vertical/channels/front-end/bandwidth","agent":"Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/1945-10-12 Firefox/37.0"} ``` ##### Parse using Apache log format (error){% #parse-using-apache-log-format-error %} Source: ```bash parse_apache_log!( s'[01/Mar/2021:12:00:19 +0000] [ab:alert] [pid 4803:tid 3814] [client 147.159.108.175:24259] I will bypass the haptic COM bandwidth, that should matrix the CSS driver!', "error" ) ``` Return: ```bash {"client":"147.159.108.175","message":"I will bypass the haptic COM bandwidth, that should matrix the CSS driver!","module":"ab","pid":4803,"port":24259,"severity":"alert","thread":"3814","timestamp":"2021-03-01T12:00:19Z"} ``` ### parse_aws_alb_log{% #parse_aws_alb_log %} Parses `value` in the [Elastic Load Balancer Access format](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#access-log-entry-examples). argumentTypeDescriptiondefaultrequired value string Access log of the Application Load Balancer. N/A yes #### Errors{% #parse_aws_alb_log-errors %} - `value` is not a properly formatted AWS ALB log. #### Examples{% #parse_aws_alb_log-examples %} ##### Parse AWS ALB log{% #parse-aws-alb-log %} Source: ```bash parse_aws_alb_log!( "http 2018-11-30T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 - 0.000 0.001 0.000 200 200 34 366 \"GET http://www.example.com:80/ HTTP/1.1\" \"curl/7.46.0\" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 \"Root=1-58337364-23a8c76965a2ef7629b185e3\" \"-\" \"-\" 0 2018-11-30T22:22:48.364000Z \"forward\" \"-\" \"-\" \"-\" \"-\" \"-\" \"-\"" ) ``` Return: ```bash {"type":"http","timestamp":"2018-11-30T22:23:00.186641Z","elb":"app/my-loadbalancer/50dc6c495c0c9188","client_host":"192.168.131.39:2817","target_host":null,"request_processing_time":0,"target_processing_time":0.001,"response_processing_time":0,"elb_status_code":"200","target_status_code":"200","received_bytes":34,"sent_bytes":366,"request_method":"GET","request_url":"http://www.example.com:80/","request_protocol":"HTTP/1.1","user_agent":"curl/7.46.0","ssl_cipher":null,"ssl_protocol":null,"target_group_arn":"arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067","trace_id":"Root=1-58337364-23a8c76965a2ef7629b185e3","traceability_id":null,"domain_name":null,"chosen_cert_arn":null,"matched_rule_priority":"0","request_creation_time":"2018-11-30T22:22:48.364000Z","actions_executed":"forward","redirect_url":null,"error_reason":null,"target_port_list":[],"target_status_code_list":[],"classification":null,"classification_reason":null} ``` ### parse_aws_cloudwatch_log_subscription_message{% #parse_aws_cloudwatch_log_subscription_message %} Parses AWS CloudWatch Logs events (configured through AWS Cloudwatch subscriptions) from the `aws_kinesis_firehose` source. argumentTypeDescriptiondefaultrequired value string The string representation of the message to parse. N/A yes #### Errors{% #parse_aws_cloudwatch_log_subscription_message-errors %} - `value` is not a properly formatted AWS CloudWatch Log subscription message. #### Examples{% #parse_aws_cloudwatch_log_subscription_message-examples %} ##### Parse AWS Cloudwatch Log subscription message{% #parse-aws-cloudwatch-log-subscription-message %} Source: ```bash parse_aws_cloudwatch_log_subscription_message!(.message) ``` Return: ```bash {"owner":"111111111111","message_type":"DATA_MESSAGE","log_group":"test","log_stream":"test","subscription_filters":["Destination"],"log_events":[{"id":"35683658089614582423604394983260738922885519999578275840","message":"{\"bytes\":26780,\"datetime\":\"14/Sep/2020:11:45:41 -0400\",\"host\":\"157.130.216.193\",\"method\":\"PUT\",\"protocol\":\"HTTP/1.0\",\"referer\":\"https://www.principalcross-platform.io/markets/ubiquitous\",\"request\":\"/expedite/convergence\",\"source_type\":\"stdin\",\"status\":301,\"user-identifier\":\"-\"}","timestamp":"2020-09-14T19:09:29.039Z"}]} ``` ### parse_aws_vpc_flow_log{% #parse_aws_vpc_flow_log %} Parses `value` in the [VPC Flow Logs format](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html). argumentTypeDescriptiondefaultrequired value string VPC Flow Log. N/A yes format string VPC Flow Log format. N/A no #### Errors{% #parse_aws_vpc_flow_log-errors %} - `value` is not a properly formatted AWS VPC Flow log. #### Examples{% #parse_aws_vpc_flow_log-examples %} ##### Parse AWS VPC Flow log (default format){% #parse-aws-vpc-flow-log-default-format %} Source: ```bash parse_aws_vpc_flow_log!("2 123456789010 eni-1235b8ca123456789 - - - - - - - 1431280876 1431280934 - NODATA") ``` Return: ```bash {"version":2,"account_id":"123456789010","interface_id":"eni-1235b8ca123456789","srcaddr":null,"dstaddr":null,"srcport":null,"dstport":null,"protocol":null,"packets":null,"bytes":null,"start":1431280876,"end":1431280934,"action":null,"log_status":"NODATA"} ``` ##### Parse AWS VPC Flow log (custom format){% #parse-aws-vpc-flow-log-custom-format %} Source: ```bash parse_aws_vpc_flow_log!( "- eni-1235b8ca123456789 10.0.1.5 10.0.0.220 10.0.1.5 203.0.113.5", "instance_id interface_id srcaddr dstaddr pkt_srcaddr pkt_dstaddr" ) ``` Return: ```bash {"instance_id":null,"interface_id":"eni-1235b8ca123456789","srcaddr":"10.0.1.5","dstaddr":"10.0.0.220","pkt_srcaddr":"10.0.1.5","pkt_dstaddr":"203.0.113.5"} ``` ##### Parse AWS VPC Flow log including v5 fields{% #parse-aws-vpc-flow-log-including-v5-fields %} Source: ```bash parse_aws_vpc_flow_log!("5 52.95.128.179 10.0.0.71 80 34210 6 1616729292 1616729349 IPv4 14 15044 123456789012 vpc-abcdefab012345678 subnet-aaaaaaaa012345678 i-0c50d5961bcb2d47b eni-1235b8ca123456789 ap-southeast-2 apse2-az3 - - ACCEPT 19 52.95.128.179 10.0.0.71 S3 - - ingress OK", format: "version srcaddr dstaddr srcport dstport protocol start end type packets bytes account_id vpc_id subnet_id instance_id interface_id region az_id sublocation_type sublocation_id action tcp_flags pkt_srcaddr pkt_dstaddr pkt_src_aws_service pkt_dst_aws_service traffic_path flow_direction log_status") ``` Return: ```bash {"account_id":"123456789012","action":"ACCEPT","az_id":"apse2-az3","bytes":15044,"dstaddr":"10.0.0.71","dstport":34210,"end":1616729349,"flow_direction":"ingress","instance_id":"i-0c50d5961bcb2d47b","interface_id":"eni-1235b8ca123456789","log_status":"OK","packets":14,"pkt_dst_aws_service":null,"pkt_dstaddr":"10.0.0.71","pkt_src_aws_service":"S3","pkt_srcaddr":"52.95.128.179","protocol":6,"region":"ap-southeast-2","srcaddr":"52.95.128.179","srcport":80,"start":1616729292,"sublocation_id":null,"sublocation_type":null,"subnet_id":"subnet-aaaaaaaa012345678","tcp_flags":19,"traffic_path":null,"type":"IPv4","version":5,"vpc_id":"vpc-abcdefab012345678"} ``` ### parse_bytes{% #parse_bytes %} Parses the `value` into a human-readable bytes format specified by `unit` and `base`. argumentTypeDescriptiondefaultrequired value string The string of the duration with either binary or SI unit. N/A yes unit string The output units for the byte. N/A yes base string The base for the byte, either 2 or 10. `2` no #### Errors{% #parse_bytes-errors %} - `value` is not a properly formatted bytes. #### Examples{% #parse_bytes-examples %} ##### Parse bytes (kilobytes){% #parse-bytes-kilobytes %} Source: ```bash parse_bytes!("1024KiB", unit: "MiB") ``` Return: ```bash 1 ``` ##### Parse bytes in SI unit (terabytes){% #parse-bytes-in-si-unit-terabytes %} Source: ```bash parse_bytes!("4TB", unit: "MB", base: "10") ``` Return: ```bash 4000000 ``` ##### Parse bytes in ambiguous unit (gigabytes){% #parse-bytes-in-ambiguous-unit-gigabytes %} Source: ```bash parse_bytes!("1GB", unit: "B", base: "2") ``` Return: ```bash 1073741824 ``` ### parse_cbor{% #parse_cbor %} Parses the `value` as [CBOR](https://cbor.io). argumentTypeDescriptiondefaultrequired value string The CBOR payload to parse. N/A yes #### Errors{% #parse_cbor-errors %} - `value` is not a valid CBOR-formatted payload. #### Examples{% #parse_cbor-examples %} ##### Parse CBOR{% #parse-cbor %} Source: ```bash parse_cbor!(decode_base64!("oWVmaWVsZGV2YWx1ZQ==")) ``` Return: ```bash {"field":"value"} ``` ### parse_cef{% #parse_cef %} Parses the `value` in CEF (Common Event Format) format. Ignores everything up to CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes translate_custom_fields boolean Toggles translation of custom field pairs to `key:value`. N/A no #### Errors{% #parse_cef-errors %} - `value` is not a properly formatted CEF string. #### Examples{% #parse_cef-examples %} ##### Parse output generated by PTA{% #parse-output-generated-by-pta %} Source: ```bash parse_cef!( "CEF:0|CyberArk|PTA|12.6|1|Suspected credentials theft|8|suser=mike2@prod1.domain.com shost=prod1.domain.com src=1.1.1.1 duser=andy@dev1.domain.com dhost=dev1.domain.com dst=2.2.2.2 cs1Label=ExtraData cs1=None cs2Label=EventID cs2=52b06812ec3500ed864c461e deviceCustomDate1Label=detectionDate deviceCustomDate1=1388577900000 cs3Label=PTAlink cs3=https://1.1.1.1/incidents/52b06812ec3500ed864c461e cs4Label=ExternalLink cs4=None" ) ``` Return: ```bash {"cefVersion":"0","deviceVendor":"CyberArk","deviceProduct":"PTA","deviceVersion":"12.6","deviceEventClassId":"1","name":"Suspected credentials theft","severity":"8","suser":"mike2@prod1.domain.com","shost":"prod1.domain.com","src":"1.1.1.1","duser":"andy@dev1.domain.com","dhost":"dev1.domain.com","dst":"2.2.2.2","cs1Label":"ExtraData","cs1":"None","cs2Label":"EventID","cs2":"52b06812ec3500ed864c461e","deviceCustomDate1Label":"detectionDate","deviceCustomDate1":"1388577900000","cs3Label":"PTAlink","cs3":"https://1.1.1.1/incidents/52b06812ec3500ed864c461e","cs4Label":"ExternalLink","cs4":"None"} ``` ##### Ignore syslog header{% #ignore-syslog-header %} Source: ```bash parse_cef!( "Sep 29 08:26:10 host CEF:1|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232" ) ``` Return: ```bash {"cefVersion":"1","deviceVendor":"Security","deviceProduct":"threatmanager","deviceVersion":"1.0","deviceEventClassId":"100","name":"worm successfully stopped","severity":"10","src":"10.0.0.1","dst":"2.1.2.2","spt":"1232"} ``` ##### Translate custom fields{% #translate-custom-fields %} Source: ```bash parse_cef!( "CEF:0|Dev|firewall|2.2|1|Connection denied|5|c6a1=2345:0425:2CA1:0000:0000:0567:5673:23b5 c6a1Label=Device IPv6 Address", translate_custom_fields: true ) ``` Return: ```bash {"cefVersion":"0","deviceVendor":"Dev","deviceProduct":"firewall","deviceVersion":"2.2","deviceEventClassId":"1","name":"Connection denied","severity":"5","Device IPv6 Address":"2345:0425:2CA1:0000:0000:0567:5673:23b5"} ``` ### parse_common_log{% #parse_common_log %} Parses the `value` using the [Common Log Format](https://httpd.apache.org/docs/current/logs.html#common) (CLF). argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes timestamp_format string The [date/time format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) to use for encoding the timestamp. `%d/%b/%Y:%T %z` no #### Errors{% #parse_common_log-errors %} - `value` does not match the Common Log Format. - `timestamp_format` is not a valid format string. - The timestamp in `value` fails to parse using the provided `timestamp_format`. #### Examples{% #parse_common_log-examples %} ##### Parse using Common Log Format (with default timestamp format){% #parse-using-common-log-format-with-default-timestamp-format %} Source: ```bash parse_common_log!("127.0.0.1 bob frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326") ``` Return: ```bash {"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326} ``` ##### Parse using Common Log Format (with custom timestamp format){% #parse-using-common-log-format-with-custom-timestamp-format %} Source: ```bash parse_common_log!( "127.0.0.1 bob frank [2000-10-10T20:55:36Z] \"GET /apache_pb.gif HTTP/1.0\" 200 2326", "%+" ) ``` Return: ```bash {"host":"127.0.0.1","identity":"bob","user":"frank","timestamp":"2000-10-10T20:55:36Z","message":"GET /apache_pb.gif HTTP/1.0","method":"GET","path":"/apache_pb.gif","protocol":"HTTP/1.0","status":200,"size":2326} ``` ### parse_csv{% #parse_csv %} Parses a single CSV formatted row. Only the first row is parsed in case of multiline input value. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes delimiter string The field delimiter to use when parsing. Must be a single-byte utf8 character. `,` no #### Errors{% #parse_csv-errors %} - The delimiter must be a single-byte UTF-8 character. - `value` is not a valid CSV string. #### Examples{% #parse_csv-examples %} ##### Parse a single CSV formatted row{% #parse-a-single-csv-formatted-row %} Source: ```bash parse_csv!("foo,bar,\"foo \"\", bar\"") ``` Return: ```bash ["foo","bar","foo \", bar"] ``` ##### Parse a single CSV formatted row with custom delimiter{% #parse-a-single-csv-formatted-row-with-custom-delimiter %} Source: ```bash parse_csv!("foo bar", delimiter: " ") ``` Return: ```bash ["foo","bar"] ``` ### parse_dnstap{% #parse_dnstap %} Parses the `value` as base64 encoded DNSTAP data. argumentTypeDescriptiondefaultrequired value string The base64 encoded representation of the DNSTAP data to parse. N/A yes lowercase_hostnames boolean Whether to turn all hostnames found in resulting data lowercase, for consistency. `false` no #### Errors{% #parse_dnstap-errors %} - `value` is not a valid base64 encoded string. - dnstap parsing failed for `value` #### Examples{% #parse_dnstap-examples %} ##### Parse dnstap query message{% #parse-dnstap-query-message %} Source: ```bash parse_dnstap!("ChVqYW1lcy1WaXJ0dWFsLU1hY2hpbmUSC0JJTkQgOS4xNi4zGgBy5wEIAxACGAEiEAAAAAAAAAAAAAAAAAAAAAAqECABBQJwlAAAAAAAAAAAADAw8+0CODVA7+zq9wVNMU3WNlI2kwIAAAABAAAAAAABCWZhY2Vib29rMQNjb20AAAEAAQAAKQIAAACAAAAMAAoACOxjCAG9zVgzWgUDY29tAGAAbQAAAAByZLM4AAAAAQAAAAAAAQJoNQdleGFtcGxlA2NvbQAABgABAAApBNABAUAAADkADwA1AAlubyBTRVAgbWF0Y2hpbmcgdGhlIERTIGZvdW5kIGZvciBkbnNzZWMtZmFpbGVkLm9yZy54AQ==") ``` Return: ```bash {"dataType":"Message","dataTypeId":1,"extraInfo":"","messageType":"ResolverQuery","messageTypeId":3,"queryZone":"com.","requestData":{"fullRcode":0,"header":{"aa":false,"ad":false,"anCount":0,"arCount":1,"cd":false,"id":37634,"nsCount":0,"opcode":0,"qdCount":1,"qr":0,"ra":false,"rcode":0,"rd":false,"tc":false},"opt":{"do":true,"ednsVersion":0,"extendedRcode":0,"options":[{"optCode":10,"optName":"Cookie","optValue":"7GMIAb3NWDM="}],"udpPayloadSize":512},"question":[{"class":"IN","domainName":"facebook1.com.","questionType":"A","questionTypeId":1}],"rcodeName":"NoError"},"responseData":{"fullRcode":16,"header":{"aa":false,"ad":false,"anCount":0,"arCount":1,"cd":false,"id":45880,"nsCount":0,"opcode":0,"qdCount":1,"qr":0,"ra":false,"rcode":16,"rd":false,"tc":false},"opt":{"do":false,"ednsVersion":1,"extendedRcode":1,"ede":[{"extraText":"no SEP matching the DS found for dnssec-failed.org.","infoCode":9,"purpose":"DNSKEY Missing"}],"udpPayloadSize":1232},"question":[{"class":"IN","domainName":"h5.example.com.","questionType":"SOA","questionTypeId":6}],"rcodeName":"BADSIG"},"responseAddress":"2001:502:7094::30","responsePort":53,"serverId":"james-Virtual-Machine","serverVersion":"BIND 9.16.3","socketFamily":"INET6","socketProtocol":"UDP","sourceAddress":"::","sourcePort":46835,"time":1593489007920014000,"timePrecision":"ns","timestamp":"2020-06-30T03:50:07.920014129Z"} ``` ### parse_duration{% #parse_duration %} Parses the `value` into a human-readable duration format specified by `unit`. argumentTypeDescriptiondefaultrequired value string The string of the duration. N/A yes unit string The output units for the duration. N/A yes #### Errors{% #parse_duration-errors %} - `value` is not a properly formatted duration. #### Examples{% #parse_duration-examples %} ##### Parse duration (milliseconds){% #parse-duration-milliseconds %} Source: ```bash parse_duration!("1005ms", unit: "s") ``` Return: ```bash 1.005 ``` ##### Parse multiple durations (seconds & milliseconds){% #parse-multiple-durations-seconds--milliseconds %} Source: ```bash parse_duration!("1s 1ms", unit: "ms") ``` Return: ```bash 1001 ``` ### parse_etld{% #parse_etld %} Parses the [eTLD](https://developer.mozilla.org/en-US/docs/Glossary/eTLD) from `value` representing domain name. argumentTypeDescriptiondefaultrequired value string The domain string. N/A yes plus_parts integer Can be provided to get additional parts of the domain name. When 1 is passed, eTLD+1 will be returned, which represents a domain registrable by a single organization. Higher numbers will return subdomains. `false` no psl string Can be provided to use a different public suffix list. By default, [https://publicsuffix.org/list/public_suffix_list.dat](https://publicsuffix.org/list/public_suffix_list.dat) is used. `false` no #### Errors{% #parse_etld-errors %} - unable to determine eTLD for `value` #### Examples{% #parse_etld-examples %} ##### Parse eTLD{% #parse-etld %} Source: ```bash parse_etld!("sub.sussex.ac.uk") ``` Return: ```bash {"etld":"ac.uk","etld_plus":"ac.uk","known_suffix":true} ``` ##### Parse eTLD+1{% #parse-etld1 %} Source: ```bash parse_etld!("sub.sussex.ac.uk", plus_parts: 1) ``` Return: ```bash {"etld":"ac.uk","etld_plus":"sussex.ac.uk","known_suffix":true} ``` ##### Parse eTLD with unknown suffix{% #parse-etld-with-unknown-suffix %} Source: ```bash parse_etld!("vector.acmecorp") ``` Return: ```bash {"etld":"acmecorp","etld_plus":"acmecorp","known_suffix":false} ``` ##### Parse eTLD with custom PSL{% #parse-etld-with-custom-psl %} Source: ```bash parse_etld!("vector.acmecorp", psl: "resources/public_suffix_list.dat") ``` Return: ```bash {"etld":"acmecorp","etld_plus":"acmecorp","known_suffix":false} ``` ### parse_glog{% #parse_glog %} Parses the `value` using the [glog (Google Logging Library)](https://github.com/google/glog) format. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes #### Errors{% #parse_glog-errors %} - `value` does not match the `glog` format. #### Examples{% #parse_glog-examples %} ##### Parse using glog{% #parse-using-glog %} Source: ```bash parse_glog!("I20210131 14:48:54.411655 15520 main.c++:9] Hello world!") ``` Return: ```bash {"level":"info","timestamp":"2021-01-31T14:48:54.411655Z","id":15520,"file":"main.c++","line":9,"message":"Hello world!"} ``` ### parse_grok{% #parse_grok %} Parses the `value` using the [`grok`](https://github.com/daschl/grok/tree/master/patterns) format. All patterns [listed here](https://github.com/daschl/grok/tree/master/patterns) are supported. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes pattern string The [Grok pattern](https://github.com/daschl/grok/tree/master/patterns). N/A yes #### Errors{% #parse_grok-errors %} - `value` fails to parse using the provided `pattern`. #### Examples{% #parse_grok-examples %} ##### Parse using Grok{% #parse-using-grok %} Source: ```bash parse_grok!( "2020-10-02T23:22:12.223222Z info Hello world", "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}" ) ``` Return: ```bash {"timestamp":"2020-10-02T23:22:12.223222Z","level":"info","message":"Hello world"} ``` ### parse_groks{% #parse_groks %} Parses the `value` using multiple [`grok`](https://github.com/daschl/grok/tree/master/patterns) patterns. All patterns [listed here](https://github.com/daschl/grok/tree/master/patterns) are supported. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes patterns array The [Grok patterns](https://github.com/daschl/grok/tree/master/patterns), which are tried in order until the first match. N/A yes aliases object The shared set of grok aliases that can be referenced in the patterns to simplify them. `true` no alias_sources string Path to the file containing aliases in a JSON format. N/A no #### Errors{% #parse_groks-errors %} - `value` fails to parse using the provided `pattern`. - `patterns` is not an array. - `aliases` is not an object. - `alias_sources` is not a string or doesn't point to a valid file. #### Examples{% #parse_groks-examples %} ##### Parse using multiple Grok patterns{% #parse-using-multiple-grok-patterns %} Source: ```bash parse_groks!( "2020-10-02T23:22:12.223222Z info Hello world", patterns: [ "%{common_prefix} %{_status} %{_message}", "%{common_prefix} %{_message}", ], aliases: { "common_prefix": "%{_timestamp} %{_loglevel}", "_timestamp": "%{TIMESTAMP_ISO8601:timestamp}", "_loglevel": "%{LOGLEVEL:level}", "_status": "%{POSINT:status}", "_message": "%{GREEDYDATA:message}" } ) ``` Return: ```bash {"timestamp":"2020-10-02T23:22:12.223222Z","level":"info","message":"Hello world"} ``` ##### Parse using aliases from file{% #parse-using-aliases-from-file %} Source: ```bash parse_groks!( "username=foo", patterns: [ "%{PATTERN_A}" ], alias_sources: [ "path/to/aliases.json" ] ) # aliases.json contents: # { # "PATTERN_A": "%{PATTERN_B}", # "PATTERN_B": "username=%{USERNAME:username}" # } ``` ### parse_influxdb{% #parse_influxdb %} Parses the `value` as an [InfluxDB line protocol](https://docs.influxdata.com/influxdb/cloud/reference/syntax/line-protocol/) string, producing a list of Vector-compatible metrics. argumentTypeDescriptiondefaultrequired value string The string representation of the InfluxDB line protocol to parse. N/A yes #### Errors{% #parse_influxdb-errors %} - `value` is not a valid InfluxDB line protocol string. - field set contains a field value of type `string`. - field set contains a `NaN` field value. #### Examples{% #parse_influxdb-examples %} ##### Parse InfluxDB line protocol{% #parse-influxdb-line-protocol %} Source: ```bash parse_influxdb!("cpu,host=A,region=us-west usage_system=64i,usage_user=10u,temperature=50.5,on=true,sleep=false 1590488773254420000") ``` Return: ```bash [{"name":"cpu_usage_system","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":64}},{"name":"cpu_usage_user","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":10}},{"name":"cpu_temperature","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":50.5}},{"name":"cpu_on","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":1}},{"name":"cpu_sleep","tags":{"host":"A","region":"us-west"},"timestamp":"2020-05-26T10:26:13.254420Z","kind":"absolute","gauge":{"value":0}}] ``` ### parse_int{% #parse_int %} Parses the string `value` representing a number in an optional base/radix to an integer. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes base integer The base the number is in. Must be between 2 and 36 (inclusive). If unspecified, the string prefix is used to determine the base: "0b", 8 for "0" or "0o", 16 for "0x", and 10 otherwise. N/A no #### Errors{% #parse_int-errors %} - The base is not between 2 and 36. - The number cannot be parsed in the base. #### Examples{% #parse_int-examples %} ##### Parse decimal{% #parse-decimal %} Source: ```bash parse_int!("-42") ``` Return: ```bash -42 ``` ##### Parse binary{% #parse-binary %} Source: ```bash parse_int!("0b1001") ``` Return: ```bash 9 ``` ##### Parse octal{% #parse-octal %} Source: ```bash parse_int!("0o42") ``` Return: ```bash 34 ``` ##### Parse hexadecimal{% #parse-hexadecimal %} Source: ```bash parse_int!("0x2a") ``` Return: ```bash 42 ``` ##### Parse explicit base{% #parse-explicit-base %} Source: ```bash parse_int!("2a", 17) ``` Return: ```bash 44 ``` ### parse_json{% #parse_json %} Parses the `value` as JSON. argumentTypeDescriptiondefaultrequired value string The string representation of the JSON to parse. N/A yes max_depth integer Number of layers to parse for nested JSON-formatted documents. The value must be in the range of 1 to 128. N/A no lossy boolean Whether to parse the JSON in a lossy manner. Replaces invalid UTF-8 characters with the Unicode character `�` (U+FFFD) if set to true, otherwise returns an error if there are any invalid UTF-8 characters present. `true` no #### Errors{% #parse_json-errors %} - `value` is not a valid JSON-formatted payload. #### Examples{% #parse_json-examples %} ##### Parse JSON{% #parse-json %} Source: ```bash parse_json!("{\"key\": \"val\"}") ``` Return: ```bash {"key":"val"} ``` ##### Parse JSON with max_depth{% #parse-json-with-max_depth %} Source: ```bash parse_json!("{\"top_level\":{\"key\": \"val\"}}", max_depth: 1) ``` Return: ```bash {"top_level":"{\"key\": \"val\"}"} ``` ### parse_key_value{% #parse_key_value %} Parses the `value` in key-value format. Also known as [logfmt](https://brandur.org/logfmt). - Keys and values can be wrapped with `"`. - `"` characters can be escaped using `\`. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes key_value_delimiter string The string that separates the key from the value. `=` no field_delimiter string The string that separates each key-value pair. N/A no whitespace string Defines the acceptance of unnecessary whitespace surrounding the configured `key_value_delimiter`. `lenient` no accept_standalone_key boolean Whether a standalone key should be accepted, the resulting object associates such keys with the boolean value `true`. `true` no #### Errors{% #parse_key_value-errors %} - `value` is not a properly formatted key-value string. #### Examples{% #parse_key_value-examples %} ##### Parse logfmt log{% #parse-logfmt-log %} Source: ```bash parse_key_value!( "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager" ) ``` Return: ```bash {"@timestamp":"Sun Jan 10 16:47:39 EST 2021","level":"info","msg":"Stopping all fetchers","tag#production":"stopping_fetchers","id":"ConsumerFetcherManager-1382721708341","module":"kafka.consumer.ConsumerFetcherManager"} ``` ##### Parse comma delimited log{% #parse-comma-delimited-log %} Source: ```bash parse_key_value!( "path:\"/cart_link\", host:store.app.com, fwd: \"102.30.171.16\", dyno: web.1, connect:0ms, service:87ms, status:304, bytes:632, protocol:https", field_delimiter: ",", key_value_delimiter: ":" ) ``` Return: ```bash {"path":"/cart_link","host":"store.app.com","fwd":"102.30.171.16","dyno":"web.1","connect":"0ms","service":"87ms","status":"304","bytes":"632","protocol":"https"} ``` ##### Parse comma delimited log with standalone keys{% #parse-comma-delimited-log-with-standalone-keys %} Source: ```bash parse_key_value!( "env:prod,service:backend,region:eu-east1,beta", field_delimiter: ",", key_value_delimiter: ":", ) ``` Return: ```bash {"env":"prod","service":"backend","region":"eu-east1","beta":true} ``` ##### Parse duplicate keys{% #parse-duplicate-keys %} Source: ```bash parse_key_value!( "at=info,method=GET,path=\"/index\",status=200,tags=dev,tags=dummy", field_delimiter: ",", key_value_delimiter: "=", ) ``` Return: ```bash {"at":"info","method":"GET","path":"/index","status":"200","tags":["dev","dummy"]} ``` ### parse_klog{% #parse_klog %} Parses the `value` using the [klog](https://github.com/kubernetes/klog) format used by Kubernetes components. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes #### Errors{% #parse_klog-errors %} - `value` does not match the `klog` format. #### Examples{% #parse_klog-examples %} ##### Parse using klog{% #parse-using-klog %} Source: ```bash parse_klog!("I0505 17:59:40.692994 28133 klog.go:70] hello from klog") ``` Return: ```bash {"file":"klog.go","id":28133,"level":"info","line":70,"message":"hello from klog","timestamp":"2025-05-05T17:59:40.692994Z"} ``` ### parse_linux_authorization{% #parse_linux_authorization %} Parses Linux authorization logs usually found under either `/var/log/auth.log` (for Debian-based systems) or `/var/log/secure` (for RedHat-based systems) according to [Syslog](https://en.wikipedia.org/wiki/Syslog) format. argumentTypeDescriptiondefaultrequired value string The text containing the message to parse. N/A yes #### Errors{% #parse_linux_authorization-errors %} - `value` is not a properly formatted Syslog message. #### Examples{% #parse_linux_authorization-examples %} ##### Parse Linux authorization event{% #parse-linux-authorization-event %} Source: ```bash parse_linux_authorization!( s'Mar 23 01:49:58 localhost sshd[1111]: Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar' ) ``` Return: ```bash {"appname":"sshd","hostname":"localhost","message":"Accepted publickey for eng from 10.1.1.1 port 8888 ssh2: RSA SHA256:foobar","procid":1111,"timestamp":"2025-03-23T01:49:58Z"} ``` ### parse_logfmt{% #parse_logfmt %} Parses the `value` in [logfmt](https://brandur.org/logfmt). - Keys and values can be wrapped using the `"` character. - `"` characters can be escaped by the `\` character. - As per this [logfmt specification](https://pkg.go.dev/github.com/kr/logfmt#section-documentation), the `parse_logfmt` function accepts standalone keys and assigns them a Boolean value of `true`. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes #### Errors{% #parse_logfmt-errors %} - `value` is not a properly formatted key-value string #### Examples{% #parse_logfmt-examples %} ##### Parse logfmt log{% #parse-logfmt-log %} Source: ```bash parse_logfmt!( "@timestamp=\"Sun Jan 10 16:47:39 EST 2021\" level=info msg=\"Stopping all fetchers\" tag#production=stopping_fetchers id=ConsumerFetcherManager-1382721708341 module=kafka.consumer.ConsumerFetcherManager" ) ``` Return: ```bash {"@timestamp":"Sun Jan 10 16:47:39 EST 2021","level":"info","msg":"Stopping all fetchers","tag#production":"stopping_fetchers","id":"ConsumerFetcherManager-1382721708341","module":"kafka.consumer.ConsumerFetcherManager"} ``` ### parse_nginx_log{% #parse_nginx_log %} ``` Parses Nginx access and error log lines. Lines can be in [`combined`](https://nginx.org/en/docs/http/ngx_http_log_module.html), [`ingress_upstreaminfo`](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/log-format/), [`main`](https://hg.nginx.org/pkg-oss/file/tip/debian/debian/nginx.conf) or [`error`](https://github.com/nginx/nginx/blob/branches/stable-1.18/src/core/ngx_log.c#L102) format. ``` argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes timestamp_format string The [date/time format](https://docs.rs/chrono/latest/chrono/format/strftime/index.html#specifiers) to use for encoding the timestamp. The time is parsed in local time if the timestamp doesn't specify a timezone. The default format is `%d/%b/%Y:%T %z` for combined logs and `%Y/%m/%d %H:%M:%S` for error logs. `%d/%b/%Y:%T %z` no format string The format to use for parsing the log. N/A yes #### Errors{% #parse_nginx_log-errors %} - `value` does not match the specified format. - `timestamp_format` is not a valid format string. - The timestamp in `value` fails to parse using the provided `timestamp_format`. #### Examples{% #parse_nginx_log-examples %} ##### Parse via Nginx log format (combined){% #parse-via-nginx-log-format-combined %} Source: ```bash parse_nginx_log!( s'172.17.0.1 - alice [01/Apr/2021:12:02:31 +0000] "POST /not-found HTTP/1.1" 404 153 "http://localhost/somewhere" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"', "combined", ) ``` Return: ```bash {"agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36","client":"172.17.0.1","compression":"2.75","referer":"http://localhost/somewhere","request":"POST /not-found HTTP/1.1","size":153,"status":404,"timestamp":"2021-04-01T12:02:31Z","user":"alice"} ``` ##### Parse via Nginx log format (error){% #parse-via-nginx-log-format-error %} Source: ```bash parse_nginx_log!( s'2021/04/01 13:02:31 [error] 31#31: *1 open() "/usr/share/nginx/html/not-found" failed (2: No such file or directory), client: 172.17.0.1, server: localhost, request: "POST /not-found HTTP/1.1", host: "localhost:8081"', "error" ) ``` Return: ```bash {"timestamp":"2021-04-01T13:02:31Z","severity":"error","pid":31,"tid":31,"cid":1,"message":"open() \"/usr/share/nginx/html/not-found\" failed (2: No such file or directory)","client":"172.17.0.1","server":"localhost","request":"POST /not-found HTTP/1.1","host":"localhost:8081"} ``` ##### Parse via Nginx log format (ingress_upstreaminfo){% #parse-via-nginx-log-format-ingress_upstreaminfo %} Source: ```bash parse_nginx_log!( s'0.0.0.0 - bob [18/Mar/2023:15:00:00 +0000] "GET /some/path HTTP/2.0" 200 12312 "https://10.0.0.1/some/referer" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" 462 0.050 [some-upstream-service-9000] [some-other-upstream-5000] 10.0.50.80:9000 19437 0.049 200 752178adb17130b291aefd8c386279e7', "ingress_upstreaminfo" ) ``` Return: ```bash {"body_bytes_size":12312,"http_referer":"https://10.0.0.1/some/referer","http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36","proxy_alternative_upstream_name":"some-other-upstream-5000","proxy_upstream_name":"some-upstream-service-9000","remote_addr":"0.0.0.0","remote_user":"bob","req_id":"752178adb17130b291aefd8c386279e7","request":"GET /some/path HTTP/2.0","request_length":462,"request_time":0.05,"status":200,"timestamp":"2023-03-18T15:00:00Z","upstream_addr":"10.0.50.80:9000","upstream_response_length":19437,"upstream_response_time":0.049,"upstream_status":200} ``` ##### Parse via Nginx log format (main){% #parse-via-nginx-log-format-main %} Source: ```bash parse_nginx_log!( s'172.24.0.3 - alice [31/Dec/2024:17:32:06 +0000] "GET / HTTP/1.1" 200 615 "https://domain.tld/path" "curl/8.11.1" "1.2.3.4, 10.10.1.1"', "main" ) ``` Return: ```bash {"body_bytes_size":615,"http_referer":"https://domain.tld/path","http_user_agent":"curl/8.11.1","http_x_forwarded_for":"1.2.3.4, 10.10.1.1","remote_addr":"172.24.0.3","remote_user":"alice","request":"GET / HTTP/1.1","status":200,"timestamp":"2024-12-31T17:32:06Z"} ``` ### parse_proto{% #parse_proto %} Parses the `value` as a protocol buffer payload. argumentTypeDescriptiondefaultrequired value string The protocol buffer payload to parse. N/A yes desc_file string The path to the protobuf descriptor set file. Must be a literal string. This file is the output of protoc -o… N/A yes message_type string The name of the message type to use for serializing. Must be a literal string. N/A yes #### Errors{% #parse_proto-errors %} - `value` is not a valid proto payload. - `desc_file` file does not exist. - `message_type` message type does not exist in the descriptor file. #### Examples{% #parse_proto-examples %} ##### Parse proto{% #parse-proto %} Source: ```bash parse_proto!(decode_base64!("Cgdzb21lb25lIggKBjEyMzQ1Ng=="), "resources/protobuf_descriptor_set.desc", "test_protobuf.Person") ``` Return: ```bash {"name":"someone","phones":[{"number":"123456"}]} ``` ### parse_query_string{% #parse_query_string %} Parses the `value` as a query string. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes #### Examples{% #parse_query_string-examples %} ##### Parse query string{% #parse-query-string %} Source: ```bash parse_query_string("foo=%2B1&bar=2&bar=3&xyz") ``` Return: ```bash {"foo":"+1","bar":["2","3"],"xyz":""} ``` ##### Parse Ruby on Rails' query string{% #parse-ruby-on-rails-query-string %} Source: ```bash parse_query_string("?foo%5b%5d=1&foo%5b%5d=2") ``` Return: ```bash {"foo[]":["1","2"]} ``` ### parse_regex{% #parse_regex %} Parses the `value` using the provided [Regex](https://en.wikipedia.org/wiki/Regular_expression) `pattern`. This function differs from the `parse_regex_all` function in that it returns only the first match. argumentTypeDescriptiondefaultrequired value string The string to search. N/A yes pattern regex The regular expression pattern to search against. N/A yes numeric_groups regex If true, the index of each group in the regular expression is also captured. Index `0` contains the whole match. `false` no #### Errors{% #parse_regex-errors %} - `value` fails to parse using the provided `pattern`. #### Examples{% #parse_regex-examples %} ##### Parse using Regex (with capture groups){% #parse-using-regex-with-capture-groups %} Source: ```bash parse_regex!("first group and second group.", r'(?P.*?) group') ``` Return: ```bash {"number":"first"} ``` ##### Parse using Regex (without capture groups){% #parse-using-regex-without-capture-groups %} Source: ```bash parse_regex!("first group and second group.", r'(\w+) group', numeric_groups: true) ``` Return: ```bash {"0":"first group","1":"first"} ``` ### parse_regex_all{% #parse_regex_all %} Parses the `value` using the provided [Regex](https://en.wikipedia.org/wiki/Regular_expression) `pattern`. This function differs from the `parse_regex` function in that it returns *all* matches, not just the first. argumentTypeDescriptiondefaultrequired value string The string to search. N/A yes pattern regex The regular expression pattern to search against. N/A yes numeric_groups regex If `true`, the index of each group in the regular expression is also captured. Index `0` contains the whole match. `false` no #### Errors{% #parse_regex_all-errors %} - `value` is not a string. - `pattern` is not a regex. #### Examples{% #parse_regex_all-examples %} ##### Parse using Regex (all matches){% #parse-using-regex-all-matches %} Source: ```bash parse_regex_all!("first group and second group.", r'(?P\w+) group', numeric_groups: true) ``` Return: ```bash [{"0":"first group","1":"first","number":"first"},{"0":"second group","1":"second","number":"second"}] ``` ### parse_ruby_hash{% #parse_ruby_hash %} Parses the `value` as ruby hash. argumentTypeDescriptiondefaultrequired value string The string representation of the ruby hash to parse. N/A yes #### Errors{% #parse_ruby_hash-errors %} - `value` is not a valid ruby hash formatted payload. #### Examples{% #parse_ruby_hash-examples %} ##### Parse ruby hash{% #parse-ruby-hash %} Source: ```bash parse_ruby_hash!(s'{ "test" => "value", "testNum" => 0.2, "testObj" => { "testBool" => true, "testNull" => nil } }') ``` Return: ```bash {"test":"value","testNum":0.2,"testObj":{"testBool":true,"testNull":null}} ``` ### parse_syslog{% #parse_syslog %} Parses the `value` in [Syslog](https://en.wikipedia.org/wiki/Syslog) format. argumentTypeDescriptiondefaultrequired value string The text containing the Syslog message to parse. N/A yes #### Errors{% #parse_syslog-errors %} - `value` is not a properly formatted Syslog message. #### Examples{% #parse_syslog-examples %} ##### Parse Syslog log (5424){% #parse-syslog-log-5424 %} Source: ```bash parse_syslog!( s'<13>1 2020-03-13T20:45:38.119Z dynamicwireless.name non 2426 ID931 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"] Try to override the THX port, maybe it will reboot the neural interface!' ) ``` Return: ```bash {"severity":"notice","facility":"user","timestamp":"2020-03-13T20:45:38.119Z","hostname":"dynamicwireless.name","appname":"non","procid":2426,"msgid":"ID931","message":"Try to override the THX port, maybe it will reboot the neural interface!","exampleSDID@32473":{"eventID":"1011","eventSource":"Application","iut":"3"},"version":1} ``` ### parse_timestamp{% #parse_timestamp %} Parses the `value` in [strptime](https://docs.rs/chrono/latest/chrono/format/strftime/index.html#specifiers) `format`. argumentTypeDescriptiondefaultrequired value string The text of the timestamp. N/A yes format string The [strptime](https://docs.rs/chrono/latest/chrono/format/strftime/index.html#specifiers) format. N/A yes timezone string The [TZ database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) format. By default, this function parses the timestamp by global `timezone` option. This argument overwrites the setting and is useful for parsing timestamps without a specified timezone, such as `16/10/2019 12:00:00`. N/A no #### Errors{% #parse_timestamp-errors %} - `value` fails to parse using the provided `format`. - `value` fails to parse using the provided `timezone`. #### Examples{% #parse_timestamp-examples %} ##### Parse timestamp{% #parse-timestamp %} Source: ```bash parse_timestamp!("10-Oct-2020 16:00+00:00", format: "%v %R %:z") ``` Return: ```bash "2020-10-10T16:00:00Z" ``` ##### Parse timestamp with timezone{% #parse-timestamp-with-timezone %} Source: ```bash parse_timestamp!("16/10/2019 12:00:00", format: "%d/%m/%Y %H:%M:%S", timezone: "Asia/Taipei") ``` Return: ```bash "2019-10-16T04:00:00Z" ``` ### parse_tokens{% #parse_tokens %} Parses the `value` in token format. A token is considered to be one of the following: - A word surrounded by whitespace. - Text delimited by double quotes: `".."`. Quotes can be included in the token if they are escaped by a backslash (`\`). - Text delimited by square brackets: `[..]`. Closing square brackets can be included in the token if they are escaped by a backslash (`\`). argumentTypeDescriptiondefaultrequired value string The string to tokenize. N/A yes #### Errors{% #parse_tokens-errors %} - `value` is not a properly formatted tokenized string. #### Examples{% #parse_tokens-examples %} ##### Parse tokens{% #parse-tokens %} Source: ```bash parse_tokens( "A sentence \"with \\\"a\\\" sentence inside\" and [some brackets]" ) ``` Return: ```bash ["A","sentence","with \\\"a\\\" sentence inside","and","some brackets"] ``` ### parse_url{% #parse_url %} Parses the `value` in [URL](https://en.wikipedia.org/wiki/URL) format. argumentTypeDescriptiondefaultrequired value string The text of the URL. N/A yes default_known_ports boolean If true and the port number is not specified in the input URL string (or matches the default port for the scheme), it is populated from well-known ports for the following schemes: `http`, `https`, `ws`, `wss`, and `ftp`. `false` no #### Errors{% #parse_url-errors %} - `value` is not a properly formatted URL. #### Examples{% #parse_url-examples %} ##### Parse URL{% #parse-url %} Source: ```bash parse_url!("ftp://foo:bar@example.com:4343/foobar?hello=world#123") ``` Return: ```bash {"scheme":"ftp","username":"foo","password":"bar","host":"example.com","port":4343,"path":"/foobar","query":{"hello":"world"},"fragment":"123"} ``` ##### Parse URL with default port{% #parse-url-with-default-port %} Source: ```bash parse_url!("https://example.com", default_known_ports: true) ``` Return: ```bash {"scheme":"https","username":"","password":"","host":"example.com","port":443,"path":"/","query":{},"fragment":null} ``` ##### Parse URL with internationalized domain name{% #parse-url-with-internationalized-domain-name %} Source: ```bash parse_url!("https://www.café.com") ``` Return: ```bash {"scheme":"https","username":"","password":"","host":"www.xn--caf-dma.com","port":null,"path":"/","query":{},"fragment":null} ``` ##### Parse URL with mixed case internationalized domain name{% #parse-url-with-mixed-case-internationalized-domain-name %} Source: ```bash parse_url!("https://www.CAFé.com") ``` Return: ```bash {"scheme":"https","username":"","password":"","host":"www.xn--caf-dma.com","port":null,"path":"/","query":{},"fragment":null} ``` ### parse_user_agent{% #parse_user_agent %} Parses the `value` as a user agent string, which has [a loosely defined format](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent) so this parser only provides best effort guarantee. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes mode string Determines performance and reliability characteristics. `fast` no #### Examples{% #parse_user_agent-examples %} ##### Fast mode{% #fast-mode %} Source: ```bash parse_user_agent( "Mozilla Firefox 1.0.1 Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.7.6) Gecko/20050223 Firefox/1.0.1" ) ``` Return: ```bash {"browser":{"family":"Firefox","version":"1.0.1"},"device":{"category":"pc"},"os":{"family":"Linux","version":null}} ``` ##### Reliable mode{% #reliable-mode %} Source: ```bash parse_user_agent( "Mozilla/4.0 (compatible; MSIE 7.66; Windows NT 5.1; SV1; .NET CLR 1.1.4322)", mode: "reliable" ) ``` Return: ```bash {"browser":{"family":"Internet Explorer","version":"7.66"},"device":{"category":"pc"},"os":{"family":"Windows XP","version":"NT 5.1"}} ``` ##### Enriched mode{% #enriched-mode %} Source: ```bash parse_user_agent( "Opera/9.80 (J2ME/MIDP; Opera Mini/4.3.24214; iPhone; CPU iPhone OS 4_2_1 like Mac OS X; AppleWebKit/24.783; U; en) Presto/2.5.25 Version/10.54", mode: "enriched" ) ``` Return: ```bash {"browser":{"family":"Opera Mini","major":"4","minor":"3","patch":"24214","version":"10.54"},"device":{"brand":"Apple","category":"smartphone","family":"iPhone","model":"iPhone"},"os":{"family":"iOS","major":"4","minor":"2","patch":"1","patch_minor":null,"version":"4.2.1"}} ``` ### parse_xml{% #parse_xml %} Parses the `value` as XML. argumentTypeDescriptiondefaultrequired value string The string representation of the XML document to parse. N/A yes include_attr boolean Include XML tag attributes in the returned object. `true` no attr_prefix string String prefix to use for XML tag attribute keys. `@` no text_key string Key name to use for expanded text nodes. `text` no always_use_text_key boolean Always return text nodes as `{"": "value"}.` `false` no parse_bool boolean Parse "true" and "false" as boolean. `true` no parse_null boolean Parse "null" as null. `true` no parse_number boolean Parse numbers as integers/floats. `true` no #### Errors{% #parse_xml-errors %} - `value` is not a valid XML document. #### Examples{% #parse_xml-examples %} ##### Parse XML{% #parse-xml %} Source: ```bash value = s'Harry PotterJ K. Rowling2005'; parse_xml!(value, text_key: "value", parse_number: false) ``` Return: ```bash {"book":{"@category":"CHILDREN","author":"J K. Rowling","title":{"@lang":"en","value":"Harry Potter"},"year":"2005"}} ``` Path Functions ### del{% #del %} Removes the field specified by the static `path` from the target. For dynamic path deletion, see the `remove` function. argumentTypeDescriptiondefaultrequired path path The path of the field to delete. N/A yes compact boolean After deletion, if `compact` is `true` and there is an empty object or array left, the empty object or array is also removed, cascading up to the root. This only applies to the path being deleted, and any parent paths. `false` no #### Examples{% #del-examples %} ##### Delete a field{% #delete-a-field %} Source: ```bash del(.field1) ``` ##### Rename a field{% #rename-a-field %} Source: ```bash .new_field = del(.old_field) ``` ### exists{% #exists %} Checks whether the `path` exists for the target. This function distinguishes between a missing path and a path with a `null` value. A regular path lookup, such as `.foo`, cannot distinguish between the two cases since it always returns `null` if the path doesn't exist. argumentTypeDescriptiondefaultrequired path path The path of the field to check. N/A yes #### Examples{% #exists-examples %} ##### Exists (field){% #exists-field %} Source: ```bash exists(.field) ``` Return: ```bash true ``` ##### Exists (array element){% #exists-array-element %} Source: ```bash exists(.array[2]) ``` Return: ```bash true ``` ### get{% #get %} Dynamically get the value of a given path. If you know the path you want to look up, use static paths such as `.foo.bar[1]` to get the value of that path. However, if you do not know the path names, use the dynamic `get` function to get the requested value. argumentTypeDescriptiondefaultrequired value object, array The object or array to query. N/A yes path array An array of path segments to look for the value. N/A yes #### Errors{% #get-errors %} - The `path` segment must be a string or an integer. #### Examples{% #get-examples %} ##### single-segment top-level field{% #single-segment-top-level-field %} Source: ```bash get!(value: { "foo": "bar" }, path: ["foo"]) ``` Return: ```bash "bar" ``` ##### multi-segment nested field{% #multi-segment-nested-field %} Source: ```bash get!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"]) ``` Return: ```bash "baz" ``` ##### array indexing{% #array-indexing %} Source: ```bash get!(value: ["foo", "bar", "baz"], path: [-2]) ``` Return: ```bash "bar" ``` ### remove{% #remove %} Dynamically remove the value for a given path. If you know the path you want to remove, use the `del` function and static paths such as `del(.foo.bar[1])` to remove the value at that path. The `del` function returns the deleted value, and is more performant than `remove`. However, if you do not know the path names, use the dynamic `remove` function to remove the value at the provided path. argumentTypeDescriptiondefaultrequired value object, array The object or array to remove data from. N/A yes path array An array of path segments to remove the value from. N/A yes compact boolean After deletion, if `compact` is `true`, any empty objects or arrays left are also removed. `false` no #### Errors{% #remove-errors %} - The `path` segment must be a string or an integer. #### Examples{% #remove-examples %} ##### single-segment top-level field{% #single-segment-top-level-field %} Source: ```bash remove!(value: { "foo": "bar" }, path: ["foo"]) ``` Return: ```bash {} ``` ##### multi-segment nested field{% #multi-segment-nested-field %} Source: ```bash remove!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"]) ``` Return: ```bash {"foo":{}} ``` ##### array indexing{% #array-indexing %} Source: ```bash remove!(value: ["foo", "bar", "baz"], path: [-2]) ``` Return: ```bash ["foo","baz"] ``` ##### compaction{% #compaction %} Source: ```bash remove!(value: { "foo": { "bar": [42], "baz": true } }, path: ["foo", "bar", 0], compact: true) ``` Return: ```bash {"foo":{"baz":true}} ``` ### set{% #set %} Dynamically insert data into the path of a given object or array. If you know the path you want to assign a value to, use static path assignments such as `.foo.bar[1] = true` for improved performance and readability. However, if you do not know the path names, use the dynamic `set` function to insert the data into the object or array. argumentTypeDescriptiondefaultrequired value object, array The object or array to insert data into. N/A yes path array An array of path segments to insert the value into. N/A yes data any The data to be inserted. N/A yes #### Errors{% #set-errors %} - The `path` segment must be a string or an integer. #### Examples{% #set-examples %} ##### single-segment top-level field{% #single-segment-top-level-field %} Source: ```bash set!(value: { "foo": "bar" }, path: ["foo"], data: "baz") ``` Return: ```bash {"foo":"baz"} ``` ##### multi-segment nested field{% #multi-segment-nested-field %} Source: ```bash set!(value: { "foo": { "bar": "baz" } }, path: ["foo", "bar"], data: "qux") ``` Return: ```bash {"foo":{"bar":"qux"}} ``` ##### array{% #array %} Source: ```bash set!(value: ["foo", "bar", "baz"], path: [-2], data: 42) ``` Return: ```bash ["foo",42,"baz"] ``` Random Functions ### random_bool{% #random_bool %} Returns a random boolean. #### Examples{% #random_bool-examples %} ##### Random boolean{% #random-boolean %} Source: ```bash is_boolean(random_bool()) ``` Return: ```bash true ``` ### random_bytes{% #random_bytes %} A cryptographically secure random number generator. Returns a string value containing the number of random bytes requested. argumentTypeDescriptiondefaultrequired length integer The number of bytes to generate. Must not be larger than 64k. N/A yes #### Errors{% #random_bytes-errors %} - `length` is negative. - `length` is larger than the maximum value (64k). #### Examples{% #random_bytes-examples %} ##### Generate random base 64 encoded bytes{% #generate-random-base-64-encoded-bytes %} Source: ```bash encode_base64(random_bytes(16)) ``` Return: ```bash "LNu0BBgUbh7XAlXbjSOomQ==" ``` ### random_float{% #random_float %} Returns a random float between [min, max). argumentTypeDescriptiondefaultrequired min float Minimum value (inclusive). N/A yes max float Maximum value (exclusive). N/A yes #### Errors{% #random_float-errors %} - `max` is not greater than `min`. #### Examples{% #random_float-examples %} ##### Random float from 0.0 to 10.0, not including 10.0{% #random-float-from-00-to-100-not-including-100 %} Source: ```bash f = random_float(0.0, 10.0) f >= 0 && f < 10 ``` Return: ```bash true ``` ### random_int{% #random_int %} Returns a random integer between [min, max). argumentTypeDescriptiondefaultrequired min integer Minimum value (inclusive). N/A yes max integer Maximum value (exclusive). N/A yes #### Errors{% #random_int-errors %} - `max` is not greater than `min`. #### Examples{% #random_int-examples %} ##### Random integer from 0 to 10, not including 10{% #random-integer-from-0-to-10-not-including-10 %} Source: ```bash i = random_int(0, 10) i >= 0 && i < 10 ``` Return: ```bash true ``` ### uuid_from_friendly_id{% #uuid_from_friendly_id %} Convert a Friendly ID (base62 encoding a 128-bit word) to a UUID. argumentTypeDescriptiondefaultrequired value timestamp A string that is a Friendly ID N/A yes #### Errors{% #uuid_from_friendly_id-errors %} - `value` is a string but the text uses characters outside of class [0-9A-Za-z]. - `value` is a base62 encoding of an integer, but the integer is greater than or equal to 2^128. #### Examples{% #uuid_from_friendly_id-examples %} ##### Convert a Friendly ID to a UUID{% #convert-a-friendly-id-to-a-uuid %} Source: ```bash uuid_from_friendly_id!("3s87yEvnmkiPBMHsj8bwwc") ``` Return: ```bash "7f41deed-d5e2-8b5e-7a13-ab4ff93cfad2" ``` ### uuid_v4{% #uuid_v4 %} Generates a random [UUIDv4](https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_%28random%29) string. #### Examples{% #uuid_v4-examples %} ##### Create a UUIDv4{% #create-a-uuidv4 %} Source: ```bash uuid_v4() ``` Return: ```bash "1d262f4f-199b-458d-879f-05fd0a5f0683" ``` ### uuid_v7{% #uuid_v7 %} Generates a random [UUIDv7](https://datatracker.ietf.org/doc/html/draft-peabody-dispatch-new-uuid-format-04#name-uuid-version-7) string. argumentTypeDescriptiondefaultrequired timestamp timestamp The timestamp used to generate the UUIDv7. `now()` no #### Examples{% #uuid_v7-examples %} ##### Create a UUIDv7 with implicit `now()`{% #create-a-uuidv7-with-implicit-now %} Source: ```bash uuid_v7() ``` Return: ```bash "06338364-8305-7b74-8000-de4963503139" ``` ##### Create a UUIDv7 with explicit `now()`{% #create-a-uuidv7-with-explicit-now %} Source: ```bash uuid_v7(now()) ``` Return: ```bash "018e29b3-0bea-7f78-8af3-d32ccb1b93c1" ``` ##### Create a UUIDv7 with custom timestamp{% #create-a-uuidv7-with-custom-timestamp %} Source: ```bash uuid_v7(t'2020-12-30T22:20:53.824727Z') ``` Return: ```bash "0176b5bd-5d19-7394-bb60-c21028c6152b" ``` String Functions ### camelcase{% #camelcase %} Takes the `value` string, and turns it into camelCase. Optionally, you can pass in the existing case of the function, or else an attempt is made to determine the case automatically. argumentTypeDescriptiondefaultrequired value string The string to convert to camelCase. N/A yes original_case string Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case N/A no #### Examples{% #camelcase-examples %} ##### camelCase a string{% #camelcase-a-string %} Source: ```bash camelcase("input-string") ``` Return: ```bash "inputString" ``` ##### camelCase a string{% #camelcase-a-string %} Source: ```bash camelcase("input-string", "kebab-case") ``` Return: ```bash "inputString" ``` ### community_id{% #community_id %} Generates an ID based on the [Community ID Spec](https://github.com/corelight/community-id-spec). argumentTypeDescriptiondefaultrequired source_ip string The source IP address. N/A yes destination_ip string The destination IP address. N/A yes protocol integer The protocol number. N/A yes source_port integer The source port or ICMP type. N/A no destination_port integer The destination port or ICMP code. N/A no seed integer The custom seed number. N/A no #### Examples{% #community_id-examples %} ##### TCP{% #tcp %} Source: ```bash community_id!(source_ip: "1.2.3.4", destination_ip: "5.6.7.8", source_port: 1122, destination_port: 3344, protocol: 6) ``` Return: ```bash "1:wCb3OG7yAFWelaUydu0D+125CLM=" ``` ### contains{% #contains %} Determines whether the `value` string contains the specified `substring`. argumentTypeDescriptiondefaultrequired value string The text to search. N/A yes substring string The substring to search for in `value`. N/A yes case_sensitive boolean Whether the match should be case sensitive. `true` no #### Examples{% #contains-examples %} ##### String contains (case sensitive){% #string-contains-case-sensitive %} Source: ```bash contains("The Needle In The Haystack", "Needle") ``` Return: ```bash true ``` ##### String contains (case insensitive){% #string-contains-case-insensitive %} Source: ```bash contains("The Needle In The Haystack", "needle", case_sensitive: false) ``` Return: ```bash true ``` ### contains_all{% #contains_all %} Determines whether the `value` string contains all the specified `substrings`. argumentTypeDescriptiondefaultrequired value string The text to search. N/A yes substrings array An array of substrings to search for in `value`. N/A yes case_sensitive boolean Whether the match should be case sensitive. N/A no #### Examples{% #contains_all-examples %} ##### String contains all{% #string-contains-all %} Source: ```bash contains_all("The Needle In The Haystack", ["Needle", "Haystack"]) ``` Return: ```bash true ``` ##### String contains all (case sensitive){% #string-contains-all-case-sensitive %} Source: ```bash contains_all("the NEEDLE in the haystack", ["needle", "haystack"]) ``` Return: ```bash false ``` ### downcase{% #downcase %} Downcases the `value` string, where downcase is defined according to the Unicode Derived Core Property Lowercase. argumentTypeDescriptiondefaultrequired value string The string to convert to lowercase. N/A yes #### Examples{% #downcase-examples %} ##### Downcase a string{% #downcase-a-string %} Source: ```bash downcase("Hello, World!") ``` Return: ```bash "hello, world!" ``` ### ends_with{% #ends_with %} Determines whether the `value` string ends with the specified `substring`. argumentTypeDescriptiondefaultrequired value string The string to search. N/A yes substring string The substring with which `value` must end. N/A yes case_sensitive boolean Whether the match should be case sensitive. `true` no #### Examples{% #ends_with-examples %} ##### String ends with (case sensitive){% #string-ends-with-case-sensitive %} Source: ```bash ends_with("The Needle In The Haystack", "The Haystack") ``` Return: ```bash true ``` ##### String ends with (case insensitive){% #string-ends-with-case-insensitive %} Source: ```bash ends_with("The Needle In The Haystack", "the haystack", case_sensitive: false) ``` Return: ```bash true ``` ### find{% #find %} Determines from left to right the start position of the first found element in `value` that matches `pattern`. Returns `-1` if not found. argumentTypeDescriptiondefaultrequired value string The string to find the pattern in. N/A yes pattern regex, string The regular expression or string pattern to match against. N/A yes from integer Offset to start searching. `0` no #### Examples{% #find-examples %} ##### Match text{% #match-text %} Source: ```bash find("foobar", "foo") ``` Return: ```bash 0 ``` ##### Match regex{% #match-regex %} Source: ```bash find("foobar", r'b.r') ``` Return: ```bash 3 ``` ##### No matches{% #no-matches %} Source: ```bash find("foobar", "baz") ``` Return: ```bash -1 ``` ##### With an offset{% #with-an-offset %} Source: ```bash find("foobarfoobarfoo", "bar", 4) ``` Return: ```bash 9 ``` ### join{% #join %} Joins each string in the `value` array into a single string, with items optionally separated from one another by a `separator`. argumentTypeDescriptiondefaultrequired value array The array of strings to join together. N/A yes separator string The string separating each original element when joined. N/A no #### Examples{% #join-examples %} ##### Join array (no separator){% #join-array-no-separator %} Source: ```bash join!(["bring", "us", "together"]) ``` Return: ```bash "bringustogether" ``` ##### Join array (comma separator){% #join-array-comma-separator %} Source: ```bash join!(["sources", "transforms", "sinks"], separator: ", ") ``` Return: ```bash "sources, transforms, sinks" ``` ### kebabcase{% #kebabcase %} Takes the `value` string, and turns it into kebab-case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically. argumentTypeDescriptiondefaultrequired value string The string to convert to kebab-case. N/A yes original_case string Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case N/A no #### Examples{% #kebabcase-examples %} ##### kebab-case a string{% #kebab-case-a-string %} Source: ```bash kebabcase("InputString") ``` Return: ```bash "input-string" ``` ##### kebab-case a string{% #kebab-case-a-string %} Source: ```bash kebabcase("InputString", "PascalCase") ``` Return: ```bash "input-string" ``` ### match{% #match %} Determines whether the `value` matches the `pattern`. argumentTypeDescriptiondefaultrequired value string The value to match. N/A yes pattern regex The regular expression pattern to match against. N/A yes #### Examples{% #match-examples %} ##### Regex match on a string{% #regex-match-on-a-string %} Source: ```bash match("I'm a little teapot", r'teapot') ``` Return: ```bash true ``` ##### String does not match the regular expression{% #string-does-not-match-the-regular-expression %} Source: ```bash match("I'm a little teapot", r'.*balloon') ``` Return: ```bash false ``` ### match_any{% #match_any %} Determines whether `value` matches any of the given `patterns`. All patterns are checked in a single pass over the target string, giving this function a potential performance advantage over the multiple calls in the `match` function. argumentTypeDescriptiondefaultrequired value string The value to match. N/A yes patterns array The array of regular expression patterns to match against. N/A yes #### Examples{% #match_any-examples %} ##### Regex match on a string{% #regex-match-on-a-string %} Source: ```bash match_any("I'm a little teapot", [r'frying pan', r'teapot']) ``` Return: ```bash true ``` ### parse_float{% #parse_float %} Parses the string `value` representing a floating point number in base 10 to a float. argumentTypeDescriptiondefaultrequired value string The string to parse. N/A yes #### Errors{% #parse_float-errors %} - `value` is not a string. #### Examples{% #parse_float-examples %} ##### Parse negative integer{% #parse-negative-integer %} Source: ```bash parse_float!("-42") ``` Return: ```bash -42 ``` ##### Parse negative integer{% #parse-negative-integer %} Source: ```bash parse_float!("42.38") ``` Return: ```bash 42.38 ``` ##### Scientific notation{% #scientific-notation %} Source: ```bash parse_float!("2.5e3") ``` Return: ```bash 2500 ``` ### pascalcase{% #pascalcase %} Takes the `value` string, and turns it into PascalCase. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically. argumentTypeDescriptiondefaultrequired value string The string to convert to PascalCase. N/A yes original_case string Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case N/A no #### Examples{% #pascalcase-examples %} ##### PascalCase a string{% #pascalcase-a-string %} Source: ```bash pascalcase("input-string") ``` Return: ```bash "InputString" ``` ##### PascalCase a string{% #pascalcase-a-string %} Source: ```bash pascalcase("input-string", "kebab-case") ``` Return: ```bash "InputString" ``` ### redact{% #redact %} Redact sensitive data in `value` such as: - [US social security card numbers](https://www.ssa.gov/history/ssn/geocard.html) - Other forms of personally identifiable information with custom patterns This can help achieve compliance by ensuring sensitive data does not leave your network. argumentTypeDescriptiondefaultrequired value string, object, array The value to redact sensitive data from. The function's behavior depends on `value`'s type: - For strings, the sensitive data is redacted and a new string is returned. - For arrays, the sensitive data is redacted in each string element. - For objects, the sensitive data in each string value is masked, but the keys are not masked. For arrays and objects, the function recurses into any nested arrays or objects. Any non-string elements are skipped. Redacted text is replaced with `[REDACTED]`. N/A yes filters array List of filters applied to `value`. Each filter can be specified in the following ways: - As a regular expression, which is used to redact text that match it. - As an object with a `type` key that corresponds to a named filter and additional keys for customizing that filter. - As a named filter, if it has no required parameters. Named filters can be a: - `pattern`: Redacts text matching any regular expressions specified in the `patterns` key, which is required. This is the expanded version of just passing a regular expression as a filter. - `us_social_security_number`: Redacts US social security card numbers. See examples for more details. This parameter must be a static expression so that the argument can be validated at compile-time to avoid runtime errors. You cannot use variables or other dynamic expressions with it. N/A yes redactor string, object Specifies what to replace the redacted strings with. It is given as an object with a "type" key specifying the type of redactor to use and additional keys depending on the type. The following types are supported: - `full`: The default. Replace with the string "[REDACTED]". - `text`: Replace with a custom string. The `replacement` key is required, and must contain the string that is used as a replacement. - `sha2`: Hash the redacted text with SHA-2 as with [`sha2`](https://en.wikipedia.org/wiki/SHA-2). Supports two optional parameters: - `variant`: The variant of the algorithm to use. Defaults to SHA-512/256. - `encoding`: How to encode the hash as text. Can be base16 or base64. Defaults to base64. - `sha3`: Hash the redacted text with SHA-3 as with [`sha3`](https://en.wikipedia.org/wiki/SHA-3). Supports two optional parameters: - `variant`: The variant of the algorithm to use. Defaults to SHA3-512. - `encoding`: How to encode the hash as text. Can be base16 or base64. Defaults to base64. As a convenience you can use a string as a shorthand for common redactor patterns: - `"full"` is equivalent to `{"type": "full"}` - `"sha2"` is equivalent to `{"type": "sha2", "variant": "SHA-512/256", "encoding": "base64"}` - `"sha3"` is equivalent to `{"type": "sha3", "variant": "SHA3-512", "encoding": "base64"}` This parameter must be a static expression so that the argument can be validated at compile-time to avoid runtime errors. You cannot use variables or other dynamic expressions with it. N/A no #### Examples{% #redact-examples %} ##### Replace text using a regex{% #replace-text-using-a-regex %} Source: ```bash redact("my id is 123456", filters: [r'\d+']) ``` Return: ```bash "my id is [REDACTED]" ``` ##### Replace us social security numbers in any field{% #replace-us-social-security-numbers-in-any-field %} Source: ```bash redact({ "name": "John Doe", "ssn": "123-12-1234"}, filters: ["us_social_security_number"]) ``` Return: ```bash {"name":"John Doe","ssn":"[REDACTED]"} ``` ##### Replace with custom text{% #replace-with-custom-text %} Source: ```bash redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "text", "replacement": "***"}) ``` Return: ```bash "my id is ***" ``` ##### Replace with SHA-2 hash{% #replace-with-sha-2-hash %} Source: ```bash redact("my id is 123456", filters: [r'\d+'], redactor: "sha2") ``` Return: ```bash "my id is GEtTedW1p6tC094dDKH+3B8P+xSnZz69AmpjaXRd63I=" ``` ##### Replace with SHA-3 hash{% #replace-with-sha-3-hash %} Source: ```bash redact("my id is 123456", filters: [r'\d+'], redactor: "sha3") ``` Return: ```bash "my id is ZNCdmTDI7PeeUTFnpYjLdUObdizo+bIupZdl8yqnTKGdLx6X3JIqPUlUWUoFBikX+yTR+OcvLtAqWO11NPlNJw==" ``` ##### Replace with SHA-256 hash using hex encoding{% #replace-with-sha-256-hash-using-hex-encoding %} Source: ```bash redact("my id is 123456", filters: [r'\d+'], redactor: {"type": "sha2", "variant": "SHA-256", "encoding": "base16"}) ``` Return: ```bash "my id is 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92" ``` ### replace{% #replace %} Replaces all matching instances of `pattern` in `value`. The `pattern` argument accepts regular expression capture groups. **Note when using capture groups**: - You will need to escape the `$` by using `$$` to avoid Vector interpreting it as an environment variable when loading configuration - If you want a literal `$` in the replacement pattern, you will also need to escape this with `$$`. When combined with environment variable interpolation in config files this means you will need to use `$$$$` to have a literal `$` in the replacement pattern. argumentTypeDescriptiondefaultrequired value string The original string. N/A yes pattern regex, string Replace all matches of this pattern. Can be a static string or a regular expression. N/A yes with string The string that the matches are replaced with. N/A yes count integer The maximum number of replacements to perform. `-1` means replace all matches. `-1` no #### Examples{% #replace-examples %} ##### Replace literal text{% #replace-literal-text %} Source: ```bash replace("Apples and Bananas", "and", "not") ``` Return: ```bash "Apples not Bananas" ``` ##### Replace using regular expression{% #replace-using-regular-expression %} Source: ```bash replace("Apples and Bananas", r'(?i)bananas', "Pineapples") ``` Return: ```bash "Apples and Pineapples" ``` ##### Replace first instance{% #replace-first-instance %} Source: ```bash replace("Bananas and Bananas", "Bananas", "Pineapples", count: 1) ``` Return: ```bash "Pineapples and Bananas" ``` ##### Replace with capture groups (Note: Use `$$num` in config files){% #replace-with-capture-groups-note-use-num-in-config-files %} Source: ```bash replace("foo123bar", r'foo(?P\d+)bar', "$num") ``` Return: ```bash "123" ``` ### screamingsnakecase{% #screamingsnakecase %} Takes the `value` string, and turns it into SCREAMING_SNAKE case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically. argumentTypeDescriptiondefaultrequired value string The string to convert to SCREAMING_SNAKE case. N/A yes original_case string Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case N/A no #### Examples{% #screamingsnakecase-examples %} ##### SCREAMING_SNAKE a string{% #screaming_snake-a-string %} Source: ```bash screamingsnakecase("input-string") ``` Return: ```bash "INPUT_STRING" ``` ##### SCREAMING_SNAKE a string{% #screaming_snake-a-string %} Source: ```bash screamingsnakecase("input-string", "kebab-case") ``` Return: ```bash "INPUT_STRING" ``` ### shannon_entropy{% #shannon_entropy %} Generates [Shannon entropy](https://en.wikipedia.org/wiki/Entropy_%28information_theory%29) from given string. It can generate it based on string bytes, codepoints, or graphemes. argumentTypeDescriptiondefaultrequired value string The input string. N/A yes segmentation string Defines how to split the string to calculate entropy, based on occurrences of segments. Byte segmentation is the fastest, but it might give undesired results when handling UTF-8 strings, while grapheme segmentation is the slowest, but most correct in these cases. `byte` no #### Examples{% #shannon_entropy-examples %} ##### Simple byte segmentation example{% #simple-byte-segmentation-example %} Source: ```bash floor(shannon_entropy("vector.dev"), precision: 4) ``` Return: ```bash 2.9219 ``` ##### UTF-8 string with bytes segmentation{% #utf-8-string-with-bytes-segmentation %} Source: ```bash floor(shannon_entropy("test123%456.فوائد.net."), precision: 4) ``` Return: ```bash 4.0784 ``` ##### UTF-8 string with grapheme segmentation{% #utf-8-string-with-grapheme-segmentation %} Source: ```bash floor(shannon_entropy("test123%456.فوائد.net.", segmentation: "grapheme"), precision: 4) ``` Return: ```bash 3.9362 ``` ### sieve{% #sieve %} Keeps only matches of `pattern` in `value`. This can be used to define patterns that are allowed in the string and remove everything else. argumentTypeDescriptiondefaultrequired value string The original string. N/A yes pattern regex Keep all matches of this pattern. N/A yes replace_single string The string to use to replace single rejected characters. N/A no replace_repeated string The string to use to replace multiple sequential instances of rejected characters. N/A no #### Examples{% #sieve-examples %} ##### Sieve with regex{% #sieve-with-regex %} Source: ```bash sieve("test123%456.فوائد.net.", r'[a-z0-9.]') ``` Return: ```bash "test123456..net." ``` ##### Custom replacements{% #custom-replacements %} Source: ```bash sieve("test123%456.فوائد.net.", r'[a-z.0-9]', replace_single: "X", replace_repeated: "") ``` Return: ```bash "test123X456..net." ``` ### slice{% #slice %} Returns a slice of `value` between the `start` and `end` positions. If the `start` and `end` parameters are negative, they refer to positions counting from the right of the string or array. If `end` refers to a position that is greater than the length of the string or array, a slice up to the end of the string or array is returned. argumentTypeDescriptiondefaultrequired value array, string The string or array to slice. N/A yes start integer The inclusive start position. A zero-based index that can be negative. N/A yes end integer The exclusive end position. A zero-based index that can be negative. `String length` no #### Examples{% #slice-examples %} ##### Slice a string (positive index){% #slice-a-string-positive-index %} Source: ```bash slice!("Supercalifragilisticexpialidocious", start: 5, end: 13) ``` Return: ```bash "califrag" ``` ##### Slice a string (negative index){% #slice-a-string-negative-index %} Source: ```bash slice!("Supercalifragilisticexpialidocious", start: 5, end: -14) ``` Return: ```bash "califragilistic" ``` ### snakecase{% #snakecase %} Takes the `value` string, and turns it into snake-case. Optionally, you can pass in the existing case of the function, or else we will try to figure out the case automatically. argumentTypeDescriptiondefaultrequired value string The string to convert to snake-case. N/A yes original_case string Optional hint on the original case type. Must be one of: kebab-case, camelCase, PascalCase, SCREAMING_SNAKE, snake_case N/A no #### Examples{% #snakecase-examples %} ##### snake-case a string{% #snake-case-a-string %} Source: ```bash snakecase("input-string") ``` Return: ```bash "input_string" ``` ##### snake-case a string{% #snake-case-a-string %} Source: ```bash snakecase("input-string", "kebab-case") ``` Return: ```bash "input_string" ``` ### split{% #split %} Splits the `value` string using `pattern`. argumentTypeDescriptiondefaultrequired value string The string to split. N/A yes pattern string, regex The string is split whenever this pattern is matched. N/A yes limit integer The maximum number of substrings to return. N/A no #### Examples{% #split-examples %} ##### Split a string (no limit){% #split-a-string-no-limit %} Source: ```bash split("apples and pears and bananas", " and ") ``` Return: ```bash ["apples","pears","bananas"] ``` ##### Split a string (with a limit){% #split-a-string-with-a-limit %} Source: ```bash split("apples and pears and bananas", " and ", limit: 2) ``` Return: ```bash ["apples","pears and bananas"] ``` ### starts_with{% #starts_with %} Determines whether `value` begins with `substring`. argumentTypeDescriptiondefaultrequired value string The string to search. N/A yes substring string The substring that the `value` must start with. N/A yes case_sensitive boolean Whether the match should be case sensitive. `true` no #### Examples{% #starts_with-examples %} ##### String starts with (case sensitive){% #string-starts-with-case-sensitive %} Source: ```bash starts_with("The Needle In The Haystack", "The Needle") ``` Return: ```bash true ``` ##### String starts with (case insensitive){% #string-starts-with-case-insensitive %} Source: ```bash starts_with("The Needle In The Haystack", "the needle", case_sensitive: false) ``` Return: ```bash true ``` ### strip_ansi_escape_codes{% #strip_ansi_escape_codes %} Strips [ANSI escape codes](https://en.wikipedia.org/wiki/ANSI_escape_code) from `value`. argumentTypeDescriptiondefaultrequired value string The string to strip. N/A yes #### Examples{% #strip_ansi_escape_codes-examples %} ##### Strip ANSI escape codes{% #strip-ansi-escape-codes %} Source: ```bash strip_ansi_escape_codes("\e[46mfoo\e[0m bar") ``` Return: ```bash "foo bar" ``` ### strip_whitespace{% #strip_whitespace %} Strips whitespace from the start and end of `value`, where whitespace is defined by the [Unicode `White_Space` property](https://en.wikipedia.org/wiki/Unicode_character_property#Whitespace). argumentTypeDescriptiondefaultrequired value string The string to trim. N/A yes #### Examples{% #strip_whitespace-examples %} ##### Strip whitespace{% #strip-whitespace %} Source: ```bash strip_whitespace(" A sentence. ") ``` Return: ```bash "A sentence." ``` ### truncate{% #truncate %} Truncates the `value` string up to the `limit` number of characters. argumentTypeDescriptiondefaultrequired value string The string to truncate. N/A yes limit integer, float The number of characters to truncate the string after. N/A yes ellipsis boolean This argument is deprecated. An ellipsis (`...`) is appended if the parameter is set to `true` *and* the `value` string is truncated because it exceeded the `limit`. N/A no suffix string A custom suffix (`...`) is appended to truncated strings. If `ellipsis` is set to `true`, this parameter is ignored for backwards compatibility. N/A no #### Examples{% #truncate-examples %} ##### Truncate a string{% #truncate-a-string %} Source: ```bash truncate("A rather long sentence.", limit: 11, suffix: "...") ``` Return: ```bash "A rather lo..." ``` ##### Truncate a string{% #truncate-a-string %} Source: ```bash truncate("A rather long sentence.", limit: 11, suffix: "[TRUNCATED]") ``` Return: ```bash "A rather lo[TRUNCATED]" ``` ### upcase{% #upcase %} Upcases `value`, where upcase is defined according to the Unicode Derived Core Property Uppercase. argumentTypeDescriptiondefaultrequired value string The string to convert to uppercase. N/A yes #### Examples{% #upcase-examples %} ##### Upcase a string{% #upcase-a-string %} Source: ```bash upcase("Hello, World!") ``` Return: ```bash "HELLO, WORLD!" ``` System Functions ### get_hostname{% #get_hostname %} Returns the local system's hostname. #### Errors{% #get_hostname-errors %} - Internal hostname resolution failed. #### Examples{% #get_hostname-examples %} ##### Get hostname{% #get-hostname %} Source: ```bash .hostname = get_hostname!() ``` ### get_timezone_name{% #get_timezone_name %} Returns the name of the timezone in the Vector configuration (see global configuration options). If the configuration is set to `local`, then it attempts to determine the name of the timezone from the host OS. If this is not possible, then it returns the fixed offset of the local timezone for the current time in the format `"[+-]HH:MM"`, for example, `"+02:00"`. #### Errors{% #get_timezone_name-errors %} - Retrieval of local timezone information failed. #### Examples{% #get_timezone_name-examples %} ##### Get the IANA name of Vector's timezone{% #get-the-iana-name-of-vectors-timezone %} Source: ```bash .vector_timezone = get_timezone_name!() ``` Timestamp Functions ### format_timestamp{% #format_timestamp %} Formats `value` into a string representation of the timestamp. argumentTypeDescriptiondefaultrequired value timestamp The timestamp to format as text. N/A yes format string The format string as described by the [Chrono library](https://docs.rs/chrono/latest/chrono/format/strftime/index.html#specifiers). N/A yes timezone string The timezone to use when formatting the timestamp. The parameter uses the TZ identifier or `local`. N/A no #### Examples{% #format_timestamp-examples %} ##### Format a timestamp (ISO8601/RFC 3339){% #format-a-timestamp-iso8601rfc-3339 %} Source: ```bash format_timestamp!(t'2020-10-21T16:00:00Z', format: "%+") ``` Return: ```bash "2020-10-21T16:00:00+00:00" ``` ##### Format a timestamp (custom){% #format-a-timestamp-custom %} Source: ```bash format_timestamp!(t'2020-10-21T16:00:00Z', format: "%v %R") ``` Return: ```bash "21-Oct-2020 16:00" ``` ### now{% #now %} Returns the current timestamp in the UTC timezone with nanosecond precision. #### Examples{% #now-examples %} ##### Generate a current timestamp{% #generate-a-current-timestamp %} Source: ```bash now() ``` Return: ```bash "2021-03-04T10:51:15.928937Z" ``` Type Functions ### array{% #array %} Returns `value` if it is an array, otherwise returns an error. This enables the type checker to guarantee that the returned value is an array and can be used in any function that expects an array. argumentTypeDescriptiondefaultrequired value any The value to check if it is an array. N/A yes #### Errors{% #array-errors %} - `value` is not an array. #### Examples{% #array-examples %} ##### Declare an array type{% #declare-an-array-type %} Source: ```bash array!(.value) ``` Return: ```bash [1,2,3] ``` ### bool{% #bool %} Returns `value` if it is a Boolean, otherwise returns an error. This enables the type checker to guarantee that the returned value is a Boolean and can be used in any function that expects a Boolean. argumentTypeDescriptiondefaultrequired value any The value to check if it is a Boolean. N/A yes #### Errors{% #bool-errors %} - `value` is not a Boolean. #### Examples{% #bool-examples %} ##### Declare a Boolean type{% #declare-a-boolean-type %} Source: ```bash bool!(.value) ``` Return: ```bash false ``` ### float{% #float %} Returns `value` if it is a float, otherwise returns an error. This enables the type checker to guarantee that the returned value is a float and can be used in any function that expects a float. argumentTypeDescriptiondefaultrequired value any The value to check if it is a float. N/A yes #### Errors{% #float-errors %} - `value` is not a float. #### Examples{% #float-examples %} ##### Declare a float type{% #declare-a-float-type %} Source: ```bash float!(.value) ``` Return: ```bash 42 ``` ### int{% #int %} Returns `value` if it is an integer, otherwise returns an error. This enables the type checker to guarantee that the returned value is an integer and can be used in any function that expects an integer. argumentTypeDescriptiondefaultrequired value any The value to check if it is an integer. N/A yes #### Errors{% #int-errors %} - `value` is not an integer. #### Examples{% #int-examples %} ##### Declare an integer type{% #declare-an-integer-type %} Source: ```bash int!(.value) ``` Return: ```bash 42 ``` ### is_array{% #is_array %} Check if the `value`'s type is an array. argumentTypeDescriptiondefaultrequired value any The value to check if it is an array. N/A yes #### Examples{% #is_array-examples %} ##### Valid array{% #valid-array %} Source: ```bash is_array([1, 2, 3]) ``` Return: ```bash true ``` ##### Non-matching type{% #non-matching-type %} Source: ```bash is_array("a string") ``` Return: ```bash false ``` ### is_boolean{% #is_boolean %} Check if the `value`'s type is a boolean. argumentTypeDescriptiondefaultrequired value any The value to check if it is a Boolean. N/A yes #### Examples{% #is_boolean-examples %} ##### Valid boolean{% #valid-boolean %} Source: ```bash is_boolean(false) ``` Return: ```bash true ``` ##### Non-matching type{% #non-matching-type %} Source: ```bash is_boolean("a string") ``` Return: ```bash false ``` ### is_empty{% #is_empty %} Check if the object, array, or string has a length of `0`. argumentTypeDescriptiondefaultrequired value object, array, string The value to check. N/A yes #### Examples{% #is_empty-examples %} ##### Empty array{% #empty-array %} Source: ```bash is_empty([]) ``` Return: ```bash true ``` ##### Non-empty string{% #non-empty-string %} Source: ```bash is_empty("a string") ``` Return: ```bash false ``` ##### Non-empty object{% #non-empty-object %} Source: ```bash is_empty({"foo": "bar"}) ``` Return: ```bash false ``` ### is_float{% #is_float %} Check if the `value`'s type is a float. argumentTypeDescriptiondefaultrequired value any The value to check if it is a float. N/A yes #### Examples{% #is_float-examples %} ##### Valid float{% #valid-float %} Source: ```bash is_float(0.577) ``` Return: ```bash true ``` ##### Non-matching type{% #non-matching-type %} Source: ```bash is_float("a string") ``` Return: ```bash false ``` ### is_integer{% #is_integer %} Check if the value`'s type is an integer. argumentTypeDescriptiondefaultrequired value any The value to check if it is an integer. N/A yes #### Examples{% #is_integer-examples %} ##### Valid integer{% #valid-integer %} Source: ```bash is_integer(1) ``` Return: ```bash true ``` ##### Non-matching type{% #non-matching-type %} Source: ```bash is_integer("a string") ``` Return: ```bash false ``` ### is_json{% #is_json %} Check if the string is a valid JSON document. argumentTypeDescriptiondefaultrequired value string The value to check if it is a valid JSON document. N/A yes variant string The variant of the JSON type to explicitly check for. N/A no #### Examples{% #is_json-examples %} ##### Valid JSON object{% #valid-json-object %} Source: ```bash is_json("{}") ``` Return: ```bash true ``` ##### Non-valid value{% #non-valid-value %} Source: ```bash is_json("{") ``` Return: ```bash false ``` ##### Exact variant{% #exact-variant %} Source: ```bash is_json("{}", variant: "object") ``` Return: ```bash true ``` ##### Non-valid exact variant{% #non-valid-exact-variant %} Source: ```bash is_json("{}", variant: "array") ``` Return: ```bash false ``` ### is_null{% #is_null %} Check if `value`'s type is `null`. For a more relaxed function, see `is_nullish`. argumentTypeDescriptiondefaultrequired value any The value to check if it is `null`. N/A yes #### Examples{% #is_null-examples %} ##### Null value{% #null-value %} Source: ```bash is_null(null) ``` Return: ```bash true ``` ##### Non-matching type{% #non-matching-type %} Source: ```bash is_null("a string") ``` Return: ```bash false ``` ### is_nullish{% #is_nullish %} Determines whether `value` is nullish. Returns `true` if the specified `value` is `null`, an empty string, a string containing only whitespace, or the string `"-"`. Returns `false` otherwise. argumentTypeDescriptiondefaultrequired value any The value to check for nullishness, for example, a useless value. N/A yes #### Examples{% #is_nullish-examples %} ##### Null detection (blank string){% #null-detection-blank-string %} Source: ```bash is_nullish("") ``` Return: ```bash true ``` ##### Null detection (dash string){% #null-detection-dash-string %} Source: ```bash is_nullish("-") ``` Return: ```bash true ``` ##### Null detection (whitespace){% #null-detection-whitespace %} Source: ```bash is_nullish(" ") ``` Return: ```bash true ``` ### is_object{% #is_object %} Check if `value`'s type is an object. argumentTypeDescriptiondefaultrequired value any The value to check if it is an object. N/A yes #### Examples{% #is_object-examples %} ##### Valid object{% #valid-object %} Source: ```bash is_object({"foo": "bar"}) ``` Return: ```bash true ``` ##### Non-matching type{% #non-matching-type %} Source: ```bash is_object("a string") ``` Return: ```bash false ``` ### is_regex{% #is_regex %} Check if `value`'s type is a regex. argumentTypeDescriptiondefaultrequired value any The value to check if it is a regex. N/A yes #### Examples{% #is_regex-examples %} ##### Valid regex{% #valid-regex %} Source: ```bash is_regex(r'pattern') ``` Return: ```bash true ``` ##### Non-matching type{% #non-matching-type %} Source: ```bash is_regex("a string") ``` Return: ```bash false ``` ### is_string{% #is_string %} Check if `value`'s type is a string. argumentTypeDescriptiondefaultrequired value any The value to check if it is a string. N/A yes #### Examples{% #is_string-examples %} ##### Valid string{% #valid-string %} Source: ```bash is_string("a string") ``` Return: ```bash true ``` ##### Non-matching type{% #non-matching-type %} Source: ```bash is_string([1, 2, 3]) ``` Return: ```bash false ``` ### is_timestamp{% #is_timestamp %} Check if `value`'s type is a timestamp. argumentTypeDescriptiondefaultrequired value any The value to check if it is a timestamp. N/A yes #### Examples{% #is_timestamp-examples %} ##### Valid timestamp{% #valid-timestamp %} Source: ```bash is_timestamp(t'2021-03-26T16:00:00Z') ``` Return: ```bash true ``` ##### Non-matching type{% #non-matching-type %} Source: ```bash is_timestamp("a string") ``` Return: ```bash false ``` ### object{% #object %} Returns `value` if it is an object, otherwise returns an error. This enables the type checker to guarantee that the returned value is an object and can be used in any function that expects an object. argumentTypeDescriptiondefaultrequired value any The value to check if it is an object. N/A yes #### Errors{% #object-errors %} - `value` is not an object. #### Examples{% #object-examples %} ##### Declare an object type{% #declare-an-object-type %} Source: ```bash object!(.value) ``` Return: ```bash {"field1":"value1","field2":"value2"} ``` ### string{% #string %} Returns `value` if it is a string, otherwise returns an error. This enables the type checker to guarantee that the returned value is a string and can be used in any function that expects a string. argumentTypeDescriptiondefaultrequired value any The value to check if it is a string. N/A yes #### Errors{% #string-errors %} - `value` is not a string. #### Examples{% #string-examples %} ##### Declare a string type{% #declare-a-string-type %} Source: ```bash string!(.message) ``` Return: ```bash "{\"field\": \"value\"}" ``` ### tag_types_externally{% #tag_types_externally %} Adds type information to all (nested) scalar values in the provided `value`. The type information is added externally, meaning that `value` has the form of `"type": value` after this transformation. argumentTypeDescriptiondefaultrequired value any The value to tag with types. N/A yes #### Examples{% #tag_types_externally-examples %} ##### Tag types externally (scalar){% #tag-types-externally-scalar %} Source: ```bash tag_types_externally(123) ``` Return: ```bash {"integer":123} ``` ##### Tag types externally (object){% #tag-types-externally-object %} Source: ```bash tag_types_externally({ "message": "Hello world", "request": { "duration_ms": 67.9 } }) ``` Return: ```bash {"message":{"string":"Hello world"},"request":{"duration_ms":{"float":67.9}}} ``` ##### Tag types externally (array){% #tag-types-externally-array %} Source: ```bash tag_types_externally(["foo", "bar"]) ``` Return: ```bash [{"string":"foo"},{"string":"bar"}] ``` ##### Tag types externally (null){% #tag-types-externally-null %} Source: ```bash tag_types_externally(null) ``` Return: ```bash null ``` ### timestamp{% #timestamp %} Returns `value` if it is a timestamp, otherwise returns an error. This enables the type checker to guarantee that the returned value is a timestamp and can be used in any function that expects a timestamp. argumentTypeDescriptiondefaultrequired value any The value to check if it is a timestamp. N/A yes #### Errors{% #timestamp-errors %} - `value` is not a timestamp. #### Examples{% #timestamp-examples %} ##### Declare a timestamp type{% #declare-a-timestamp-type %} Source: ```bash timestamp(t'2020-10-10T16:00:00Z') ``` Return: ```bash "2020-10-10T16:00:00Z" ``` ## Further reading{% #further-reading %} - [Remap reserved attributes](https://docs.datadoghq.com/observability_pipelines/guide/remap_reserved_attributes/) - [Writing Effective Grok Parsing Rules with Regular Expressions](https://docs.datadoghq.com/logs/guide/regex_log_parsing/)