Pipelines de logs

Pipelines de logs

Les pipelines et les processeurs fonctionnent sur les logs entrants : ils effectuent le parsing de ces logs et les transforment en attributs structurés pour faciliter les requêtes.

  • Consultez la [page de configuration des pipelines] (https://app.datadoghq.com/logs/pipelines) pour obtenir une liste des pipelines et des processeurs actuellement configurés dans notre interface utilisateur.

  • Vous trouverez de plus amples informations concernant l’API pour les processeurs dans la documentation relative aux processeurs.

  • Pour en savoir plus sur les pipelines, consultez la documentation à ce sujet.

  • Remarque : ces endpoints sont uniquement disponibles pour les utilisateurs admin. Veillez à utiliser une clé d’application créée par un admin.

Les règles de parsing Grok peuvent affecter la sortie JSON et nécessitent de configurer les données renvoyées avant leur utilisation dans une requête. Par exemple, si vous utilisez les données renvoyées par une requête dans un autre corps de requête, et que vous avez une règle de parsing qui utilise une expression regex comme \s pour les espaces, vous devrez configurer tous les espaces échappés en tant que %{space} pour utiliser les données.

GET https://api.datadoghq.eu/api/v1/logs/config/pipeline-orderhttps://api.ddog-gov.com/api/v1/logs/config/pipeline-orderhttps://api.datadoghq.com/api/v1/logs/config/pipeline-orderhttps://api.us3.datadoghq.com/api/v1/logs/config/pipeline-order

Présentation

Récupérez la séquence actuelle de vos pipelines. Cet endpoint ne prend aucun argument JSON.

Réponse

OK

Object containing the ordered list of pipeline IDs.

Expand All

Champ

Type

Description

pipeline_ids [required]

[string]

Ordered Array of <PIPELINE_ID> strings, the order of pipeline IDs in the array define the overall Pipelines order for Datadog.

{
  "pipeline_ids": [
    "tags",
    "org_ids",
    "products"
  ]
}

Forbidden

Error response object.

Expand All

Champ

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Exemple de code


# Curl command
curl -X GET "https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com/api/v1/logs/config/pipeline-order" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}"
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "os"
    datadog "github.com/DataDog/datadog-api-client-go/api/v1/datadog"
)

func main() {
    ctx := datadog.NewDefaultContext(context.Background())


    configuration := datadog.NewConfiguration()

    apiClient := datadog.NewAPIClient(configuration)
    resp, r, err := apiClient.LogsPipelinesApi.GetLogsPipelineOrder(ctx)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.GetLogsPipelineOrder`: %v\n", err)
        fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
    }
    // response from `GetLogsPipelineOrder`: LogsPipelinesOrder
    responseContent, _ := json.MarshalIndent(resp, "", "  ")
    fmt.Fprintf(os.Stdout, "Response from LogsPipelinesApi.GetLogsPipelineOrder:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" go run "main.go"
import os
from dateutil.parser import parse as dateutil_parser
from datadog_api_client.v1 import ApiClient, ApiException, Configuration
from datadog_api_client.v1.api import logs_pipelines_api
from datadog_api_client.v1.models import *
from pprint import pprint
# See configuration.py for a list of all supported configuration parameters.
configuration = Configuration()

# Enter a context with an instance of the API client
with ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = logs_pipelines_api.LogsPipelinesApi(api_client)

    # example, this endpoint has no required or optional parameters
    try:
        # Get pipeline order
        api_response = api_instance.get_logs_pipeline_order()
        pprint(api_response)
    except ApiException as e:
        print("Exception when calling LogsPipelinesApi->get_logs_pipeline_order: %s\n" % e)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" python3 "example.py"
require 'datadog_api_client'
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new

begin
  # Get pipeline order
  result = api_instance.get_logs_pipeline_order
  p result
rescue DatadogAPIClient::V1::APIError => e
  puts "Error when calling LogsPipelinesAPI->get_logs_pipeline_order: #{e}"
end

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" rb "example.rb"
import { v1 } from '@datadog/datadog-api-client';
import * as fs from 'fs';

const configuration = v1.createConfiguration();
const apiInstance = new v1.LogsPipelinesApi(configuration);


apiInstance.getLogsPipelineOrder().then((data:any) => {
  console.log('API called successfully. Returned data: ' + JSON.stringify(data));
}).catch((error:any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" tsc "example.ts"
import java.util.*;
import com.datadog.api.v1.client.ApiClient;
import com.datadog.api.v1.client.ApiException;
import com.datadog.api.v1.client.Configuration;
import com.datadog.api.v1.client.model.*;
import com.datadog.api.v1.client.api.LogsPipelinesApi;

public class Example {
    public static void main(String[] args) {
        ApiClient defaultClient = Configuration.getDefaultApiClient();

        LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);
        try {
            LogsPipelinesOrder result = apiInstance.getLogsPipelineOrder();
            System.out.println(result);
        } catch (ApiException e) {
            System.err.println("Exception when calling LogsPipelinesApi#getLogsPipelineOrder");
            System.err.println("Status code: " + e.getCode());
            System.err.println("Reason: " + e.getResponseBody());
            System.err.println("Response headers: " + e.getResponseHeaders());
            e.printStackTrace();
        }
    }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" java "Example.java"

PUT https://api.datadoghq.eu/api/v1/logs/config/pipeline-orderhttps://api.ddog-gov.com/api/v1/logs/config/pipeline-orderhttps://api.datadoghq.com/api/v1/logs/config/pipeline-orderhttps://api.us3.datadoghq.com/api/v1/logs/config/pipeline-order

Présentation

Mettez à jour la séquence de vos pipelines. Les logs étant traités de manière séquentielle, la réorganisation d’un pipeline peut changer la structure et le contenu des données traitées par les autres pipelines et leurs processeurs.

Remarque : la méthode PUT permet de mettre à jour la séquence des pipelines en remplaçant votre séquence actuelle par la nouvelle, envoyée à votre organisation Datadog.

Requête

Body Data (required)

Objet contenant la nouvelle liste triée des ID de pipeline.

Expand All

Champ

Type

Description

pipeline_ids [required]

[string]

Ordered Array of <PIPELINE_ID> strings, the order of pipeline IDs in the array define the overall Pipelines order for Datadog.

{
  "pipeline_ids": [
    "tags",
    "org_ids",
    "products"
  ]
}

Réponse

OK

Object containing the ordered list of pipeline IDs.

Expand All

Champ

Type

Description

pipeline_ids [required]

[string]

Ordered Array of <PIPELINE_ID> strings, the order of pipeline IDs in the array define the overall Pipelines order for Datadog.

{
  "pipeline_ids": [
    "tags",
    "org_ids",
    "products"
  ]
}

Bad Request

Response returned by the Logs API when errors occur.

Expand All

Champ

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Forbidden

Error response object.

Expand All

Champ

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Unprocessable Entity

Response returned by the Logs API when errors occur.

Expand All

Champ

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Exemple de code


# Curl command
curl -X PUT "https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com/api/v1/logs/config/pipeline-order" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
-d @- << EOF
{
  "pipeline_ids": [
    "tags",
    "org_ids",
    "products"
  ]
}
EOF
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "os"
    datadog "github.com/DataDog/datadog-api-client-go/api/v1/datadog"
)

func main() {
    ctx := datadog.NewDefaultContext(context.Background())

    body := *datadog.NewLogsPipelinesOrder([]string{"PipelineIds_example"}) // LogsPipelinesOrder | Object containing the new ordered list of pipeline IDs.

    configuration := datadog.NewConfiguration()

    apiClient := datadog.NewAPIClient(configuration)
    resp, r, err := apiClient.LogsPipelinesApi.UpdateLogsPipelineOrder(ctx, body)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.UpdateLogsPipelineOrder`: %v\n", err)
        fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
    }
    // response from `UpdateLogsPipelineOrder`: LogsPipelinesOrder
    responseContent, _ := json.MarshalIndent(resp, "", "  ")
    fmt.Fprintf(os.Stdout, "Response from LogsPipelinesApi.UpdateLogsPipelineOrder:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" go run "main.go"
import os
from dateutil.parser import parse as dateutil_parser
from datadog_api_client.v1 import ApiClient, ApiException, Configuration
from datadog_api_client.v1.api import logs_pipelines_api
from datadog_api_client.v1.models import *
from pprint import pprint
# See configuration.py for a list of all supported configuration parameters.
configuration = Configuration()

# Enter a context with an instance of the API client
with ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = logs_pipelines_api.LogsPipelinesApi(api_client)
    body = LogsPipelinesOrder(
        pipeline_ids=["tags","org_ids","products"],
    )  # LogsPipelinesOrder | Object containing the new ordered list of pipeline IDs.

    # example passing only required values which don't have defaults set
    try:
        # Update pipeline order
        api_response = api_instance.update_logs_pipeline_order(body)
        pprint(api_response)
    except ApiException as e:
        print("Exception when calling LogsPipelinesApi->update_logs_pipeline_order: %s\n" % e)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" python3 "example.py"
require 'datadog_api_client'
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new
body = DatadogAPIClient::V1::LogsPipelinesOrder.new({pipeline_ids: ['pipeline_ids_example']}) # LogsPipelinesOrder | Object containing the new ordered list of pipeline IDs.

begin
  # Update pipeline order
  result = api_instance.update_logs_pipeline_order(body)
  p result
rescue DatadogAPIClient::V1::APIError => e
  puts "Error when calling LogsPipelinesAPI->update_logs_pipeline_order: #{e}"
end

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" rb "example.rb"
import { v1 } from '@datadog/datadog-api-client';
import * as fs from 'fs';

const configuration = v1.createConfiguration();
const apiInstance = new v1.LogsPipelinesApi(configuration);

let params:v1.LogsPipelinesApiUpdateLogsPipelineOrderRequest = {
  // LogsPipelinesOrder | Object containing the new ordered list of pipeline IDs.
  body: {
    pipelineIds: ["tags","org_ids","products"],
  },
};

apiInstance.updateLogsPipelineOrder(params).then((data:any) => {
  console.log('API called successfully. Returned data: ' + JSON.stringify(data));
}).catch((error:any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" tsc "example.ts"
import java.util.*;
import com.datadog.api.v1.client.ApiClient;
import com.datadog.api.v1.client.ApiException;
import com.datadog.api.v1.client.Configuration;
import com.datadog.api.v1.client.model.*;
import com.datadog.api.v1.client.api.LogsPipelinesApi;

public class Example {
    public static void main(String[] args) {
        ApiClient defaultClient = Configuration.getDefaultApiClient();

        LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);
        LogsPipelinesOrder body = new LogsPipelinesOrder(); // LogsPipelinesOrder | Object containing the new ordered list of pipeline IDs.
        try {
            LogsPipelinesOrder result = apiInstance.updateLogsPipelineOrder(body);
            System.out.println(result);
        } catch (ApiException e) {
            System.err.println("Exception when calling LogsPipelinesApi#updateLogsPipelineOrder");
            System.err.println("Status code: " + e.getCode());
            System.err.println("Reason: " + e.getResponseBody());
            System.err.println("Response headers: " + e.getResponseHeaders());
            e.printStackTrace();
        }
    }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" java "Example.java"

GET https://api.datadoghq.eu/api/v1/logs/config/pipelineshttps://api.ddog-gov.com/api/v1/logs/config/pipelineshttps://api.datadoghq.com/api/v1/logs/config/pipelineshttps://api.us3.datadoghq.com/api/v1/logs/config/pipelines

Présentation

Récupérez tous les pipelines de votre organisation. Cet endpoint ne prend aucun argument JSON.

Réponse

OK

Array of pipeline ID strings.

Expand All

Champ

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name

string

Name of the pipeline.

processors

[object <oneOf>]

Ordered list of processors in this pipeline.

Option 1

object

Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section.

grok [required]

object

Set of rules for the grok parser.

match_rules [required]

string

List of match rules for the grok parser, separated by a new line.

support_rules

string

List of support rules for the grok parser, separated by a new line.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

samples

[string]

List of sample logs to test this grok parser.

source [required]

string

Name of the log attribute to parse.

type [required]

enum

Type of logs grok parser. Allowed enum values: grok-parser

Option 2

object

As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.

  • timestamp

  • date

  • _timestamp

  • Timestamp

  • eventTime

  • published_date

    If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp. The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.

Note: If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.

If multiple log date remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs date remapper. Allowed enum values: date-remapper

Option 3

object

Use this Processor if you want to assign some attributes as the official status.

Each incoming status value is mapped as follows.

  • Integers from 0 to 7 map to the Syslog severity standards
  • Strings beginning with emerg or f (case-insensitive) map to emerg (0)
  • Strings beginning with a (case-insensitive) map to alert (1)
  • Strings beginning with c (case-insensitive) map to critical (2)
  • Strings beginning with err (case-insensitive) map to error (3)
  • Strings beginning with w (case-insensitive) map to warning (4)
  • Strings beginning with n (case-insensitive) map to notice (5)
  • Strings beginning with i (case-insensitive) map to info (6)
  • Strings beginning with d, trace or verbose (case-insensitive) map to debug (7)
  • Strings beginning with o or matching OK or Success (case-insensitive) map to OK
  • All others map to info (6)

Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs status remapper. Allowed enum values: status-remapper

Option 4

object

Use this processor if you want to assign one or more attributes as the official service.

Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs service remapper. Allowed enum values: service-remapper

Option 5

object

The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.

Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs message remapper. Allowed enum values: message-remapper

Option 6

object

The remapper processor remaps any source attribute(s) or tag to another target attribute or tag. Constraints on the tag/attribute name are explained in the Tag Best Practice documentation. Some additional constraints are applied as : or , are not allowed in the target tag/attribute name.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

override_on_conflict

boolean

Override or not the target element if already set,

preserve_source

boolean

Remove or preserve the remapped source element.

source_type

string

Defines if the sources are from log attribute or tag.

sources [required]

[string]

Array of source attributes.

target [required]

string

Final attribute or tag name to remap the sources to.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

target_type

string

Defines if the final attribute or tag name is from log attribute or tag.

type [required]

enum

Type of logs attribute remapper. Allowed enum values: attribute-remapper

Option 7

object

This processor extracts query parameters and other important parameters from a URL.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

normalize_ending_slashes

boolean

Normalize the ending slashes or not.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs URL parser. Allowed enum values: url-parser

Option 8

object

The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.

is_enabled

boolean

Whether or not the processor is enabled.

is_encoded

boolean

Define if the source attribute is URL encoded or not.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs User-Agent parser. Allowed enum values: user-agent-parser

Option 9

object

Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view. For example, URL groups, machine groups, environments, and response time buckets.

Notes:

  • The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
  • Once the log has matched one of the Processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
  • The names of the categories must be unique.
  • Once defined in the Category Processor, you can map categories to log status using the Log Status Remapper.

categories [required]

[object]

Array of filters to match or not a log and their corresponding nameto assign a custom value to the log.

filter

object

Filter for logs.

query

string

The filter query.

name

string

Value to assign to the target attribute.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

target [required]

string

Name of the target attribute which value is defined by the matching category.

type [required]

enum

Type of logs category processor. Allowed enum values: category-processor

Option 10

object

Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.

The formula can use parentheses and the basic arithmetic operators -, +, *, /.

By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.

Notes:

  • The operator - needs to be space split in the formula as it can also be contained in attribute names.
  • If the target attribute already exists, it is overwritten by the result of the formula.
  • Results are rounded up to the 9th decimal. For example, if the result of the formula is 0.1234567891, the actual value stored for the attribute is 0.123456789.
  • If you need to scale a unit of measure, see Scale Filter.

expression [required]

string

Arithmetic operation between one or more log attributes.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of expression by 0, false skip the operation if an attribute is missing.

name

string

Name of the processor.

target [required]

string

Name of the attribute that contains the result of the arithmetic operation.

type [required]

enum

Type of logs arithmetic processor. Allowed enum values: arithmetic-processor

Option 11

object

Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.

The template is defined by both raw text and blocks with the syntax %{attribute_path}.

Notes:

  • The processor only accepts attributes with values or an array of values in the blocks.
  • If an attribute cannot be used (object or array of object), it is replaced by an empty string or the entire operation is skipped depending on your selection.
  • If the target attribute already exists, it is overwritten by the result of the template.
  • Results of the template cannot exceed 256 characters.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of template by an empty string. If false (default), skips the operation for missing attributes.

name

string

Name of the processor.

target [required]

string

The name of the attribute that contains the result of the template.

template [required]

string

A formula with one or more attributes and raw text.

type [required]

enum

Type of logs string builder processor. Allowed enum values: string-builder-processor

Option 12

object

Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps. For example, first use a high-level filtering such as team and then a second level of filtering based on the integration, service, or any other tag or attribute.

A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.

filter

object

Filter for logs.

query

string

The filter query.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

processors

[object]

Ordered list of processors in this pipeline.

type [required]

enum

Type of logs pipeline processor. Allowed enum values: pipeline

Option 13

object

The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of GeoIP parser. Allowed enum values: geo-ip-parser

Option 14

object

Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

default_lookup

string

Value to set the target attribute if the source value is not found in the list.

is_enabled

boolean

Whether or not the processor is enabled.

lookup_table [required]

[string]

Mapping table of values for the source attribute and their associated target attribute values, formatted as ["source_key1,target_value1", "source_key2,target_value2"]

name

string

Name of the processor.

source [required]

string

Source attribute used to perform the lookup.

target [required]

string

Name of the attribute that contains the corresponding value in the mapping list or the default_lookup if not found in the mapping list.

type [required]

enum

Type of logs lookup processor. Allowed enum values: lookup-processor

Option 15

object

There are two ways to improve correlation between application traces and logs.

  1. Follow the documentation on how to inject a trace ID in the application logs and by default log integrations take care of all the rest of the setup.

  2. Use the Trace remapper processor to define a log attribute as its associated trace ID.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources

[string]

Array of source attributes.

type [required]

enum

Type of logs trace remapper. Allowed enum values: trace-id-remapper

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "id": "string",
  "is_enabled": false,
  "is_read_only": false,
  "name": "",
  "processors": [
    {
      "grok": {
        "match_rules": "rule_name_1 foo\nrule_name_2 bar\n",
        "support_rules": "rule_name_1 foo\nrule_name_2 bar\n"
      },
      "is_enabled": false,
      "name": "string",
      "samples": [],
      "source": "message",
      "type": "grok-parser"
    }
  ],
  "type": "pipeline"
}

Forbidden

Error response object.

Expand All

Champ

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Exemple de code


# Curl command
curl -X GET "https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com/api/v1/logs/config/pipelines" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}"
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "os"
    datadog "github.com/DataDog/datadog-api-client-go/api/v1/datadog"
)

func main() {
    ctx := datadog.NewDefaultContext(context.Background())


    configuration := datadog.NewConfiguration()

    apiClient := datadog.NewAPIClient(configuration)
    resp, r, err := apiClient.LogsPipelinesApi.ListLogsPipelines(ctx)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.ListLogsPipelines`: %v\n", err)
        fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
    }
    // response from `ListLogsPipelines`: []LogsPipeline
    responseContent, _ := json.MarshalIndent(resp, "", "  ")
    fmt.Fprintf(os.Stdout, "Response from LogsPipelinesApi.ListLogsPipelines:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" go run "main.go"
import os
from dateutil.parser import parse as dateutil_parser
from datadog_api_client.v1 import ApiClient, ApiException, Configuration
from datadog_api_client.v1.api import logs_pipelines_api
from datadog_api_client.v1.models import *
from pprint import pprint
# See configuration.py for a list of all supported configuration parameters.
configuration = Configuration()

# Enter a context with an instance of the API client
with ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = logs_pipelines_api.LogsPipelinesApi(api_client)

    # example, this endpoint has no required or optional parameters
    try:
        # Get all pipelines
        api_response = api_instance.list_logs_pipelines()
        pprint(api_response)
    except ApiException as e:
        print("Exception when calling LogsPipelinesApi->list_logs_pipelines: %s\n" % e)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" python3 "example.py"
require 'datadog_api_client'
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new

begin
  # Get all pipelines
  result = api_instance.list_logs_pipelines
  p result
rescue DatadogAPIClient::V1::APIError => e
  puts "Error when calling LogsPipelinesAPI->list_logs_pipelines: #{e}"
end

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" rb "example.rb"
import { v1 } from '@datadog/datadog-api-client';
import * as fs from 'fs';

const configuration = v1.createConfiguration();
const apiInstance = new v1.LogsPipelinesApi(configuration);


apiInstance.listLogsPipelines().then((data:any) => {
  console.log('API called successfully. Returned data: ' + JSON.stringify(data));
}).catch((error:any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" tsc "example.ts"
import java.util.*;
import com.datadog.api.v1.client.ApiClient;
import com.datadog.api.v1.client.ApiException;
import com.datadog.api.v1.client.Configuration;
import com.datadog.api.v1.client.model.*;
import com.datadog.api.v1.client.api.LogsPipelinesApi;

public class Example {
    public static void main(String[] args) {
        ApiClient defaultClient = Configuration.getDefaultApiClient();

        LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);
        try {
            List<LogsPipeline> result = apiInstance.listLogsPipelines();
            System.out.println(result);
        } catch (ApiException e) {
            System.err.println("Exception when calling LogsPipelinesApi#listLogsPipelines");
            System.err.println("Status code: " + e.getCode());
            System.err.println("Reason: " + e.getResponseBody());
            System.err.println("Response headers: " + e.getResponseHeaders());
            e.printStackTrace();
        }
    }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" java "Example.java"

POST https://api.datadoghq.eu/api/v1/logs/config/pipelineshttps://api.ddog-gov.com/api/v1/logs/config/pipelineshttps://api.datadoghq.com/api/v1/logs/config/pipelineshttps://api.us3.datadoghq.com/api/v1/logs/config/pipelines

Présentation

Créez un pipeline dans votre organisation.

Requête

Body Data (required)

Définition du nouveau pipeline.

Expand All

Champ

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[object <oneOf>]

Ordered list of processors in this pipeline.

Option 1

object

Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section.

grok [required]

object

Set of rules for the grok parser.

match_rules [required]

string

List of match rules for the grok parser, separated by a new line.

support_rules

string

List of support rules for the grok parser, separated by a new line.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

samples

[string]

List of sample logs to test this grok parser.

source [required]

string

Name of the log attribute to parse.

type [required]

enum

Type of logs grok parser. Allowed enum values: grok-parser

Option 2

object

As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.

  • timestamp

  • date

  • _timestamp

  • Timestamp

  • eventTime

  • published_date

    If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp. The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.

Note: If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.

If multiple log date remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs date remapper. Allowed enum values: date-remapper

Option 3

object

Use this Processor if you want to assign some attributes as the official status.

Each incoming status value is mapped as follows.

  • Integers from 0 to 7 map to the Syslog severity standards
  • Strings beginning with emerg or f (case-insensitive) map to emerg (0)
  • Strings beginning with a (case-insensitive) map to alert (1)
  • Strings beginning with c (case-insensitive) map to critical (2)
  • Strings beginning with err (case-insensitive) map to error (3)
  • Strings beginning with w (case-insensitive) map to warning (4)
  • Strings beginning with n (case-insensitive) map to notice (5)
  • Strings beginning with i (case-insensitive) map to info (6)
  • Strings beginning with d, trace or verbose (case-insensitive) map to debug (7)
  • Strings beginning with o or matching OK or Success (case-insensitive) map to OK
  • All others map to info (6)

Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs status remapper. Allowed enum values: status-remapper

Option 4

object

Use this processor if you want to assign one or more attributes as the official service.

Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs service remapper. Allowed enum values: service-remapper

Option 5

object

The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.

Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs message remapper. Allowed enum values: message-remapper

Option 6

object

The remapper processor remaps any source attribute(s) or tag to another target attribute or tag. Constraints on the tag/attribute name are explained in the Tag Best Practice documentation. Some additional constraints are applied as : or , are not allowed in the target tag/attribute name.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

override_on_conflict

boolean

Override or not the target element if already set,

preserve_source

boolean

Remove or preserve the remapped source element.

source_type

string

Defines if the sources are from log attribute or tag.

sources [required]

[string]

Array of source attributes.

target [required]

string

Final attribute or tag name to remap the sources to.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

target_type

string

Defines if the final attribute or tag name is from log attribute or tag.

type [required]

enum

Type of logs attribute remapper. Allowed enum values: attribute-remapper

Option 7

object

This processor extracts query parameters and other important parameters from a URL.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

normalize_ending_slashes

boolean

Normalize the ending slashes or not.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs URL parser. Allowed enum values: url-parser

Option 8

object

The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.

is_enabled

boolean

Whether or not the processor is enabled.

is_encoded

boolean

Define if the source attribute is URL encoded or not.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs User-Agent parser. Allowed enum values: user-agent-parser

Option 9

object

Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view. For example, URL groups, machine groups, environments, and response time buckets.

Notes:

  • The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
  • Once the log has matched one of the Processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
  • The names of the categories must be unique.
  • Once defined in the Category Processor, you can map categories to log status using the Log Status Remapper.

categories [required]

[object]

Array of filters to match or not a log and their corresponding nameto assign a custom value to the log.

filter

object

Filter for logs.

query

string

The filter query.

name

string

Value to assign to the target attribute.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

target [required]

string

Name of the target attribute which value is defined by the matching category.

type [required]

enum

Type of logs category processor. Allowed enum values: category-processor

Option 10

object

Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.

The formula can use parentheses and the basic arithmetic operators -, +, *, /.

By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.

Notes:

  • The operator - needs to be space split in the formula as it can also be contained in attribute names.
  • If the target attribute already exists, it is overwritten by the result of the formula.
  • Results are rounded up to the 9th decimal. For example, if the result of the formula is 0.1234567891, the actual value stored for the attribute is 0.123456789.
  • If you need to scale a unit of measure, see Scale Filter.

expression [required]

string

Arithmetic operation between one or more log attributes.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of expression by 0, false skip the operation if an attribute is missing.

name

string

Name of the processor.

target [required]

string

Name of the attribute that contains the result of the arithmetic operation.

type [required]

enum

Type of logs arithmetic processor. Allowed enum values: arithmetic-processor

Option 11

object

Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.

The template is defined by both raw text and blocks with the syntax %{attribute_path}.

Notes:

  • The processor only accepts attributes with values or an array of values in the blocks.
  • If an attribute cannot be used (object or array of object), it is replaced by an empty string or the entire operation is skipped depending on your selection.
  • If the target attribute already exists, it is overwritten by the result of the template.
  • Results of the template cannot exceed 256 characters.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of template by an empty string. If false (default), skips the operation for missing attributes.

name

string

Name of the processor.

target [required]

string

The name of the attribute that contains the result of the template.

template [required]

string

A formula with one or more attributes and raw text.

type [required]

enum

Type of logs string builder processor. Allowed enum values: string-builder-processor

Option 12

object

Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps. For example, first use a high-level filtering such as team and then a second level of filtering based on the integration, service, or any other tag or attribute.

A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.

filter

object

Filter for logs.

query

string

The filter query.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

processors

[object]

Ordered list of processors in this pipeline.

type [required]

enum

Type of logs pipeline processor. Allowed enum values: pipeline

Option 13

object

The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of GeoIP parser. Allowed enum values: geo-ip-parser

Option 14

object

Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

default_lookup

string

Value to set the target attribute if the source value is not found in the list.

is_enabled

boolean

Whether or not the processor is enabled.

lookup_table [required]

[string]

Mapping table of values for the source attribute and their associated target attribute values, formatted as ["source_key1,target_value1", "source_key2,target_value2"]

name

string

Name of the processor.

source [required]

string

Source attribute used to perform the lookup.

target [required]

string

Name of the attribute that contains the corresponding value in the mapping list or the default_lookup if not found in the mapping list.

type [required]

enum

Type of logs lookup processor. Allowed enum values: lookup-processor

Option 15

object

There are two ways to improve correlation between application traces and logs.

  1. Follow the documentation on how to inject a trace ID in the application logs and by default log integrations take care of all the rest of the setup.

  2. Use the Trace remapper processor to define a log attribute as its associated trace ID.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources

[string]

Array of source attributes.

type [required]

enum

Type of logs trace remapper. Allowed enum values: trace-id-remapper

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "is_enabled": false,
  "name": "",
  "processors": [
    {
      "grok": {
        "match_rules": "rule_name_1 foo\nrule_name_2 bar\n",
        "support_rules": "rule_name_1 foo\nrule_name_2 bar\n"
      },
      "is_enabled": false,
      "name": "string",
      "samples": [],
      "source": "message",
      "type": "grok-parser"
    }
  ]
}

Réponse

OK

Pipelines and processors operate on incoming logs, parsing and transforming them into structured attributes for easier querying.

Note: These endpoints are only available for admin users. Make sure to use an application key created by an admin.

Expand All

Champ

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[object <oneOf>]

Ordered list of processors in this pipeline.

Option 1

object

Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section.

grok [required]

object

Set of rules for the grok parser.

match_rules [required]

string

List of match rules for the grok parser, separated by a new line.

support_rules

string

List of support rules for the grok parser, separated by a new line.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

samples

[string]

List of sample logs to test this grok parser.

source [required]

string

Name of the log attribute to parse.

type [required]

enum

Type of logs grok parser. Allowed enum values: grok-parser

Option 2

object

As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.

  • timestamp

  • date

  • _timestamp

  • Timestamp

  • eventTime

  • published_date

    If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp. The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.

Note: If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.

If multiple log date remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs date remapper. Allowed enum values: date-remapper

Option 3

object

Use this Processor if you want to assign some attributes as the official status.

Each incoming status value is mapped as follows.

  • Integers from 0 to 7 map to the Syslog severity standards
  • Strings beginning with emerg or f (case-insensitive) map to emerg (0)
  • Strings beginning with a (case-insensitive) map to alert (1)
  • Strings beginning with c (case-insensitive) map to critical (2)
  • Strings beginning with err (case-insensitive) map to error (3)
  • Strings beginning with w (case-insensitive) map to warning (4)
  • Strings beginning with n (case-insensitive) map to notice (5)
  • Strings beginning with i (case-insensitive) map to info (6)
  • Strings beginning with d, trace or verbose (case-insensitive) map to debug (7)
  • Strings beginning with o or matching OK or Success (case-insensitive) map to OK
  • All others map to info (6)

Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs status remapper. Allowed enum values: status-remapper

Option 4

object

Use this processor if you want to assign one or more attributes as the official service.

Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs service remapper. Allowed enum values: service-remapper

Option 5

object

The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.

Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs message remapper. Allowed enum values: message-remapper

Option 6

object

The remapper processor remaps any source attribute(s) or tag to another target attribute or tag. Constraints on the tag/attribute name are explained in the Tag Best Practice documentation. Some additional constraints are applied as : or , are not allowed in the target tag/attribute name.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

override_on_conflict

boolean

Override or not the target element if already set,

preserve_source

boolean

Remove or preserve the remapped source element.

source_type

string

Defines if the sources are from log attribute or tag.

sources [required]

[string]

Array of source attributes.

target [required]

string

Final attribute or tag name to remap the sources to.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

target_type

string

Defines if the final attribute or tag name is from log attribute or tag.

type [required]

enum

Type of logs attribute remapper. Allowed enum values: attribute-remapper

Option 7

object

This processor extracts query parameters and other important parameters from a URL.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

normalize_ending_slashes

boolean

Normalize the ending slashes or not.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs URL parser. Allowed enum values: url-parser

Option 8

object

The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.

is_enabled

boolean

Whether or not the processor is enabled.

is_encoded

boolean

Define if the source attribute is URL encoded or not.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs User-Agent parser. Allowed enum values: user-agent-parser

Option 9

object

Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view. For example, URL groups, machine groups, environments, and response time buckets.

Notes:

  • The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
  • Once the log has matched one of the Processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
  • The names of the categories must be unique.
  • Once defined in the Category Processor, you can map categories to log status using the Log Status Remapper.

categories [required]

[object]

Array of filters to match or not a log and their corresponding nameto assign a custom value to the log.

filter

object

Filter for logs.

query

string

The filter query.

name

string

Value to assign to the target attribute.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

target [required]

string

Name of the target attribute which value is defined by the matching category.

type [required]

enum

Type of logs category processor. Allowed enum values: category-processor

Option 10

object

Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.

The formula can use parentheses and the basic arithmetic operators -, +, *, /.

By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.

Notes:

  • The operator - needs to be space split in the formula as it can also be contained in attribute names.
  • If the target attribute already exists, it is overwritten by the result of the formula.
  • Results are rounded up to the 9th decimal. For example, if the result of the formula is 0.1234567891, the actual value stored for the attribute is 0.123456789.
  • If you need to scale a unit of measure, see Scale Filter.

expression [required]

string

Arithmetic operation between one or more log attributes.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of expression by 0, false skip the operation if an attribute is missing.

name

string

Name of the processor.

target [required]

string

Name of the attribute that contains the result of the arithmetic operation.

type [required]

enum

Type of logs arithmetic processor. Allowed enum values: arithmetic-processor

Option 11

object

Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.

The template is defined by both raw text and blocks with the syntax %{attribute_path}.

Notes:

  • The processor only accepts attributes with values or an array of values in the blocks.
  • If an attribute cannot be used (object or array of object), it is replaced by an empty string or the entire operation is skipped depending on your selection.
  • If the target attribute already exists, it is overwritten by the result of the template.
  • Results of the template cannot exceed 256 characters.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of template by an empty string. If false (default), skips the operation for missing attributes.

name

string

Name of the processor.

target [required]

string

The name of the attribute that contains the result of the template.

template [required]

string

A formula with one or more attributes and raw text.

type [required]

enum

Type of logs string builder processor. Allowed enum values: string-builder-processor

Option 12

object

Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps. For example, first use a high-level filtering such as team and then a second level of filtering based on the integration, service, or any other tag or attribute.

A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.

filter

object

Filter for logs.

query

string

The filter query.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

processors

[object]

Ordered list of processors in this pipeline.

type [required]

enum

Type of logs pipeline processor. Allowed enum values: pipeline

Option 13

object

The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of GeoIP parser. Allowed enum values: geo-ip-parser

Option 14

object

Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

default_lookup

string

Value to set the target attribute if the source value is not found in the list.

is_enabled

boolean

Whether or not the processor is enabled.

lookup_table [required]

[string]

Mapping table of values for the source attribute and their associated target attribute values, formatted as ["source_key1,target_value1", "source_key2,target_value2"]

name

string

Name of the processor.

source [required]

string

Source attribute used to perform the lookup.

target [required]

string

Name of the attribute that contains the corresponding value in the mapping list or the default_lookup if not found in the mapping list.

type [required]

enum

Type of logs lookup processor. Allowed enum values: lookup-processor

Option 15

object

There are two ways to improve correlation between application traces and logs.

  1. Follow the documentation on how to inject a trace ID in the application logs and by default log integrations take care of all the rest of the setup.

  2. Use the Trace remapper processor to define a log attribute as its associated trace ID.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources

[string]

Array of source attributes.

type [required]

enum

Type of logs trace remapper. Allowed enum values: trace-id-remapper

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "id": "string",
  "is_enabled": false,
  "is_read_only": false,
  "name": "",
  "processors": [
    {
      "grok": {
        "match_rules": "rule_name_1 foo\nrule_name_2 bar\n",
        "support_rules": "rule_name_1 foo\nrule_name_2 bar\n"
      },
      "is_enabled": false,
      "name": "string",
      "samples": [],
      "source": "message",
      "type": "grok-parser"
    }
  ],
  "type": "pipeline"
}

Bad Request

Response returned by the Logs API when errors occur.

Expand All

Champ

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Forbidden

Error response object.

Expand All

Champ

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Exemple de code


# Curl command
curl -X POST "https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com/api/v1/logs/config/pipelines" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
-d @- << EOF
{
  "name": "",
  "processors": [
    {
      "grok": {
        "match_rules": "rule_name_1 foo\nrule_name_2 bar\n"
      }
    }
  ]
}
EOF
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "os"
    datadog "github.com/DataDog/datadog-api-client-go/api/v1/datadog"
)

func main() {
    ctx := datadog.NewDefaultContext(context.Background())

    body := *datadog.NewLogsPipeline("Name_example") // LogsPipeline | Definition of the new pipeline.

    configuration := datadog.NewConfiguration()

    apiClient := datadog.NewAPIClient(configuration)
    resp, r, err := apiClient.LogsPipelinesApi.CreateLogsPipeline(ctx, body)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.CreateLogsPipeline`: %v\n", err)
        fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
    }
    // response from `CreateLogsPipeline`: LogsPipeline
    responseContent, _ := json.MarshalIndent(resp, "", "  ")
    fmt.Fprintf(os.Stdout, "Response from LogsPipelinesApi.CreateLogsPipeline:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" go run "main.go"
import os
from dateutil.parser import parse as dateutil_parser
from datadog_api_client.v1 import ApiClient, ApiException, Configuration
from datadog_api_client.v1.api import logs_pipelines_api
from datadog_api_client.v1.models import *
from pprint import pprint
# See configuration.py for a list of all supported configuration parameters.
configuration = Configuration()

# Enter a context with an instance of the API client
with ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = logs_pipelines_api.LogsPipelinesApi(api_client)
    body = LogsPipeline(
        filter=LogsFilter(
            query="source:python",
        ),
        id="id_example",
        is_enabled=True,
        is_read_only=True,
        name="",
        processors=[
            LogsProcessor(),
        ],
        type="pipeline",
    )  # LogsPipeline | Definition of the new pipeline.

    # example passing only required values which don't have defaults set
    try:
        # Create a pipeline
        api_response = api_instance.create_logs_pipeline(body)
        pprint(api_response)
    except ApiException as e:
        print("Exception when calling LogsPipelinesApi->create_logs_pipeline: %s\n" % e)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" python3 "example.py"
require 'datadog_api_client'
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new
body = DatadogAPIClient::V1::LogsPipeline.new({name: 'name_example'}) # LogsPipeline | Definition of the new pipeline.

begin
  # Create a pipeline
  result = api_instance.create_logs_pipeline(body)
  p result
rescue DatadogAPIClient::V1::APIError => e
  puts "Error when calling LogsPipelinesAPI->create_logs_pipeline: #{e}"
end

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" rb "example.rb"
import { v1 } from '@datadog/datadog-api-client';
import * as fs from 'fs';

const configuration = v1.createConfiguration();
const apiInstance = new v1.LogsPipelinesApi(configuration);

let params:v1.LogsPipelinesApiCreateLogsPipelineRequest = {
  // LogsPipeline | Definition of the new pipeline.
  body: {
    filter: {
      query: "source:python",
    },
    isEnabled: true,
    name: "",
    processors: [
      {
    grok: {
      matchRules: `rule_name_1 foo
rule_name_2 bar
`,
      supportRules: `rule_name_1 foo
rule_name_2 bar
`,
    },
    isEnabled: false,
    name: "name_example",
    samples: [
      "samples_example",
    ],
    source: "message",
    type: "grok-parser",
  },
    ],
  },
};

apiInstance.createLogsPipeline(params).then((data:any) => {
  console.log('API called successfully. Returned data: ' + JSON.stringify(data));
}).catch((error:any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" tsc "example.ts"
import java.util.*;
import com.datadog.api.v1.client.ApiClient;
import com.datadog.api.v1.client.ApiException;
import com.datadog.api.v1.client.Configuration;
import com.datadog.api.v1.client.model.*;
import com.datadog.api.v1.client.api.LogsPipelinesApi;

public class Example {
    public static void main(String[] args) {
        ApiClient defaultClient = Configuration.getDefaultApiClient();

        LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);
        LogsPipeline body = new LogsPipeline(); // LogsPipeline | Definition of the new pipeline.
        try {
            LogsPipeline result = apiInstance.createLogsPipeline(body);
            System.out.println(result);
        } catch (ApiException e) {
            System.err.println("Exception when calling LogsPipelinesApi#createLogsPipeline");
            System.err.println("Status code: " + e.getCode());
            System.err.println("Reason: " + e.getResponseBody());
            System.err.println("Response headers: " + e.getResponseHeaders());
            e.printStackTrace();
        }
    }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" java "Example.java"

GET https://api.datadoghq.eu/api/v1/logs/config/pipelines/{pipeline_id}https://api.ddog-gov.com/api/v1/logs/config/pipelines/{pipeline_id}https://api.datadoghq.com/api/v1/logs/config/pipelines/{pipeline_id}https://api.us3.datadoghq.com/api/v1/logs/config/pipelines/{pipeline_id}

Présentation

Récupérez un pipeline spécifique de votre organisation. Cet endpoint ne prend aucun argument JSON.

Arguments

Paramètres du chemin

Nom

Type

Description

pipeline_id [required]

string

ID of the pipeline to get.

Réponse

OK

Pipelines and processors operate on incoming logs, parsing and transforming them into structured attributes for easier querying.

Note: These endpoints are only available for admin users. Make sure to use an application key created by an admin.

Expand All

Champ

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[object <oneOf>]

Ordered list of processors in this pipeline.

Option 1

object

Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section.

grok [required]

object

Set of rules for the grok parser.

match_rules [required]

string

List of match rules for the grok parser, separated by a new line.

support_rules

string

List of support rules for the grok parser, separated by a new line.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

samples

[string]

List of sample logs to test this grok parser.

source [required]

string

Name of the log attribute to parse.

type [required]

enum

Type of logs grok parser. Allowed enum values: grok-parser

Option 2

object

As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.

  • timestamp

  • date

  • _timestamp

  • Timestamp

  • eventTime

  • published_date

    If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp. The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.

Note: If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.

If multiple log date remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs date remapper. Allowed enum values: date-remapper

Option 3

object

Use this Processor if you want to assign some attributes as the official status.

Each incoming status value is mapped as follows.

  • Integers from 0 to 7 map to the Syslog severity standards
  • Strings beginning with emerg or f (case-insensitive) map to emerg (0)
  • Strings beginning with a (case-insensitive) map to alert (1)
  • Strings beginning with c (case-insensitive) map to critical (2)
  • Strings beginning with err (case-insensitive) map to error (3)
  • Strings beginning with w (case-insensitive) map to warning (4)
  • Strings beginning with n (case-insensitive) map to notice (5)
  • Strings beginning with i (case-insensitive) map to info (6)
  • Strings beginning with d, trace or verbose (case-insensitive) map to debug (7)
  • Strings beginning with o or matching OK or Success (case-insensitive) map to OK
  • All others map to info (6)

Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs status remapper. Allowed enum values: status-remapper

Option 4

object

Use this processor if you want to assign one or more attributes as the official service.

Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs service remapper. Allowed enum values: service-remapper

Option 5

object

The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.

Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs message remapper. Allowed enum values: message-remapper

Option 6

object

The remapper processor remaps any source attribute(s) or tag to another target attribute or tag. Constraints on the tag/attribute name are explained in the Tag Best Practice documentation. Some additional constraints are applied as : or , are not allowed in the target tag/attribute name.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

override_on_conflict

boolean

Override or not the target element if already set,

preserve_source

boolean

Remove or preserve the remapped source element.

source_type

string

Defines if the sources are from log attribute or tag.

sources [required]

[string]

Array of source attributes.

target [required]

string

Final attribute or tag name to remap the sources to.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

target_type

string

Defines if the final attribute or tag name is from log attribute or tag.

type [required]

enum

Type of logs attribute remapper. Allowed enum values: attribute-remapper

Option 7

object

This processor extracts query parameters and other important parameters from a URL.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

normalize_ending_slashes

boolean

Normalize the ending slashes or not.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs URL parser. Allowed enum values: url-parser

Option 8

object

The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.

is_enabled

boolean

Whether or not the processor is enabled.

is_encoded

boolean

Define if the source attribute is URL encoded or not.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs User-Agent parser. Allowed enum values: user-agent-parser

Option 9

object

Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view. For example, URL groups, machine groups, environments, and response time buckets.

Notes:

  • The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
  • Once the log has matched one of the Processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
  • The names of the categories must be unique.
  • Once defined in the Category Processor, you can map categories to log status using the Log Status Remapper.

categories [required]

[object]

Array of filters to match or not a log and their corresponding nameto assign a custom value to the log.

filter

object

Filter for logs.

query

string

The filter query.

name

string

Value to assign to the target attribute.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

target [required]

string

Name of the target attribute which value is defined by the matching category.

type [required]

enum

Type of logs category processor. Allowed enum values: category-processor

Option 10

object

Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.

The formula can use parentheses and the basic arithmetic operators -, +, *, /.

By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.

Notes:

  • The operator - needs to be space split in the formula as it can also be contained in attribute names.
  • If the target attribute already exists, it is overwritten by the result of the formula.
  • Results are rounded up to the 9th decimal. For example, if the result of the formula is 0.1234567891, the actual value stored for the attribute is 0.123456789.
  • If you need to scale a unit of measure, see Scale Filter.

expression [required]

string

Arithmetic operation between one or more log attributes.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of expression by 0, false skip the operation if an attribute is missing.

name

string

Name of the processor.

target [required]

string

Name of the attribute that contains the result of the arithmetic operation.

type [required]

enum

Type of logs arithmetic processor. Allowed enum values: arithmetic-processor

Option 11

object

Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.

The template is defined by both raw text and blocks with the syntax %{attribute_path}.

Notes:

  • The processor only accepts attributes with values or an array of values in the blocks.
  • If an attribute cannot be used (object or array of object), it is replaced by an empty string or the entire operation is skipped depending on your selection.
  • If the target attribute already exists, it is overwritten by the result of the template.
  • Results of the template cannot exceed 256 characters.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of template by an empty string. If false (default), skips the operation for missing attributes.

name

string

Name of the processor.

target [required]

string

The name of the attribute that contains the result of the template.

template [required]

string

A formula with one or more attributes and raw text.

type [required]

enum

Type of logs string builder processor. Allowed enum values: string-builder-processor

Option 12

object

Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps. For example, first use a high-level filtering such as team and then a second level of filtering based on the integration, service, or any other tag or attribute.

A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.

filter

object

Filter for logs.

query

string

The filter query.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

processors

[object]

Ordered list of processors in this pipeline.

type [required]

enum

Type of logs pipeline processor. Allowed enum values: pipeline

Option 13

object

The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of GeoIP parser. Allowed enum values: geo-ip-parser

Option 14

object

Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

default_lookup

string

Value to set the target attribute if the source value is not found in the list.

is_enabled

boolean

Whether or not the processor is enabled.

lookup_table [required]

[string]

Mapping table of values for the source attribute and their associated target attribute values, formatted as ["source_key1,target_value1", "source_key2,target_value2"]

name

string

Name of the processor.

source [required]

string

Source attribute used to perform the lookup.

target [required]

string

Name of the attribute that contains the corresponding value in the mapping list or the default_lookup if not found in the mapping list.

type [required]

enum

Type of logs lookup processor. Allowed enum values: lookup-processor

Option 15

object

There are two ways to improve correlation between application traces and logs.

  1. Follow the documentation on how to inject a trace ID in the application logs and by default log integrations take care of all the rest of the setup.

  2. Use the Trace remapper processor to define a log attribute as its associated trace ID.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources

[string]

Array of source attributes.

type [required]

enum

Type of logs trace remapper. Allowed enum values: trace-id-remapper

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "id": "string",
  "is_enabled": false,
  "is_read_only": false,
  "name": "",
  "processors": [
    {
      "grok": {
        "match_rules": "rule_name_1 foo\nrule_name_2 bar\n",
        "support_rules": "rule_name_1 foo\nrule_name_2 bar\n"
      },
      "is_enabled": false,
      "name": "string",
      "samples": [],
      "source": "message",
      "type": "grok-parser"
    }
  ],
  "type": "pipeline"
}

Bad Request

Response returned by the Logs API when errors occur.

Expand All

Champ

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Forbidden

Error response object.

Expand All

Champ

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Exemple de code


# Path parameters
export pipeline_id="CHANGE_ME"
# Curl command curl -X GET "https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com/api/v1/logs/config/pipelines/${pipeline_id}" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}"
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "os"
    datadog "github.com/DataDog/datadog-api-client-go/api/v1/datadog"
)

func main() {
    ctx := datadog.NewDefaultContext(context.Background())

    pipelineId := "pipelineId_example" // string | ID of the pipeline to get.

    configuration := datadog.NewConfiguration()

    apiClient := datadog.NewAPIClient(configuration)
    resp, r, err := apiClient.LogsPipelinesApi.GetLogsPipeline(ctx, pipelineId)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.GetLogsPipeline`: %v\n", err)
        fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
    }
    // response from `GetLogsPipeline`: LogsPipeline
    responseContent, _ := json.MarshalIndent(resp, "", "  ")
    fmt.Fprintf(os.Stdout, "Response from LogsPipelinesApi.GetLogsPipeline:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" go run "main.go"
import os
from dateutil.parser import parse as dateutil_parser
from datadog_api_client.v1 import ApiClient, ApiException, Configuration
from datadog_api_client.v1.api import logs_pipelines_api
from datadog_api_client.v1.models import *
from pprint import pprint
# See configuration.py for a list of all supported configuration parameters.
configuration = Configuration()

# Enter a context with an instance of the API client
with ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = logs_pipelines_api.LogsPipelinesApi(api_client)
    pipeline_id = "pipeline_id_example"  # str | ID of the pipeline to get.

    # example passing only required values which don't have defaults set
    try:
        # Get a pipeline
        api_response = api_instance.get_logs_pipeline(pipeline_id)
        pprint(api_response)
    except ApiException as e:
        print("Exception when calling LogsPipelinesApi->get_logs_pipeline: %s\n" % e)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" python3 "example.py"
require 'datadog_api_client'
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new
pipeline_id = 'pipeline_id_example' # String | ID of the pipeline to get.

begin
  # Get a pipeline
  result = api_instance.get_logs_pipeline(pipeline_id)
  p result
rescue DatadogAPIClient::V1::APIError => e
  puts "Error when calling LogsPipelinesAPI->get_logs_pipeline: #{e}"
end

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" rb "example.rb"
import { v1 } from '@datadog/datadog-api-client';
import * as fs from 'fs';

const configuration = v1.createConfiguration();
const apiInstance = new v1.LogsPipelinesApi(configuration);

let params:v1.LogsPipelinesApiGetLogsPipelineRequest = {
  // string | ID of the pipeline to get.
  pipelineId: "pipeline_id_example",
};

apiInstance.getLogsPipeline(params).then((data:any) => {
  console.log('API called successfully. Returned data: ' + JSON.stringify(data));
}).catch((error:any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" tsc "example.ts"
import java.util.*;
import com.datadog.api.v1.client.ApiClient;
import com.datadog.api.v1.client.ApiException;
import com.datadog.api.v1.client.Configuration;
import com.datadog.api.v1.client.model.*;
import com.datadog.api.v1.client.api.LogsPipelinesApi;

public class Example {
    public static void main(String[] args) {
        ApiClient defaultClient = Configuration.getDefaultApiClient();

        LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);
        String pipelineId = "pipelineId_example"; // String | ID of the pipeline to get.
        try {
            LogsPipeline result = apiInstance.getLogsPipeline(pipelineId);
            System.out.println(result);
        } catch (ApiException e) {
            System.err.println("Exception when calling LogsPipelinesApi#getLogsPipeline");
            System.err.println("Status code: " + e.getCode());
            System.err.println("Reason: " + e.getResponseBody());
            System.err.println("Response headers: " + e.getResponseHeaders());
            e.printStackTrace();
        }
    }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" java "Example.java"

DELETE https://api.datadoghq.eu/api/v1/logs/config/pipelines/{pipeline_id}https://api.ddog-gov.com/api/v1/logs/config/pipelines/{pipeline_id}https://api.datadoghq.com/api/v1/logs/config/pipelines/{pipeline_id}https://api.us3.datadoghq.com/api/v1/logs/config/pipelines/{pipeline_id}

Présentation

Supprimez un pipeline donné de votre organisation. Cet endpoint ne prend aucun argument JSON.

Arguments

Paramètres du chemin

Nom

Type

Description

pipeline_id [required]

string

ID of the pipeline to delete.

Réponse

OK

Bad Request

Response returned by the Logs API when errors occur.

Expand All

Champ

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Forbidden

Error response object.

Expand All

Champ

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Exemple de code


# Path parameters
export pipeline_id="CHANGE_ME"
# Curl command curl -X DELETE "https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com/api/v1/logs/config/pipelines/${pipeline_id}" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}"
package main

import (
    "context"
    "fmt"
    "os"
    datadog "github.com/DataDog/datadog-api-client-go/api/v1/datadog"
)

func main() {
    ctx := datadog.NewDefaultContext(context.Background())

    pipelineId := "pipelineId_example" // string | ID of the pipeline to delete.

    configuration := datadog.NewConfiguration()

    apiClient := datadog.NewAPIClient(configuration)
    r, err := apiClient.LogsPipelinesApi.DeleteLogsPipeline(ctx, pipelineId)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.DeleteLogsPipeline`: %v\n", err)
        fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
    }
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" go run "main.go"
import os
from dateutil.parser import parse as dateutil_parser
from datadog_api_client.v1 import ApiClient, ApiException, Configuration
from datadog_api_client.v1.api import logs_pipelines_api
from datadog_api_client.v1.models import *
from pprint import pprint
# See configuration.py for a list of all supported configuration parameters.
configuration = Configuration()

# Enter a context with an instance of the API client
with ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = logs_pipelines_api.LogsPipelinesApi(api_client)
    pipeline_id = "pipeline_id_example"  # str | ID of the pipeline to delete.

    # example passing only required values which don't have defaults set
    try:
        # Delete a pipeline
        api_instance.delete_logs_pipeline(pipeline_id)
    except ApiException as e:
        print("Exception when calling LogsPipelinesApi->delete_logs_pipeline: %s\n" % e)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" python3 "example.py"
require 'datadog_api_client'
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new
pipeline_id = 'pipeline_id_example' # String | ID of the pipeline to delete.

begin
  # Delete a pipeline
  api_instance.delete_logs_pipeline(pipeline_id)
rescue DatadogAPIClient::V1::APIError => e
  puts "Error when calling LogsPipelinesAPI->delete_logs_pipeline: #{e}"
end

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" rb "example.rb"
import { v1 } from '@datadog/datadog-api-client';
import * as fs from 'fs';

const configuration = v1.createConfiguration();
const apiInstance = new v1.LogsPipelinesApi(configuration);

let params:v1.LogsPipelinesApiDeleteLogsPipelineRequest = {
  // string | ID of the pipeline to delete.
  pipelineId: "pipeline_id_example",
};

apiInstance.deleteLogsPipeline(params).then((data:any) => {
  console.log('API called successfully. Returned data: ' + JSON.stringify(data));
}).catch((error:any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" tsc "example.ts"
import java.util.*;
import com.datadog.api.v1.client.ApiClient;
import com.datadog.api.v1.client.ApiException;
import com.datadog.api.v1.client.Configuration;
import com.datadog.api.v1.client.model.*;
import com.datadog.api.v1.client.api.LogsPipelinesApi;

public class Example {
    public static void main(String[] args) {
        ApiClient defaultClient = Configuration.getDefaultApiClient();

        LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);
        String pipelineId = "pipelineId_example"; // String | ID of the pipeline to delete.
        try {
            apiInstance.deleteLogsPipeline(pipelineId);
        } catch (ApiException e) {
            System.err.println("Exception when calling LogsPipelinesApi#deleteLogsPipeline");
            System.err.println("Status code: " + e.getCode());
            System.err.println("Reason: " + e.getResponseBody());
            System.err.println("Response headers: " + e.getResponseHeaders());
            e.printStackTrace();
        }
    }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" java "Example.java"

PUT https://api.datadoghq.eu/api/v1/logs/config/pipelines/{pipeline_id}https://api.ddog-gov.com/api/v1/logs/config/pipelines/{pipeline_id}https://api.datadoghq.com/api/v1/logs/config/pipelines/{pipeline_id}https://api.us3.datadoghq.com/api/v1/logs/config/pipelines/{pipeline_id}

Présentation

Mettez à jour une configuration de pipeline donnée pour modifier ses processeurs ou sa séquence.

Remarque : cette méthode permet de mettre à jour la configuration de votre pipeline en remplaçant votre configuration actuelle par la nouvelle, envoyée à votre organisation Datadog.

Arguments

Paramètres du chemin

Nom

Type

Description

pipeline_id [required]

string

ID of the pipeline to delete.

Requête

Body Data (required)

Nouvelle définition du pipeline.

Expand All

Champ

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[object <oneOf>]

Ordered list of processors in this pipeline.

Option 1

object

Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section.

grok [required]

object

Set of rules for the grok parser.

match_rules [required]

string

List of match rules for the grok parser, separated by a new line.

support_rules

string

List of support rules for the grok parser, separated by a new line.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

samples

[string]

List of sample logs to test this grok parser.

source [required]

string

Name of the log attribute to parse.

type [required]

enum

Type of logs grok parser. Allowed enum values: grok-parser

Option 2

object

As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.

  • timestamp

  • date

  • _timestamp

  • Timestamp

  • eventTime

  • published_date

    If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp. The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.

Note: If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.

If multiple log date remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs date remapper. Allowed enum values: date-remapper

Option 3

object

Use this Processor if you want to assign some attributes as the official status.

Each incoming status value is mapped as follows.

  • Integers from 0 to 7 map to the Syslog severity standards
  • Strings beginning with emerg or f (case-insensitive) map to emerg (0)
  • Strings beginning with a (case-insensitive) map to alert (1)
  • Strings beginning with c (case-insensitive) map to critical (2)
  • Strings beginning with err (case-insensitive) map to error (3)
  • Strings beginning with w (case-insensitive) map to warning (4)
  • Strings beginning with n (case-insensitive) map to notice (5)
  • Strings beginning with i (case-insensitive) map to info (6)
  • Strings beginning with d, trace or verbose (case-insensitive) map to debug (7)
  • Strings beginning with o or matching OK or Success (case-insensitive) map to OK
  • All others map to info (6)

Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs status remapper. Allowed enum values: status-remapper

Option 4

object

Use this processor if you want to assign one or more attributes as the official service.

Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs service remapper. Allowed enum values: service-remapper

Option 5

object

The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.

Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs message remapper. Allowed enum values: message-remapper

Option 6

object

The remapper processor remaps any source attribute(s) or tag to another target attribute or tag. Constraints on the tag/attribute name are explained in the Tag Best Practice documentation. Some additional constraints are applied as : or , are not allowed in the target tag/attribute name.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

override_on_conflict

boolean

Override or not the target element if already set,

preserve_source

boolean

Remove or preserve the remapped source element.

source_type

string

Defines if the sources are from log attribute or tag.

sources [required]

[string]

Array of source attributes.

target [required]

string

Final attribute or tag name to remap the sources to.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

target_type

string

Defines if the final attribute or tag name is from log attribute or tag.

type [required]

enum

Type of logs attribute remapper. Allowed enum values: attribute-remapper

Option 7

object

This processor extracts query parameters and other important parameters from a URL.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

normalize_ending_slashes

boolean

Normalize the ending slashes or not.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs URL parser. Allowed enum values: url-parser

Option 8

object

The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.

is_enabled

boolean

Whether or not the processor is enabled.

is_encoded

boolean

Define if the source attribute is URL encoded or not.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs User-Agent parser. Allowed enum values: user-agent-parser

Option 9

object

Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view. For example, URL groups, machine groups, environments, and response time buckets.

Notes:

  • The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
  • Once the log has matched one of the Processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
  • The names of the categories must be unique.
  • Once defined in the Category Processor, you can map categories to log status using the Log Status Remapper.

categories [required]

[object]

Array of filters to match or not a log and their corresponding nameto assign a custom value to the log.

filter

object

Filter for logs.

query

string

The filter query.

name

string

Value to assign to the target attribute.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

target [required]

string

Name of the target attribute which value is defined by the matching category.

type [required]

enum

Type of logs category processor. Allowed enum values: category-processor

Option 10

object

Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.

The formula can use parentheses and the basic arithmetic operators -, +, *, /.

By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.

Notes:

  • The operator - needs to be space split in the formula as it can also be contained in attribute names.
  • If the target attribute already exists, it is overwritten by the result of the formula.
  • Results are rounded up to the 9th decimal. For example, if the result of the formula is 0.1234567891, the actual value stored for the attribute is 0.123456789.
  • If you need to scale a unit of measure, see Scale Filter.

expression [required]

string

Arithmetic operation between one or more log attributes.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of expression by 0, false skip the operation if an attribute is missing.

name

string

Name of the processor.

target [required]

string

Name of the attribute that contains the result of the arithmetic operation.

type [required]

enum

Type of logs arithmetic processor. Allowed enum values: arithmetic-processor

Option 11

object

Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.

The template is defined by both raw text and blocks with the syntax %{attribute_path}.

Notes:

  • The processor only accepts attributes with values or an array of values in the blocks.
  • If an attribute cannot be used (object or array of object), it is replaced by an empty string or the entire operation is skipped depending on your selection.
  • If the target attribute already exists, it is overwritten by the result of the template.
  • Results of the template cannot exceed 256 characters.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of template by an empty string. If false (default), skips the operation for missing attributes.

name

string

Name of the processor.

target [required]

string

The name of the attribute that contains the result of the template.

template [required]

string

A formula with one or more attributes and raw text.

type [required]

enum

Type of logs string builder processor. Allowed enum values: string-builder-processor

Option 12

object

Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps. For example, first use a high-level filtering such as team and then a second level of filtering based on the integration, service, or any other tag or attribute.

A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.

filter

object

Filter for logs.

query

string

The filter query.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

processors

[object]

Ordered list of processors in this pipeline.

type [required]

enum

Type of logs pipeline processor. Allowed enum values: pipeline

Option 13

object

The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of GeoIP parser. Allowed enum values: geo-ip-parser

Option 14

object

Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

default_lookup

string

Value to set the target attribute if the source value is not found in the list.

is_enabled

boolean

Whether or not the processor is enabled.

lookup_table [required]

[string]

Mapping table of values for the source attribute and their associated target attribute values, formatted as ["source_key1,target_value1", "source_key2,target_value2"]

name

string

Name of the processor.

source [required]

string

Source attribute used to perform the lookup.

target [required]

string

Name of the attribute that contains the corresponding value in the mapping list or the default_lookup if not found in the mapping list.

type [required]

enum

Type of logs lookup processor. Allowed enum values: lookup-processor

Option 15

object

There are two ways to improve correlation between application traces and logs.

  1. Follow the documentation on how to inject a trace ID in the application logs and by default log integrations take care of all the rest of the setup.

  2. Use the Trace remapper processor to define a log attribute as its associated trace ID.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources

[string]

Array of source attributes.

type [required]

enum

Type of logs trace remapper. Allowed enum values: trace-id-remapper

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "is_enabled": false,
  "name": "",
  "processors": [
    {
      "grok": {
        "match_rules": "rule_name_1 foo\nrule_name_2 bar\n",
        "support_rules": "rule_name_1 foo\nrule_name_2 bar\n"
      },
      "is_enabled": false,
      "name": "string",
      "samples": [],
      "source": "message",
      "type": "grok-parser"
    }
  ]
}

Réponse

OK

Pipelines and processors operate on incoming logs, parsing and transforming them into structured attributes for easier querying.

Note: These endpoints are only available for admin users. Make sure to use an application key created by an admin.

Expand All

Champ

Type

Description

filter

object

Filter for logs.

query

string

The filter query.

id

string

ID of the pipeline.

is_enabled

boolean

Whether or not the pipeline is enabled.

is_read_only

boolean

Whether or not the pipeline can be edited.

name [required]

string

Name of the pipeline.

processors

[object <oneOf>]

Ordered list of processors in this pipeline.

Option 1

object

Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the parsing section.

grok [required]

object

Set of rules for the grok parser.

match_rules [required]

string

List of match rules for the grok parser, separated by a new line.

support_rules

string

List of support rules for the grok parser, separated by a new line.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

samples

[string]

List of sample logs to test this grok parser.

source [required]

string

Name of the log attribute to parse.

type [required]

enum

Type of logs grok parser. Allowed enum values: grok-parser

Option 2

object

As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.

  • timestamp

  • date

  • _timestamp

  • Timestamp

  • eventTime

  • published_date

    If your logs put their dates in an attribute not in this list, use the log date Remapper Processor to define their date attribute as the official log timestamp. The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.

Note: If your logs don’t contain any of the default attributes and you haven’t defined your own date attribute, Datadog timestamps the logs with the date it received them.

If multiple log date remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs date remapper. Allowed enum values: date-remapper

Option 3

object

Use this Processor if you want to assign some attributes as the official status.

Each incoming status value is mapped as follows.

  • Integers from 0 to 7 map to the Syslog severity standards
  • Strings beginning with emerg or f (case-insensitive) map to emerg (0)
  • Strings beginning with a (case-insensitive) map to alert (1)
  • Strings beginning with c (case-insensitive) map to critical (2)
  • Strings beginning with err (case-insensitive) map to error (3)
  • Strings beginning with w (case-insensitive) map to warning (4)
  • Strings beginning with n (case-insensitive) map to notice (5)
  • Strings beginning with i (case-insensitive) map to info (6)
  • Strings beginning with d, trace or verbose (case-insensitive) map to debug (7)
  • Strings beginning with o or matching OK or Success (case-insensitive) map to OK
  • All others map to info (6)

Note: If multiple log status remapper processors can be applied to a given log, only the first one (according to the pipelines order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs status remapper. Allowed enum values: status-remapper

Option 4

object

Use this processor if you want to assign one or more attributes as the official service.

Note: If multiple service remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs service remapper. Allowed enum values: service-remapper

Option 5

object

The message is a key attribute in Datadog. It is displayed in the message column of the Log Explorer and you can do full string search on it. Use this Processor to define one or more attributes as the official log message.

Note: If multiple log message remapper processors can be applied to a given log, only the first one (according to the pipeline order) is taken into account.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

type [required]

enum

Type of logs message remapper. Allowed enum values: message-remapper

Option 6

object

The remapper processor remaps any source attribute(s) or tag to another target attribute or tag. Constraints on the tag/attribute name are explained in the Tag Best Practice documentation. Some additional constraints are applied as : or , are not allowed in the target tag/attribute name.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

override_on_conflict

boolean

Override or not the target element if already set,

preserve_source

boolean

Remove or preserve the remapped source element.

source_type

string

Defines if the sources are from log attribute or tag.

sources [required]

[string]

Array of source attributes.

target [required]

string

Final attribute or tag name to remap the sources to.

target_format

enum

If the target_type of the remapper is attribute, try to cast the value to a new specific type. If the cast is not possible, the original type is kept. string, integer, or double are the possible types. If the target_type is tag, this parameter may not be specified. Allowed enum values: auto,string,integer,double

target_type

string

Defines if the final attribute or tag name is from log attribute or tag.

type [required]

enum

Type of logs attribute remapper. Allowed enum values: attribute-remapper

Option 7

object

This processor extracts query parameters and other important parameters from a URL.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

normalize_ending_slashes

boolean

Normalize the ending slashes or not.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs URL parser. Allowed enum values: url-parser

Option 8

object

The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data. It recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.

is_enabled

boolean

Whether or not the processor is enabled.

is_encoded

boolean

Define if the source attribute is URL encoded or not.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of logs User-Agent parser. Allowed enum values: user-agent-parser

Option 9

object

Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log matching a provided search query. Use categories to create groups for an analytical view. For example, URL groups, machine groups, environments, and response time buckets.

Notes:

  • The syntax of the query is the one of Logs Explorer search bar. The query can be done on any log attribute or tag, whether it is a facet or not. Wildcards can also be used inside your query.
  • Once the log has matched one of the Processor queries, it stops. Make sure they are properly ordered in case a log could match several queries.
  • The names of the categories must be unique.
  • Once defined in the Category Processor, you can map categories to log status using the Log Status Remapper.

categories [required]

[object]

Array of filters to match or not a log and their corresponding nameto assign a custom value to the log.

filter

object

Filter for logs.

query

string

The filter query.

name

string

Value to assign to the target attribute.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

target [required]

string

Name of the target attribute which value is defined by the matching category.

type [required]

enum

Type of logs category processor. Allowed enum values: category-processor

Option 10

object

Use the Arithmetic Processor to add a new attribute (without spaces or special characters in the new attribute name) to a log with the result of the provided formula. This enables you to remap different time attributes with different units into a single attribute, or to compute operations on attributes within the same log.

The formula can use parentheses and the basic arithmetic operators -, +, *, /.

By default, the calculation is skipped if an attribute is missing. Select “Replace missing attribute by 0” to automatically populate missing attribute values with 0 to ensure that the calculation is done. An attribute is missing if it is not found in the log attributes, or if it cannot be converted to a number.

Notes:

  • The operator - needs to be space split in the formula as it can also be contained in attribute names.
  • If the target attribute already exists, it is overwritten by the result of the formula.
  • Results are rounded up to the 9th decimal. For example, if the result of the formula is 0.1234567891, the actual value stored for the attribute is 0.123456789.
  • If you need to scale a unit of measure, see Scale Filter.

expression [required]

string

Arithmetic operation between one or more log attributes.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of expression by 0, false skip the operation if an attribute is missing.

name

string

Name of the processor.

target [required]

string

Name of the attribute that contains the result of the arithmetic operation.

type [required]

enum

Type of logs arithmetic processor. Allowed enum values: arithmetic-processor

Option 11

object

Use the string builder processor to add a new attribute (without spaces or special characters) to a log with the result of the provided template. This enables aggregation of different attributes or raw strings into a single attribute.

The template is defined by both raw text and blocks with the syntax %{attribute_path}.

Notes:

  • The processor only accepts attributes with values or an array of values in the blocks.
  • If an attribute cannot be used (object or array of object), it is replaced by an empty string or the entire operation is skipped depending on your selection.
  • If the target attribute already exists, it is overwritten by the result of the template.
  • Results of the template cannot exceed 256 characters.

is_enabled

boolean

Whether or not the processor is enabled.

is_replace_missing

boolean

If true, it replaces all missing attributes of template by an empty string. If false (default), skips the operation for missing attributes.

name

string

Name of the processor.

target [required]

string

The name of the attribute that contains the result of the template.

template [required]

string

A formula with one or more attributes and raw text.

type [required]

enum

Type of logs string builder processor. Allowed enum values: string-builder-processor

Option 12

object

Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps. For example, first use a high-level filtering such as team and then a second level of filtering based on the integration, service, or any other tag or attribute.

A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.

filter

object

Filter for logs.

query

string

The filter query.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

processors

[object]

Ordered list of processors in this pipeline.

type [required]

enum

Type of logs pipeline processor. Allowed enum values: pipeline

Option 13

object

The GeoIP parser takes an IP address attribute and extracts if available the Continent, Country, Subdivision, and City information in the target attribute path.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources [required]

[string]

Array of source attributes.

target [required]

string

Name of the parent attribute that contains all the extracted details from the sources.

type [required]

enum

Type of GeoIP parser. Allowed enum values: geo-ip-parser

Option 14

object

Use the Lookup Processor to define a mapping between a log attribute and a human readable value saved in the processors mapping table. For example, you can use the Lookup Processor to map an internal service ID into a human readable service name. Alternatively, you could also use it to check if the MAC address that just attempted to connect to the production environment belongs to your list of stolen machines.

default_lookup

string

Value to set the target attribute if the source value is not found in the list.

is_enabled

boolean

Whether or not the processor is enabled.

lookup_table [required]

[string]

Mapping table of values for the source attribute and their associated target attribute values, formatted as ["source_key1,target_value1", "source_key2,target_value2"]

name

string

Name of the processor.

source [required]

string

Source attribute used to perform the lookup.

target [required]

string

Name of the attribute that contains the corresponding value in the mapping list or the default_lookup if not found in the mapping list.

type [required]

enum

Type of logs lookup processor. Allowed enum values: lookup-processor

Option 15

object

There are two ways to improve correlation between application traces and logs.

  1. Follow the documentation on how to inject a trace ID in the application logs and by default log integrations take care of all the rest of the setup.

  2. Use the Trace remapper processor to define a log attribute as its associated trace ID.

is_enabled

boolean

Whether or not the processor is enabled.

name

string

Name of the processor.

sources

[string]

Array of source attributes.

type [required]

enum

Type of logs trace remapper. Allowed enum values: trace-id-remapper

type

string

Type of pipeline.

{
  "filter": {
    "query": "source:python"
  },
  "id": "string",
  "is_enabled": false,
  "is_read_only": false,
  "name": "",
  "processors": [
    {
      "grok": {
        "match_rules": "rule_name_1 foo\nrule_name_2 bar\n",
        "support_rules": "rule_name_1 foo\nrule_name_2 bar\n"
      },
      "is_enabled": false,
      "name": "string",
      "samples": [],
      "source": "message",
      "type": "grok-parser"
    }
  ],
  "type": "pipeline"
}

Bad Request

Response returned by the Logs API when errors occur.

Expand All

Champ

Type

Description

error

object

Error returned by the Logs API

code

string

Code identifying the error

details

[object]

Additional error details

message

string

Error message

{
  "error": {
    "code": "string",
    "details": [],
    "message": "string"
  }
}

Forbidden

Error response object.

Expand All

Champ

Type

Description

errors [required]

[string]

Array of errors returned by the API.

{
  "errors": [
    "Bad Request"
  ]
}

Exemple de code


# Path parameters
export pipeline_id="CHANGE_ME"
# Curl command curl -X PUT "https://api.datadoghq.eu"https://api.ddog-gov.com"https://api.datadoghq.com"https://api.us3.datadoghq.com/api/v1/logs/config/pipelines/${pipeline_id}" \ -H "Content-Type: application/json" \ -H "DD-API-KEY: ${DD_API_KEY}" \ -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \ -d @- << EOF { "name": "", "processors": [ { "grok": { "match_rules": "rule_name_1 foo\nrule_name_2 bar\n" } } ] } EOF
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "os"
    datadog "github.com/DataDog/datadog-api-client-go/api/v1/datadog"
)

func main() {
    ctx := datadog.NewDefaultContext(context.Background())

    pipelineId := "pipelineId_example" // string | ID of the pipeline to delete.
    body := *datadog.NewLogsPipeline("Name_example") // LogsPipeline | New definition of the pipeline.

    configuration := datadog.NewConfiguration()

    apiClient := datadog.NewAPIClient(configuration)
    resp, r, err := apiClient.LogsPipelinesApi.UpdateLogsPipeline(ctx, pipelineId, body)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error when calling `LogsPipelinesApi.UpdateLogsPipeline`: %v\n", err)
        fmt.Fprintf(os.Stderr, "Full HTTP response: %v\n", r)
    }
    // response from `UpdateLogsPipeline`: LogsPipeline
    responseContent, _ := json.MarshalIndent(resp, "", "  ")
    fmt.Fprintf(os.Stdout, "Response from LogsPipelinesApi.UpdateLogsPipeline:\n%s\n", responseContent)
}

Instructions

First install the library and its dependencies and then save the example to main.go and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" go run "main.go"
import os
from dateutil.parser import parse as dateutil_parser
from datadog_api_client.v1 import ApiClient, ApiException, Configuration
from datadog_api_client.v1.api import logs_pipelines_api
from datadog_api_client.v1.models import *
from pprint import pprint
# See configuration.py for a list of all supported configuration parameters.
configuration = Configuration()

# Enter a context with an instance of the API client
with ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = logs_pipelines_api.LogsPipelinesApi(api_client)
    pipeline_id = "pipeline_id_example"  # str | ID of the pipeline to delete.
    body = LogsPipeline(
        filter=LogsFilter(
            query="source:python",
        ),
        id="id_example",
        is_enabled=True,
        is_read_only=True,
        name="",
        processors=[
            LogsProcessor(),
        ],
        type="pipeline",
    )  # LogsPipeline | New definition of the pipeline.

    # example passing only required values which don't have defaults set
    try:
        # Update a pipeline
        api_response = api_instance.update_logs_pipeline(pipeline_id, body)
        pprint(api_response)
    except ApiException as e:
        print("Exception when calling LogsPipelinesApi->update_logs_pipeline: %s\n" % e)

Instructions

First install the library and its dependencies and then save the example to example.py and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" python3 "example.py"
require 'datadog_api_client'
api_instance = DatadogAPIClient::V1::LogsPipelinesAPI.new
pipeline_id = 'pipeline_id_example' # String | ID of the pipeline to delete.
body = DatadogAPIClient::V1::LogsPipeline.new({name: 'name_example'}) # LogsPipeline | New definition of the pipeline.

begin
  # Update a pipeline
  result = api_instance.update_logs_pipeline(pipeline_id, body)
  p result
rescue DatadogAPIClient::V1::APIError => e
  puts "Error when calling LogsPipelinesAPI->update_logs_pipeline: #{e}"
end

Instructions

First install the library and its dependencies and then save the example to example.rb and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" rb "example.rb"
import { v1 } from '@datadog/datadog-api-client';
import * as fs from 'fs';

const configuration = v1.createConfiguration();
const apiInstance = new v1.LogsPipelinesApi(configuration);

let params:v1.LogsPipelinesApiUpdateLogsPipelineRequest = {
  // string | ID of the pipeline to delete.
  pipelineId: "pipeline_id_example",
  // LogsPipeline | New definition of the pipeline.
  body: {
    filter: {
      query: "source:python",
    },
    isEnabled: true,
    name: "",
    processors: [
      {
    grok: {
      matchRules: `rule_name_1 foo
rule_name_2 bar
`,
      supportRules: `rule_name_1 foo
rule_name_2 bar
`,
    },
    isEnabled: false,
    name: "name_example",
    samples: [
      "samples_example",
    ],
    source: "message",
    type: "grok-parser",
  },
    ],
  },
};

apiInstance.updateLogsPipeline(params).then((data:any) => {
  console.log('API called successfully. Returned data: ' + JSON.stringify(data));
}).catch((error:any) => console.error(error));

Instructions

First install the library and its dependencies and then save the example to example.ts and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" tsc "example.ts"
import java.util.*;
import com.datadog.api.v1.client.ApiClient;
import com.datadog.api.v1.client.ApiException;
import com.datadog.api.v1.client.Configuration;
import com.datadog.api.v1.client.model.*;
import com.datadog.api.v1.client.api.LogsPipelinesApi;

public class Example {
    public static void main(String[] args) {
        ApiClient defaultClient = Configuration.getDefaultApiClient();

        LogsPipelinesApi apiInstance = new LogsPipelinesApi(defaultClient);
        String pipelineId = "pipelineId_example"; // String | ID of the pipeline to delete.
        LogsPipeline body = new LogsPipeline(); // LogsPipeline | New definition of the pipeline.
        try {
            LogsPipeline result = apiInstance.updateLogsPipeline(pipelineId, body);
            System.out.println(result);
        } catch (ApiException e) {
            System.err.println("Exception when calling LogsPipelinesApi#updateLogsPipeline");
            System.err.println("Status code: " + e.getCode());
            System.err.println("Reason: " + e.getResponseBody());
            System.err.println("Response headers: " + e.getResponseHeaders());
            e.printStackTrace();
        }
    }
}

Instructions

First install the library and its dependencies and then save the example to Example.java and run following commands:

              
export DD_SITE="datadoghq.comus3.datadoghq.comdatadoghq.euddog-gov.com" DD_API_KEY="<API-KEY>" DD_APP_KEY="<APP-KEY>" java "Example.java"