Overview

Cloud SIEM analyzes ingested logs to detect threats using detection rules. Most organizations rely on Content Packs to manage which logs Cloud SIEM analyzes; each Content Pack includes prebuilt filters for the log source it covers. If you need more granular control (for example, to exclude an environment or to restrict analysis to a custom log set), configure log filtering directly.

Filtering tools and configuration methods

The filtering tools and configuration methods available depend on your Cloud SIEM product:

If you’re not sure which Cloud SIEM product your organization uses, see Determine the Cloud SIEM product your organization is using.

Cloud SIEM Standalone: Filter the Cloud SIEM index

Cloud SIEM Standalone does not use Security Filters. Standalone organizations control which logs Cloud SIEM analyzes by configuring an index filter on the Cloud SIEM log index.

An index filter is a query that specifies which logs enter the index. Logs that do not match the filter are not indexed, and therefore are not analyzed by Cloud SIEM. The query syntax is the same as the standard Datadog log search syntax.

For full configuration instructions, see Indexes filters in the Log Configuration documentation.

This guide does not apply to Cloud SIEM Standalone.

How Security Filters work

Security Filters specify which ingested logs Cloud SIEM analyzes. Cloud SIEM evaluates every ingested log against all enabled Security Filters.

Security Filters are inclusive: Datadog analyzes a log if the log matches at least one enabled Security Filter and is not removed by an exclusion on that filter. A Security Filter has three parts:

• A query that selects the logs the filter matches.
• Zero or more exclusion filters, each with its own query. A log matched by the main query is removed from analysis if it also matches any exclusion query.
• A status (active or inactive). Inactive filters are ignored by Cloud SIEM but remain saved for later use.

Cloud SIEM tags every ingested log with datadog.cloud_siem:true or datadog.cloud_siem:false, based on whether Security Filters selected the log for analysis. You can use this tag in Logs Explorer to verify the effect of your Security Filters on any specific log.

Limitations on Security Filter queries

Datadog evaluates Security Filters before Cloud SIEM applies geoIP, threat intel, and OCSF observable enrichment. The fields that enrichment populates are not present on the log when Security Filters evaluate it, so Security Filter queries cannot reliably reference them. The constraint applies in two distinct ways:

• Fields not present at filter-evaluation time. These fields are populated only by Cloud SIEM’s out-of-the-box enrichment, which runs after Security Filters. A Security Filter query that references one of these fields matches no logs.

  • network.ip.list
  • network.ip.attributes
  • network.client.is_private_network_ip
  • network.destination.is_private_network_ip
  • ocsf.observables
  • ocsf.type_uid

  To filter on these categories of data, rewrite the query using attributes present at ingestion time, such as source, service, host, tags, and parsed log attributes.

• Fields whose availability depends on Pipeline configuration. These fields can be populated either by a Logs Pipeline processor (before Security Filters), or by Cloud SIEM’s out-of-the-box enrichment (after Security Filters). Whether a Security Filter query against one of these fields works depends on whether the corresponding Pipeline processor is configured.

  • network.client.geoip (and sub-attributes such as network.client.geoip.country.iso_code)
  • threat_intel.results
  • threat_intel.indicators_matched

  To make these fields available to Security Filter queries, configure the corresponding GeoIP or Threat Intel Enrichment processor in your Logs Pipeline. With the processor in place, the fields get populated before Security Filters run.

Cloud SIEM’s geoIP enrichment uses the same GeoIPFilter and IPinfo database as a Pipeline GeoIP processor, so the geoIP data is identical regardless of which path populates it. The threat intel workaround is different: a Pipeline Threat Intel Enrichment processor draws on your own threat intel sources (see Bring your own threat intelligence for setup guidance), which produces a different set of matches than Cloud SIEM’s out-of-the-box, Datadog-managed threat intel. If you rely on Cloud SIEM’s out-of-the-box threat intel, the matching cannot be moved upstream of Security Filters.

The pre-enrichment evaluation applies to Security Filters only. Detection rules, suppression rules, and signal queries run against the post-enrichment log and can continue to reference these fields.

Configure Security Filters in the UI (Add-on with Flex Logs)

Applies to: Cloud SIEM Add-on with Flex Logs. Cloud SIEM Legacy organizations must configure Security Filters through the API.

The Security Filters Configuration section lists every Security Filter, its query, its exclusions, and whether it is currently active. A Preview pane at the bottom of the section streams the logs that reach Cloud SIEM analysis after every enabled filter is applied. Use the Preview pane to validate that your filter configuration produces the expected result.

Create a Security Filter

1. In Datadog, go to Security > Cloud SIEM > Settings.
2. Scroll to the Security Filters Configuration section and click it to expand it.
3. In the Security Filters Configuration section, click + New Security Filter. The New Security Filter window opens.
4. In the New Security Filter window, enter:
   • Name: A short, descriptive name (for example, AWS CloudTrail).
   • Query: The log query that selects the logs Cloud SIEM should analyze (for example, source:cloudtrail).
   • Exclusion filters (optional): One or more exclusion queries that subtract logs from the filter’s match. To add one, click + New Exclusion Filter and enter an exclusion query.
   • Filter status: Set to Active to enable the filter on save.
5. Review the Preview pane to confirm the filter matches the logs you expect.
6. Click Save.

Edit or disable a Security Filter

1. In the Security Filters Configuration section, click the Edit icon on the filter row.
2. In the Edit Security Filter dialog, update the name, query, exclusions, or filter status.
3. Click Save.

To disable a filter without deleting it, set Filter status to inactive. The filter remains saved so you can re-enable it later.

Delete a Security Filter

In the Security Filters Configuration section, click the Delete icon on the filter row, then confirm the deletion.

The built-in all ingested logs filter cannot be deleted, but you can disable it.

Configure Security Filters with the API

Applies to: Cloud SIEM Add-on with Flex Logs and Cloud SIEM Legacy.

When working with the API, you must understand which Cloud SIEM version you are using. The versions affect the logic of the security filters, and Cloud SIEM only analyzes logs based on the logic of the filters present.

• For Add-on with Flex Logs, there are no security filters present when you first onboard. Each content pack you enable adds a security filter so Cloud SIEM can analyze those logs.
• Legacy operates differently. It comes with a default filter that allows all logs to be analyzed. Datadog recommends adding exclusion filters to the default filter to reduce your log volumes.

Prerequisites for the Cloud SIEM API

• An API key and an application key from an admin user. Both are available on your Datadog account API key page. Replace <DATADOG_API_KEY> and <DATADOG_APP_KEY> in the examples in this section with your API key and application key.
• curl installed on the machine running the examples. For examples in other languages, see the Security Monitoring API documentation.

Create a custom Security Filter

Custom Security Filters tell Cloud SIEM which logs it should analyze. This example call creates a filter that matches source:cloudtrail, so Cloud SIEM analyzes logs from AWS CloudTrail.

curl -L -X POST 'https://api./api/v2/security_monitoring/configuration/security_filters' \
--header 'Content-Type: application/json' \
--header 'DD-API-KEY: <DATADOG_API_KEY>' \
--header 'DD-APPLICATION-KEY: <DATADOG_APP_KEY>' \
--data-raw '{
    "data": {
        "type": "security_filters",
        "attributes": {
            "is_enabled": true,
            "name": "cloudtrail",
            "exclusion_filters": [],
            "filtered_data_type": "logs",
            "query": "source:cloudtrail"
        }
    }
}'

Example response:

{
    "data": {
        "attributes": {
            "is_enabled": true,
            "is_builtin": false,
            "name": "cloudtrail",
            "filtered_data_type": "logs",
            "exclusion_filters": [],
            "version": 1,
            "query": "source:cloudtrail"
        },
        "type": "security_filters",
        "id": "qa6-tzm-rp7"
    }
}

Add an exclusion to an existing Security Filter

You can refine existing Security Filters using exclusion filters. These are useful to remove logs that would the main filter query would otherwise process. This example excludes logs tagged env:staging.

1. GET existing Security Filters to find the filter’s id:

   curl -L -X GET 'https://api./api/v2/security_monitoring/configuration/security_filters' \
   --header 'Content-Type: application/json' \
   --header 'DD-API-KEY: <DATADOG_API_KEY>' \
   --header 'DD-APPLICATION-KEY: <DATADOG_APP_KEY>'
   

   Example response:

   {
       "data": [
           {
               "attributes": {
                   "is_enabled": true,
                   "is_builtin": true,
                   "name": "all ingested logs",
                   "filtered_data_type": "logs",
                   "exclusion_filters": [],
                   "version": 1,
                   "query": "*"
               },
               "type": "security_filters",
               "id": "l6l-rmx-mqx"
           }
       ]
   }
   

   In the example response, the filter id is l6l-rmx-mqx. Every organization has its own unique filter IDs, so always retrieve the current id before sending a PATCH.

2. PATCH the filter to add the exclusion. Replace <FILTER_ID> with the id returned by the list request:

   curl -L -X PATCH 'https://api./api/v2/security_monitoring/configuration/security_filters/<FILTER_ID>' \
   --header 'Content-Type: application/json' \
   --header 'DD-API-KEY: <DATADOG_API_KEY>' \
   --header 'DD-APPLICATION-KEY: <DATADOG_APP_KEY>' \
   --data-raw '{
       "data": {
           "attributes": {
                "exclusion_filters": [
                   {
                       "query": "env:staging",
                       "name": "exclude env:staging"
                   }
               ],
               "version": 1
           },
           "type": "security_filters"
       }
   }'
   

   Example response:

   {
       "data": {
           "attributes": {
               "is_enabled": true,
               "is_builtin": true,
               "name": "all ingested logs",
               "filtered_data_type": "logs",
               "exclusion_filters": [
                   {
                       "query": "env:staging",
                       "name": "exclude env:staging"
                   }
               ],
               "version": 2,
               "query": "*"
           },
           "type": "security_filters",
           "id": "<FILTER_ID>"
       }
   }
   

Disable a Security Filter

You can enable and disable Security Filters to control whether their associated logs are analyzed. You can disable a filter but keep it in your list by setting its is_enabled attribute to false:

curl -L -X PATCH 'https://api./api/v2/security_monitoring/configuration/security_filters/<FILTER_ID>' \
--header 'Content-Type: application/json' \
--header 'DD-API-KEY: <DATADOG_API_KEY>' \
--header 'DD-APPLICATION-KEY: <DATADOG_APP_KEY>' \
--data-raw '{
    "data": {
        "attributes": {
            "is_enabled": false
        },
        "type": "security_filters"
    }
}'

Example response:

{
    "data": {
        "attributes": {
            "is_enabled": false,
            "is_builtin": true,
            "name": "all ingested logs",
            "filtered_data_type": "logs",
            "exclusion_filters": [
                {
                    "query": "env:staging",
                    "name": "exclude env:staging"
                }
            ],
            "version": 3,
            "query": "*"
        },
        "type": "security_filters",
        "id": "l6l-rmx-mqx"
    }
}

Delete a Security Filter

To delete a custom Security Filter, send a DELETE request with the filter id:

curl -L -X DELETE 'https://api./api/v2/security_monitoring/configuration/security_filters/<FILTER_ID>' \
--header 'Content-Type: application/json' \
--header 'DD-API-KEY: <DATADOG_API_KEY>' \
--header 'DD-APPLICATION-KEY: <DATADOG_APP_KEY>'

A successful request returns HTTP 204 No Content.

Troubleshoot Security Filters

If a log you expect Cloud SIEM to analyze is not producing signals, work through these checks in order:

1. Verify that Security Filters selected the log for analysis. Open the log in Logs Explorer and check the value of datadog.cloud_siem.
   • If the tag is false, the log was excluded by Security Filters. Review your filter set and confirm that at least one enabled filter matches the log, and that no exclusion removes it.
   • If the tag is true, the log was selected for analysis and the issue is downstream of Security Filters (for example, no detection rule matched).
2. Check whether your filter query references an unsupported field. See Limitations on Security Filter queries.
   • A query against a field not present at filter-evaluation time returns zero matches.
   • A query against a field whose availability depends on Pipeline configuration returns matches only when the corresponding Pipeline processor is configured.
3. Validate the combined effect of all filters.
   • In the UI, expand the Preview pane in the Security Filters Configuration section to stream the logs that reach analysis after every enabled filter is applied.
   • With the API, send a GET request to list all filters and review the combined query and exclusion_filters payloads.

Useful tags and attributes for Security Filter queries

If you restrict analysis to specific categories of logs, take care not to exclude logs that contain valuable security-relevant users, entities, or sources. The tables in this section list common queries you can use or adapt.

Key users and entities for Security Filter queries

Key security sources for Security Filter queries

Further reading

Additional helpful documentation, links, and articles:

Detection RulesDOCUMENTATION Content PacksDOCUMENTATION Determine the Cloud SIEM product your organization is usingDOCUMENTATION