--- title: Customize which logs Cloud SIEM analyzes description: >- Customize which ingested logs Cloud SIEM analyzes by configuring Security Filters or log index filters. Covers the Cloud SIEM product that applies to your organization, UI and API configuration, supported and unsupported filter fields, and common patterns for AWS, Azure, Google Cloud, identity providers, and more. breadcrumbs: >- Docs > Datadog Security > Cloud SIEM > Cloud SIEM Guides > Customize which logs Cloud SIEM analyzes --- # Customize which logs Cloud SIEM analyzes ## Overview{% #overview %} Cloud SIEM analyzes ingested logs to detect threats using [detection rules](https://docs.datadoghq.com/security/default_rules.md#cat-cloud-siem). Most organizations rely on [Content Packs](https://docs.datadoghq.com/security/cloud_siem/ingest_and_enrich/content_packs.md) to manage which logs Cloud SIEM analyzes; each Content Pack includes prebuilt filters for the log source it covers. If you need more granular control (for example, to exclude an environment or to restrict analysis to a custom log set), configure log filtering directly. ## Filtering tools and configuration methods{% #filtering-tools-and-configuration-methods %} The filtering tools and configuration methods available depend on your Cloud SIEM product: | Cloud SIEM product | Filtering tool | Configuration methods | | --------------------- | ---------------------------------------- | --------------------- | | Standalone | Log index filters (not Security Filters) | Log Configuration UI | | Add-on with Flex Logs | Security Filters | UI and API | | Legacy | Security Filters | API | If you're not sure which Cloud SIEM product your organization uses, see [Determine the Cloud SIEM product your organization is using](https://docs.datadoghq.com/security/cloud_siem/guide/determine-cloud-siem-product.md). {% alert level="info" %} Security Filters and log index filters control only the logs analyzed by Cloud SIEM. Logs generated by the Datadog Agent for Workload Protection (`source:runtime-security-agent`) and Cloud Security Misconfigurations (`source:compliance-agent`) are not billed as analyzed logs regardless of filter configuration. {% /alert %} ### Cloud SIEM Standalone: Filter the Cloud SIEM index{% #cloud-siem-standalone-filter-the-cloud-siem-index %} Cloud SIEM Standalone does not use Security Filters. Standalone organizations control which logs Cloud SIEM analyzes by configuring an index filter on the Cloud SIEM log index. An index filter is a query that specifies which logs enter the index. Logs that do not match the filter are not indexed, and therefore are not analyzed by Cloud SIEM. The query syntax is the same as the standard Datadog log search syntax. For full configuration instructions, see [Indexes filters](https://docs.datadoghq.com/logs/log_configuration/indexes.md#indexes-filters) in the Log Configuration documentation. **This guide does not apply to Cloud SIEM Standalone.** ## How Security Filters work{% #how-security-filters-work %} Security Filters specify which ingested logs Cloud SIEM analyzes. Cloud SIEM evaluates every ingested log against all enabled Security Filters. Security Filters are **inclusive**: Datadog analyzes a log if the log matches at least one enabled Security Filter and is not removed by an exclusion on that filter. A Security Filter has three parts: - A **query** that selects the logs the filter matches. - Zero or more **exclusion filters**, each with its own query. A log matched by the main query is removed from analysis if it also matches any exclusion query. - A **status** (active or inactive). Inactive filters are ignored by Cloud SIEM but remain saved for later use. Cloud SIEM tags every ingested log with `datadog.cloud_siem:true` or `datadog.cloud_siem:false`, based on whether Security Filters selected the log for analysis. You can use this tag in Logs Explorer to verify the effect of your Security Filters on any specific log. ### Limitations on Security Filter queries{% #limitations-on-security-filter-queries %} Datadog evaluates Security Filters before Cloud SIEM applies geoIP, threat intel, and OCSF observable enrichment. The fields that enrichment populates are not present on the log when Security Filters evaluate it, so Security Filter queries cannot reliably reference them. The constraint applies in two distinct ways: - **Fields not present at filter-evaluation time.** These fields are populated only by Cloud SIEM's out-of-the-box enrichment, which runs after Security Filters. A Security Filter query that references one of these fields matches no logs. - `network.ip.list` - `network.ip.attributes` - `network.client.is_private_network_ip` - `network.destination.is_private_network_ip` - `ocsf.observables` - `ocsf.type_uid` To filter on these categories of data, rewrite the query using attributes present at ingestion time, such as `source`, `service`, `host`, tags, and parsed log attributes. - **Fields whose availability depends on Pipeline configuration.** These fields can be populated either by a Logs Pipeline processor (before Security Filters), or by Cloud SIEM's out-of-the-box enrichment (after Security Filters). Whether a Security Filter query against one of these fields works depends on whether the corresponding Pipeline processor is configured. - `network.client.geoip` (and sub-attributes such as `network.client.geoip.country.iso_code`) - `threat_intel.results` - `threat_intel.indicators_matched` To make these fields available to Security Filter queries, configure the corresponding GeoIP or Threat Intel Enrichment processor in your [Logs Pipeline](https://docs.datadoghq.com/logs/log_configuration/pipelines.md). With the processor in place, the fields get populated before Security Filters run. Cloud SIEM's geoIP enrichment uses the same GeoIPFilter and IPinfo database as a Pipeline GeoIP processor, so the geoIP data is identical regardless of which path populates it. The threat intel workaround is different: a Pipeline Threat Intel Enrichment processor draws on your own threat intel sources (see [Bring your own threat intelligence](https://docs.datadoghq.com/security/cloud_siem/ingest_and_enrich/threat_intelligence.md) for setup guidance), which produces a different set of matches than Cloud SIEM's out-of-the-box, Datadog-managed threat intel. If you rely on Cloud SIEM's out-of-the-box threat intel, the matching cannot be moved upstream of Security Filters. The pre-enrichment evaluation applies to Security Filters only. Detection rules, suppression rules, and signal queries run against the post-enrichment log and can continue to reference these fields. ## Configure Security Filters in the UI (Add-on with Flex Logs){% #configure-security-filters-in-the-ui-add-on-with-flex-logs %} **Applies to:** Cloud SIEM Add-on with Flex Logs. Cloud SIEM Legacy organizations must configure Security Filters through the API. {% alert level="info" %} For most use cases, the easiest way to manage which logs Cloud SIEM analyzes is by enabling or disabling [Content Packs](https://docs.datadoghq.com/security/cloud_siem/ingest_and_enrich/content_packs.md). Each Content Pack manages its own Security Filter, so toggling a Content Pack adds or removes the corresponding filter automatically. The Security Filters UI described below is provided for fine tuning and custom log control beyond what Content Packs cover. {% /alert %} The Security Filters Configuration section lists every Security Filter, its query, its exclusions, and whether it is currently active. A Preview pane at the bottom of the section streams the logs that reach Cloud SIEM analysis after every enabled filter is applied. Use the Preview pane to validate that your filter configuration produces the expected result. {% image source="https://docs.dd-static.net/images/security/cloud_siem/guide/siem_settings_page_security_filters.b1c03efd4530384d60f2d8abace79200.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/security/cloud_siem/guide/siem_settings_page_security_filters.b1c03efd4530384d60f2d8abace79200.png?auto=format&fit=max&w=850&dpr=2 2x" alt="The Security Filters Configuration section of the Cloud SIEM settings page, showing a list of configured filters with columns for Name, Filters, and Active, followed by a Preview pane streaming matching logs" /%} ### Create a Security Filter{% #create-a-security-filter %} 1. In Datadog, go to **Security** > **Cloud SIEM** > [**Settings**](https://app.datadoghq.com/security/configuration/siem/setup). 1. Scroll to the **Security Filters Configuration** section and click it to expand it. 1. In the **Security Filters Configuration** section, click **+ New Security Filter**. The New Security Filter window opens. 1. In the **New Security Filter** window, enter: - **Name:** A short, descriptive name (for example, `AWS CloudTrail`). - **Query:** The log query that selects the logs Cloud SIEM should analyze (for example, `source:cloudtrail`). - **Exclusion filters** (optional): One or more exclusion queries that subtract logs from the filter's match. To add one, click **+ New Exclusion Filter** and enter an exclusion query. - **Filter status:** Set to **Active** to enable the filter on save. 1. Review the **Preview** pane to confirm the filter matches the logs you expect. 1. Click **Save**. {% image source="https://docs.dd-static.net/images/security/cloud_siem/guide/siem_settings_add_security-filter_empty.7b79d45fbf6d8d670c4040a571a44374.png?auto=format&fit=max&w=850 1x, https://docs.dd-static.net/images/security/cloud_siem/guide/siem_settings_add_security-filter_empty.7b79d45fbf6d8d670c4040a571a44374.png?auto=format&fit=max&w=850&dpr=2 2x" alt="The New Security Filter dialog with empty Name and Query fields, no exclusion filters, Filter status set to Active, and a Preview pane showing live log ingestion statistics" /%} ### Edit or disable a Security Filter{% #edit-or-disable-a-security-filter %} 1. In the **Security Filters Configuration** section, click the **Edit** icon on the filter row. 1. In the **Edit Security Filter** dialog, update the name, query, exclusions, or filter status. 1. Click **Save**. To disable a filter without deleting it, set **Filter status** to inactive. The filter remains saved so you can re-enable it later. ### Delete a Security Filter{% #delete-a-security-filter %} In the **Security Filters Configuration** section, click the **Delete** icon on the filter row, then confirm the deletion. The built-in `all ingested logs` filter cannot be deleted, but you can disable it. ## Configure Security Filters with the API{% #configure-security-filters-with-the-api %} **Applies to:** Cloud SIEM Add-on with Flex Logs and Cloud SIEM Legacy. When working with the API, you must understand which Cloud SIEM version you are using. The versions affect the logic of the security filters, and Cloud SIEM only analyzes logs based on the logic of the filters present. - For **Add-on with Flex Logs**, there are no security filters present when you first onboard. Each content pack you enable adds a security filter so Cloud SIEM can analyze those logs. - **Legacy** operates differently. It comes with a default filter that allows all logs to be analyzed. Datadog recommends adding exclusion filters to the default filter to reduce your log volumes. ### Prerequisites for the Cloud SIEM API{% #prerequisites-for-the-cloud-siem-api %} - An API key and an application key from an admin user. Both are available on your [Datadog account API key page](https://docs.datadoghq.com/account_management/api-app-keys.md). Replace `` and `` in the examples in this section with your API key and application key. - `curl` installed on the machine running the examples. For examples in other languages, see the [Security Monitoring API documentation](https://docs.datadoghq.com/api/latest/security-monitoring.md#get-all-security-filters). {% alert level="info" %} Several of the Security Filter API payloads include a `version` attribute. `version` is optional: if you omit it, the API updates the latest version of the filter. Include `version` when you want the request to succeed only against a specific known version (for example, to avoid overwriting concurrent changes). {% /alert %} ### Create a custom Security Filter{% #create-a-custom-security-filter %} Custom Security Filters tell Cloud SIEM which logs it should analyze. This example call creates a filter that matches `source:cloudtrail`, so Cloud SIEM analyzes logs from AWS CloudTrail. ```bash curl -L -X POST 'https://api./api/v2/security_monitoring/configuration/security_filters' \ --header 'Content-Type: application/json' \ --header 'DD-API-KEY: ' \ --header 'DD-APPLICATION-KEY: ' \ --data-raw '{ "data": { "type": "security_filters", "attributes": { "is_enabled": true, "name": "cloudtrail", "exclusion_filters": [], "filtered_data_type": "logs", "query": "source:cloudtrail" } } }' ``` Example response: ```json { "data": { "attributes": { "is_enabled": true, "is_builtin": false, "name": "cloudtrail", "filtered_data_type": "logs", "exclusion_filters": [], "version": 1, "query": "source:cloudtrail" }, "type": "security_filters", "id": "qa6-tzm-rp7" } } ``` ### Add an exclusion to an existing Security Filter{% #add-an-exclusion-to-an-existing-security-filter %} You can refine existing Security Filters using exclusion filters. These are useful to remove logs that would the main filter query would otherwise process. This example excludes logs tagged `env:staging`. 1. GET existing Security Filters to find the filter's `id`: ```bash curl -L -X GET 'https://api./api/v2/security_monitoring/configuration/security_filters' \ --header 'Content-Type: application/json' \ --header 'DD-API-KEY: ' \ --header 'DD-APPLICATION-KEY: ' ``` Example response: ```json { "data": [ { "attributes": { "is_enabled": true, "is_builtin": true, "name": "all ingested logs", "filtered_data_type": "logs", "exclusion_filters": [], "version": 1, "query": "*" }, "type": "security_filters", "id": "l6l-rmx-mqx" } ] } ``` In the example response, the filter `id` is `l6l-rmx-mqx`. Every organization has its own unique filter IDs, so always retrieve the current `id` before sending a PATCH. 1. PATCH the filter to add the exclusion. Replace `` with the `id` returned by the list request: ```bash curl -L -X PATCH 'https://api./api/v2/security_monitoring/configuration/security_filters/' \ --header 'Content-Type: application/json' \ --header 'DD-API-KEY: ' \ --header 'DD-APPLICATION-KEY: ' \ --data-raw '{ "data": { "attributes": { "exclusion_filters": [ { "query": "env:staging", "name": "exclude env:staging" } ], "version": 1 }, "type": "security_filters" } }' ``` Example response: ```json { "data": { "attributes": { "is_enabled": true, "is_builtin": true, "name": "all ingested logs", "filtered_data_type": "logs", "exclusion_filters": [ { "query": "env:staging", "name": "exclude env:staging" } ], "version": 2, "query": "*" }, "type": "security_filters", "id": "" } } ``` ### Disable a Security Filter{% #disable-a-security-filter %} You can enable and disable Security Filters to control whether their associated logs are analyzed. You can disable a filter but keep it in your list by setting its `is_enabled` attribute to `false`: ```bash curl -L -X PATCH 'https://api./api/v2/security_monitoring/configuration/security_filters/' \ --header 'Content-Type: application/json' \ --header 'DD-API-KEY: ' \ --header 'DD-APPLICATION-KEY: ' \ --data-raw '{ "data": { "attributes": { "is_enabled": false }, "type": "security_filters" } }' ``` Example response: ```json { "data": { "attributes": { "is_enabled": false, "is_builtin": true, "name": "all ingested logs", "filtered_data_type": "logs", "exclusion_filters": [ { "query": "env:staging", "name": "exclude env:staging" } ], "version": 3, "query": "*" }, "type": "security_filters", "id": "l6l-rmx-mqx" } } ``` ### Delete a Security Filter{% #delete-a-security-filter-1 %} To delete a custom Security Filter, send a DELETE request with the filter `id`: ```bash curl -L -X DELETE 'https://api./api/v2/security_monitoring/configuration/security_filters/' \ --header 'Content-Type: application/json' \ --header 'DD-API-KEY: ' \ --header 'DD-APPLICATION-KEY: ' ``` A successful request returns `HTTP 204 No Content`. ## Troubleshoot Security Filters{% #troubleshoot-security-filters %} If a log you expect Cloud SIEM to analyze is not producing signals, work through these checks in order: 1. **Verify that Security Filters selected the log for analysis.** Open the log in Logs Explorer and check the value of `datadog.cloud_siem`. - If the tag is `false`, the log was excluded by Security Filters. Review your filter set and confirm that at least one enabled filter matches the log, and that no exclusion removes it. - If the tag is `true`, the log was selected for analysis and the issue is downstream of Security Filters (for example, no detection rule matched). 1. **Check whether your filter query references an unsupported field.** See Limitations on Security Filter queries. - A query against a field not present at filter-evaluation time returns zero matches. - A query against a field whose availability depends on Pipeline configuration returns matches only when the corresponding Pipeline processor is configured. 1. **Validate the combined effect of all filters.** - In the UI, expand the **Preview** pane in the Security Filters Configuration section to stream the logs that reach analysis after every enabled filter is applied. - With the API, send a `GET` request to list all filters and review the combined `query` and `exclusion_filters` payloads. ## Useful tags and attributes for Security Filter queries{% #useful-tags-and-attributes-for-security-filter-queries %} If you restrict analysis to specific categories of logs, take care not to exclude logs that contain valuable security-relevant users, entities, or sources. The tables in this section list common queries you can use or adapt. ### Key users and entities for Security Filter queries{% #key-users-and-entities-for-security-filter-queries %} | Name | Query | | ------------------- | ------------------------------------------ | | All named events | `@evt.name:*` | | All client IPs | `@network.client.ip:*` | | All destination IPs | `@network.destination.ip:*` | | All users | `@usr.id:* OR @usr.name:* OR @usr.email:*` | | All hosts | `host:* OR instance-id:*` | ### Key security sources for Security Filter queries{% #key-security-sources-for-security-filter-queries %} | Name | Query | | ---------------------- | --------------------------------------------- | | AWS Security Logs | `source:(cloudtrail OR guardduty OR route53)` | | AWS Network Logs | `source:(vpc OR waf OR elb OR alb)` | | Google Cloud Logs | `source:gcp*` | | Azure Logs | `source:azure*` | | Kubernetes Audit Logs | `source:kubernetes.audit` | | Identity Provider Logs | `source:(okta OR gsuite OR auth0)` | | CDN Logs | `source:(cloudfront OR cloudflare OR fastly)` | | Web Server Logs | `source:(nginx* OR apache OR iis)` | ## Further reading{% #further-reading %} - [Detection Rules](https://docs.datadoghq.com/security/detection_rules.md) - [Content Packs](https://docs.datadoghq.com/security/cloud_siem/ingest_and_enrich/content_packs.md) - [Determine the Cloud SIEM product your organization is using](https://docs.datadoghq.com/security/cloud_siem/guide/determine-cloud-siem-product.md)