Create Alerts for Sensitive Data

Overview

After you configure Sensitive Data Scanner to detect sensitive information in your telemetry data, you can create alerts when matches are detected. Use these alerts to:

  • Notify security and compliance teams about potential data exposure
  • Track trends in sensitive data volume over time
  • Identify services or teams that need additional security guidance

For log findings, Datadog recommends security notification rules. For APM and RUM, create monitors using the tags that Sensitive Data Scanner adds when it detects matches.

Prerequisites

Before creating alerts for sensitive data, verify you have:

Create alerts by data type

Sensitive Data Scanner works across multiple Datadog products. The alerting flow differs depending on which data type you’re monitoring.

Logs

Although you can create log monitors for sensitive data, notification rules are recommended because they alert directly on findings and include finding context.

To create a notification rule for log findings:

  1. On the Log Findings explorer, filter findings to the scope you want to alert on. For example, filter by severity, service, env, team, category, or scanning rule.
  2. Click Create Notification Rule from the explorer or from a finding side panel.
  3. Review or adjust the prefilled filters. The Preview of Matching Findings panel shows which findings match the rule.
  4. Choose whether to aggregate matching findings over a time frame or trigger immediately for each individual finding.
  5. Under Destination, add notification recipients, such as teams, users, cases, Jira, PagerDuty, Slack, Microsoft Teams, webhooks, or other integrations.
  6. Click Save.

Sensitive Data Scanner finding notifications include finding context, such as the rule, severity, service, environment, team, and a link to the finding. Notifications do not include log samples.

Notify on high-severity findings in production

To alert on high-severity findings in production, filter the Log Findings explorer by severity High, Critical, or both, and env:prod. Then click Create Notification Rule.

APM

To create an APM monitor for sensitive data in traces:

  1. Navigate to Monitors > New Monitor > APM.
  2. Select Trace Analytics as the monitor type.
  3. In the Define the search query section, add the sensitive_data:* tag to scope your monitor to traces containing sensitive data.
    • Use sensitive_data_category:* tags to scope to specific categories of sensitive data.
  4. Configure the alert conditions:
    • Threshold: Set appropriate thresholds based on your expected trace volume.
    • Group by: Group by service, resource, or env to identify which services or endpoints are transmitting sensitive data.
  5. Configure notifications and add a monitor message with remediation guidance.

Monitor for API keys in APM spans

Use this query to monitor for credentials detected in APM spans: sensitive_data_category:credentials env:prod.

If you want to scope to specific credential types, such as API keys or AWS access keys, use this query: (sensitive_data:aws_access_key OR sensitive_data:api_key) service:checkout-service

RUM

To create a RUM monitor for sensitive data in events:

  1. Navigate to Monitors > New Monitor > RUM.
  2. In the Define the search query section, add the sensitive_data:* tag to scope your monitor to RUM events containing sensitive data.
  3. Configure the alert conditions:
    • Threshold: Set appropriate thresholds for sensitive data matches in user sessions.
    • Group by: Group by application.name, view.name, or geo.country to identify where sensitive data is being captured.
  4. Configure notifications and add a monitor message.

Monitor for email addresses in RUM events

Use this query to monitor for email addresses detected in RUM events: sensitive_data:email_address application.name:checkout-app

Best practices

Scope alerts to high-value findings

To avoid alert fatigue from false positives, start with findings that represent the highest risk for your organization. For example:

  • Financial services: Alert on payment card findings.
  • PII: Alert on personally identifiable information findings.
  • SaaS applications: Alert on credentials, API keys, and tokens.

For log finding alerts, use notification rule filters such as severity, service, env, team, category, or scanning rule. For APM and RUM monitors, use domain-specific tags such as sensitive_data_category:payment_card, sensitive_data_category:pii, or sensitive_data_category:credentials.

Combine with service and team context

Combine sensitive data conditions with ownership or runtime context. For example, scope alerts to production services or teams responsible for customer-facing applications.

This approach helps you:

  • Reduce noise by focusing on high-risk services
  • Route alerts to the appropriate teams
  • Identify patterns in which services are leaking or transmitting sensitive data

Set appropriate thresholds for monitors

For APM and RUM monitors, start with conservative thresholds and adjust based on your baseline:

  1. Monitor sensitive data matches for 1-2 weeks without alerts to establish a baseline.
  2. Set thresholds above your normal baseline to catch anomalies.
  3. For critical data types (for example, payment cards and SSNs), consider setting thresholds lower or alerting on any match.

Further Reading

Additional helpful documentation, links, and articles:

Set up Sensitive Data ScannerDOCUMENTATION more more Learn more about security notification rulesDOCUMENTATION more more Learn more about out-of-the-box library rulesDOCUMENTATION more more Learn to set up APM monitorsDOCUMENTATION more more Learn to set up RUM monitorsDOCUMENTATION more more