Información general

Los trabajos históricos son consultas ejecutables de una sola vez en logs históricos que se utilizan para realizar tests retrospectivos de las reglas de detección y evaluar su eficacia en datos pasados. Los resultados generados en el trabajo son versiones ligeras de señales que proporcionan información sobre posibles amenazas y anomalías en los logs históricos. Después de revisar los resultados, puedes convertir los resultados que necesitan acción inmediata en señales.

Crear una regla

  1. Para crear una regla de detección de umbral o trabajo, navega hasta la página Create a New Detection (Crear una nueva detección).
  2. Selecciona Historical Job (Trabajo historico).

Definir tu trabajo histórico

  1. Selecciona el índice de logs y el intervalo de tiempo para el trabajo.
  2. Selecciona el método de detección que deseas utilizar para crear señales.

Definir consultas de búsqueda

Definir la consulta de búsqueda
  1. Para buscar eventos de Audit Trail o eventos de Events Management, haz clic en la flecha hacia abajo junto a Logs y selecciona Audit Trail o Events (Eventos).

  2. Crea una consulta de búsqueda para tus logs o eventos utilizando la sintaxis de búsqueda del Log Explorer.

  3. (Optional) In the Count dropdown menu, select attributes whose unique values are counted over the specified time frame.

  4. (Optional) In the group by dropdown menu, select attributes you want to group by.

    • The defined group by generates a signal for each group by value.
    • Typically, the group by is an entity (like user, or IP). The group by is also used to join the queries together.
    • Joining logs that span a time frame can increase the confidence or severity of the security signal. For example, to detect a successful brute force attack, both successful and unsuccessful authentication logs must be correlated for a user.
  5. (Optional) To create calculated fields that transform your logs during query time:

    1. Click Add and select Calculated fields.
    2. In Name your field, enter a descriptive name that indicates the purpose of the calculated field.
      • For example, if you want to combine users’ first and last name into one field, you might name the calculated field fullName.
    3. In the Define your formula field, enter a formula or expression, which determines the result to be computed and stored as the value of the calculated field for each log event.
  6. (Optional) To filter logs using reference tables:

    1. Click the Add button next to the query editor and select Join with Reference Table.
    2. In the Inner join with reference table dropdown menu, select your reference table.
    3. In the where field dropdown menu, select the log field to join on.
    4. Select the IN or NOT IN operator to filter in or filter out matching logs.
    5. In the column dropdown menu, select the column of the reference table to join on.
  7. (Optional) To test your rules against sample logs, click Unit Test.

    1. To construct a sample log, you can:
      1. Navigate to Log Explorer in a new window.
      2. In the search bar, enter the query you are using for the detection rule.
    2. Select one of the logs.
    3. Click the export button at the top right side of the log side panel, and then select Copy.
    4. Navigate back to the Unit Test modal, and then paste the log into the text box. Edit the sample as needed for your use case.
    5. Toggle the switch for Query is expected to match based on the example event to fit your use case.
    6. Click Run Query Test.
Definir la consulta de búsqueda
  1. Para buscar eventos de Audit Trail o eventos de Events Management, haz clic en la flecha hacia abajo junto a Logs y selecciona Audit Trail o Events (Eventos).

  2. Crea una consulta de búsqueda para tus logs o eventos utilizando la sintaxis de búsqueda del Log Explorer.

  3. In the Detect new value dropdown menu, select the attributes you want to detect.

    • For example, if you create a query for successful user authentication with the following settings:
      • Detect new value is country
      • group by is user
      • Learning duration is after 7 days
        Then, logs coming in over the next 7 days are evaluated with those configured values. If a log comes in with a new value after the learning duration (7 days), a signal is generated, and the new value is learned to prevent future signals with this value.
    • You can also identify users and entities using multiple Detect new value attributes in a single query.
      • For example, if you want to detect when a user signs in from a new device and from a country that they’ve never signed in from before, add device_id and country_name to the Detect new value field.
  4. (Optional) Define a signal grouping in the group by dropdown menu.

    • The defined group by generates a signal for each group by value.
    • Typically, the group by is an entity (like user or IP address).
  5. In the dropdown menu to the right of group by, select the learning duration.

  6. (Optional) Define a signal grouping in the group by dropdown menu.

    • The defined group by generates a signal for each group by value.
    • Typically, the group by is an entity (like user or IP address).
  7. In the dropdown menu to the right of group by, select the learning duration.

  8. (Optional) To create calculated fields that transform your logs during query time:

    1. Click Add and select Calculated fields.
    2. In Name your field, enter a descriptive name that indicates the purpose of the calculated field.
      • For example, if you want to combine users’ first and last name into one field, you might name the calculated field fullName.
    3. In the Define your formula field, enter a formula or expression, which determines the result to be computed and stored as the value of the calculated field for each log event.
  9. (Optional) To filter logs using reference tables:

    1. Click the Add button next to the query editor and select Join with Reference Table.
    2. In the Inner join with reference table dropdown menu, select your reference table.
    3. In the where field dropdown menu, select the log field to join on.
    4. Select the IN or NOT IN operator to filter in or filter out matching logs.
    5. In the column dropdown menu, select the column of the reference table to join on.
  10. (Optional) To test your rules against sample logs, click Unit Test.

    1. To construct a sample log, you can:
      1. Navigate to Log Explorer in a new window.
      2. In the search bar, enter the query you are using for the detection rule.
    2. Select one of the logs.
    3. Click the export button at the top right side of the log side panel, and then select Copy.
    4. Navigate back to the Unit Test modal, and then paste the log into the text box. Edit the sample as needed for your use case.
    5. Toggle the switch for Query is expected to match based on the example event to fit your use case.
    6. Click Run Query Test.
Definir la consulta de búsqueda
  1. Para buscar eventos de Audit Trail o eventos de Events Management, haz clic en la flecha hacia abajo junto a Logs y selecciona Audit Trail o Events (Eventos).

  2. Crea una consulta de búsqueda para tus logs o eventos utilizando la sintaxis de búsqueda del Log Explorer.

  3. (Optional) In the Count dropdown menu, select attributes whose unique values you want to count during the specified time frame.

  4. (Optional) In the group by dropdown menu, select attributes you want to group by.

    • The defined group by generates a signal for each group by value.
    • Typically, the group by is an entity (like user or IP). The group by can also join the queries together.
    • Joining logs that span a time frame can increase the confidence or severity of the security signal. For example, if you want to detect a successful brute force attack, both successful and unsuccessful authentication logs must be correlated for a user.
    • Anomaly detection inspects how the group by attribute has behaved in the past. If a group by attribute is seen for the first time (for example, the first time an IP is communicating with your system) and is anomalous, it does not generate a security signal because the anomaly detection algorithm has no historical data to compare with.
  5. (Optional) To create calculated fields that transform your logs during query time:

    1. Click Add and select Calculated fields.
    2. In Name your field, enter a descriptive name that indicates the purpose of the calculated field.
      • For example, if you want to combine users’ first and last name into one field, you might name the calculated field fullName.
    3. In the Define your formula field, enter a formula or expression, which determines the result to be computed and stored as the value of the calculated field for each log event.
  6. (Optional) To filter logs using reference tables:

    1. Click the Add button next to the query editor and select Join with Reference Table.
    2. In the Inner join with reference table dropdown menu, select your reference table.
    3. In the where field dropdown menu, select the log field to join on.
    4. Select the IN or NOT IN operator to filter in or filter out matching logs.
    5. In the column dropdown menu, select the column of the reference table to join on.
  7. (Optional) To test your rules against sample logs, click Unit Test.

    1. To construct a sample log, you can:
      1. Navigate to Log Explorer in a new window.
      2. In the search bar, enter the query you are using for the detection rule.
    2. Select one of the logs.
    3. Click the export button at the top right side of the log side panel, and then select Copy.
    4. Navigate back to the Unit Test modal, and then paste the log into the text box. Edit the sample as needed for your use case.
    5. Toggle the switch for Query is expected to match based on the example event to fit your use case.
    6. Click Run Query Test.
Definir la consulta de búsqueda
  1. Para buscar eventos de Audit Trail o eventos de Events Management, haz clic en la flecha hacia abajo junto a Logs y selecciona Audit Trail o Events (Eventos).

  2. Crea una consulta de búsqueda para tus logs o eventos utilizando la sintaxis de búsqueda del Log Explorer.

  3. In the Detect anomaly field, specify the fields whose values you want to analyze.

  4. In the group by field, specify the fields you want to group by.

    • The defined group by generates a signal for each group by value.
    • Typically, the group by is an entity (like user or IP). The group by can also join the queries together.
    • Joining logs that span a time frame can increase the confidence or severity of the security signal. For example, to detect a successful brute force attack, both successful and unsuccessful authentication logs must be correlated for a user.
  5. In the Learn for dropdown menu, select the number of days for the learning period. During the learning period, the rule sets a baseline of normal field values and does not generate any signals.

    • Note: If the detection rule is modified, the learning period restarts at day 0.
  6. (Optional) To create calculated fields that transform your logs during query time:

    1. Click Add and select Calculated fields.
    2. In Name your field, enter a descriptive name that indicates the purpose of the calculated field.
      • For example, if you want to combine users’ first and last name into one field, you might name the calculated field fullName.
    3. In the Define your formula field, enter a formula or expression, which determines the result to be computed and stored as the value of the calculated field for each log event.
  7. (Optional) To filter logs using reference tables:

    1. Click the Add button next to the query editor and select Join with Reference Table.
    2. In the Inner join with reference table dropdown menu, select your reference table.
    3. In the where field dropdown menu, select the log field to join on.
    4. Select the IN or NOT IN operator to filter in or filter out matching logs.
    5. In the column dropdown menu, select the column of the reference table to join on.
  8. (Optional) To test your rules against sample logs, click Unit Test.

    1. To construct a sample log, you can:
      1. Navigate to Log Explorer in a new window.
      2. In the search bar, enter the query you are using for the detection rule.
    2. Select one of the logs.
    3. Click the export button at the top right side of the log side panel, and then select Copy.
    4. Navigate back to the Unit Test modal, and then paste the log into the text box. Edit the sample as needed for your use case.
    5. Toggle the switch for Query is expected to match based on the example event to fit your use case.
    6. Click Run Query Test.
Definir la consulta de búsqueda
  1. Para buscar eventos de Audit Trail o eventos de Events Management, haz clic en la flecha hacia abajo junto a Logs y selecciona Audit Trail o Events (Eventos).
  2. Crea una consulta de búsqueda para tus logs o eventos utilizando la sintaxis de búsqueda del Log Explorer.
  3. In the User attribute dropdown menu, select the log attribute that contains the user ID. This can be an identifier like an email address, user name, or account identifier.
  4. The Location attribute value is automatically set to @network.client.geoip.
    • The location attribute specifies which field holds the geographic information for a log.
    • The only supported value is @network.client.geoip, which is enriched by the GeoIP parser to give a log location information based on the client’s IP address.
  5. Click the Baseline user locations checkbox if you want Datadog to learn regular access locations before triggering a signal.
    • When selected, signals are suppressed for the first 24 hours. During that time, Datadog learns the user’s regular access locations. This can be helpful to reduce noise and infer VPN usage or credentialed API access.
    • See How the impossible detection method works for more information.
  1. (Optional) To create calculated fields that transform your logs during query time:

    1. Click Add and select Calculated fields.
    2. In Name your field, enter a descriptive name that indicates the purpose of the calculated field.
      • For example, if you want to combine users’ first and last name into one field, you might name the calculated field fullName.
    3. In the Define your formula field, enter a formula or expression, which determines the result to be computed and stored as the value of the calculated field for each log event.
  2. (Optional) To filter logs using reference tables:

    1. Click the Add button next to the query editor and select Join with Reference Table.
    2. In the Inner join with reference table dropdown menu, select your reference table.
    3. In the where field dropdown menu, select the log field to join on.
    4. Select the IN or NOT IN operator to filter in or filter out matching logs.
    5. In the column dropdown menu, select the column of the reference table to join on.
  3. (Optional) To test your rules against sample logs, click Unit Test.

    1. To construct a sample log, you can:
      1. Navigate to Log Explorer in a new window.
      2. In the search bar, enter the query you are using for the detection rule.
    2. Select one of the logs.
    3. Click the export button at the top right side of the log side panel, and then select Copy.
    4. Navigate back to the Unit Test modal, and then paste the log into the text box. Edit the sample as needed for your use case.
    5. Toggle the switch for Query is expected to match based on the example event to fit your use case.
    6. Click Run Query Test.

Nota: Todos los logs y eventos que coincidan con esta consulta se analizan para un posible recorrido imposible.

Definir la consulta de búsqueda
  1. Para buscar eventos de Audit Trail o eventos de Events Management, haz clic en la flecha hacia abajo junto a Logs y selecciona Audit Trail o Events (Eventos).

  2. Crea una consulta raíz para tus logs o eventos utilizando la sintaxis de búsqueda del Log Explorer.

  3. En el menú desplegable Trigger for each new (Activar para cada nuevo), selecciona los atributos en los que cada atributo genera una señal para cada nuevo valor de atributo durante el periodo fijo de 24 horas.

  4. (Optional) To create calculated fields that transform your logs during query time:

    1. Click Add and select Calculated fields.
    2. In Name your field, enter a descriptive name that indicates the purpose of the calculated field.
      • For example, if you want to combine users’ first and last name into one field, you might name the calculated field fullName.
    3. In the Define your formula field, enter a formula or expression, which determines the result to be computed and stored as the value of the calculated field for each log event.
    • Consulta [Lenguaje de expresiones de campos calculados][3] para obtener información sobre la sintaxis y las construcciones del lenguaje.
  5. (Optional) To create calculated fields that transform your logs during query time:

    1. Click Add and select Calculated fields.
    2. In Name your field, enter a descriptive name that indicates the purpose of the calculated field.
      • For example, if you want to combine users’ first and last name into one field, you might name the calculated field fullName.
    3. In the Define your formula field, enter a formula or expression, which determines the result to be computed and stored as the value of the calculated field for each log event.
  6. (Optional) To filter logs using reference tables:

    1. Click the Add button next to the query editor and select Join with Reference Table.
    2. In the Inner join with reference table dropdown menu, select your reference table.
    3. In the where field dropdown menu, select the log field to join on.
    4. Select the IN or NOT IN operator to filter in or filter out matching logs.
    5. In the column dropdown menu, select the column of the reference table to join on.
  7. (Optional) To test your rules against sample logs, click Unit Test.

    1. To construct a sample log, you can:
      1. Navigate to Log Explorer in a new window.
      2. In the search bar, enter the query you are using for the detection rule.
    2. Select one of the logs.
    3. Click the export button at the top right side of the log side panel, and then select Copy.
    4. Navigate back to the Unit Test modal, and then paste the log into the text box. Edit the sample as needed for your use case.
    5. Toggle the switch for Query is expected to match based on the example event to fit your use case.
    6. Click Run Query Test.

Haz clic en Add Root Query (Añadir consulta raíz) para añadir consultas adicionales.

Establecer condiciones

Configura tus condiciones, gravedad y destinatarios de notificaciones
  1. If you have a single query, skip to step 2. If you have multiple queries, you can create a Simple condition or Then condition.
    • If you want to create a simple condition, leave the selection as is.
    • If you want to create a then condition, click THEN condition.
      • Use the Then condition when you want to trigger a signal if query A occurs and then query B occurs.
      • Note: The then operator can only be used on a single rule condition.
  2. (Optional) Click the pencil icon next to Condition 1 if you want to rename the condition. This name is appended to the rule name when a signal is generated.
  3. In the Set severity to dropdown menu, select the appropriate severity level (INFO, LOW, MEDIUM, HIGH, CRITICAL).
  4. If you are creating a Simple condition, enter the condition when a signal should be created. If you are creating a Then condition, enter the conditions required for a signal to be generated.
    • All rule conditions are evaluated as condition statements. Thus, the order of the conditions affects which notifications are sent because the first condition to match generates the signal. Click and drag your rule conditions to change their order.
    • A rule condition contains logical operations (>, >=, <, &&, ||) to determine if a signal should be generated based on the event counts in the previously defined queries.
    • The ASCII lowercase query labels are referenced in this section. An example rule condition for query a is a > 3.
    • Note: The query label must precede the operator. For example, a > 3 is allowed; 3 < a is not allowed.
  5. (Optional) In the Add notify section, click Add Recipient to configure notification targets.
    • You can also create notification rules to avoid manual edits to notification preferences for individual detection rules.

Otros parámetros

1. Multiactivación de trabajo

In the Job multi-triggering behavior section, configure how often to keep updating the same signal when new values are detected within a specified time frame. For example, the same signal updates when any new value is detected within 1 hour, for a maximum duration of 24 hours.

  • An evaluation window defines a sliding period in which at least one case evaluates as true and assesses cases in real time.
  • After a signal is generated, the signal remains “open” if a case is matched at least once within the keep alive window. Each time a new event matches any of the cases, the last updated timestamp is updated for the signal.
  • A signal closes after the time exceeds the maximum signal duration, regardless of the query being matched. This time is calculated from the first seen timestamp.
  • Note: The evaluation window must be less than or equal to the keep alive and maximum signal duration.

2. Habilitar agrupación opcional

Toggle Enable Optional Group By section if you want to group events even when values are missing. If there is a missing value, a sample value is generated so that the log does not get excluded.

Otros parámetros

1. Olvidar valor

In the Forget Value dropdown, select the number of days (1-30 days) after which the value is forgotten.

2. Comportamiento de multiactivación de trabajos

In the Job multi-triggering behavior section, configure how often to keep updating the same signal when new values are detected within a specified time frame. For example, the same signal updates when any new value is detected within 1 hour, for a maximum duration of 24 hours.

  • An evaluation window defines a sliding period in which at least one case evaluates as true and assesses cases in real time.
  • After a signal is generated, the signal remains “open” if a case is matched at least once within the keep alive window. Each time a new event matches any of the cases, the last updated timestamp is updated for the signal.
  • A signal closes after the time exceeds the maximum signal duration, regardless of the query being matched. This time is calculated from the first seen timestamp.
  • Note: The evaluation window must be less than or equal to the keep alive and maximum signal duration.

3. Habilitar agrupación opcional

Toggle Enable Optional Group By section if you want to group events even when values are missing. If there is a missing value, a sample value is generated so that the log does not get excluded.

4. Activar referencia instantánea

Toggle Enable instantaneous baseline if you want to build the baseline based on past events for the first event received.

Otros parámetros

1. Multiactivación de trabajo

In the Job multi-triggering behavior section, configure how often to keep updating the same signal when new values are detected within a specified time frame. For example, the same signal updates when any new value is detected within 1 hour, for a maximum duration of 24 hours.

  • An evaluation window defines a sliding period in which at least one case evaluates as true and assesses cases in real time.
  • After a signal is generated, the signal remains “open” if a case is matched at least once within the keep alive window. Each time a new event matches any of the cases, the last updated timestamp is updated for the signal.
  • A signal closes after the time exceeds the maximum signal duration, regardless of the query being matched. This time is calculated from the first seen timestamp.
  • Note: The evaluation window must be less than or equal to the keep alive and maximum signal duration.

2. Habilitar agrupación opcional

Toggle Enable Optional Group By section if you want to group events even when values are missing. If there is a missing value, a sample value is generated so that the log does not get excluded.

Configura tus condiciones, gravedad y destinatarios de notificaciones
  1. (Opcional) Haz clic en el icono del lápiz situado junto a Condition 1 (Condición 1) si deseas cambiar el nombre de la condición. Este nombre se añade al nombre de la regla cuando se genera una señal.
  2. En el campo Anomaly count (Recuento de anomalías), introduce la condición de cuántos logs anómalos dentro de la ventana especificada son necesarios para activar una señal.
    • Por ejemplo, si la condición es a >= 3 donde a es la consulta, se activa una señal si hay al menos tres logs anómalos dentro de la ventana de evaluación.
    • Todas las condiciones de las reglas se evalúan como sentencias de condición. Por lo tanto, el orden de las condiciones afecta a las notificaciones que se envían, ya que la primera condición que coincide genera la señal. Haz clic y arrastra las condiciones de tus reglas para cambiar su orden.
    • Una condición de regla contiene operaciones lógicas (>, >=, &&, ||) para determinar si debe generarse una señal en función de los recuentos de eventos en las consultas definidas previamente.
    • En esta sección, se hace referencia a las etiquetas de consulta ASCII en minúsculas. Un ejemplo de condición de regla para la consulta a es a > 3.
    • Nota: La etiqueta (label) de la consulta debe situarse por delante del operador. Por ejemplo, a > 3 es válido y 3 < a no es válido.
  3. En el menú desplegable within a window of (dentro de una ventana de), selecciona el periodo de tiempo durante el cual se activa una señal si se cumple la condición.
    • Se especifica una evaluation window para que coincida cuando al menos uno de los casos coincida con true (verdadero). Se trata de una ventana variable y evalúa los casos en tiempo real.

Otros parámetros

1. Detección de anomalías de contenido

In the Content anomaly detection options section, specify the parameters to assess whether a log is anomalous or not.

  • Content anomaly detection balances precision and sensitivity using several rule parameters that you can set:
    1. Similarity threshold: Defines how dissimilar a field value must be to be considered anomalous (default: 70%).
    2. Minimum similar items: Sets how many similar historical logs must exist for a value to be considered normal (default: 1).
    3. Evaluation window: The time frame during which anomalies are counted toward a signal (for example, a 10-minute time frame).
  • These parameters help to identify field content that is both unusual and rare, filtering out minor or common variations.
  • See Anomaly detection parameters for more information.

2. Comportamiento de multiactivación de trabajos

Configure how often you want to keep updating the same signal if new values are detected within a specified time frame. For example, the same signal updates if any new value is detected within 1 hour, for a maximum duration of 24 hours.

  • After a signal is generated, the signal remains “open” if a case is matched at least once within the keep alive window. Each time a new event matches any of the cases, the last updated timestamp is updated for the signal.
  • A signal closes after the time exceeds the maximum signal duration, regardless of the query being matched. This time is calculated from the first seen timestamp.
  • Note: The evaluation window must be less than or equal to the keep alive and maximum signal duration.

3. Habilitar agrupación opcional

Toggle Enable Optional Group By section if you want to group events even when values are missing. If there is a missing value, a sample value is generated so that the log does not get excluded.

Otros parámetros

1. Multiactivación de trabajo

In the Job multi-triggering behavior section, configure how often to keep updating the same signal when new values are detected within a specified time frame. For example, the same signal updates when any new value is detected within 1 hour, for a maximum duration of 24 hours.

  • An evaluation window defines a sliding period in which at least one case evaluates as true and assesses cases in real time.
  • After a signal is generated, the signal remains “open” if a case is matched at least once within the keep alive window. Each time a new event matches any of the cases, the last updated timestamp is updated for the signal.
  • A signal closes after the time exceeds the maximum signal duration, regardless of the query being matched. This time is calculated from the first seen timestamp.
  • Note: The evaluation window must be less than or equal to the keep alive and maximum signal duration.

2. Habilitar agrupación opcional

Toggle Enable Optional Group By section if you want to group events even when values are missing. If there is a missing value, a sample value is generated so that the log does not get excluded.

Configura tus condiciones, gravedad y destinatarios de notificaciones
  1. (Opcional) Haz clic en el icono del lápiz situado junto a Condition 1 (Condición 1) si deseas cambiar el nombre de la condición. Este nombre se añade al nombre de la regla cuando se genera una señal.
  2. En el campo Query (Consulta), introduce las etiquetas de un log que quieras que active una señal.
    • Por ejemplo, si deseas que logs con la etiqueta dev:demo activen señales con una gravedad de INFO, introduce dev:demo en el campo de consulta. Del mismo modo, si deseas que logs con la etiqueta dev:prod activen señales con una gravedad de MEDIUM, introduce dev:prod en el campo de consulta.

Otros parámetros

Toggle Enable Optional Group By section if you want to group events even when values are missing. If there is a missing value, a sample value is generated so that the log does not get excluded.

Notificar cuando el trabajo esté completo

(Opcional) Haz clic en Add Recipient (Añadir destinatario) para enviar notificaciones al finalizar el análisis de trabajo. Consulta Canales de notificación para obtener más información.

Describe tu manual

  1. Enter a Rule name. The name appears in the detection rules list view and the title of the security signal.
  2. In the Rule message section, use notification variables and Markdown to customize the notifications sent when a signal is generated.
  3. Use the Tag resulting signals dropdown menu to add tags to your signals. For example, security:attack or technique:T1110-brute-force.
    • Note: the tag security is special. This tag is used to classify the security signal. The recommended options are: attack, threat-intel, compliance, anomaly, and data-leak.