Use Observability Pipelines’ Microsoft Sentinel destination to send logs to Microsoft Sentinel. See Logs Ingestion API for API call limits in Microsoft Sentinel.

Setup

Set up the Microsoft Sentinel destination and its environment variables when you set up a pipeline. The information below is configured in the pipelines UI, except for Prerequisites which provides instructions on how to find the information you need in Microsoft Azure.

Set up the destination

Prerequisites

To set up the Microsoft Sentinel destination, you need to create a Workspace in Azure if you haven’t already. In that workspace:

1. Add Microsoft Sentinel to the workspace.
2. Create a Data Collection Endpoint (DCE).
3. Create a Log Analytics Workspace in the workspace if you haven’t already.
4. Follow the instructions for the type of table to which you want to send data.
   1. Create a JSON file for your Data Collection Rule (DCR) parameters. See Data collection rule (DCR) for more information and Supported Azure Tables for all available tables to which you can send data.

      • In the streamDeclarations property, you must list all log fields you want mapped to the corresponding Azure table column. See Stream declarations for more information.

      • In the transformKql property, you must list all fields on the log that are dropped and not mapped to the table. See Data flow properties for more information.

      • Note: Each log field must be listed in one of these properties: either streamDeclarations or transformKql; otherwise the log is dropped. See Monitor DCR data collection in Azure Monitor on how to set up an alert when logs are dropped.

      • For example, this JSON file (dcr-commonsecuritylog.json) adds the log fields to be mapped to the CommonSecurityLog table:

        {
            "location": "eastus",
            "kind": "Direct",
            "properties": {
            "dataCollectionEndpointId": "<DCE_RESOURCE_ID>",
            "streamDeclarations": {
                "Custom-CommonSecurityLog": {
                "columns": [
                    { "name": "TimeGenerated",      "type": "datetime" },
                    { "name": "DeviceVendor",       "type": "string"   },
                    { "name": "DeviceProduct",      "type": "string"   },
                    { "name": "DeviceVersion",      "type": "string"   },
                    { "name": "DeviceEventClassID", "type": "string"   },
                    { "name": "Activity",           "type": "string"   },
                    { "name": "LogSeverity",        "type": "string"   },
                    { "name": "SourceIP",           "type": "string"   },
                    { "name": "DestinationIP",      "type": "string"   },
                    { "name": "Message",            "type": "string"   },
                    { "name": "source_type",        "type": "string"   },
                    { "name": "path",               "type": "string"   },
                    { "name": "timestamp",          "type": "string"   }
                ]
                }
            },
            "destinations": {
                "logAnalytics": [
                {
                    "workspaceResourceId": "<WORKSPACE_RESOURCE_ID>",
                    "name": "LogAnalyticsDest"
                }
                ]
            },
            "dataFlows": [
                {
                "streams":      ["Custom-CommonSecurityLog"],
                "destinations": ["LogAnalyticsDest"],
                "transformKql": "source | project-away source_type, path, timestamp",
                "outputStream": "Microsoft-CommonSecurityLog"
                }
            ]
            }
            ```
        

      • Replace the placeholders:

        • <DCE_RESOURCE_ID> with the ID of the DCE resource you created in step 2. Run the az monitor data-collection endpoint show command to get the DCE resource ID. For example:

          az monitor data-collection endpoint show \
          --name "<DCE_NAME>" \
          --resource-group <RESOURCE_GROUP> \
          --subscription <SUBSCRIPTION_ID> \
          --query "id"
          

        • <WORKSPACE_RESOURCE_ID> with the ID of the Logs Analytics Workspace you created in step 3. Run the az monitor log-analytics workspace show command to get the Workspace resource ID. For example:

          az monitor log-analytics workspace show \
          --workspace-name "<DCE_NAME>" \
          --resource-group <RESOURCE_GROUP> \
          --subscription <SUBSCRIPTION_ID> \
          --query "id"
          

      • See CommonSecurityLog Columns for a full list of commonsecuritylog table columns.

   2. Run the az monitor data-collection rule create Azure CLI command to create a DCR with the JSON file you created in the previous step. For example, with the dcr-commonsecuritylog.json example file:

      az monitor data-collection rule create \
          --resource-group "myResourceGroup" \
          --location "eastus" \
          --name "myCollectionRule" \
          --subscription "mysubscription" \
          --rule-file "\path\to\json\dcr-commonsecuritylog.json"
      

   1. In the Log Analytics Workspace, navigate to Settings > Tables.
   2. Click + Create.
   3. Define a custom table (for example, MyOPWLogs).
      • Notes:
        - After the table is configured, the prefix Custom- and suffix _CL are automatically appended to the table name. For example, if you defined the table name in Azure to be MyOPWLogs, the full table name is stored as Custom-MyOPWLogs_CL. You must use the full table name when you set up the Observability Pipelines Microsoft Sentinel destination.
        -The full table name can be found in the resource JSON of the DCR under streamDeclarations.
   4. Select New Custom Log (DCR-based).
   5. Click Create a new data collection rule and select the DCE you created earlier.
   6. Click Next.
   7. Upload a sample JSON Log. For this example, the following JSON is used for the Schema and Transformation, where TimeGenerated is required:

      {
          "TimeGenerated": "2024-07-22T11:47:51Z",
          "event": {}
      }
      

   8. Click Create.
5. In Azure, navigate to Microsoft Entra ID.
   1. Click Add > App Registration.
   2. Click Create.
   3. On the overview page, click Client credentials: Add a certificate or secret.
   4. Click New client secret.
   5. Enter a name for the secret and click Add. Note: Make sure to take note of the client secret, which gets obfuscated after 10 minutes.
   6. Also take note of the Tenant ID and Client ID. You need this information, along with the client secret, when you set up the Observability Pipelines Microsoft Sentinel destination.
6. In Azure Portal’s Data Collection Rules page, search for and select the DCR you created earlier.
   1. Click Access Control (IAM) in the left nav.
   2. Click Add and select Add role assignment.
   3. Add the Monitoring Metrics Publisher role.
   4. On the Members page, select User, group, or service principal.
   5. Click Select Members and search for the application you created in the app registration step.
   6. Click Review + Assign. Note: It can take up to 10 minutes for the IAM change to take effect.

The table below summarizes the Azure and Microsoft Sentinel information you need when you set up the Observability Pipelines Microsoft Sentinel destination:

Set up the destination in Observability Pipelines

To set up the Microsoft Sentinel destination in Observability Pipelines:

1. Enter the identifier for your Microsoft Sentinel client secret. If you leave it blank, the default is used.
2. Enter the identifier for your Microsoft Sentinel Data Collection endpoint. If you leave it blank, the default is used.
3. Enter the client ID for your application, such as 550e8400-e29b-41d4-a716-446655440000.
4. Enter the directory ID for your tenant, such as 72f988bf-86f1-41af-91ab-2d7cd011db47. This is the Azure AD tenant ID.
5. Enter the full table name to which you are sending logs. An example table name: Custom-MyOPWLogs_CL.
6. Enter the Data Collection Rule (DCR) immutable ID, such as dcr-000a00a000a00000a000000aa000a0aa.

Optional buffering

Toggle the switch to enable Buffering Options. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn’t create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing logs to disk, ensuring buffered logs persist through a Worker restart. See Destination buffers for more information.

• If left unconfigured, your destination uses a memory buffer with a capacity of 500 events.
• To configure a buffer on your destination:
  1. Select the buffer type you want to set (Memory or Disk).
  2. Enter the buffer size and select the unit.
     1. Maximum memory buffer size is 128 GB.
     2. Maximum disk buffer size is 500 GB.
  3. In the Behavior on full buffer dropdown menu, select whether you want to block events or drop new events when the buffer is full.

Set secrets

These are the defaults used for secret identifiers and environment variables.

Note: If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with DD_OP. For example, if you entered PASSWORD_1 for a password identifier, the environment variable for that password is DD_OP_PASSWORD_1.

How the destination works

Event batching

A batch of events is flushed when one of these parameters is met. See event batching for more information.