Use Observability Pipelines’ Microsoft Sentinel destination to send logs to Microsoft Sentinel.

Setup

Set up the Microsoft Sentinel destination and its environment variables when you set up a pipeline. The information below is configured in the pipelines UI, except for Prerequisites which provides instructions on how to find the information you need in Microsoft Azure.

Set up the destination

Prerequisites

To set up the Microsoft Sentinel destination, you need to create a Workspace in Azure if you haven’t already. In that workspace:

1. Add Microsoft Sentinel to the workspace.
2. Create a Data Collection Endpoint (DCE).
3. Create a Log Analytics Workspace in the workspace if you haven’t already.
4. In the Log Analytics Workspace, navigate to Settings > Tables.
   1. Click + Create.
   2. Define a custom table (for example, Custom-MyLogs_CL).
      • Notes:
        - For custom tables, the table name must start with Custom-. CL is automatically appended to the end of the table name. You need the table name to set up the Observability Pipelines Microsoft Sentinel destination.
        - You can also use an Azure Table instead of a custom table.
   3. Select New Custom Log (DCR-based).
   4. Click Create a new data collection rule and select the DCE you created earlier.
   5. Click Next.
   6. Upload a sample JSON Log. For this example, the following JSON is used for the Schema and Transformation, where TimeGenerated is required:

      {
          "TimeGenerated": "2024-07-22T11:47:51Z",
          "event": {}
      }
      

   7. Click Create.
5. In Azure, navigate to Microsoft Entra ID.
   1. Click Add > App Registration.
   2. Click Create.
   3. On the overview page, click Client credentials: Add a certificate or secret.
   4. Click New client secret.
   5. Enter a name for the secret and click Add. Note: Make sure to take note of the client secret, which gets obfuscated after 10 minutes.
   6. Also take note of the Tenant ID and Client ID. You need this information, along with the client secret, when you set up the Observability Pipelines Microsoft Sentinel destination.
6. In Azure Portal’s Data Collection Rules page, search for and select the DCR you created earlier.
   1. Click Access Control (IAM) in the left nav.
   2. Click Add and select Add role assignment.
   3. Add the Monitoring Metrics Publisher role.
   4. On the Members page, select User, group, or service principal.
   5. Click Select Members and search for the application you created in the app registration step.
   6. Click Review + Assign. Note: It can take up to 10 minutes for the IAM change to take effect.

The table below summarizes the Azure and Microsoft Sentinel information you need when you set up the Observability Pipelines Microsoft Sentinel destination:

Set up the destination in Observability Pipelines

To set up the Microsoft Sentinel destination in Observability Pipelines:

1. Enter the client ID for your application, such as 550e8400-e29b-41d4-a716-446655440000.
2. Enter the directory ID for your tenant, such as 72f988bf-86f1-41af-91ab-2d7cd011db47. This is the Azure AD tenant ID.
3. Enter the name of the table to which you are sending logs. An example table name: Custom-MyLogs_CL.
4. Enter the Data Collection Rule (DCR) immutable ID, such as dcr-000a00a000a00000a000000aa000a0aa.

Set the environment variables

• Data collection endpoint (DCE)
  • The DCE endpoint URL is shown as the Logs Ingestion Endpoint or Data Collection Endpoint on the DCR Overview page. An example URL: https://<DCE-ID>.ingest.monitor.azure.com.
  • Stored as the environment variable DD_OP_DESTINATION_MICROSOFT_SENTINEL_DCE_URI
• Client secret
  • This is the Azure AD application’s client secret, such as 550e8400-e29b-41d4-a716-446655440000.
  • Stored as the environment variable DD_OP_DESTINATION_MICROSOFT_SENTINEL_CLIENT_SECRET

How the destination works

Event batching

A batch of events is flushed when one of these parameters is met. See event batching for more information.