--- title: Microsoft Sentinel Destination description: Datadog, the leading service for cloud-scale monitoring. breadcrumbs: Docs > Observability Pipelines > Destinations > Microsoft Sentinel Destination --- # Microsoft Sentinel Destination {% callout %} # Important note for users on the following Datadog sites: app.ddog-gov.com {% alert level="danger" %} This product is not supported for your selected [Datadog site](https://docs.datadoghq.com/getting_started/site). (). {% /alert %} {% /callout %} Available for: {% icon name="icon-logs" /%} Logs Use Observability Pipelines' Microsoft Sentinel destination to send logs to Microsoft Sentinel. See [Logs Ingestion API](https://learn.microsoft.com/en-us/azure/azure-monitor/fundamentals/service-limits#logs-ingestion-api) for API call limits in Microsoft Sentinel. ## Setup{% #setup %} Set up the Microsoft Sentinel destination and its environment variables when you [set up a pipeline](https://app.datadoghq.com/observability-pipelines). The information below is configured in the pipelines UI, except for Prerequisites which provides instructions on how to find the information you need in Microsoft Azure. ### Set up the destination{% #set-up-the-destination %} #### Prerequisites{% #prerequisites %} To set up the Microsoft Sentinel destination, you need to create a Workspace in Azure if you haven't already. In that workspace: 1. [Add Microsoft Sentinel](https://portal.azure.com/#browse/microsoft.securityinsightsarg%2Fsentinel) to the workspace. 1. [Create a Data Collection Endpoint (DCE)](https://portal.azure.com/#view/HubsExtension/BrowseResource.ReactView/resourceType/microsoft.insights%2Fdatacollectionendpoints). 1. [Create a Log Analytics Workspace](https://portal.azure.com/#create/Microsoft.LogAnalyticsOMS) in the workspace if you haven't already. 1. Follow the instructions for the type of table to which you want to send data. {% tab title="Azure Table" %} 1. Create a JSON file for your Data Collection Rule (DCR) parameters. See [Data collection rule (DCR)](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview#data-collection-rule-dcr) for more information and [Supported Azure Tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview#supported-tables) for all available tables to which you can send data. - In the `streamDeclarations` property, you must list all log fields you want mapped to the corresponding Azure table column. See [Stream declarations](https://learn.microsoft.com/en-us/azure/azure-monitor/data-collection/data-collection-rule-structure#stream-declarations) for more information. - In the `transformKql` property, you must list all fields on the log that are dropped and not mapped to the table. See [Data flow properties](https://learn.microsoft.com/en-us/azure/azure-monitor/data-collection/data-collection-rule-structure#data-flow-properties) for more information. - **Note**: Each log field must be listed in one of these properties: either `streamDeclarations` or `transformKql`; otherwise the log is dropped. See [Monitor DCR data collection in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/data-collection/data-collection-monitor) on how to set up an alert when logs are dropped. - For example, this JSON file (`dcr-commonsecuritylog.json`) adds the log fields to be mapped to the [`CommonSecurityLog`](https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/commonsecuritylog) table: ````bash { "location": "eastus", "kind": "Direct", "properties": { "dataCollectionEndpointId": "", "streamDeclarations": { "Custom-CommonSecurityLog": { "columns": [ { "name": "TimeGenerated", "type": "datetime" }, { "name": "DeviceVendor", "type": "string" }, { "name": "DeviceProduct", "type": "string" }, { "name": "DeviceVersion", "type": "string" }, { "name": "DeviceEventClassID", "type": "string" }, { "name": "Activity", "type": "string" }, { "name": "LogSeverity", "type": "string" }, { "name": "SourceIP", "type": "string" }, { "name": "DestinationIP", "type": "string" }, { "name": "Message", "type": "string" }, { "name": "source_type", "type": "string" }, { "name": "path", "type": "string" }, { "name": "timestamp", "type": "string" } ] } }, "destinations": { "logAnalytics": [ { "workspaceResourceId": "", "name": "LogAnalyticsDest" } ] }, "dataFlows": [ { "streams": ["Custom-CommonSecurityLog"], "destinations": ["LogAnalyticsDest"], "transformKql": "source | project-away source_type, path, timestamp", "outputStream": "Microsoft-CommonSecurityLog" } ] } ``` ```` - Replace the placeholders: - `` with the ID of the DCE resource you created in step 2. Run the [`az monitor data-collection endpoint show`](https://learn.microsoft.com/en-us/cli/azure/monitor/data-collection/endpoint?view=azure-cli-latest#az-monitor-data-collection-endpoint-show) command to get the DCE resource ID. For example: ``` az monitor data-collection endpoint show \ --name "" \ --resource-group \ --subscription \ --query "id" ``` - `` with the ID of the Logs Analytics Workspace you created in step 3. Run the [`az monitor log-analytics workspace show`](https://learn.microsoft.com/en-us/cli/azure/monitor/log-analytics/workspace?view=azure-cli-latest#az-monitor-log-analytics-workspace-show) command to get the Workspace resource ID. For example: ``` az monitor log-analytics workspace show \ --workspace-name "" \ --resource-group \ --subscription \ --query "id" ``` - See [CommonSecurityLog Columns](https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/commonsecuritylog#columns) for a full list of `commonsecuritylog` table columns. 1. Run the [`az monitor data-collection rule create`](https://learn.microsoft.com/en-us/cli/azure/monitor/data-collection/rule?view=azure-cli-latest#az-monitor-data-collection-rule-create) Azure CLI command to create a DCR with the JSON file you created in the previous step. For example, with the `dcr-commonsecuritylog.json` example file: ```bash az monitor data-collection rule create \ --resource-group "myResourceGroup" \ --location "eastus" \ --name "myCollectionRule" \ --subscription "mysubscription" \ --rule-file "\path\to\json\dcr-commonsecuritylog.json" ``` {% /tab %} {% tab title="Custom table" %} 1. In the Log Analytics Workspace, navigate to **Settings** > **Tables**. 1. Click **+ Create**. 1. Define a custom table (for example, `MyOPWLogs`). - **Notes**:- After the table is configured, the prefix `Custom-` and suffix `_CL` are automatically appended to the table name. For example, if you defined the table name in Azure to be `MyOPWLogs`, the full table name is stored as `Custom-MyOPWLogs_CL`. You must use the full table name when you set up the Observability Pipelines Microsoft Sentinel destination.-The full table name can be found in the resource JSON of the DCR under `streamDeclarations`. 1. Select **New Custom Log (DCR-based)**. 1. Click **Create a new data collection rule** and select the DCE you created earlier. 1. Click **Next**. 1. Upload a sample JSON Log. For this example, the following JSON is used for the **Schema and Transformation**, where `TimeGenerated` is required: ```json { "TimeGenerated": "2024-07-22T11:47:51Z", "event": {} } ``` 1. Click **Create**. {% /tab %} 1. In Azure, navigate to **Microsoft Entra ID**. 1. Click **Add** > **App Registration**. 1. Click **Create**. 1. On the overview page, click **Client credentials: Add a certificate or secret**. 1. Click **New client secret**. 1. Enter a name for the secret and click **Add**. **Note**: Make sure to take note of the client secret, which gets obfuscated after 10 minutes. 1. Also take note of the **Tenant ID** and **Client ID**. You need this information, along with the client secret, when you set up the Observability Pipelines Microsoft Sentinel destination. 1. In Azure Portal's [Data Collection Rules](https://portal.azure.com/#view/HubsExtension/BrowseResource.ReactView/resourceType/microsoft.insights%2Fdatacollectionrules) page, search for and select the DCR you created earlier. 1. Click **Access Control (IAM)** in the left nav. 1. Click **Add** and select **Add role assignment**. 1. Add the **Monitoring Metrics Publisher** role. 1. On the Members page, select **User, group, or service principal**. 1. Click **Select Members** and search for the application you created in the app registration step. 1. Click **Review + Assign**. **Note**: It can take up to 10 minutes for the IAM change to take effect. The table below summarizes the Azure and Microsoft Sentinel information you need when you set up the Observability Pipelines Microsoft Sentinel destination: | Name | Description | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Application (client) ID | The Azure Active Directory (AD) application's client ID. See [Register an application in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app?tabs=certificate%2Cexpose-a-web-api) for more information.**Example**: `550e8400-e29b-41d4-a716-446655440000` | | Directory (tenant) ID | The Azure AD tenant ID. See [Register an application in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app?tabs=certificate%2Cexpose-a-web-api) for more information.**Example**: `72f988bf-86f1-41af-91ab-2d7cd011db47` | | Table (Stream) Name | The name of the stream which matches the table chosen when configuring the Data Collection Rule (DCR). **Note**: The full table name can be found in the resource JSON of the DCR under `streamDeclarations`.**Example**: `Custom-MyOPWLogs_CL` | | Data Collection Rule (DCR) immutable ID | This is the immutable ID of the DCR where logging routes are defined. It is the **Immutable ID** shown on the DCR Overview page.**Note**: Ensure the Monitoring Metrics Publisher role is assigned in the DCR IAM settings.**Example**: `dcr-000a00a000a00000a000000aa000a0aa`See [Data collection rules (DCRs) in Azure Monitor](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview) to learn more about creating or viewing DCRs. | #### Set up the destination in Observability Pipelines{% #set-up-the-destination-in-observability-pipelines %} To set up the Microsoft Sentinel destination in Observability Pipelines: {% alert level="danger" %} Only enter the identifiers for the Microsoft Sentinel client secret and Data Collection Endpoint. Do not enter the actual values. {% /alert %} 1. Enter the identifier for your Microsoft Sentinel client secret. If you leave it blank, the default is used. 1. Enter the identifier for your Microsoft Sentinel Data Collection endpoint. If you leave it blank, the default is used. 1. Enter the client ID for your application, such as `550e8400-e29b-41d4-a716-446655440000`. 1. Enter the directory ID for your tenant, such as `72f988bf-86f1-41af-91ab-2d7cd011db47`. This is the Azure AD tenant ID. 1. Enter the full table name to which you are sending logs. An example table name: `Custom-MyOPWLogs_CL`. 1. Enter the Data Collection Rule (DCR) immutable ID, such as `dcr-000a00a000a00000a000000aa000a0aa`. #### Optional buffering{% #optional-buffering %} Toggle the switch to enable **Buffering Options**. Enable a configurable buffer on your destination to ensure intermittent latency or an outage at the destination doesn't create immediate backpressure, and allow events to continue to be ingested from your source. Disk buffers can also increase pipeline durability by writing data to disk, ensuring buffered data persists through a Worker restart. See [Destination buffers](https://docs.datadoghq.com/observability_pipelines/scaling_and_performance/buffering_and_backpressure/#destination-buffers) for more information. - If left unconfigured, your destination uses a memory buffer with a capacity of 500 events. - To configure a buffer on your destination: 1. Select the buffer type you want to set (**Memory** or **Disk**). 1. Enter the buffer size and select the unit. 1. Maximum memory buffer size is 128 GB. 1. Maximum disk buffer size is 500 GB. 1. In the **Behavior on full buffer** dropdown menu, select whether you want to **block** events or **drop new events** when the buffer is full. ### Set secrets{% #set-secrets %} These are the defaults used for secret identifiers and environment variables. **Note**: If you enter secret identifiers and then choose to use environment variables, the environment variable is the identifier entered and prepended with `DD_OP`. For example, if you entered `PASSWORD_1` for a password identifier, the environment variable for that password is `DD_OP_PASSWORD_1`. {% tab title="Secrets Management" %} - Microsoft Sentinel client secret identifier: - References the DCE endpoint URL shown as the **Logs Ingestion Endpoint** or **Data Collection Endpoint** on the DCR Overview page. An example URL: `https://.ingest.monitor.azure.com`. - The default identifier is `DESTINATION_MICROSOFT_SENTINEL_CLIENT_SECRET`. - Microsoft Sentinel Data Collection endpoint identifier: - References the Azure AD application's client secret, such as `550e8400-e29b-41d4-a716-446655440000`. - The default identifier is `DESTINATION_MICROSOFT_SENTINEL_DCE_URI`. {% /tab %} {% tab title="Environment Variables" %} - Data collection endpoint (DCE) - The DCE endpoint URL is shown as the **Logs Ingestion Endpoint** or **Data Collection Endpoint** on the DCR Overview page. An example URL: `https://.ingest.monitor.azure.com`. - The default environment variable is `DD_OP_DESTINATION_MICROSOFT_SENTINEL_DCE_URI` - Client secret - This is the Azure AD application's client secret, such as `550e8400-e29b-41d4-a716-446655440000`. - The default environment variable is `DD_OP_DESTINATION_MICROSOFT_SENTINEL_CLIENT_SECRET` {% /tab %} ## How the destination works{% #how-the-destination-works %} ### Event batching{% #event-batching %} A batch of events is flushed when one of these parameters is met. See [event batching](https://docs.datadoghq.com/observability_pipelines/destinations/#event-batching) for more information. | Maximum Events | Maximum Size (MB) | Timeout (seconds) | | -------------- | ----------------- | ----------------- | | None | 10 | 1 |