Azure Automated Log Forwarding Architecture and Configuration
Overview
Datadog’s automated log forwarding setup for the Azure integration offers several benefits over other setup methods:
Simplified setup of the required Azure infrastructure across all of your subscriptions and regions, allowing you to start forwarding logs from across your Azure environment in minutes. New resources added to your environment in the future are automatically set up to forward logs to Datadog.
Automatic scaling to match the volume of logs in your Azure environment. New log forwarders are spun up as needed to meet the demand, and are automatically removed as demand subsides.
Reduced Azure costs by using more cost-effective Azure services, and by scaling down in periods of low volume.
This log forwarding method is available to all Datadog sites and most Azure regions (any region that supports Container Apps). Datadog recommends this method to set up log forwarding from your Azure environment.
Supported logs
This setup supports forwarding of any logs available through diagnostic settings, including activity logs and resource logs. For non-platform logs, Datadog recommends using the Datadog Agent to forward logs.
Architecture
Services used
- Azure Function apps are used to discover resources in your Azure subscriptions, scale log forwarders, and configure diagnostic settings on the detected resources.
- Azure Container Apps are used to collect resource logs generated by diagnostic settings, track which logs have been processed already, and submit them to Datadog.
- Azure Storage Accounts are used to store logs generated by your resources, as well as a small cache of metadata such as subscription IDs, resource IDs, and regions.
High-level architecture
The deployment template sets up a control plane and log forwarders in your selected subscriptions.
Control plane
The control plane is a set of Azure Function apps and a storage account for caching. One control plane is deployed in your chosen subscription and performs the following tasks:
- Discovery of resources in your chosen subscriptions that are able to log through diagnostic settings.
- Automatic configuration of diagnostic settings on discovered resources to flow logs into a storage account that the log forwarders are tracking.
- Scaling of log forwarders in regions where your resources are located, enabling them to match log volume dynamically.
Log forwarders
Log forwarders consist of an Azure Container Apps job and storage account for logs. They are deployed by the control plane in each subscription you select for log forwarding. The number of log forwarders deployed per subscription scales according to the volume of logs generated by your resources. Log forwarders perform the following tasks:
- Temporarily store logs generated from your resources’ diagnostic settings in a storage account.
- Process the stored logs and forward them to Datadog.
In Azure, a resource’s diagnostic settings can only target storage accounts within the same region. As such, the forwarders are spun up in each region where resources with diagnostic settings exist.
See Azure’s Diagnostic settings in Azure Monitor page for more information.
Detailed architecture
Setup
Setup is completed through an Azure Resource Manager (ARM) template. This allows you to update configuration options such as the API Key, Datadog Site, or monitored subscriptions through redeployments with the same template.
Upon accessing the ARM template, you select the management group, region, and subscriptions where you would like to automate log forwarding. The template then deploys a control plane, which in turn deploys log forwarders in each of the subscriptions you selected.
See the Azure Automated Log Forwarding Setup guide for a step-by-step walkthrough of deploying the ARM template through the Azure Portal.
Security and permissions
The ARM template grants the control plane only the permissions needed to manage the forwarders and place diagnostic settings on your resources. To achieve this, resource groups are created and permissions are granted during the ARM template deployment. After this, you can add permissions for more subscriptions by redeploying the ARM template.
Permissions used
Monitoring Contributor role at the subscription level for the selected subscriptions.
- This is needed to discover resources with available diagnostic settings and enable log output to storage.
Contributor role at the resource group level, for the log-forwarding resource groups in the selected subscriptions.
- This is needed to manage (create and delete) forwarder storage accounts and Container Apps jobs.
Website Contributor role at the control plane resource group level, for updating the control plane function apps.
No information about your resources is exported. Datadog only requests the information required to enable log output, and the only output of this architecture is the logs sent to Datadog.
Note: Optionally, you can generate metrics, logs, and events about the health of the control plane and send them to Datadog for debugging purposes. This is enabled through a feature flag.
Help and feedback
To get help or provide feedback, email azure-log-forwarding@datadoghq.com.
Additional helpful documentation, links, and articles:
