Use this guide to automate your Azure log forwarding setup with an Azure Resource Manager (ARM) template.
The ARM template deploys resources from a series of Azure services (storage accounts and function apps) into your subscriptions, which collect and forward logs to Datadog. These services automatically scale up or down to match log volume. Scaling is managed by a control plane, which is a set of function apps deployed to a subscription and region of your choice. Storage accounts and function apps are deployed in each of the subscriptions forwarding logs to Datadog.
All sites: Automated log forwarding is available to use on all Datadog sites.
How to choose between automated and manual setup
Choose the manual setup method if you want to:
apply custom tags to your resources
Use the automated setup method if you want to:
automate deployment through the Azure portal
manage your infrastructure through declarative templates
centrally control access, tags, and billing
redeploy your resources in the correct order and in a consistent way
save costs by using a storage account rather than an event hub
Setup
Begin by opening the Azure Log Forwarding ARM template corresponding to your Azure environment, or by clicking + Add Log Collection in the Azure integration tile.
The sections below provide instructions for completing each page of the template.
Basics
Under Project details, select the management group. This is needed for the ARM template to grant permissions to the subscriptions you select for automated log forwarding.
Under Instance details, select values for:
Region. This is where the control plane is deployed.
Subscriptions to Forward Logs. These are the subscriptions to be configured for log forwarding.
Control Plane Subscription. This is the subscription that the control plane is deployed to.
Resource Group Name. This is the resource group to be used by the control plane. It is recommended to choose a new, unused resource group name to simplify management of control plane services.
Click the checkbox to acknowledge the deployment warnings.
Click Review + create.
Review + create
Review the finalized deployment details.
Click Create.
Architecture
Services used
Azure Function apps are used to discover resources in your Azure subscriptions, scale log forwarders, and configure diagnostic settings on the detected resources.
Azure Container Apps are used to collect resource logs generated by diagnostic settings, track which logs have been processed already, and submit them to Datadog.
Azure Storage Accounts are used to store logs generated by your resources, as well as a small cache of metadata such as subscription IDs, resource IDs, and regions.
The control plane is a set of Azure Function apps and a storage account for caching. One control plane is deployed in your chosen subscription and performs the following tasks:
Discovery of resources in your chosen subscriptions that are able to log through diagnostic settings.
Automatic configuration of diagnostic settings on discovered resources to flow logs into a storage account that the log forwarders are tracking.
Scaling of log forwarders in regions where your resources are located, enabling them to match log volume dynamically.
Log forwarders
Log forwarders consist of an Azure Container Apps job and storage account for logs. They are deployed by the control plane in each subscription you select for log forwarding. The number of log forwarders deployed per subscription scales according to the volume of logs generated by your resources. Log forwarders perform the following tasks:
Temporarily store logs generated from your resources’ diagnostic settings in a storage account.
Process the stored logs and forward them to Datadog.
In Azure, a resource’s diagnostic settings can only target storage accounts within the same region. As such, the forwarders are spun up in each region where resources with diagnostic settings exist.
The ARM template grants the control plane only the permissions needed to manage the forwarders and place diagnostic settings on your resources. To achieve this, resource groups are created and permissions are granted during the ARM template deployment. After this, you can add permissions for more subscriptions by redeploying the ARM template.
This is needed to discover resources with available diagnostic settings and enable log output to storage.
Contributor role at the resource group level, for the log-forwarding resource groups in the selected subscriptions.
This is needed to manage (create and delete) forwarder storage accounts and Container Apps jobs.
Website Contributor role at the control plane resource group level, for updating the control plane function apps.
No information about your resources is exported. Datadog only requests the information required to enable log output, and the only output of this architecture is the logs sent to Datadog.
Note: Optionally, you can generate metrics, logs, and events about the health of the control plane and send them to Datadog for debugging purposes. This is enabled through a feature flag.
Log archiving
Archiving logs to Azure Blob Storage requires an App Registration. If you haven’t already, follow the automatic or manual setup instructions to configure the integration using an App Registration. App Registrations created for archiving purposes do not need the Monitoring Reader role.
After configuring an App Registration, create a log archive that writes to Azure Blob Storage.
Note: If your storage bucket is in a subscription being monitored through the Azure Native integration, a warning is displayed in the Azure Integration Tile about the App Registration being redundant. You can ignore this warning.
Uninstall
Begin by opening an Azure Cloud Shell, and ensure it is running in Azure CLI/Bash, not PowerShell.
The script first discovers any instances running in each subscription, then prompts you to select the instance(s) to uninstall. Confirm the resource deletions, and wait for the resources to be deleted.
