For AI agents: A markdown version of this page is available at https://docs.datadoghq.com/logs/guide/azure-automated-log-forwarding.md.
A documentation index is available at /llms.txt.
Use this guide to set up and manage Azure automated log forwarding. You can configure log forwarding directly in Datadog or deploy it with an Azure Resource Manager (ARM) template.
The ARM template deploys resources from a series of Azure services (storage accounts and function apps) into your subscriptions, which collect and forward logs to Datadog. These services automatically scale up or down to match log volume. Scaling is managed by a control plane, which is a set of function apps deployed to a subscription and region of your choice. Storage accounts and function apps are deployed in each of the subscriptions forwarding logs to Datadog.
All sites: Automated log forwarding is available to use on all Datadog sites.
Supported Azure environments: Automated log forwarding supports the Azure commercial (public) cloud only. Azure Government and Azure China are not supported. If you use Datadog government sites, you can only use this feature with workloads in Azure commercial cloud.
How to choose between automated and manual setup
Choose the manual setup method if you want to:
apply custom tags to your resources
Use the automated setup method if you want to:
automate deployment through the Azure portal
manage your infrastructure through declarative templates
centrally control access, tags, and billing
redeploy your resources in the correct order and in a consistent way
save costs by using a storage account rather than an event hub
Setup
Configure Log Forwarding
Use the Configure Log Forwarding flow to set up new or manage existing log forwarders directly in Datadog. You can use this flow to deploy automated log forwarding from scratch or update an existing setup, such as adding or removing subscriptions or modifying log filters.
Choose to deploy a new setup or update an existing one.
Copy the provided command and paste it in your Azure Cloud Shell.
Select the subscriptions to forward logs from.
Optionally, add or remove log filters.
Click Confirm.
ARM template
Alternatively, you can deploy automated log forwarding with an Azure Public ARM template. The sections below provide instructions for completing each page of the template.
Basics
Under Project details, select the management group. This is needed for the ARM template to grant permissions to the subscriptions you select for automated log forwarding.
Under Instance details, select values for:
Region. This is where the control plane is deployed.
Subscriptions to Forward Logs. These are the subscriptions to be configured for log forwarding.
Control Plane Subscription. This is the subscription that the control plane is deployed to.
Resource Group Name. This is the resource group to be used by the control plane. It is recommended to choose a new, unused resource group name to simplify management of control plane services.
Click the checkbox to acknowledge the deployment warnings.
Click Review + create.
Review + create
Review the finalized deployment details.
Click Create.
Resource tag filtering
You can use tag filters to control which Azure resources have their logs forwarded to Datadog. For tag filter syntax, wildcard support, and examples, see Resource tag filtering in the Azure getting started guide.
Log Analytics Workspaces
You can forward logs from Azure Log Analytics Workspaces (LAWs) to Datadog through the automated log forwarder. Previously, Datadog only supported diagnostic setting logs from LAWs. With data export rules, you can also forward logs from LAW Log Tables to Datadog.
Restrictions
You can only set up forwarding for LAW resources within the same region as the log forwarder.
You can have a maximum of 10 data export rules on a LAW. If the LAW has no remaining capacity for a Data Export Rule, delete an existing rule to make room.
Not all log tables can be exported. See Microsoft’s list of unsupported tables.
Forward logs from a Log Analytics Workspace
If you haven’t already created an automated log forwarder, follow the Setup instructions. If you already have a log forwarder, make sure it is updated to the latest version.
In the Azure Portal, navigate to the desired Log Analytics Workspace.
Under Settings, click Data export.
Click New export rule.
Name the rule, check Enable upon creation, and click Next.
Select the tables to export. You can modify this selection later by editing the data export rule. Click Next.
For Destination type, select Storage Account. Select the subscription containing your log forwarder, and choose a log forwarder storage account. These accounts typically have the prefix ddlogstorage. Click Next.
Review the rule and click Create. Logs from the LAW start appearing in Datadog within a few minutes.
Troubleshooting
Logs are not appearing in Datadog
If you have created a data export rule but do not see logs in Datadog:
Verify the data export rule is enabled.
Verify the destination storage account is one created by the automated log forwarder (the name typically starts with ddlogstorage).
In the storage account, inspect the containers. Containers with the am- prefix indicate LAW exports. If you only see containers with the insights- prefix, the data export rule may be improperly configured.
Verify the LAW has collected new logs within the past two hours.
The data export rule allows you to specify which log tables from your Log Analytics Workspace are exported. Edit the data export rule to add or remove tables.
Expected latency
LAW logs typically appear in Datadog within two to five minutes, but may take up to 20 minutes to first appear. LAW logs may have different properties from non-LAW logs.
Architecture
Services used
Azure Function apps are used to discover resources in your Azure subscriptions, scale log forwarders, and configure diagnostic settings on the detected resources.
Azure Container Apps are used to collect resource logs generated by diagnostic settings, track which logs have been processed already, and submit them to Datadog.
Azure Storage Accounts are used to store logs generated by your resources, as well as a small cache of metadata such as subscription IDs, resource IDs, and regions.
The control plane is a set of Azure Function apps and a storage account for caching. One control plane is deployed in your chosen subscription and performs the following tasks:
Discovery of resources in your chosen subscriptions that are able to log through diagnostic settings.
Automatic configuration of diagnostic settings on discovered resources to flow logs into a storage account that the log forwarders are tracking.
Scaling of log forwarders in regions where your resources are located, enabling them to match log volume dynamically.
Log forwarders
Log forwarders consist of an Azure Container Apps job and storage account for logs. They are deployed by the control plane in each subscription you select for log forwarding. The number of log forwarders deployed per subscription scales according to the volume of logs generated by your resources. Log forwarders perform the following tasks:
Temporarily store logs generated from your resources’ diagnostic settings in a storage account.
Process the stored logs and forward them to Datadog.
In Azure, a resource’s diagnostic settings can only target storage accounts within the same region. As such, the forwarders are spun up in each region where resources with diagnostic settings exist.
The ARM template grants the control plane only the permissions needed to manage the forwarders and place diagnostic settings on your resources. To achieve this, resource groups are created and permissions are granted during the ARM template deployment. After this, you can add permissions for more subscriptions by redeploying the ARM template.
This is needed to discover resources with available diagnostic settings and enable log output to storage.
Contributor role at the resource group level, for the log-forwarding resource groups in the selected subscriptions.
This is needed to manage (create and delete) forwarder storage accounts and Container Apps jobs.
Website Contributor role at the control plane resource group level, for updating the control plane function apps.
No information about your resources is exported. Datadog only requests the information required to enable log output, and the only output of this architecture is the logs sent to Datadog.
Note: Optionally, you can enable the control plane to submit its own health metrics, logs, and events to Datadog for debugging purposes. To do this, set the environment variable DD_TELEMETRY=true on any Function App or Container App in the control plane.
Log archiving
Archiving logs to Azure Blob Storage requires an App Registration. If you haven’t already, follow the automatic or manual setup instructions to configure the integration using an App Registration. App Registrations created for archiving purposes do not need the Monitoring Reader role.
After configuring an App Registration, create a log archive that writes to Azure Blob Storage.
Note: If your storage bucket is in a subscription being monitored through the Azure Native integration, a redundancy warning appears in the Azure integration tile. This warning can be safely ignored for log archiving.
Uninstall
Begin by opening an Azure Cloud Shell, and ensure it is running in Azure CLI/Bash, not PowerShell.
The script first discovers any instances running in each subscription, then prompts you to select the instance(s) to uninstall. Confirm the resource deletions, and wait for the resources to be deleted.
Further reading
Additional helpful documentation, links, and articles:
