--- title: Azure Advanced Configuration description: Advanced configuration and architecture of the Datadog Azure integration breadcrumbs: Docs > Integrations > Integration Guides > Azure Advanced Configuration --- # Azure Advanced Configuration ## Overview{% #overview %} This guide provides advanced configuration options and reference architectures for users configuring Datadog's Azure integration, as well as alternative configuration options for specific use cases. ### Reference architectures{% #reference-architectures %} The diagrams in this guide provide a visual representation of the configuration process and outcome when following the steps in [Getting Started with Azure](https://docs.datadoghq.com/getting_started/integrations/azure/). This guide provides a detailed overview of Datadog's interaction with your Azure environment and answers common security, compliance, and governance questions. ### Alternate configurations{% #alternate-configurations %} The setup processes documented in [Getting Started with Azure](https://docs.datadoghq.com/getting_started/integrations/azure/) are the recommended steps and result in the ideal configuration for the majority of users. Alternate configuration options in this document may be preferable for certain use cases. Any trade-offs in performance, features, or ease-of-management are outlined as needed. ## Metric and data collection{% #metric-and-data-collection %} Enabling Datadog's Azure integration allows Datadog to: - Discover and monitor all resources in all subscriptions within the given scope - Automatically update discovered metric definitions, to ensure that all of the metrics available from Azure Monitor are collected - Ingest a range of both general and resource-specific metadata (including custom Azure tags), and apply it to the associated resource metrics in Datadog as tags - Query Azure metadata APIs and use the responses to [generate useful metrics in Datadog](https://www.datadoghq.com/blog/datadog-generated-metrics-azure/) for insights that Azure Monitor does not support The Azure APIs used and data collected are identical regardless of whether you use the standard or Azure Native version of the integration. The diagram below outlines the process and resulting architecture of the Azure integration configuration described in [Getting Started with Azure](https://docs.datadoghq.com/getting_started/integrations/azure/). {% image source="https://datadog-docs.imgix.net/images/integrations/guide/azure_architecture_and_configuration/app_registration_integration_setup.fc7202e08aed596cf1b32c66f8a920e4.png?auto=format" alt="Workflow diagram showing Azure App Registration integration setup: create app registration with service principal and client secrets in Azure, assign monitoring reader role to subscription or management resources, then configure integration with metric tag filters in the Datadog backend." /%} After this is completed, data collection begins automatically. The app registration allows Datadog to [request a token from Azure Active Directory](https://learn.microsoft.com/azure/databricks/dev-tools/api/latest/aad/) (AD). Datadog uses this token as the authorization for API calls to various Azure APIs, to discover resources within the scope provided, and collect data. This continuous process runs with two-minute intervals by default, and is used to discover and collect data from your Azure environment. The data collection process is pictured below. {% image source="https://datadog-docs.imgix.net/images/integrations/guide/azure_architecture_and_configuration/app_registration_metric_collection.a2a7e3bb3614d8da281cddf8391a5c48.png?auto=format" alt="Workflow diagram showing Azure metric collection process: the Datadog backend reads configuration, authenticates through service principal to Azure Active Directory, collects subscription and resource metadata using RBAC permissions, filters resources by tags, then retrieves metrics from Azure Monitor for ingestion into Datadog." /%} ## Log collection{% #log-collection %} The diagram below provides a reference architecture for forwarding logs from Azure to Datadog, as described in the [Event Hub log forwarding guide](https://docs.datadoghq.com/logs/guide/azure-event-hub-log-forwarding). {% image source="https://datadog-docs.imgix.net/images/integrations/guide/azure_architecture_and_configuration/manual_log_forwarding.918da6f9cb9001fd1b7d00082de81818.png?auto=format" alt="Architecture diagram showing manual Azure log forwarding setup across two regions where Azure resources use diagnostic settings to send logs through Event Hubs and Log Forwarding Functions to Datadog Logs Ingestion." /%} ### Alternate configuration options for log forwarding{% #alternate-configuration-options-for-log-forwarding %} The default architecture above is suitable for most users. Depending on the scale and composition of your Azure environment, as well as the methods your organization uses to implement this architecture, the sections below detail additional considerations that may be relevant. #### Using the provided templates{% #using-the-provided-templates %} The **Deploy to Azure** button in the main Azure [Event Hub Logging guide](https://docs.datadoghq.com/logs/guide/azure-event-hub-log-forwarding/) provides a template for creating an Event Hub and forwarder function pair. In addition to using this template to deploy directly, you can use the underlying ARM templates as a starting point for your own infrastructure as code deployments. These templates do not add diagnostic settings, apart from one optional diagnostic setting for activity logs. For resource logs, Datadog recommends utilizing ARM templates or Terraform to add diagnostic settings to your resources programmatically. These diagnostic settings must be added to every resource that needs to send resource logs to Datadog. #### Region considerations{% #region-considerations %} Diagnostic settings can only send resource logs to Event Hubs in the same region as the resource. Add an Event Hub and forwarder function pair in each region that contains resources for which you want to send resource logs to Datadog. However, diagnostic settings are not limited to sending logs to Event Hubs in the same subscription as the resource. If you have multiple subscriptions within your Azure tenant, they can share a single Event Hub and forwarder function within the same region. #### High-volume log considerations{% #high-volume-log-considerations %} As the volume of logs scales, you may see bottlenecks, typically arising in the Event Hubs. If you plan to submit high log volumes, you may want to consider adding additional partitions or using a Premium or Dedicated tier. For especially high log volumes, you may consider adding additional Event Hub and forwarder function pairs within the same region, and splitting traffic between them. {% collapsible-section #azure-datadog-extension-commands %} #### Commands to install the Azure Datadog Extension ## Install on Azure{% #install-on-azure %} Datadog provides an Azure extension to assist with Agent deployment on Azure instances: - [Introducing Azure monitoring with one-click Datadog deployment](https://www.datadoghq.com/blog/introducing-azure-monitoring-with-one-click-datadog-deployment) - [Azure integration](https://docs.datadoghq.com/getting_started/integrations/azure/#install-the-agent-for-greater-visibility-into-your-application) *All sites* - [Azure Native integration](https://docs.datadoghq.com/integrations/guide/azure-native-integration/#agent-extensions) *US3 only* An alternative to the GUI installation is the command line. To run the Datadog Agent in your Azure instances as an extension, use the command that matches your environment. Replace `` with your Datadog account **site parameter** value in the [Datadog sites page](https://docs.datadoghq.com/getting_started/site/#access-the-datadog-site), and `` with your [Datadog API key](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys). **Note**: Python 3.9+ is required for installing Agent version 7+. {% tab title="Windows" %} ```powershell Set-AzVMExtension -Name "DatadogAgent" -Publisher "Datadog.Agent" -Type "DatadogWindowsAgent" -TypeHandlerVersion "7.0" -Settings @{"site" = ""; "agentVersion" = "latest"} -ProtectedSettings @{"api_key" = ""} -DisableAutoUpgradeMinorVersion ``` More information on the syntax to set Azure instance extensions can be found in the [Azure Extension Set-AzVMExtension documentation](https://learn.microsoft.com/powershell/module/az.compute/set-azvmextension). The Azure Extension can accept both normal settings and protected settings. The normal settings include: | Variable | Type | Description | | ---------------------------- | ------ | -------------------------------------------------------------------------------------------------------- | | `site` | String | Set the Datadog intake site, for example: `SITE=` | | `agentVersion` | String | The Agent version to install, following the format `x.y.z` or `latest` | | `agentConfiguration` | URI | (optional) URI to the Azure blob containing the Agent configuration as a ZIP file. | | `agentConfigurationChecksum` | String | The SHA256 checksum of the Agent configuration ZIP file, mandatory if `agentConfiguration` is specified. | The protected settings include: | Variable | Type | Description | | --------- | ------ | --------------------------------------------------- | | `api_key` | String | Adds the Datadog API KEY to the configuration file. | **Note**: If `agentConfiguration` and `api_key` are specified at the same time, the API key found in the `agentConfiguration` takes precedence. Also note that if an API key is set on the target machine, it's not possible to change it with `Set-AzVMExtension`. ### Specify a configuration URI{% #specify-a-configuration-uri %} This example shows how to specify a configuration for the Datadog Agent to use. The Datadog Agent configuration URI must be an Azure blob storage URI. The Datadog Windows Agent Azure Extension checks that the `agentConfiguration` URI comes from the `.blob.core.windows.net` domain. The Datataog Agent configuration should be created from the `%PROGRAMDATA%\Datadog` folder (see the [sample `config_template.yaml` file](https://github.com/DataDog/datadog-agent/blob/master/pkg/config/config_template.yaml) for all available configuration options). {% alert level="info" %} To reuse the configuration of an existing Agent: 1. Remove any extra installation artifacts or files from the existing Agent's `%ProgramData%\Datadog` folder so that it only contains the `datadog.yaml` file and `conf.d` folder with your integration configurations. 1. Save the sanitized `%ProgramData%\Datadog` folder as a ZIP file. {% /alert %} #### Generate a SHA256 checksum from the Agent configuration file{% #generate-a-sha256-checksum-from-the-agent-configuration-file %} ```powershell Get-FileHash -Path "C:\path\to\yourfile.zip" -Algorithm SHA256 ``` #### Install the Agent with your custom configuration{% #install-the-agent-with-your-custom-configuration %} Reference the blob storage URL with the `agentConfiguration` parameter and the checksum with the `agentConfigurationChecksum` parameter to create the VM extension. ```powershell Set-AzVMExtension -Name "DatadogAgent" -Publisher "Datadog.Agent" -Type "DatadogWindowsAgent" -TypeHandlerVersion "7.0" -Settings @{"site" = ""; "agentConfiguration" = "https://.blob.core.windows.net/.zip"; "agentConfigurationChecksum" = ""} -DisableAutoUpgradeMinorVersion ``` **Note**: After the Datadog Agent is installed, the configuration can only be changed when upgrading to a newer version. ### Set a specific version of the Agent{% #set-a-specific-version-of-the-agent %} This example shows how to specify a version of the Agent to install. By default the Datadog Windows Agent Azure Extension installs the latest version of the Datadog Agent. **Note**: Downgrades are *not* supported, so it's not possible to install a *lower* version of the Datadog Agent than the one currently installed on the target machine. To install a lower version of the Datadog Agent, uninstall the previous version first by removing the Datadog Windows Agent Azure Extension on the target machine. Removing the Datadog Windows Agent Azure Extension does not remove the Datadog Agent configuration. ```powershell Set-AzVMExtension -Name "DatadogAgent" -Publisher "Datadog.Agent" -Type "DatadogWindowsAgent" -TypeHandlerVersion "7.0" -Settings @{"site" = ""; "agentVersion" = "latest"} -ProtectedSettings @{"api_key" = ""} -DisableAutoUpgradeMinorVersion ``` {% /tab %} {% tab title="Linux" %} ```shell az vm extension set --publisher "Datadog.Agent" --name "DatadogLinuxAgent" --version 7.0 --settings '{"site":"datadoghq.com", "agentVersion":"latest"}' --protected-settings '{"api_key":""}' --no-auto-upgrade-minor-version ``` More information on the syntax to set Azure instance extensions can be found in the [Azure Extension CLI reference](https://learn.microsoft.com/cli/azure/vm/extension). The Azure Extension can accept both normal settings and protected settings. The normal settings include: | Variable | Type | Description | | ---------------------------- | ------ | -------------------------------------------------------------------------------------------------------- | | `site` | String | Set the Datadog intake site, for example: `SITE=` | | `agentVersion` | String | The Agent version to install, following the format `x.y.z` or `latest` | | `agentConfiguration` | URI | (optional) URI to the Azure blob containing the Agent configuration as a zip file. | | `agentConfigurationChecksum` | String | The SHA256 checksum of the Agent configuration ZIP file, mandatory if `agentConfiguration` is specified. | The protected settings include: | Variable | Type | Description | | --------- | ------ | --------------------------------------------------- | | `api_key` | String | Adds the Datadog API KEY to the configuration file. | **Note**: If `agentConfiguration` and `api_key` are specified at the same time, the API key found in the `agentConfiguration` takes precedence. If an API key is set on the target machine, it's not possible to change it with the `api_key` setting. ### Specify a configuration URI{% #specify-a-configuration-uri %} This example shows how to specify a configuration for the Datadog Agent to use. - The Datadog Agent configuration URI must be an Azure blob storage URI. - The Datadog Linux Agent Azure Extension checks that the `agentConfiguration` URI comes from the `.blob.core.windows.net` domain. - The Datataog Agent configuration should be created from the `/etc/datadog-agent/` folder (see the [sample `config_template.yaml` file](https://github.com/DataDog/datadog-agent/blob/master/pkg/config/config_template.yaml) for all available configuration options). {% alert level="info" %} To reuse the configuration of an existing Agent by saving its `/etc/datadog-agent` folder as a ZIP file: `zip -r datadog_config.zip /etc/datadog-agent`. {% /alert %} #### Generate a SHA256 checksum from the Agent configuration file{% #generate-a-sha256-checksum-from-the-agent-configuration-file %} ```shell sha256sum .zip ``` #### Install the Agent with your custom configuration{% #install-the-agent-with-your-custom-configuration %} Reference the blob storage URL with the `agentConfiguration` parameter and the checksum with the `agentConfigurationChecksum` parameter to create the VM extension. ```shell az vm extension set --publisher "Datadog.Agent" --name "DatadogLinuxAgent" --version 7.0 --settings '{"site":"datadoghq.com", "agentVersion":"latest", "agentConfiguration":"https://.blob.core.windows.net/.zip", "agentConfigurationChecksum":""}' --protected-settings '{"api_key":""}' --no-auto-upgrade-minor-version ``` {% /tab %} ## Install on Azure Arc{% #install-on-azure-arc %} To run the Datadog Agent in your [Azure Arc](https://docs.datadoghq.com/integrations/azure_arc/) instances as an extension, use the command that matches your environment. {% tab title="Windows" %} ```powershell az connectedmachine extension create --name --machine-name -g --publisher Datadog.Agent --type DatadogWindowsAgent --location --settings '{"site":""}' --protected-settings '{"api_key":""}' ``` {% /tab %} {% tab title="Linux" %} ```bash az connectedmachine extension create --name --machine-name -g --publisher Datadog.Agent --type DatadogLinuxAgent --location --settings '{"site":""}' --protected-settings '{"api_key":""}' ``` {% /tab %} More information on the syntax to set Azure `connectedmachine` extensions can be found in the [az connectedmachine extension](https://learn.microsoft.com/cli/azure/connectedmachine/extension) page. {% /collapsible-section %} {% collapsible-section #azure-count-metrics %} #### Azure count metrics Datadog generates an additional metric for each resource monitored with the [Azure integration](https://docs.datadoghq.com/integrations/azure/): `azure.*.count`. For example, Azure Virtual Machines monitored by Datadog reports `azure.vm.count`. The `azure.*.count` metric is an improvement over `azure.*.status`, which is deprecated. ## Count metric{% #count-metric %} The `azure.*.count` metric provides two fundamental pieces of information: - The number of resources of that type. - The status of each resource as reported by Azure. The `azure.*.count` metric is created in the same namespace as the other metrics for that resource type, for example: `azure.network_loadbalancers.count`. It includes all of the same metadata tags as the other metrics in that namespace, plus as additional tag for `status`. ### Use cases{% #use-cases %} Use the `azure.*.count` metric to: - Create a view of the number of Virtual Machines broken out by their status over time by graphing `azure.vm.count` over everything and summing by `status`. - Create query widgets in dashboards to display the number of a given resource type. Use any available tags to scope the count to a relevant aggregation such as region, resource group, kind, or status. - Create monitors to alert you about the status of different Azure resources. **Note**: In some cases, the default visualization settings can make it appear as though resources are being double counted intermittently in charts or query widgets. This does not affect monitors or widgets scoped to a specific status. You can reduce this effect by turning off [interpolation](https://docs.datadoghq.com/metrics/guide/interpolation-the-fill-modifier-explained/) in charts or query widgets by setting Interpolation > none or using `.fill(null)`. For most resource types, the possible statuses are: - Running - Unavailable - Unknown - Degraded - Failed Virtual machines have more detailed statuses, including: - Running - Stopped_deallocated - Stopped - Unknown - Unavailable - Degraded - Failed If you see a status of `query_failed` you need to enable the Resource Health provider in Azure. {% /collapsible-section %} ## Azure integration troubleshooting{% #azure-integration-troubleshooting %} {% collapsible-section #troubleshooting-find-your-tenant-name %} #### Find your tenant name 1. Navigate to [portal.azure.com](https://portal.azure.com). 1. In the left sidebar, select **Azure Active Directory**. 1. Under **Basic information**, find the **Name** value. {% /collapsible-section %} {% collapsible-section #troubleshooting-unable-to-login %} #### Unable to log in If you experience an error logging in while trying to install the Azure integration, contact [Datadog support](https://docs.datadoghq.com/help/). When possible, attach a screenshot. {% /collapsible-section %} {% collapsible-section #troubleshooting-missing-metrics %} #### Missing metrics Ensure you completed the installation process, which includes giving read permissions to the Azure application for the subscriptions you want to monitor. For ARM deployed virtual machines, you must also turn on Diagnostics and select the VM metrics you would like to collect. See **Enable Diagnostics** below for instructions. For other missing metrics, contact [Datadog support](https://docs.datadoghq.com/help/) with the following information about the metric: - dimensions - resource group - resource name - subscription ID or subscription name Attach a screenshot of a graph from Azure Monitor that shows a graph of the metric. **Important**: Graph 1-minute datapoints in the screenshot. #### Enable diagnostics{% #enable-diagnostics %} Turning on Diagnostics allows ARM deployed VMs to collect logging information which includes metrics for CPU, Network, etc. Follow these instructions: 1. Navigate to the [Azure portal](https://portal.azure.com) and locate your VM. 1. Click on **Diagnostics settings** under the **Monitoring** section. 1. Pick a storage account and click **Enable guest-level monitoring**. 1. By default, basic metrics and logs are enabled. Adjust based on your preferences. 1. Click **Save** to save any changes. {% image source="https://datadog-docs.imgix.net/images/integrations/guide/azure_troubleshooting/azure_enable_diagnostics.1ef26274bd5f43993bb0605dab29d190.png?auto=format" alt="azure diagnostics settings overview displayed with No storage account highlighted under Pick a storage account and enable guest level monitoring enabled" /%} {% /collapsible-section %} {% collapsible-section #troubleshooting-metric-discrepancy %} #### Discrepancy between your data in Azure and Datadog Datadog's Azure integration collects all metrics from [Azure Monitor](https://learn.microsoft.com/azure/azure-monitor/reference/metrics-index). Metrics are collected with all available dimensions (which are mapped to tags in Datadog), the highest time granularity, and the primary aggregation type. The sections below describe two important distinctions to be aware of, as well as steps to reconcile the discrepancy. #### 1. Time aggregation Datadog displays raw data from Azure in per-second values, regardless of the time frame selected in Azure. That can make Datadog's value appear lower than the value Azure displays. See [Time aggregation](https://docs.datadoghq.com/metrics/#time-aggregation) in the metric documentation for more information. #### 2. Space aggregation [Space aggregation](https://docs.datadoghq.com/metrics/#space-aggregation) in Datadog corresponds to the [primary aggregation type](https://learn.microsoft.com/azure/azure-monitor/metrics/metrics-aggregation-explained#aggregation-types) of the metric in Azure Monitor. You can find the primary aggregation type from Azure's [Metric Definitions - List](https://learn.microsoft.com/rest/api/monitor/metric-definitions/list?view=rest-monitor-2023-10-01) API, in the field `primaryAggregationType`. #### Reconcile the discrepancy{% #reconcile-the-discrepancy %} 1. Graph the metric in [Azure Monitor Metrics Explorer](https://portal.azure.com/#view/Microsoft_Azure_Monitoring/AzureMonitoringBrowseBlade/~/metrics), or by going to the resource in Azure and clicking **Monitoring** and then **Metrics** in the left panel. 1. Graph the metric in the [Datadog Metrics Explorer](https://app.datadoghq.com/metric/explorer). 1. Confirm that the query in Azure is scoped identically to the query in Datadog: - Any dimensions used in the Azure metric query should match tags used in the Datadog metric query - The primary aggregation type used in the query should match the Datadog [space aggregator](https://docs.datadoghq.com/metrics/#space-aggregation) - The time frame should match the time frame in the Datadog Metric Explorer 1. Hover over a datapoint on the graph to display the timestamp and value. {% image source="https://datadog-docs.imgix.net/images/integrations/guide/azure_advanced_configuration/azure_metric_explorer.6a0476ed86a5c65bdd14f2aaef3afda7.png?auto=format" alt="The Azure Metrics Explorer with the cursor hovering over a point in the graph, and the metric value and timestamp highlighted" /%} Find the same point in time in the Datadog graph and compare the values. If the values are equal, the original discrepancy was due to differences in either time or space aggregation between the two graphs. {% /collapsible-section %} ### Log collection troubleshooting{% #log-collection-troubleshooting %} To troubleshoot log forwarding issues, begin with the general troubleshooting section. If those steps do not resolve your issue, look at the following setup-specific guidance. - Automated - Blob Storage - Native integration - Event Hub #### General troubleshooting applicable to all setup methods{% #general-troubleshooting-applicable-to-all-setup-methods %} {% dl %} {% dt %} Exceeding the daily log quota: {% /dt %} {% dd %} Ensure that you have not exceeded your [daily quota](https://docs.datadoghq.com/logs/log_configuration/indexes/#set-daily-quota) for log retention. {% /dd %} {% dt %} Excluding logs with exclusion filters: {% /dt %} {% dd %} Ensure that your logs are not excluded by [exclusion filters](https://docs.datadoghq.com/logs/log_configuration/indexes/#exclusion-filters). {% /dd %} {% dt %} Incorrect syntax used to search logs: {% /dt %} {% dd %} Ensure that you used the correct [search syntax](https://docs.datadoghq.com/logs/explorer/search_syntax/) and that your search contains the correct attributes or tags. {% /dd %} {% dt %} Diagnostic setting not present: {% /dt %} {% dd %} Ensure that a [diagnostic setting](https://learn.microsoft.com/azure/azure-monitor/platform/diagnostic-settings?tabs=portal) has been added to the resources, Activity Logs, and Entra ID logs you want to send to Datadog. {% /dd %} {% /dl %} **Note**: You can search for Entra ID logs with the query `source:azure.activedirectory`. #### Automated log forwarding troubleshooting{% #automated-log-forwarding-troubleshooting %} {% collapsible-section #troubleshooting-logs-naming-conflicts %} #### Check for naming conflicts If you have Azure resources with the same resource name as one of the default parameters, it can lead to naming conflicts. Azure does not allow resources to share resource names within an individual subscription. Datadog recommends renaming the default parameter with a unique name that does not already exist within your environment. **Note**: If you are rerunning the template due to this failure, Datadog recommends also removing the entire resource group to create a fresh deployment. {% /collapsible-section %} {% collapsible-section #troubleshooting-logs-unregistered-resource-provider %} #### Ensure the resource provider is registered If your template deployment is failing due to the error `The subscription is not registered to use namespace 'Microsoft.EventHub'`: Azure has resource providers for each of its services, such as `Microsoft.EventHub` for Azure Event Hubs. If your Azure subscription is not registered to a required resource provider, the script fails. Remediate this by registering with the appropriate resource provider. Run the example command below in CloudShell, substituting with the name of the resource provider (for example, `Microsoft.EventHub`). ```shell az provider register --namespace ``` {% /collapsible-section %} {% collapsible-section #troubleshooting-logs-check-site-and-api-key-alf %} #### Check DD_SITE and DD_API_KEY values If all logs are missing, verify your [Datadog site](https://docs.datadoghq.com/getting_started/site/#access-the-datadog-site) and API key: 1. In the resource group where the automated log forwarding resources were deployed, go to **Settings > Deployments**. 1. Select the most recent deployment. 1. Click **Redeploy** to check and modify the values. {% /collapsible-section %} {% collapsible-section #troubleshooting-logs-verify-logs-in-storage-account %} #### Verify logs are present in the storage account container For missing Azure resource logs, find the ARM-deployed storage account in the resource's region and check the container with the expected logs. {% /collapsible-section %} {% collapsible-section #troubleshooting-logs-inspect-forwarder-job-logs %} #### Inspect the containerAppConsole logs of the forwarder jobs [Container App Console logs](https://learn.microsoft.com/azure/azure-monitor/reference/tables/containerappconsolelogs) help you diagnose application errors and exceptions. To inspect the logs, enable logging within the Container Apps environment in the same region as your resources that are missing logs. {% /collapsible-section %} **Note**: When using the automated log forwarder to send logs, you must manually set Microsoft Entra diagnostic settings. See [Configure Microsoft Entra diagnostic settings for activity logs](https://learn.microsoft.com/entra/identity/monitoring-health/howto-configure-diagnostic-settings) for more information. #### Blob Storage log forwarding troubleshooting{% #blob-storage-log-forwarding-troubleshooting %} {% collapsible-section #troubleshooting-logs-check-site-and-api-key-bs %} #### Check DD_SITE and DD_API_KEY values If you are missing all logs, ensure that: - The [selected Datadog site](https://docs.datadoghq.com/getting_started/site/#access-the-datadog-site) is correct - The configured [API key](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys) belongs to the correct Datadog organization {% /collapsible-section %} {% collapsible-section #troubleshooting-logs-verify-forwarder-config %} #### Verify the forwarder config Ensure that the forwarder is using the latest version, running Node.js 22 LTS or later on Windows OS. The code is publicly available at [index.js](https://github.com/DataDog/datadog-serverless-functions/blob/master/azure/blobs_logs_monitoring/index.js) {% /collapsible-section %} {% collapsible-section #troubleshooting-logs-inspect-blob-trigger %} #### Inspect the Blob Trigger Verify the Blob Storage trigger is configured with the following options: - **Binding Type**: `Azure Blob Storage` - **Blob parameter name**: `blobContent` - **Path**: The path to your log data within the storage account - **Storage account connection**: The name of the application setting containing the storage account connection string {% /collapsible-section %} {% collapsible-section #troubleshooting-logs-inspect-blob-metrics %} #### Inspect Blob Storage metrics {% dl %} {% dt %} Ensure that Blob Storage is receiving data: {% /dt %} {% dd %} `azure.storage_storageaccounts_blobservices.ingress` {% /dd %} {% dt %} Ensure that Blob Storage is sending data: {% /dt %} {% dd %} `azure.storage_storageaccounts_blobservices.egress` {% /dd %} {% dt %} Ensure that requests to Blob Storage are successful: {% /dt %} {% dd %} `azure.storage_storageaccounts_blobservices.transactions` {% /dd %} {% /dl %} For the `azure.storage_storageaccounts_blobservices.transactions` metric, you can use the following metric queries to view the number of successful or failed `PUT` operations, respectively: - `azure.storage_storageaccounts_blobservices.transactions{responsetype:success , apiname:put*} by {apiname}` - `azure.storage_storageaccounts_blobservices.transactions{!responsetype:success , apiname:put*} by {responsetype}` {% /collapsible-section %} {% collapsible-section #troubleshooting-logs-inspect-function-metrics-bs %} #### Inspect function metrics Check if the function app is encountering execution issues, data receipt or transmission issues, or error status codes by inspecting the following metrics: {% dl %} {% dt %} Ensure that the function is receiving data: {% /dt %} {% dd %} `azure.functions.bytes_received` {% /dd %} {% dt %} Ensure that the function is sending data: {% /dt %} {% dd %} `azure.functions.bytes_sent` {% /dd %} {% dt %} Ensure that the function's requests are receiving successful status codes: {% /dt %} {% dd %} `azure.functions.http2xx` {% /dd %} {% dt %} Ensure that the function's requests are not receiving error status codes: {% /dt %} {% dd %} `azure.functions.http4xx` {% /dd %} {% dd %} `azure.functions.http5xx` {% /dd %} {% /dl %} {% /collapsible-section %} {% collapsible-section #troubleshooting-logs-diagnostic-settings-not-added %} #### Diagnostic settings not automatically added to resources or Activity Logs If a diagnostic setting has not been added automatically to your resources or Activity Logs, check if: - The [maximum number of diagnostic settings](https://learn.microsoft.com/troubleshoot/azure/partner-solutions/log-limitations#check-diagnostic-settings-limit) has been reached - A lock exists on the resources that prevents modifications {% /collapsible-section %} **Note**: When new logs are added to a Blob Storage file, the function is triggered to send the full contents of the file, not only the new additions. #### Azure Native integration log forwarding troubleshooting{% #azure-native-integration-log-forwarding-troubleshooting %} {% collapsible-section #troubleshooting-logs-check-resource-logs-tag-rules %} #### Check resource logs tag rules Check that any tag rules in the Datadog resource configuration match the tags set on your resource logs. If there are no issues with the tag rules, contact [Datadog Support](https://docs.datadoghq.com/help/) and share the following information: - **Tenant ID**: Find the Tenant ID value nn Entra ID, under **Basic information** - **Subscription ID**: The subscription ID of any subscriptions that logs are missing from - **Datadog Resource ID**: Find the Datadog Resource ID by going to **Settings > Properties** on the Datadog resource blade. The Resource ID uses this format: `/subscriptions//resourceGroups/myresourcegroup/providers/Microsoft.Datadog/monitors/ with the name of the resource provider (for example, `Microsoft.EventHub`). ```shell az provider register --namespace ``` {% /collapsible-section %} {% collapsible-section #troubleshooting-logs-check-site-and-api-key-eh %} #### Check DD_SITE and DD_API_KEY values If you are missing all logs, ensure that: - The [selected Datadog site](https://docs.datadoghq.com/getting_started/site/#access-the-datadog-site) is correct - The configured [API key](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys) belongs to the correct Datadog organization **Note**: if the Datadog site and API key have been set in the function app settings, your app may restart if they are modified. {% /collapsible-section %} {% collapsible-section #troubleshooting-logs-check-event-hub-bottlenecks %} #### Identify potential Event Hub bottlenecks A spike in incoming messages with a drop in outgoing ones suggests a bottleneck. Use these metrics to investigate: {% dl %} {% dt %} Check for incoming messages: {% /dt %} {% dd %} `azure.eventhub_namespaces.incoming_messages` {% /dd %} {% dt %} Check for incoming data: {% /dt %} {% dd %} `azure.eventhub_namespaces.incoming_bytes` {% /dd %} {% dt %} Check for outgoing messages: {% /dt %} {% dd %} `azure.eventhub_namespaces.outgoing_messages` {% /dd %} {% dt %} Check for outgoing data: {% /dt %} {% dd %} `azure.eventhub_namespaces.outgoing_bytes` {% /dd %} {% dt %} Check for Microsoft.EventHub throttled requests: {% /dt %} {% dd %} `azure.eventhub_namespaces.throttled_requests` {% /dd %} {% dt %} Check for Microsoft.EventHub server errors: {% /dt %} {% dd %} `azure.eventhub_namespaces.server_errors` {% /dd %} {% /dl %} **Note**: Increasing log delays can cause data loss, as [logs older than 18 hours](https://docs.datadoghq.com/logs/troubleshooting/#missing-logs---timestamp-outside-of-the-ingestion-window) are dropped. To remediate the bottleneck, you can either: - [Scale up your Event Hub partitions](https://learn.microsoft.com/azure/event-hubs/event-hubs-scalability) to handle higher throughput (maximum 32). Because in-place scaling is only available for Premium or Dedicated tiers, you must create a new Event Hub with the desired partition count and then reconfigure its diagnostic settings and forwarder trigger. Consult [Azure support](https://azure.microsoft.com/support) for scaling advice. - Split the log forwarder into multiple pipelines in Datadog, with each one processing a specific subset of resources based on rules such as priority. This increases the number of logs that can be handled simultaneously. {% /collapsible-section %} {% collapsible-section #inspect-function-metrics-eh %} #### Inspect function metrics Ensure the function app is executing by looking at the following function metrics: {% dl %} {% dt %} Check the function execution count: {% /dt %} {% dd %} `azure.functions.function_execution_count` {% /dd %} {% dt %} Check that the function is receiving data: {% /dt %} {% dd %} `azure.functions.bytes_received` {% /dd %} {% dt %} Check that the function is receiving data: {% /dt %} {% dd %} `azure.functions.bytes_sent` {% /dd %} {% dt %} Check that the function's requests are returning successful status codes: {% /dt %} {% dd %} `azure.functions.http2xx` {% /dd %} {% dt %} Check that the function's requests are not returning error status codes: {% /dt %} {% dd %} `azure.functions.http4xx` {% /dd %} {% dd %} `azure.functions.http5xx` {% /dd %} {% /dl %} **Notes**: - The log forwarder uses the [Azure Functions V4 Programming model](https://techcommunity.microsoft.com/blog/appsonazureblog/azure-functions-node-js-v4-programming-model-is-generally-available/3929217), which is package-based and disables direct code editing in the Azure portal. To update or customize a function that was previously set up manually, contact [Datadog support](https://docs.datadoghq.com/help/) for instructions. - For more reliable, efficient, and cost-effective log collection, transition to the [Automated Log Forwarder](https://www.datadoghq.com/blog/azure-log-forwarding/). It fully automates the process of forwarding logs from all your Azure resources to Datadog. {% /collapsible-section %} ## Monitoring multiple app registrations{% #monitoring-multiple-app-registrations %} Subscriptions monitored by multiple app registrations can introduce overlapping access configurations. This setup is not recommended and may result in integration issues or system conflicts, and may also increase your Azure Monitor costs. ## Powered-down Azure VMs on the infrastructure list{% #powered-down-azure-vms-on-the-infrastructure-list %} When you power down your VMs in Azure, the Datadog Azure integration still collects the metric `azure.vm.status` for that VM. This metric is tagged with `status:running`, `status:not_running`, or `status:unknown`. This is intended, but causes the VM to remain on your infrastructure list. If your VM reports only this metric, it does not count towards your billable host-count. For more billing information, see the Datadog [billing page](https://docs.datadoghq.com/account_management/billing/). If you destroy your Azure VM, it phases out of your infrastructure list within 3 hours. ## Azure VMs appear in the app without metrics{% #azure-vms-appear-in-the-app-without-metrics %} After properly installing the Azure Integration within Datadog, metrics from you Azure VMs and other services should begin to flow in about 15 minutes. If after this time you see Azure VMs in your infrastructure list but no metrics are being reported, a few things can be happening. 1. Make sure you are looking for the right metrics. **Classic** virtual machine metrics begin with the azure.vm namespace and ARM deployed virtual machine metrics begin with the `azure.compute_virtualmachines` namespace. 1. If neither of these namespaces are returning metrics, make sure **Diagnostics** is turned on for the Virtual Machines within the Azure Portal. Only Boot diagnostics and Basic metrics are required. - For **Classic** VMs: {% image source="https://datadog-docs.imgix.net/images/integrations/guide/azure_vms_appearing_in_the_app_without_metrics/classic_vm.573548d33d64b5697139ada6d3073819.png?auto=format" alt="The azure portal showing the diagnostics view of a classic virtual machine with status set to on" /%} - For ARM deployed VMs: {% image source="https://datadog-docs.imgix.net/images/integrations/guide/azure_vms_appearing_in_the_app_without_metrics/arm_deployed_vm.5658d7f9d67b2a1195e37678fa3b88e1.png?auto=format" alt="The azure portal showing the diagnostics settings view of a virtual machine with status set to on" /%} 1. Make sure the Virtual machine is running. The integration does not collect performance metrics for stopped or deallocated machines. Use the `azure.vm.count` metric and the `status` tag values of `running`, `stopped`, and `stopped_deallocated` to determine the status of your hosts. Make sure the host in question has `status:running`, and is running in the Azure portal. {% image source="https://datadog-docs.imgix.net/images/integrations/guide/azure_vms_appearing_in_the_app_without_metrics/azure_vm_running_2025-05-02.2303da8c56448100a6038c6caa5c5fd6.png?auto=format" alt="A graph of the azure.vm.count metric from status:running" /%} ## Issues with `azure.*.count` metrics{% #issues-with-azurecount-metrics %} If your Azure integration is reporting metrics but not `azure.*.count`, or `azure.*.count` is returning `status:query_failed`, your Azure subscription needs to register the Azure Resource Health provider. Using the Azure Command Line Interface: ```bash azure login # Login to the Azure user associated with your Datadog account azure config mode arm azure provider register Microsoft.ResourceHealth ``` The `azure.*.count` metric should show in Datadog within 5 - 10 minutes. ## Further reading{% #further-reading %} - [Azure Integration](https://docs.datadoghq.com/integrations/azure/)