--- title: Azure Machine Learning Services description: >- Track workspace runs, node utilization, and online endpoint traffic and latency for Azure Machine Learning. breadcrumbs: Docs > Integrations > Azure Machine Learning Services --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # Azure Machine Learning Services {% callout %} # Important note for users on the following Datadog sites: us2.ddog-gov.com {% alert level="info" %} To find out if this integration is available in your organization, see your [Datadog Integrations](https://app.datadoghq.com/integrations) page or ask your organization administrator. To initiate an exception request to enable this integration for your organization, email [support@ddog-gov.com](mailto:support@ddog-gov.com). {% /alert %} {% /callout %} ## Overview{% #overview %} The Azure Machine Learning service empowers developers and data scientists with a wide range of productive experiences for building, training, and deploying machine learning models faster. Use Datadog to monitor your Azure Machine Learning performance and utilization in context with the rest of your applications and infrastructure. Get metrics from Azure Machine Learning to: - Track the number and status of runs and model deployments. - Monitor the utilization of your machine learning nodes. - Track request traffic and latency for online endpoints and their deployments. - Optimize performance vs. cost. ## Setup{% #setup %} ### Installation{% #installation %} If you haven't already, set up the [Microsoft Azure integration](https://docs.datadoghq.com/integrations/azure.md) first. There are no other installation steps. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **azure.machinelearningservices_workspaces.count**(gauge) | The count of Azure Machine Learning workspaces. | | **azure.machinelearningservices_workspaces_onlineendpoints.count**(gauge) | The count of Azure Machine Learning online endpoints. | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.count**(gauge) | The count of Azure Machine Learning online endpoint deployments. | | **azure.machinelearningservices_workspaces.completed_runs**(count) | The number of runs completed successfully for this workspace.*Shown as operation* | | **azure.machinelearningservices_workspaces.started_runs**(count) | The number of runs started for this workspace.*Shown as operation* | | **azure.machinelearningservices_workspaces.failed_runs**(count) | The number of runs failed for this workspace.*Shown as operation* | | **azure.machinelearningservices_workspaces.model_register_succeeded**(gauge) | The number of model registrations that succeeded in this workspace.*Shown as operation* | | **azure.machinelearningservices_workspaces.model_register_failed**(gauge) | The number of model registrations that failed in this workspace.*Shown as operation* | | **azure.machinelearningservices_workspaces.model_deploy_started**(gauge) | The number of model deployments started in this workspace.*Shown as operation* | | **azure.machinelearningservices_workspaces.model_deploy_succeeded**(gauge) | The number of model deployments that succeeded in this workspace.*Shown as operation* | | **azure.machinelearningservices_workspaces.model_deploy_failed**(gauge) | The number of model deployments that failed in this workspace.*Shown as operation* | | **azure.machinelearningservices_workspaces.total_nodes**(gauge) | The number of total nodes. This total includes some of Active Nodes, Idle Nodes, Unusable Nodes, Premepted Nodes, Leaving Nodes.*Shown as node* | | **azure.machinelearningservices_workspaces.active_nodes**(gauge) | The number of Active nodes. These are the nodes which are actively running a job.*Shown as node* | | **azure.machinelearningservices_workspaces.idle_nodes**(gauge) | The number of idle nodes. Idle nodes are the nodes which are not running any jobs but can accept new jobs if available.*Shown as node* | | **azure.machinelearningservices_workspaces.unusable_nodes**(gauge) | The number of unusable nodes. Unusable nodes are not functional due to some unresolvable issue. Azure will recycle these nodes.*Shown as node* | | **azure.machinelearningservices_workspaces.preempted_nodes**(gauge) | The number of preempted nodes. These nodes are the low priority nodes which are taken away from the available node pool.*Shown as node* | | **azure.machinelearningservices_workspaces.leaving_nodes**(gauge) | The number of leaving nodes. Leaving nodes are the nodes which just finished processing a job and will go to Idle state.*Shown as node* | | **azure.machinelearningservices_workspaces.total_cores**(gauge) | The number of total cores.*Shown as core* | | **azure.machinelearningservices_workspaces.active_cores**(gauge) | The number of active cores.*Shown as core* | | **azure.machinelearningservices_workspaces.idle_cores**(gauge) | The number of idle cores.*Shown as core* | | **azure.machinelearningservices_workspaces.unusable_cores**(gauge) | The number of unusable cores.*Shown as core* | | **azure.machinelearningservices_workspaces.preempted_cores**(gauge) | The number of preempted cores.*Shown as core* | | **azure.machinelearningservices_workspaces.leaving_cores**(gauge) | The number of leaving cores.*Shown as core* | | **azure.machinelearningservices_workspaces.quota_utilization_percentage**(gauge) | The percent of quota utilized.*Shown as percent* | | **azure.machinelearningservices_workspaces.agents**(gauge) | Number of events for AI agents in this workspace. | | **azure.machinelearningservices_workspaces.indexed_files**(gauge) | Number of files indexed for file search in this workspace. | | **azure.machinelearningservices_workspaces.messages**(gauge) | Number of events for AI agent messages in this workspace. | | **azure.machinelearningservices_workspaces.runs**(gauge) | Number of runs by AI agents in this workspace. | | **azure.machinelearningservices_workspaces.threads**(gauge) | Number of events for AI agent threads in this workspace. | | **azure.machinelearningservices_workspaces.tokens**(gauge) | Count of tokens by AI agents in this workspace. | | **azure.machinelearningservices_workspaces.tool_calls**(gauge) | Tool calls made by AI agents in this workspace. | | **azure.machinelearningservices_workspaces.cpu_capacity_millicores**(gauge) | Maximum capacity of a CPU node in millicores. Capacity is aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.cpu_memory_capacity_megabytes**(gauge) | Maximum memory utilization of a CPU node in megabytes. Utilization is aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.cpu_memory_utilization_megabytes**(gauge) | Memory utilization of a CPU node in megabytes. Utilization is aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.cpu_memory_utilization_percentage**(gauge) | Memory utilization percentage of a CPU node. Utilization is aggregated in one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces.cpu_utilization**(gauge) | Percentage of utilization on a CPU node. Utilization is reported at one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces.cpu_utilization_millicores**(gauge) | Utilization of a CPU node in millicores. Utilization is aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.cpu_utilization_percentage**(gauge) | Utilization percentage of a CPU node. Utilization is aggregated in one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces.disk_avail_megabytes**(gauge) | Available disk space in megabytes. Metrics are aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.disk_read_megabytes**(gauge) | Data read from disk in megabytes. Metrics are aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.disk_used_megabytes**(gauge) | Used disk space in megabytes. Metrics are aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.disk_write_megabytes**(gauge) | Data written into disk in megabytes. Metrics are aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.gpu_capacity_milli_gp_us**(gauge) | Maximum capacity of a GPU device in milli-gpus. Capacity is aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.gpu_energy_joules**(gauge) | Interval energy in joules on a GPU node. Energy is reported at one minute intervals. | | **azure.machinelearningservices_workspaces.gpu_memory_capacity_megabytes**(gauge) | Maximum memory capacity of a GPU device in megabytes. Capacity aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.gpu_memory_utilization**(gauge) | Percentage of memory utilization on a GPU node. Utilization is reported at one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces.gpu_memory_utilization_megabytes**(gauge) | Memory utilization of a GPU device in megabytes. Utilization aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.gpu_memory_utilization_percentage**(gauge) | Memory utilization percentage of a GPU device. Utilization aggregated in one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces.gpu_utilization**(gauge) | Percentage of utilization on a GPU node. Utilization is reported at one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces.gpu_utilization_milli_gp_us**(gauge) | Utilization of a GPU device in milli-gpus. Utilization is aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.gpu_utilization_percentage**(gauge) | Utilization percentage of a GPU device. Utilization is aggregated in one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces.ib_receive_megabytes**(gauge) | Network data received over infiniband in megabytes. Metrics are aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.ib_transmit_megabytes**(gauge) | Network data sent over infiniband in megabytes. Metrics are aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.network_input_megabytes**(gauge) | Network data received in megabytes. Metrics are aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.network_output_megabytes**(gauge) | Network data sent in megabytes. Metrics are aggregated in one minute intervals. | | **azure.machinelearningservices_workspaces.storage_api_failure_count**(gauge) | Azure blob storage API calls failure count. | | **azure.machinelearningservices_workspaces.storage_api_success_count**(gauge) | Azure blob storage API calls success count. | | **azure.machinelearningservices_workspaces.cancel_requested_runs**(gauge) | Number of runs where cancel was requested for this workspace. Count is updated when cancellation request has been received for a run. | | **azure.machinelearningservices_workspaces.cancelled_runs**(gauge) | Number of runs cancelled for this workspace. Count is updated when a run is successfully cancelled. | | **azure.machinelearningservices_workspaces.errors**(gauge) | Number of run errors in this workspace. Count is updated whenever run encounters an error. | | **azure.machinelearningservices_workspaces.finalizing_runs**(gauge) | Number of runs entered finalizing state for this workspace. Count is updated when a run has completed but output collection still in progress. | | **azure.machinelearningservices_workspaces.not_responding_runs**(gauge) | Number of runs not responding for this workspace. Count is updated when a run enters not responding state. | | **azure.machinelearningservices_workspaces.not_started_runs**(gauge) | Number of runs in not started state for this workspace. Count is updated when a request is received to create a run but run information has not yet been populated. | | **azure.machinelearningservices_workspaces.preparing_runs**(gauge) | Number of runs that are preparing for this workspace. Count is updated when a run enters preparing state while the run environment is being prepared. | | **azure.machinelearningservices_workspaces.provisioning_runs**(gauge) | Number of runs that are provisioning for this workspace. Count is updated when a run is waiting on compute target creation or provisioning. | | **azure.machinelearningservices_workspaces.queued_runs**(count) | Number of runs that are queued for this workspace. Count is updated when a run is queued in compute target. Can occur when waiting for required compute nodes to be ready. | | **azure.machinelearningservices_workspaces.starting_runs**(gauge) | Number of runs started for this workspace. Count is updated after request to create run and run info, such as the run ID, has been populated. | | **azure.machinelearningservices_workspaces.warnings**(gauge) | Number of run warnings in this workspace. Count is updated whenever a run encounters a warning. | | **azure.machinelearningservices_workspaces_onlineendpoints.connections_active**(gauge) | Total number of concurrent TCP connections active from clients.*Shown as connection* | | **azure.machinelearningservices_workspaces_onlineendpoints.data_collection_errors_per_minute**(gauge) | Number of data collection events dropped per minute.*Shown as event* | | **azure.machinelearningservices_workspaces_onlineendpoints.data_collection_events_per_minute**(gauge) | Number of data collection events processed per minute.*Shown as event* | | **azure.machinelearningservices_workspaces_onlineendpoints.network_bytes**(gauge) | Bytes per second served for the endpoint.*Shown as byte* | | **azure.machinelearningservices_workspaces_onlineendpoints.new_connections_per_second**(gauge) | Average number of new TCP connections per second established from clients.*Shown as connection* | | **azure.machinelearningservices_workspaces_onlineendpoints.request_latency**(gauge) | Average complete interval of time taken for a request to be responded in milliseconds.*Shown as millisecond* | | **azure.machinelearningservices_workspaces_onlineendpoints.request_latency_p50**(gauge) | Average P50 request latency aggregated by all request latency values collected over the selected time period.*Shown as millisecond* | | **azure.machinelearningservices_workspaces_onlineendpoints.request_latency_p90**(gauge) | Average P90 request latency aggregated by all request latency values collected over the selected time period.*Shown as millisecond* | | **azure.machinelearningservices_workspaces_onlineendpoints.request_latency_p95**(gauge) | Average P95 request latency aggregated by all request latency values collected over the selected time period.*Shown as millisecond* | | **azure.machinelearningservices_workspaces_onlineendpoints.request_latency_p99**(gauge) | Average P99 request latency aggregated by all request latency values collected over the selected time period.*Shown as millisecond* | | **azure.machinelearningservices_workspaces_onlineendpoints.requests_per_minute**(gauge) | Number of requests sent to online endpoint within a minute.*Shown as request* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.cpu_memory_utilization_percentage**(gauge) | Percentage of memory utilization on an instance. Utilization is reported at one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.cpu_utilization_percentage**(gauge) | Percentage of CPU utilization on an instance. Utilization is reported at one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.data_collection_errors_per_minute**(gauge) | Number of data collection events dropped per minute.*Shown as event* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.data_collection_events_per_minute**(gauge) | Number of data collection events processed per minute.*Shown as event* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.deployment_capacity**(gauge) | Number of instances in the deployment.*Shown as instance* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.disk_utilization**(gauge) | Percentage of disk utilization on an instance. Utilization is reported at one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.gpu_energy_joules**(gauge) | Interval energy in joules on a GPU node. Energy is reported at one minute intervals.*Shown as joule* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.gpu_memory_utilization_percentage**(gauge) | Percentage of GPU memory utilization on an instance. Utilization is reported at one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.gpu_utilization_percentage**(gauge) | Percentage of GPU utilization on an instance. Utilization is reported at one minute intervals.*Shown as percent* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.request_latency_p50**(gauge) | Average P50 request latency aggregated by all request latency values collected over the selected time period.*Shown as millisecond* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.request_latency_p90**(gauge) | Average P90 request latency aggregated by all request latency values collected over the selected time period.*Shown as millisecond* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.request_latency_p95**(gauge) | Average P95 request latency aggregated by all request latency values collected over the selected time period.*Shown as millisecond* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.request_latency_p99**(gauge) | Average P99 request latency aggregated by all request latency values collected over the selected time period.*Shown as millisecond* | | **azure.machinelearningservices_workspaces_onlineendpoints_deployments.requests_per_minute**(gauge) | Number of requests sent to online deployment within a minute.*Shown as request* | ### Events{% #events %} The Azure Machine Learning integration does not include any events. ### Service Checks{% #service-checks %} The Azure Machine Learning integration does not include any service checks. ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/). ## Further Reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Machine learning model monitoring: Best practices](https://www.datadoghq.com/blog/ml-model-monitoring-in-production-best-practices/)