--- title: vLLM description: vLLM is a library for LLM inference and serving breadcrumbs: Docs > Integrations > vLLM --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # vLLM Supported OS Integration version3.4.1 {% callout %} # Important note for users on the following Datadog sites: us2.ddog-gov.com {% alert level="info" %} To find out if this integration is available in your organization, see your [Datadog Integrations](https://app.datadoghq.com/integrations) page or ask your organization administrator. To initiate an exception request to enable this integration for your organization, email [support@ddog-gov.com](mailto:support@ddog-gov.com). {% /alert %} {% /callout %} ## Overview{% #overview %} This check monitors [vLLM](https://docs.vllm.ai/en/stable/) through the Datadog Agent. **Minimum Agent version:** 7.56.0 ## Setup{% #setup %} Follow the instructions below to install and configure this check for an Agent running on a host. ### Installation{% #installation %} The vLLM check is included in the [Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest) package. No additional installation is needed on your server. ### Configuration{% #configuration %} 1. Edit the `vllm.d/conf.yaml` file, in the `conf.d/` folder at the root of your Agent's configuration directory to start collecting your vllm performance data. See the [sample vllm.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/vllm/datadog_checks/vllm/data/conf.yaml.example) for all available configuration options. 1. [Restart the Agent](https://docs.datadoghq.com/agent/guide/agent-commands.md#start-stop-and-restart-the-agent). ### Validation{% #validation %} [Run the Agent's status subcommand](https://docs.datadoghq.com/agent/guide/agent-commands.md#agent-status-and-information) and look for `vllm` under the Checks section. ## Data Collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **vllm.avg.generation_throughput.toks_per_s**(gauge) | Average generation throughput in tokens/s | | **vllm.avg.prompt.throughput.toks_per_s**(gauge) | Average prefill throughput in tokens/s | | **vllm.cache_config_info**(gauge) | Information on cache config | | **vllm.cpu_cache_usage_perc**(gauge) | CPU KV-cache usage. 1 means 100 percent usage*Shown as percent* | | **vllm.e2e_request_latency.seconds.bucket**(count) | The observations of end to end request latency bucketed by seconds. | | **vllm.e2e_request_latency.seconds.count**(count) | The total number of observations of end to end request latency. | | **vllm.e2e_request_latency.seconds.sum**(count) | The sum of end to end request latency in seconds.*Shown as second* | | **vllm.estimated_flops_per_gpu.count**(count) | Estimated floating-point operations per GPU.*Shown as operation* | | **vllm.external_prefix_cache.hits.count**(count) | External prefix cache hits measured in cached tokens.*Shown as item* | | **vllm.external_prefix_cache.queries.count**(count) | External prefix cache queries measured in queried tokens.*Shown as item* | | **vllm.generation_tokens.count**(count) | Number of generation tokens processed. | | **vllm.gpu_cache_usage_perc**(gauge) | GPU KV-cache usage. 1 means 100 percent usage*Shown as percent* | | **vllm.num_preemptions.count**(count) | Cumulative number of preemption from the engine. | | **vllm.num_requests.running**(gauge) | Number of requests currently running on GPU. | | **vllm.num_requests.swapped**(gauge) | Number of requests swapped to CPU. | | **vllm.num_requests.waiting**(gauge) | Number of requests waiting. | | **vllm.process.cpu_seconds.count**(count) | Total user and system CPU time spent in seconds.*Shown as second* | | **vllm.process.max_fds**(gauge) | Maximum number of open file descriptors.*Shown as file* | | **vllm.process.open_fds**(gauge) | Number of open file descriptors.*Shown as file* | | **vllm.process.resident_memory_bytes**(gauge) | Resident memory size in bytes.*Shown as byte* | | **vllm.process.start_time_seconds**(gauge) | Start time of the process since unix epoch in seconds.*Shown as second* | | **vllm.process.virtual_memory_bytes**(gauge) | Virtual memory size in bytes.*Shown as byte* | | **vllm.prompt_tokens.cached.count**(count) | Number of cached prompt tokens from local and external caches.*Shown as item* | | **vllm.prompt_tokens.count**(count) | Number of prefill tokens processed. | | **vllm.python.gc.collections.count**(count) | Number of times this generation was collected | | **vllm.python.gc.objects.collected.count**(count) | Objects collected during gc | | **vllm.python.gc.objects.uncollectable.count**(count) | Uncollectable objects found during GC | | **vllm.python.info**(gauge) | Python platform information | | **vllm.request.decode_time.seconds.bucket**(count) | Request decode-time histogram buckets. | | **vllm.request.decode_time.seconds.count**(count) | Number of request decode-time observations. | | **vllm.request.decode_time.seconds.sum**(count) | Total request decode time.*Shown as second* | | **vllm.request.generation_tokens.bucket**(count) | Number of generation tokens processed. | | **vllm.request.generation_tokens.count**(count) | Number of generation tokens processed. | | **vllm.request.generation_tokens.sum**(count) | Number of generation tokens processed. | | **vllm.request.params.best_of.bucket**(count) | Histogram of the best_of request parameter. | | **vllm.request.params.best_of.count**(count) | Histogram of the best_of request parameter. | | **vllm.request.params.best_of.sum**(count) | Histogram of the best_of request parameter. | | **vllm.request.params.n.bucket**(count) | Histogram of the n request parameter. | | **vllm.request.params.n.count**(count) | Histogram of the n request parameter. | | **vllm.request.params.n.sum**(count) | Histogram of the n request parameter. | | **vllm.request.prefill_kv_computed_tokens.bucket**(count) | Histogram buckets for new KV tokens computed during prefill. | | **vllm.request.prefill_kv_computed_tokens.count**(count) | Number of prefill KV token observations. | | **vllm.request.prefill_kv_computed_tokens.sum**(count) | Total new KV tokens computed during prefill.*Shown as item* | | **vllm.request.prefill_time.seconds.bucket**(count) | Request prefill-time histogram buckets. | | **vllm.request.prefill_time.seconds.count**(count) | Number of request prefill-time observations. | | **vllm.request.prefill_time.seconds.sum**(count) | Total request prefill time.*Shown as second* | | **vllm.request.prompt_tokens.bucket**(count) | Number of prefill tokens processed. | | **vllm.request.prompt_tokens.count**(count) | Number of prefill tokens processed. | | **vllm.request.prompt_tokens.sum**(count) | Number of prefill tokens processed. | | **vllm.request.queue_time.seconds.bucket**(count) | Request queue-time histogram buckets. | | **vllm.request.queue_time.seconds.count**(count) | Number of request queue-time observations. | | **vllm.request.queue_time.seconds.sum**(count) | Total request queue time.*Shown as second* | | **vllm.request.success.count**(count) | Count of successfully processed requests. | | **vllm.spec_decode.num_accepted_tokens.count**(count) | Number of speculative decode tokens accepted.*Shown as item* | | **vllm.spec_decode.num_accepted_tokens_per_pos.count**(count) | Number of accepted speculative decode tokens by draft position.*Shown as item* | | **vllm.spec_decode.num_draft_tokens.count**(count) | Number of speculative decode draft tokens.*Shown as item* | | **vllm.time_per_output_token.seconds.bucket**(count) | The observations of time per output token bucketed by seconds. | | **vllm.time_per_output_token.seconds.count**(count) | The total number of observations of time per output token. | | **vllm.time_per_output_token.seconds.sum**(count) | The sum of time per output token in seconds.*Shown as second* | | **vllm.time_to_first_token.seconds.bucket**(count) | The observations of time to first token bucketed by seconds. | | **vllm.time_to_first_token.seconds.count**(count) | The total number of observations of time to first token. | | **vllm.time_to_first_token.seconds.sum**(count) | The sum of time to first token in seconds.*Shown as second* | The following metrics are collected only when [GPU Monitoring](https://docs.datadoghq.com/gpu_monitoring.md) is enabled: - `vllm.estimated_flops_per_gpu` - `vllm.external_prefix_cache` - `vllm.prompt_tokens.cached` - `vllm.request.decode_time` - `vllm.request.prefill_kv_computed_tokens` - `vllm.request.prefill_time` - `vllm.request.queue_time` - `vllm.spec_decode` ### Events{% #events %} The vLLM integration does not include any events. ### Service Checks{% #service-checks %} The vLLM integration does not include any service checks. **vllm.openmetrics.health** Returns `CRITICAL` if the Agent is unable to connect to the vLLM OpenMetrics endpoint, otherwise returns `OK`. *Statuses: ok, critical* ### Logs{% #logs %} Log collection is disabled by default in the Datadog Agent. If you are running your Agent as a container, see [container installation](https://docs.datadoghq.com/containers/docker/log.md?tab=containerinstallation#installation) to enable log collection. If you are running a host Agent, see [host Agent](https://docs.datadoghq.com/containers/docker/log.md?tab=hostagent#installation) instead. In either case, make sure that the `source` value for your logs is `vllm`. This setting ensures that the built-in processing pipeline finds your logs. To set your log configuration for a container, see [log integrations](https://docs.datadoghq.com/containers/docker/log.md?tab=dockerfile#log-integrations). ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/). ## Further Reading{% #further-reading %} Additional helpful documentation, links, and articles: - [Optimize LLM application performance with Datadog's vLLM integration](https://www.datadoghq.com/blog/vllm-integration/)