--- title: SGLang description: Monitor SGLang request performance, token throughput, and KV cache efficiency breadcrumbs: Docs > Integrations > SGLang --- > For the complete documentation index, see [llms.txt](https://docs.datadoghq.com/llms.txt). # SGLang Supported OS Integration version1.0.0 {% callout %} # Important note for users on the following Datadog sites: us2.ddog-gov.com {% alert level="info" %} To find out if this integration is available in your organization, see your [Datadog Integrations](https://app.datadoghq.com/integrations) page or ask your organization administrator. To initiate an exception request to enable this integration for your organization, email [support@ddog-gov.com](mailto:support@ddog-gov.com). {% /alert %} {% /callout %} ## Overview{% #overview %} [SGLang](https://docs.sglang.ai/) is a high-performance serving framework for large language models and multimodal models. This check collects Prometheus metrics from an SGLang server for: - Request and token throughput - Time to first token, inter-token latency, and end-to-end latency - Queue depth - KV cache utilization and hit rate - Speculative decoding efficiency - Process resource usage Histogram buckets are submitted as Datadog distributions by default, so percentile aggregations are available for request latency and token-count metrics. This integration is part of [GPU Monitoring](https://docs.datadoghq.com/gpu_monitoring.md) and only runs when GPU Monitoring is enabled. **Minimum Agent version:** 7.85.0 ## Setup{% #setup %} Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery integration templates](https://docs.datadoghq.com/containers/kubernetes/integrations.md). ### Prerequisites{% #prerequisites %} Enable metrics when starting the SGLang server: ```shell sglang serve --model-path --enable-metrics ``` SGLang exposes metrics at `http://localhost:30000/metrics` by default. Enable GPU Monitoring in the Datadog Agent configuration: ```yaml gpu: enabled: true ``` The equivalent environment variable is `DD_GPU_ENABLED=true`. When GPU Monitoring is disabled, the Agent skips every configured SGLang instance. ### Installation{% #installation %} The SGLang check is included in the [Datadog Agent](https://app.datadoghq.com/account/settings/agent/latest) package. No additional installation is needed. ### Configuration{% #configuration %} 1. Edit `sglang.d/conf.yaml` in the `conf.d/` folder at the root of the Agent configuration directory. Set `openmetrics_endpoint` to the SGLang server's `/metrics` endpoint. See the [sample sglang.d/conf.yaml](https://github.com/DataDog/integrations-core/blob/master/sglang/datadog_checks/sglang/data/conf.yaml.example) for all available configuration options. 1. [Restart the Agent](https://docs.datadoghq.com/agent/guide/agent-commands.md#start-stop-and-restart-the-agent). ### Validation{% #validation %} Run the [Agent status command](https://docs.datadoghq.com/agent/guide/agent-commands.md#agent-status-and-information) and look for `sglang` under the Checks section. ## Data collected{% #data-collected %} ### Metrics{% #metrics %} | | | | | **sglang.cache_config_info**(gauge) | Cache configuration information. | | **sglang.cached_tokens.count**(count) | Total number of cached prompt tokens.*Shown as token* | | **sglang.cuda_graph.active**(gauge) | Whether CUDA graph execution is active. | | **sglang.cuda_graph.passes.count**(count) | Total number of CUDA graph execution passes. | | **sglang.decode.sequence_length_sum**(gauge) | Sum of sequence lengths currently being decoded.*Shown as token* | | **sglang.disaggregation.bootstrap_failures.count**(count) | Total number of failed bootstrap requests. | | **sglang.disaggregation.decode_preallocation_queue**(gauge) | Requests waiting for decode preallocation. | | **sglang.disaggregation.decode_transfer_queue**(gauge) | Requests waiting for decode transfer. | | **sglang.disaggregation.kv_transfer.allocation_milliseconds**(gauge) | Time spent allocating memory for KV transfers.*Shown as millisecond* | | **sglang.disaggregation.kv_transfer.bootstrap_milliseconds**(gauge) | Time spent bootstrapping KV transfers.*Shown as millisecond* | | **sglang.disaggregation.kv_transfer.gigabytes_per_second**(gauge) | KV cache transfer throughput.*Shown as gigabyte* | | **sglang.disaggregation.kv_transfer.latency_milliseconds**(gauge) | KV cache transfer latency.*Shown as millisecond* | | **sglang.disaggregation.kv_transfer.megabytes**(gauge) | KV cache data transferred.*Shown as megabyte* | | **sglang.disaggregation.prefill_bootstrap_queue**(gauge) | Requests in the prefill bootstrap queue. | | **sglang.disaggregation.prefill_inflight_queue**(gauge) | Requests in the prefill inflight queue. | | **sglang.disaggregation.prefill_preallocation_queue**(gauge) | Requests waiting for prefill preallocation. | | **sglang.disaggregation.prefill_retries.count**(count) | Total number of retried prefill requests. | | **sglang.disaggregation.transfer_failures.count**(count) | Total number of failed KV transfer requests. | | **sglang.e2e_request_latency.seconds**(gauge) | End-to-end request latency (distribution).*Shown as second* | | **sglang.e2e_request_latency.seconds.count**(count) | End-to-end request latency observation count. | | **sglang.e2e_request_latency.seconds.sum**(count) | Sum of end-to-end request latency.*Shown as second* | | **sglang.estimated_flops_per_gpu.count**(count) | Estimated floating-point operations per GPU.*Shown as operation* | | **sglang.estimated_read_bytes_per_gpu.count**(count) | Estimated bytes read per GPU.*Shown as byte* | | **sglang.estimated_write_bytes_per_gpu.count**(count) | Estimated bytes written per GPU.*Shown as byte* | | **sglang.forward_execution.seconds.count**(count) | Total model forward execution time.*Shown as second* | | **sglang.generation_throughput.tokens_per_second**(gauge) | Generation throughput.*Shown as token* | | **sglang.generation_tokens.count**(count) | Total number of generation tokens processed.*Shown as token* | | **sglang.http.requests.active**(gauge) | Currently active HTTP requests. | | **sglang.http.requests.count**(count) | Total number of HTTP requests.*Shown as request* | | **sglang.http.responses.count**(count) | Total number of HTTP responses.*Shown as response* | | **sglang.inter_token_latency.seconds**(gauge) | Inter-token latency (distribution).*Shown as second* | | **sglang.inter_token_latency.seconds.count**(count) | Inter-token latency observation count. | | **sglang.inter_token_latency.seconds.sum**(count) | Sum of inter-token latency.*Shown as second* | | **sglang.kv_cache.full_attention_usage**(gauge) | Full-attention KV cache usage as a fraction.*Shown as fraction* | | **sglang.kv_cache.hit_rate**(gauge) | Prefix KV cache hit rate.*Shown as fraction* | | **sglang.kv_cache.max_tokens**(gauge) | Maximum number of tokens in the KV cache pool.*Shown as token* | | **sglang.kv_cache.pending_preallocation_usage**(gauge) | KV cache usage reserved by pending preallocations.*Shown as fraction* | | **sglang.kv_cache.swa_usage**(gauge) | Sliding-window-attention KV cache usage as a fraction.*Shown as fraction* | | **sglang.kv_cache.usage**(gauge) | Overall KV cache usage as a fraction.*Shown as fraction* | | **sglang.kv_cache.used_tokens**(gauge) | Number of tokens currently stored in the KV cache.*Shown as token* | | **sglang.mamba_pool.usage**(gauge) | Mamba state pool usage as a fraction.*Shown as fraction* | | **sglang.num_requests.grammar_waiting**(gauge) | Requests waiting for grammar compilation.*Shown as request* | | **sglang.num_requests.offline_batch_running**(gauge) | Running low-priority offline batch requests.*Shown as request* | | **sglang.num_requests.running**(gauge) | Number of running inference requests.*Shown as request* | | **sglang.num_requests.waiting**(gauge) | Number of inference requests in the waiting queue.*Shown as request* | | **sglang.process.cpu_seconds.count**(count) | Total CPU time consumed by the SGLang process.*Shown as second* | | **sglang.process.max_fds**(gauge) | Maximum number of file descriptors. | | **sglang.process.open_fds**(gauge) | Number of open file descriptors. | | **sglang.process.resident_memory_bytes**(gauge) | Resident memory used by the SGLang process.*Shown as byte* | | **sglang.process.start_time_seconds**(gauge) | Unix timestamp when the SGLang process started.*Shown as second* | | **sglang.process.virtual_memory_bytes**(gauge) | Virtual memory used by the SGLang process.*Shown as byte* | | **sglang.prompt_tokens.count**(count) | Total number of prompt tokens processed.*Shown as token* | | **sglang.python.gc.collections.count**(count) | Python garbage collection runs. | | **sglang.python.gc.objects.collected.count**(count) | Python objects collected by garbage collection.*Shown as object* | | **sglang.python.gc.objects.uncollectable.count**(count) | Python objects that garbage collection could not collect.*Shown as object* | | **sglang.python.info**(gauge) | Python runtime information. | | **sglang.queue_time.seconds**(gauge) | Time requests spend waiting in a queue (distribution).*Shown as second* | | **sglang.queue_time.seconds.count**(count) | Request queue-time observation count. | | **sglang.queue_time.seconds.sum**(count) | Sum of request queue time.*Shown as second* | | **sglang.realtime_tokens.count**(count) | Tokens processed in real time.*Shown as token* | | **sglang.request.generation_tokens**(gauge) | Generation tokens per request (distribution).*Shown as token* | | **sglang.request.generation_tokens.count**(count) | Generation-token histogram observation count. | | **sglang.request.generation_tokens.sum**(count) | Sum of generation tokens observed per request.*Shown as token* | | **sglang.request.prompt_tokens**(gauge) | Prompt tokens per request (distribution).*Shown as token* | | **sglang.request.prompt_tokens.count**(count) | Prompt-token histogram observation count. | | **sglang.request.prompt_tokens.sum**(count) | Sum of prompt tokens observed per request.*Shown as token* | | **sglang.request.stage_latency.seconds**(gauge) | Per-stage request latency (distribution).*Shown as second* | | **sglang.request.stage_latency.seconds.count**(count) | Per-stage request latency observation count. | | **sglang.request.stage_latency.seconds.sum**(count) | Sum of per-stage request latency.*Shown as second* | | **sglang.request.uncached_prompt_tokens**(gauge) | Uncached prompt tokens per request (distribution).*Shown as token* | | **sglang.request.uncached_prompt_tokens.count**(count) | Uncached prompt-token histogram observation count. | | **sglang.request.uncached_prompt_tokens.sum**(count) | Sum of uncached prompt tokens observed per request.*Shown as token* | | **sglang.requests.aborted.count**(count) | Total number of aborted requests.*Shown as request* | | **sglang.requests.count**(count) | Total number of inference requests processed.*Shown as request* | | **sglang.requests.paused**(gauge) | Number of paused requests.*Shown as request* | | **sglang.requests.retracted**(gauge) | Number of currently retracted requests.*Shown as request* | | **sglang.requests.retracted_total.count**(count) | Total number of retracted requests.*Shown as request* | | **sglang.retracted_input_tokens.count**(count) | Input tokens belonging to retracted requests.*Shown as token* | | **sglang.retracted_output_tokens.count**(count) | Output tokens belonging to retracted requests.*Shown as token* | | **sglang.routing_keys.active**(gauge) | Number of routing keys with active requests. | | **sglang.scheduler.idle.seconds.count**(count) | Total time the scheduler spent idle.*Shown as second* | | **sglang.scheduler.new_token_ratio**(gauge) | Ratio used by the scheduler when allocating new tokens.*Shown as fraction* | | **sglang.scheduler.process_cpu.seconds.count**(count) | Total scheduler CPU processing time.*Shown as second* | | **sglang.scheduler.utilization**(gauge) | Scheduler utilization as a fraction.*Shown as fraction* | | **sglang.spec_decode.accept_length**(gauge) | Average speculative decoding acceptance length.*Shown as token* | | **sglang.spec_decode.accept_rate**(gauge) | Average speculative decoding acceptance rate.*Shown as fraction* | | **sglang.spec_decode.block_accept_length**(gauge) | Average accepted speculative tokens per block.*Shown as token* | | **sglang.spec_decode.capture_length**(gauge) | Speculative decoding capture length.*Shown as token* | | **sglang.spec_decode.draft_tokens**(gauge) | Number of draft tokens per speculative decoding step.*Shown as token* | | **sglang.spec_decode.steps**(gauge) | Number of speculative decoding steps. | | **sglang.spec_decode.verify_calls.count**(count) | Total speculative decoding verification calls. | | **sglang.startup.cuda_graph.seconds**(gauge) | Time spent capturing CUDA graphs during startup.*Shown as second* | | **sglang.startup.phase.seconds**(gauge) | Startup duration for the phase identified by the phase tag.*Shown as second* | | **sglang.startup.seconds**(gauge) | Time taken for the SGLang engine to start.*Shown as second* | | **sglang.time_to_first_token.seconds**(gauge) | Time from request receipt to first generated token (distribution).*Shown as second* | | **sglang.time_to_first_token.seconds.count**(count) | Time-to-first-token observation count. | | **sglang.time_to_first_token.seconds.sum**(count) | Sum of time to first token.*Shown as second* | | **sglang.weight_load.seconds**(gauge) | Time taken to load model weights.*Shown as second* | | **sglang.weight_update.seconds**(gauge) | Duration of the most recent weight update from the source identified by the source tag.*Shown as second* | The integration renames SGLang's `endpoint` metric label to the `http_endpoint` Datadog tag. This avoids colliding with the `endpoint` tag that identifies the configured OpenMetrics endpoint. ### Events{% #events %} The SGLang integration does not include any events. ### Service checks{% #service-checks %} **sglang.openmetrics.health** Returns `CRITICAL` if the Agent is unable to connect to the SGLang OpenMetrics endpoint, otherwise returns `OK`. *Statuses: ok, critical* ## Troubleshooting{% #troubleshooting %} Need help? Contact [Datadog support](https://docs.datadoghq.com/help/).