sglang.cache_config_info
(gauge) | Cache configuration information. |
sglang.cached_tokens.count
(count) | Total number of cached prompt tokens.
Shown as token |
sglang.cuda_graph.active
(gauge) | Whether CUDA graph execution is active. |
sglang.cuda_graph.passes.count
(count) | Total number of CUDA graph execution passes. |
sglang.decode.sequence_length_sum
(gauge) | Sum of sequence lengths currently being decoded.
Shown as token |
sglang.disaggregation.bootstrap_failures.count
(count) | Total number of failed bootstrap requests. |
sglang.disaggregation.decode_preallocation_queue
(gauge) | Requests waiting for decode preallocation. |
sglang.disaggregation.decode_transfer_queue
(gauge) | Requests waiting for decode transfer. |
sglang.disaggregation.kv_transfer.allocation_milliseconds
(gauge) | Time spent allocating memory for KV transfers.
Shown as millisecond |
sglang.disaggregation.kv_transfer.bootstrap_milliseconds
(gauge) | Time spent bootstrapping KV transfers.
Shown as millisecond |
sglang.disaggregation.kv_transfer.gigabytes_per_second
(gauge) | KV cache transfer throughput.
Shown as gigabyte |
sglang.disaggregation.kv_transfer.latency_milliseconds
(gauge) | KV cache transfer latency.
Shown as millisecond |
sglang.disaggregation.kv_transfer.megabytes
(gauge) | KV cache data transferred.
Shown as megabyte |
sglang.disaggregation.prefill_bootstrap_queue
(gauge) | Requests in the prefill bootstrap queue. |
sglang.disaggregation.prefill_inflight_queue
(gauge) | Requests in the prefill inflight queue. |
sglang.disaggregation.prefill_preallocation_queue
(gauge) | Requests waiting for prefill preallocation. |
sglang.disaggregation.prefill_retries.count
(count) | Total number of retried prefill requests. |
sglang.disaggregation.transfer_failures.count
(count) | Total number of failed KV transfer requests. |
sglang.e2e_request_latency.seconds
(gauge) | End-to-end request latency (distribution).
Shown as second |
sglang.e2e_request_latency.seconds.count
(count) | End-to-end request latency observation count. |
sglang.e2e_request_latency.seconds.sum
(count) | Sum of end-to-end request latency.
Shown as second |
sglang.estimated_flops_per_gpu.count
(count) | Estimated floating-point operations per GPU.
Shown as operation |
sglang.estimated_read_bytes_per_gpu.count
(count) | Estimated bytes read per GPU.
Shown as byte |
sglang.estimated_write_bytes_per_gpu.count
(count) | Estimated bytes written per GPU.
Shown as byte |
sglang.forward_execution.seconds.count
(count) | Total model forward execution time.
Shown as second |
sglang.generation_throughput.tokens_per_second
(gauge) | Generation throughput.
Shown as token |
sglang.generation_tokens.count
(count) | Total number of generation tokens processed.
Shown as token |
sglang.http.requests.active
(gauge) | Currently active HTTP requests. |
sglang.http.requests.count
(count) | Total number of HTTP requests.
Shown as request |
sglang.http.responses.count
(count) | Total number of HTTP responses.
Shown as response |
sglang.inter_token_latency.seconds
(gauge) | Inter-token latency (distribution).
Shown as second |
sglang.inter_token_latency.seconds.count
(count) | Inter-token latency observation count. |
sglang.inter_token_latency.seconds.sum
(count) | Sum of inter-token latency.
Shown as second |
sglang.kv_cache.full_attention_usage
(gauge) | Full-attention KV cache usage as a fraction.
Shown as fraction |
sglang.kv_cache.hit_rate
(gauge) | Prefix KV cache hit rate.
Shown as fraction |
sglang.kv_cache.max_tokens
(gauge) | Maximum number of tokens in the KV cache pool.
Shown as token |
sglang.kv_cache.pending_preallocation_usage
(gauge) | KV cache usage reserved by pending preallocations.
Shown as fraction |
sglang.kv_cache.swa_usage
(gauge) | Sliding-window-attention KV cache usage as a fraction.
Shown as fraction |
sglang.kv_cache.usage
(gauge) | Overall KV cache usage as a fraction.
Shown as fraction |
sglang.kv_cache.used_tokens
(gauge) | Number of tokens currently stored in the KV cache.
Shown as token |
sglang.mamba_pool.usage
(gauge) | Mamba state pool usage as a fraction.
Shown as fraction |
sglang.num_requests.grammar_waiting
(gauge) | Requests waiting for grammar compilation.
Shown as request |
sglang.num_requests.offline_batch_running
(gauge) | Running low-priority offline batch requests.
Shown as request |
sglang.num_requests.running
(gauge) | Number of running inference requests.
Shown as request |
sglang.num_requests.waiting
(gauge) | Number of inference requests in the waiting queue.
Shown as request |
sglang.process.cpu_seconds.count
(count) | Total CPU time consumed by the SGLang process.
Shown as second |
sglang.process.max_fds
(gauge) | Maximum number of file descriptors. |
sglang.process.open_fds
(gauge) | Number of open file descriptors. |
sglang.process.resident_memory_bytes
(gauge) | Resident memory used by the SGLang process.
Shown as byte |
sglang.process.start_time_seconds
(gauge) | Unix timestamp when the SGLang process started.
Shown as second |
sglang.process.virtual_memory_bytes
(gauge) | Virtual memory used by the SGLang process.
Shown as byte |
sglang.prompt_tokens.count
(count) | Total number of prompt tokens processed.
Shown as token |
sglang.python.gc.collections.count
(count) | Python garbage collection runs. |
sglang.python.gc.objects.collected.count
(count) | Python objects collected by garbage collection.
Shown as object |
sglang.python.gc.objects.uncollectable.count
(count) | Python objects that garbage collection could not collect.
Shown as object |
sglang.python.info
(gauge) | Python runtime information. |
sglang.queue_time.seconds
(gauge) | Time requests spend waiting in a queue (distribution).
Shown as second |
sglang.queue_time.seconds.count
(count) | Request queue-time observation count. |
sglang.queue_time.seconds.sum
(count) | Sum of request queue time.
Shown as second |
sglang.realtime_tokens.count
(count) | Tokens processed in real time.
Shown as token |
sglang.request.generation_tokens
(gauge) | Generation tokens per request (distribution).
Shown as token |
sglang.request.generation_tokens.count
(count) | Generation-token histogram observation count. |
sglang.request.generation_tokens.sum
(count) | Sum of generation tokens observed per request.
Shown as token |
sglang.request.prompt_tokens
(gauge) | Prompt tokens per request (distribution).
Shown as token |
sglang.request.prompt_tokens.count
(count) | Prompt-token histogram observation count. |
sglang.request.prompt_tokens.sum
(count) | Sum of prompt tokens observed per request.
Shown as token |
sglang.request.stage_latency.seconds
(gauge) | Per-stage request latency (distribution).
Shown as second |
sglang.request.stage_latency.seconds.count
(count) | Per-stage request latency observation count. |
sglang.request.stage_latency.seconds.sum
(count) | Sum of per-stage request latency.
Shown as second |
sglang.request.uncached_prompt_tokens
(gauge) | Uncached prompt tokens per request (distribution).
Shown as token |
sglang.request.uncached_prompt_tokens.count
(count) | Uncached prompt-token histogram observation count. |
sglang.request.uncached_prompt_tokens.sum
(count) | Sum of uncached prompt tokens observed per request.
Shown as token |
sglang.requests.aborted.count
(count) | Total number of aborted requests.
Shown as request |
sglang.requests.count
(count) | Total number of inference requests processed.
Shown as request |
sglang.requests.paused
(gauge) | Number of paused requests.
Shown as request |
sglang.requests.retracted
(gauge) | Number of currently retracted requests.
Shown as request |
sglang.requests.retracted_total.count
(count) | Total number of retracted requests.
Shown as request |
sglang.retracted_input_tokens.count
(count) | Input tokens belonging to retracted requests.
Shown as token |
sglang.retracted_output_tokens.count
(count) | Output tokens belonging to retracted requests.
Shown as token |
sglang.routing_keys.active
(gauge) | Number of routing keys with active requests. |
sglang.scheduler.idle.seconds.count
(count) | Total time the scheduler spent idle.
Shown as second |
sglang.scheduler.new_token_ratio
(gauge) | Ratio used by the scheduler when allocating new tokens.
Shown as fraction |
sglang.scheduler.process_cpu.seconds.count
(count) | Total scheduler CPU processing time.
Shown as second |
sglang.scheduler.utilization
(gauge) | Scheduler utilization as a fraction.
Shown as fraction |
sglang.spec_decode.accept_length
(gauge) | Average speculative decoding acceptance length.
Shown as token |
sglang.spec_decode.accept_rate
(gauge) | Average speculative decoding acceptance rate.
Shown as fraction |
sglang.spec_decode.block_accept_length
(gauge) | Average accepted speculative tokens per block.
Shown as token |
sglang.spec_decode.capture_length
(gauge) | Speculative decoding capture length.
Shown as token |
sglang.spec_decode.draft_tokens
(gauge) | Number of draft tokens per speculative decoding step.
Shown as token |
sglang.spec_decode.steps
(gauge) | Number of speculative decoding steps. |
sglang.spec_decode.verify_calls.count
(count) | Total speculative decoding verification calls. |
sglang.startup.cuda_graph.seconds
(gauge) | Time spent capturing CUDA graphs during startup.
Shown as second |
sglang.startup.phase.seconds
(gauge) | Startup duration for the phase identified by the phase tag.
Shown as second |
sglang.startup.seconds
(gauge) | Time taken for the SGLang engine to start.
Shown as second |
sglang.time_to_first_token.seconds
(gauge) | Time from request receipt to first generated token (distribution).
Shown as second |
sglang.time_to_first_token.seconds.count
(count) | Time-to-first-token observation count. |
sglang.time_to_first_token.seconds.sum
(count) | Sum of time to first token.
Shown as second |
sglang.weight_load.seconds
(gauge) | Time taken to load model weights.
Shown as second |
sglang.weight_update.seconds
(gauge) | Duration of the most recent weight update from the source identified by the source tag.
Shown as second |
