Why vLLM throughput is far below expected, and how to tune it
Avg generation throughput: 12.3 tokens/s (far below expected for the GPU and model combination)
Also appears as
- GPU utilization stuck around 20-40% while requests queue up in vLLM logs
- Time to first token and inter-token latency both much higher than benchmark numbers for the same model and GPU
Short answer
Low vLLM throughput almost always traces back to max-num-seqs capping concurrent batching too low, chunked prefill being disabled so long prompts stall the decode batch, an unintended dtype that doesn't use tensor cores efficiently, CPU-bound tokenization or preprocessing, or requests spilling into swap. Diagnose with nvidia-smi and vLLM's own throughput logs before changing anything.
Affects: vLLM 0.4 and later, any GPU; most common when serving with default flags on a model or workload that needs tuning for concurrency, prefill, or dtype
Tuning pass, in order of impact
- 1Raise --max-num-seqs, since the default is often conservative, so more requests can be batched concurrently for the same forward pass.
- 2Enable chunked prefill with --enable-chunked-prefill so long prompt prefill doesn't block decode steps for other in-flight requests.
- 3Confirm --dtype matches the checkpoint's native precision, such as bfloat16 or float16, rather than falling back to float32.
- 4Check GPU utilization with nvidia-smi during load; if it's low while requests queue, the bottleneck is likely request arrival rate, tokenization, or network, not the model itself.
- 5If VRAM allows, raise --gpu-memory-utilization to grow the KV cache pool and support more concurrent sequences.
How to confirm this is your problem
- Tokens per second reported in vLLM logs or metrics is far below published benchmarks for the same model and GPU pairing.
- nvidia-smi shows GPU utilization well under 80-90 percent even under sustained load.
- Latency increases sharply once more than a handful of concurrent requests arrive.
- Long-prompt requests appear to stall throughput for all other concurrent users.
Root causes and fixes
max-num-seqs or max-num-batched-tokens capped too low for available VRAM
vLLM's continuous batching only helps if enough sequences are actually admitted into the running batch at once. A conservative default or manually-set low cap means the GPU sits idle between small batches instead of saturating with a larger concurrent batch, wasting the parallelism the paged KV cache was built to exploit.
Fix: Raise --max-num-seqs, and optionally --max-num-batched-tokens, toward the largest value your KV cache pool can support, checking GPU utilization as you increase it.
vllm serve MODEL_ID --max-num-seqs 256 nvidia-smi dmon -s u
Chunked prefill disabled, so long prompts block decode steps
Without chunked prefill, a long incoming prompt's entire prefill computation runs as one blocking step, delaying token generation for every other sequence already being decoded in that batch. This creates head-of-line blocking that tanks aggregate throughput under mixed short and long prompt traffic.
Fix: Enable --enable-chunked-prefill so long prefills are split into pieces interleaved with ongoing decode steps.
vllm serve MODEL_ID --enable-chunked-prefill
Wrong or fallback dtype not using tensor cores efficiently
If --dtype isn't explicitly set to match the checkpoint, or an unsupported combination silently falls back toward slower compute paths, matrix multiplications run far slower because they don't hit the GPU's fp16 or bf16 tensor core throughput, even though the model still works correctly.
Fix: Explicitly set --dtype bfloat16 or --dtype float16 matching the checkpoint's training precision, and confirm it's actually applied in the startup logs.
vllm serve MODEL_ID --dtype bfloat16
CPU-bound tokenization or request preprocessing ahead of the GPU
With very high request rates or a slow tokenizer implementation, the CPU-side path, including tokenizing, detokenizing, and JSON serialization in the OpenAI-compatible API layer, can become the bottleneck, leaving the GPU underfed even though the model itself could run faster.
Fix: Profile CPU usage during load; if a single core is pegged while the GPU sits idle, add more API server workers or check for a slow, non-fast tokenizer being loaded.
top
python -c "from transformers import AutoTokenizer; t = AutoTokenizer.from_pretrained('MODEL_ID'); print(t.is_fast)"KV cache pool too small, forcing requests into swap or aggressive preemption
When the KV cache pool fills up under high concurrency, vLLM either swaps blocks to CPU RAM or preempts and recomputes lower-priority sequences. Both are far slower than serving purely from GPU memory and show up as throughput cliffs precisely when concurrency rises.
Fix: Increase --gpu-memory-utilization or reduce --max-model-len to grow the effective KV cache pool so fewer requests need swap or preemption under load.
vllm serve MODEL_ID --gpu-memory-utilization 0.92
Diagnostic commands
Watch GPU utilization during a load test
nvidia-smi dmon -s u
Sustained utilization well under 80-90 percent during heavy concurrent load points to a CPU, batching, or network bottleneck rather than the model being compute-bound.
Check vLLM's own throughput and latency logging
vllm serve MODEL_ID ... 2>&1 | grep -i "throughput"
Compare the logged tokens per second against published benchmark numbers for the same model and GPU; a large gap confirms a config issue rather than a hardware ceiling.
Confirm the dtype actually in use
vllm serve MODEL_ID 2>&1 | grep -i dtype
If it doesn't match the checkpoint's native precision, fix --dtype explicitly and relaunch.
Check current batching and concurrency settings
vllm serve --help | grep -i "max-num-seqs"
Confirm these flags are set to values appropriate for your VRAM headroom, not left at overly conservative defaults.
Stopping it from happening again
- Benchmark each new model and GPU combination against published or self-measured baseline throughput before production rollout, so regressions are caught immediately.
- Load-test with a realistic mix of short and long prompts, not just uniform synthetic requests, to catch prefill head-of-line blocking early.
- Document the tuned flags, including max-num-seqs, chunked-prefill, and dtype, per model in your deployment manifests so they survive redeploys.
- Monitor GPU utilization and per-request latency continuously in production to catch throughput regressions from config drift.
When this becomes an architecture problem
If you've tuned batching, prefill, and dtype and GPU utilization is genuinely pinned near 100 percent with throughput still below product requirements, that's no longer a configuration problem: it's a capacity problem needing more or faster GPUs, a smaller or quantized model, or speculative decoding, all of which are architecture decisions worth planning deliberately.
Frequently asked questions
What's the single highest-leverage flag for throughput?
For most under-tuned deployments it's max-num-seqs: raising it lets vLLM's continuous batching admit more concurrent sequences per forward pass, directly increasing GPU utilization and aggregate tokens per second, as long as your KV cache pool has room.
Does chunked prefill help single-request latency too?
It mainly helps aggregate throughput and fairness under concurrent mixed-length traffic by preventing long prompts from blocking others. For a single isolated request it has minimal effect on that request's own latency.
How do I know if I'm CPU-bound versus GPU-bound?
Watch nvidia-smi during a load test: if GPU utilization stays low while requests queue and a CPU core is pegged, you're CPU-bound on tokenization or API serialization. If GPU utilization is high and throughput is still low, look at batching and dtype settings instead.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
Free ToolLLM Serving Capacity Planner
Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.
Free ToolSpeculative Decoding Speedup Calculator
Model the real decode speedup speculative decoding delivers from your draft model's acceptance rate, proposal depth, and per-step overhead.
Related problems
vLLM runs out of memory during startup, before serving any requests
vLLM's startup OOMs happen because it preallocates a KV cache pool sized against gpu_memory_utilization right after loading weights, so the failure point is engine initialization, not user traffic. Fix it by lowering gpu_memory_utilization if it's set too aggressively for actual free VRAM, lowering max_model_len, or reducing weight footprint with quantization or more GPUs.
vLLM fails to start because there is not enough memory for the KV cache
vLLM reserves a fixed pool of GPU memory (gpu_memory_utilization, default 0.9) for weights plus KV cache, and if the weights already consume most of that budget there is nothing left for even one sequence's KV cache blocks. The fix is to raise gpu_memory_utilization toward the physical limit, lower max_model_len so each sequence's KV cache is smaller, or serve a quantized checkpoint so more of the budget is available for cache.
LLM serving throughput collapses once load increases past a certain point
Throughput collapsing past a load threshold is almost always KV cache exhaustion: once in-flight requests' combined KV cache exceeds available GPU memory, the scheduler preempts some sequences, discarding their KV cache and forcing a full recompute when they resume, which burns GPU cycles on redundant work instead of new tokens. The fix is admission control that keeps the server below its true KV cache-limited concurrency, not just retrying harder or adding a bigger queue.
GPU utilization stays low during LLM inference even under load
Low GPU utilization during inference almost always means the GPU is waiting on something else: request concurrency is too low for the batching scheduler to fill, the client code is calling the server synchronously one request at a time, tokenization or network I/O is serialized in front of the GPU call, or max-num-seqs is set too low to admit enough concurrent sequences. Raising effective concurrency, either by fixing the client or the server's admission limits, is almost always the fix, not more GPU compute.
GuideLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideThe LLM Inference Cost Optimization Playbook
Cut LLM inference costs with a practical playbook: quantization, batching, GPU right-sizing, caching, and the on-prem vs API breakeven math for 2026.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.