Why vLLM says there is no memory for the KV cache, and how to fix it
ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine
Also appears as
- RuntimeError: To serve at least one request with the models max seq len, (X GiB KV cache is needed
- torch.OutOfMemoryError: KV cache allocation failed
Short answer
vLLM reserves a fixed pool of GPU memory (gpu_memory_utilization, default 0.9) for weights plus KV cache, and if the weights already consume most of that budget there is nothing left for even one sequence's KV cache blocks. The fix is to raise gpu_memory_utilization toward the physical limit, lower max_model_len so each sequence's KV cache is smaller, or serve a quantized checkpoint so more of the budget is available for cache.
Affects: vLLM 0.4 and later on any CUDA GPU, most common serving 13B+ models at long max_model_len on 24-48 GB cards
Get vLLM serving again in one restart
- 1Raise the memory budget: add --gpu-memory-utilization 0.95 (from the 0.9 default) so vLLM reserves more of the physical VRAM for weights plus KV cache.
- 2Lower --max-model-len to the context length you actually need in production, not the model's architectural maximum, since KV cache size scales directly with it.
- 3If the model is large relative to the GPU, switch to a quantized checkpoint (AWQ, GPTQ, or FP8) with --quantization so more VRAM is freed for the cache pool.
- 4For multi-GPU boxes, add --tensor-parallel-size N so weight memory is sharded and each GPU has a larger share left over for KV cache blocks.
- 5Restart and confirm in the startup logs that the reported number of KV cache blocks is greater than zero.
How to confirm this is your problem
- vllm serve process exits during startup, before it ever accepts a request
- Log explicitly states there is no memory available for cache blocks or that even one request cannot be served at max_model_len
- Same model serves fine with a shorter --max-model-len on the identical GPU
- Reducing gpu_memory_utilization makes the error worse, raising it toward 0.9-0.95 resolves it
Root causes and fixes
max_model_len is set to the model's full architectural context (often 32k-128k) which requires far more KV cache than the GPU has left after weights
KV cache size per sequence scales as 2 x num_layers x num_kv_heads x head_dim x max_model_len x bytes_per_element. A model with a 128k native context can require tens of gigabytes of KV cache for a single maximum-length sequence, which frequently exceeds what remains after loading the weights themselves.
Fix: Set --max-model-len to the longest context your application actually uses, for example 8192 or 16384, rather than accepting the model's default maximum.
vllm serve MODEL --max-model-len 8192 --gpu-memory-utilization 0.9
gpu_memory_utilization is left at a conservative default while the physical GPU has more headroom
vLLM's default of 0.9 intentionally leaves 10 percent of VRAM unreserved as a safety margin for CUDA context and fragmentation. On GPUs shared with other processes or with driver overhead already present, that margin is fine, but on a dedicated inference GPU it can be the difference between zero and enough KV cache blocks.
Fix: Increase --gpu-memory-utilization toward 0.93-0.95 on a GPU dedicated solely to this vLLM process, watching nvidia-smi to confirm no other process needs the freed headroom.
vllm serve MODEL --gpu-memory-utilization 0.95
Full-precision weights leave too little of the memory budget for any KV cache
Weights and KV cache draw from the same gpu_memory_utilization pool. If fp16 weights alone consume 80-90 percent of VRAM, there is arithmetically almost nothing left for cache blocks, regardless of how max_model_len is tuned.
Fix: Serve a 4-bit or FP8 quantized checkpoint so weight memory shrinks by roughly 2-4x, freeing proportionally more of the budget for KV cache.
vllm serve MODEL --quantization awq
Tensor-parallel size is smaller than it should be for the model and GPU count available
Without tensor parallelism, all weights and all KV cache for every request live on one GPU. If additional GPUs are present but --tensor-parallel-size is left at 1, the single GPU is starved for cache space while the others sit unused.
Fix: Set --tensor-parallel-size to match the number of GPUs allocated to this model so both weights and KV cache are sharded, multiplying the effective memory budget.
vllm serve MODEL --tensor-parallel-size 2 --gpu-memory-utilization 0.9
Another process on the same GPU (a monitoring agent, a second model server, or a stale vLLM worker) is holding memory outside vLLM's accounting
vLLM's gpu_memory_utilization is computed against total device memory at process start, but memory already held by unrelated processes reduces what is genuinely free, so vLLM's internal math can look correct while the real headroom is smaller than assumed.
Fix: Confirm with nvidia-smi that vLLM's process is the only significant consumer on the GPU before tuning utilization further.
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
Diagnostic commands
Read the exact KV cache requirement vLLM reports
vllm serve MODEL --max-model-len 8192 2>&1 | grep -i 'kv cache\|gpu blocks'
The startup log states the number of GPU blocks available and the number needed per sequence; zero or a very low block count confirms this is a capacity problem, not a bug.
Check real free memory versus what vLLM assumes
nvidia-smi --query-gpu=memory.used,memory.total,memory.free --format=csv
Compare memory.free against (1 - gpu_memory_utilization) x memory.total; if actual free memory is lower than expected, another process is consuming the headroom vLLM planned around.
Estimate KV cache size for your target context
python -c "layers=32; kv_heads=8; head_dim=128; ctx=8192; bytes_elem=2; print(2*layers*kv_heads*head_dim*ctx*bytes_elem/1e9, 'GB per sequence')"
Multiply this per-sequence figure by your target concurrent request count to see the true KV cache budget you need, then compare against what is left after weights on your GPU.
Stopping it from happening again
- Set --max-model-len explicitly to your application's real requirement in every deployment config; never leave it at the model default.
- Size gpu_memory_utilization per GPU model as part of your deployment template, not per incident.
- Load-test with your expected concurrent request count before production cutover, since KV cache exhaustion under load looks different from startup failure.
- Track free VRAM per GPU node in monitoring so a co-located process cannot silently eat vLLM's planned headroom.
When this becomes an architecture problem
If your production concurrency and context length genuinely require more KV cache than any single GPU can provide even after quantization and a tuned max_model_len, that is a multi-GPU tensor-parallel or multi-node serving architecture decision, not a flag to tune further.
Frequently asked questions
What does gpu_memory_utilization actually control?
It sets the fraction of total GPU memory vLLM is allowed to reserve for the combination of model weights and the KV cache block pool, default 0.9. vLLM computes available cache memory as (gpu_memory_utilization x total_memory) minus the memory the weights already consume, then allocates that remainder as KV cache blocks.
Why does lowering max_model_len fix an out-of-memory startup failure?
KV cache memory per sequence is directly proportional to max_model_len. Halving max_model_len roughly halves the KV cache memory vLLM must reserve for even a single maximum-length request, which is often enough to bring the requirement back under the available budget without touching hardware.
Is it safe to set gpu_memory_utilization close to 1.0?
Not recommended. Values above roughly 0.95 leave too little headroom for CUDA context overhead and short-lived allocations, risking a runtime OOM later under load even though startup succeeds. 0.9 to 0.95 is the practical safe range for a GPU dedicated to one vLLM process.
Does quantization reduce KV cache memory too, or only weight memory?
Standard AWQ/GPTQ/FP8 quantization reduces weight memory only; KV cache dtype is controlled separately (vLLM supports FP8 KV cache on supported hardware). Quantizing weights still helps indirectly because it frees more of the shared gpu_memory_utilization budget for the cache pool.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
KV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolvLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
Related problems
vLLM runs out of memory during startup, before serving any requests
vLLM's startup OOMs happen because it preallocates a KV cache pool sized against gpu_memory_utilization right after loading weights, so the failure point is engine initialization, not user traffic. Fix it by lowering gpu_memory_utilization if it's set too aggressively for actual free VRAM, lowering max_model_len, or reducing weight footprint with quantization or more GPUs.
vLLM: model's max seq len is larger than the KV cache can hold
vLLM preallocates a fixed KV cache pool sized by gpu_memory_utilization and refuses to start a context length whose worst case (batch x max sequence length) doesn't fit in that pool. Fix it by raising --gpu-memory-utilization toward 0.9-0.95, lowering --max-model-len to what you actually need, or adding a GPU/quantizing weights to leave more headroom for cache.
CUDA out of memory when loading an LLM
This happens because model weights alone require roughly 2 bytes per parameter in fp16/bf16 (a 70B model needs about 140 GB before you even run inference), and that number does not fit your GPU. The fix is to either quantize the weights (AWQ, GPTQ, FP8, or GGUF), split the model across multiple GPUs with tensor parallelism, or pick a GPU with enough VRAM for the parameter count you are loading.
vLLM throughput is far below expected tokens per second
Low vLLM throughput almost always traces back to max-num-seqs capping concurrent batching too low, chunked prefill being disabled so long prompts stall the decode batch, an unintended dtype that doesn't use tensor cores efficiently, CPU-bound tokenization or preprocessing, or requests spilling into swap. Diagnose with nvidia-smi and vLLM's own throughput logs before changing anything.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.