KV Cache Optimization: Prefix Caching and Chunked Prefill in Production
The KV cache, not model weights, is usually the binding constraint on how many concurrent users a GPU can serve. Every token generated stores a key and value vector per attention layer, and for a 70B model at 8k context that cache can consume more VRAM per active user than a naive read of GPU specs suggests. Getting KV cache management right, through PagedAttention's memory efficiency, prefix caching's reuse of shared prompt segments, and chunked prefill's handling of long-context requests, is what separates a deployment that serves 5 concurrent users from one that serves 50 on the same hardware. This guide covers the mechanisms and the tuning decisions that actually move the needle.
Why KV Cache Dominates Memory Budgets
KV cache size scales with sequence length, number of layers, number of attention heads, head dimension, and batch size, multiplied by two for key and value and by two again for FP16 storage. For a 70B-class dense model with 80 layers, cache for a single 8k-token sequence can run into several gigabytes, meaning that on an 80GB GPU, model weights alone (roughly 40GB at FP16, or less quantized) leave a fixed remainder that directly caps concurrent users. This is the calculation vLLM's --gpu-memory-utilization flag and PagedAttention are managing under the hood, and it is why quantizing weights to free memory for more KV cache capacity is often a bigger throughput lever than raw compute optimization. Grouped-query attention (GQA), used in most current open models, reduces KV cache size substantially versus older multi-head attention by sharing key/value heads across query heads, which is part of why modern models serve more efficiently than their 2023-era counterparts at the same parameter count.
Prefix Caching: Reusing Work Across Requests
Prefix caching reuses computed KV cache blocks across requests that share an identical prompt prefix, which is enormously valuable for workloads with a long, static system prompt, a shared RAG template, or repeated few-shot examples. Instead of recomputing attention for the shared prefix on every request, vLLM (with prefix caching enabled, on by default in current versions) hashes prompt blocks and serves cached ones directly, cutting time-to-first-token dramatically for the cache-hit portion. The practical implication for prompt engineering: structure prompts with the static, shared content first and the unique, per-request content last, since PagedAttention's block-based caching only benefits from matching prefixes, not matching suffixes or middles. A RAG pipeline that puts the retrieved document before a fixed instruction template gets none of this benefit; one that puts the fixed instruction and system prompt first gets a large one.
- Structure prompts static-content-first, unique-content-last to maximize cache hit rate
- Long, repeated system prompts and RAG instruction templates are the highest-value prefix caching targets
- Cache hit rate is a metric worth exposing on your monitoring dashboard, since it directly predicts TTFT improvement
- Prefix caching benefit shrinks fast if every request personalizes the prompt early, so keep personalization late in the prompt structure
Chunked Prefill: Preventing Long Prompts From Blocking the Batch
Without chunked prefill, a single long prompt's prefill phase (the initial pass processing the full input before generation starts) runs as one large compute step that can stall the decode step for every other sequence in the batch, since prefill and decode compete for the same GPU. Chunked prefill splits a long prompt's prefill into smaller pieces that interleave with ongoing decode steps for other requests, keeping tail latency stable for concurrent users even when someone sends a 20k-token document alongside short chat requests. This matters most in mixed workloads, RAG systems where retrieved context length varies widely, or any deployment fielding both short interactive queries and long document-processing requests on the same server. Enable it explicitly and verify with your vLLM version's defaults, since behavior has changed across releases.
Sizing KV Cache for Concurrent Users
To size a deployment, calculate expected KV cache memory per average request (context length times per-token cache size for your specific model and attention architecture), multiply by your target concurrent user count, and confirm it fits within GPU memory after subtracting quantized weight size and a safety margin. Then load test to validate, because theoretical calculations miss real effects like prefix cache hit rate and request-length variance. A common sizing mistake is provisioning for average context length rather than the p95 or p99, which causes intermittent OOM-driven preemption exactly during your highest-value traffic (long, complex user requests) rather than during easy ones. Track KV cache utilization as a first-class metric, not just GPU memory utilization, since the two diverge meaningfully once prefix caching is active.
- Size for p95/p99 context length, not average, or preemption hits your most complex requests first
- Quantized weights free proportionally more room for KV cache, often the highest-leverage capacity lever available
- GQA models need less KV cache per token than older MHA architectures at equivalent parameter count
- Monitor KV cache utilization separately from raw GPU memory utilization once prefix caching is active
How Netray Tunes KV Cache for On-Prem Deployments
Netray sizes KV cache capacity against a client's real prompt-length distribution pulled from production logs or a representative traffic sample, not published benchmark numbers, before recommending GPU count or quantization strategy. We enable and validate prefix caching against the client's actual prompt structure, often restructuring RAG templates to move static content earlier and materially improve cache hit rate, and we tune chunked prefill settings for mixed short-and-long-context workloads common in document-heavy enterprise use cases. This sizing work is part of our GPU sizing and on-prem inference deployment engagements, and it is usually the difference between a client's initial hardware estimate and what they actually need to buy.
Frequently Asked Questions
What is prefix caching in LLM serving and when does it help most?
Prefix caching reuses computed KV cache blocks across requests that share an identical prompt prefix, avoiding recomputation of shared content like system prompts or RAG instruction templates. It helps most in workloads with long, static, shared prompt content: chat systems with a fixed system prompt, or RAG pipelines with a consistent instruction template. To benefit, structure prompts with static content first and unique per-request content last, since caching only matches identical prefixes.
How much KV cache memory does a 70B model need per user?
It depends on context length, attention architecture, and precision, but for a 70B-class model at 8k tokens with grouped-query attention, expect roughly 1 to 3 GB per concurrent sequence depending on the specific model. This is why KV cache, not model weights, usually caps concurrent users on a given GPU. Calculate for your specific model's layer count and head configuration, and always size against p95/p99 context length rather than average.
What does chunked prefill do and why does it matter for mixed workloads?
Chunked prefill splits a long prompt's initial processing pass into smaller pieces that interleave with ongoing decode steps for other concurrent requests, rather than running as one large blocking compute step. Without it, a single long document upload can stall response latency for every other user's short chat request sharing the same GPU. It matters most in RAG systems and any deployment serving both short interactive queries and long document-processing requests together.
Key Takeaways
- 1Why KV Cache Dominates Memory Budgets: KV cache size scales with sequence length, number of layers, number of attention heads, head dimension, and batch size, multiplied by two for key and value and by two again for FP16 storage. For a 70B-class dense model with 80 layers, cache for a single 8k-token sequence can run into several gigabytes, meaning that on an 80GB GPU, model weights alone (roughly 40GB at FP16, or less quantized) leave a fixed remainder that directly caps concurrent users.
- 2Prefix Caching: Reusing Work Across Requests: Prefix caching reuses computed KV cache blocks across requests that share an identical prompt prefix, which is enormously valuable for workloads with a long, static system prompt, a shared RAG template, or repeated few-shot examples. Instead of recomputing attention for the shared prefix on every request, vLLM (with prefix caching enabled, on by default in current versions) hashes prompt blocks and serves cached ones directly, cutting time-to-first-token dramatically for the cache-hit portion.
- 3Chunked Prefill: Preventing Long Prompts From Blocking the Batch: Without chunked prefill, a single long prompt's prefill phase (the initial pass processing the full input before generation starts) runs as one large compute step that can stall the decode step for every other sequence in the batch, since prefill and decode compete for the same GPU. Chunked prefill splits a long prompt's prefill into smaller pieces that interleave with ongoing decode steps for other requests, keeping tail latency stable for concurrent users even when someone sends a 20k-token document alongside short chat requests.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
KV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Terms used in this article
Sizing GPU memory for concurrent LLM users? Netray will calculate real KV cache requirements against your traffic pattern before you commit to hardware.
Related Resources
vLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
AI & AutomationMulti-GPU LLM Serving: Tensor vs Pipeline Parallelism
Multi-GPU LLM serving explained: tensor parallelism vs pipeline parallelism, NCCL interconnect requirements, and when to split a model across GPUs.
AI & AutomationLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.