KV Cache Memory Calculator: Size Your Attention Cache Precisely
This free KV cache memory calculator applies the standard attention cache formula, two times layers times KV heads times head dimension times precision times context length, to tell you exactly how much GPU memory your model's attention cache consumes. It is built for platform engineers who need to know why VRAM disappears after weights load and how many concurrent users a given GPU configuration can actually support. Enter your model's architecture parameters and target context length, and the tool returns cache size per token, per sequence, and the maximum concurrent sequences your available VRAM supports.
Your numbers
Depth of the model. A 70B-class dense model commonly has 80 layers; smaller models have fewer.
Grouped-query attention shares KV heads across query heads, which is why this is often far fewer than total attention heads.
Size of each attention head, commonly 64-128 across current open-weight model families.
FP8 KV cache quantization roughly halves cache memory with a small accuracy cost.
Maximum tokens (prompt plus generated) the cache must hold for one active sequence.
How many requests are actively decoding at once and each need their own KV cache slot.
Total VRAM minus model weights and runtime overhead, the space left for KV cache blocks.
Your results
Estimates use the standard 2 x layers x kv_heads x head_dim x precision formula per token. Real deployments rarely fill every sequence to maximum context, so PagedAttention typically supports higher practical concurrency than this worst-case figure.
Get your full KV cache capacity report
We will email you a personalized cache sizing breakdown across context lengths and concurrency targets, and a Netray inference specialist will follow up with tuning recommendations.
No spam. Your results stay private. Unsubscribe anytime.
Why KV cache dominates memory at scale
Model weights are a fixed, one-time memory cost. KV cache is not: it grows linearly with context length and with the number of concurrent sequences, which means a chat application with long conversations and many simultaneous users can consume far more memory in cache than the model weights themselves ever did. Grouped-query attention, used by essentially every current open-weight model family, shrinks this cost dramatically by sharing key/value projections across multiple query heads, which is why the KV head count in this calculator is often 4-16 rather than the 32-64 total attention heads a model might have.
- The formula counts 2x for storing both keys and values separately at every layer.
- Grouped-query attention (GQA) cuts KV cache size by using far fewer KV heads than query heads, often 4-8x fewer.
- Cache grows linearly with context length, so doubling your context window doubles cache memory per sequence.
- FP8 KV cache quantization roughly halves cache memory with typically small quality impact on long-context recall.
Interpreting per-sequence and total cache size
Per-sequence cache size tells you the worst case for one fully-populated context, which matters for setting a hard concurrency ceiling. Total cache size at your target batch size tells you what to actually provision for. The gap between these two numbers and your available VRAM is your real headroom: if total cache exceeds available memory, requests will queue, get rejected, or in poorly configured deployments crash the server outright. PagedAttention, the technique vLLM introduced, allocates cache in fixed-size blocks rather than reserving worst-case memory per sequence up front, which is why real deployments often support meaningfully higher concurrency than a naive worst-case calculation implies.
Levers for reducing KV cache pressure
When max concurrent sequences comes back too low for your traffic target, you have several real options beyond just buying more VRAM. Reducing default context length for use cases that do not need it is the highest-leverage lever, since cache scales linearly with it. FP8 KV cache quantization is close to a free win on supporting hardware. Sliding-window attention variants and cache eviction policies for older tokens in long conversations trade some long-range recall for materially lower steady-state memory.
- Cap default context length per use case rather than always reserving the model's maximum window.
- Enable FP8 or INT8 KV cache quantization where your serving engine and hardware support it.
- Use PagedAttention-style block allocation instead of static per-sequence reservation to raise effective concurrency.
- Summarize or truncate long conversation history rather than replaying full context on every turn.
How Netray sizes serving capacity around KV cache
Netray designs on-prem inference platforms where getting concurrency right the first time matters, because ITAR and CMMC environments cannot simply burst to a public cloud API when capacity runs short. We model KV cache pressure against your real conversation length and concurrency patterns, not synthetic benchmarks, and tune context policies and quantization to hit your target user count on the hardware you have. Engagements typically start with a capacity model built from your actual usage logs.
Frequently Asked Questions
Why does grouped-query attention matter so much for memory?
Standard multi-head attention stores a separate key and value projection for every attention head, which scales cache size directly with head count. Grouped-query attention shares KV projections across groups of query heads, so a model with 32 query heads might only maintain 8 KV heads, cutting cache memory by 4x with minimal quality impact. Nearly every current open-weight model family uses GQA specifically because unconstrained KV cache growth made long-context serving impractical.
Does PagedAttention actually change the total memory required?
Not the theoretical maximum, but it dramatically improves practical utilization. Without it, serving engines reserve worst-case contiguous memory per sequence up front, wasting large amounts of VRAM on sequences that end up shorter than their reservation. PagedAttention allocates cache in small fixed-size blocks on demand, similar to virtual memory paging in an operating system, which lets a GPU serve substantially more real-world concurrent sequences from the same VRAM budget.
How much does long context cost in KV cache memory?
Linearly and often surprisingly. For a 70B-class model with 80 layers and 8 KV heads at FP16, one sequence at 8K context needs roughly 1.3 GB of cache; the same sequence at 128K context needs roughly 21 GB, more memory than many entire GPUs. This is why long-context deployments either require far fewer concurrent users per GPU, aggressive KV quantization, or both.
Should I quantize KV cache separately from model weights?
Yes, they are independent decisions. You can serve FP16 weights with an FP8 KV cache, or a quantized model with a full-precision cache, depending on which constraint binds harder for your workload. KV cache quantization tends to have a smaller and more predictable quality impact than weight quantization because attention scores are more tolerant of reduced precision than the weights that generate them, but validate on long-context tasks specifically since that is where degradation shows up first.
Get a KV cache and concurrency model built from your real traffic patterns, not a worst-case estimate.
Related Tools
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
On-Prem AIConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
On-Prem AIModel Context Window Planner
Allocate a fixed context window across system prompt, retrieved chunks, conversation history, and reserved output, then see exactly how much retrieval headroom is left.
Go Deeper
KV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
vLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
On-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.