vLLM Throughput Estimator: Model Continuous Batching Performance
This free vLLM throughput estimator translates model size, GPU class, and batch depth into a realistic tokens-per-second figure for a self-hosted deployment, and it is built for platform engineers sizing on-prem inference clusters. Continuous batching is the single biggest throughput lever in modern serving stacks like vLLM, TensorRT-LLM, and SGLang, because it lets the GPU work on dozens of requests simultaneously instead of idling between decode steps for one user at a time. Enter your model footprint, GPU class, and how many sequences you expect in flight, and the tool returns single-stream speed, aggregate cluster throughput, and sustained requests per minute.
Your numbers
Weight memory footprint at FP16. Quantized deployments should use a smaller equivalent GB figure.
Memory bandwidth per GPU, the dominant factor in decode speed for a memory-bound workload.
Tensor-parallel degree for one model replica behind the vLLM scheduler.
How many requests vLLM's continuous batching scheduler is actively decoding at once.
Generated response length, used to convert aggregate throughput into requests per minute.
Real vLLM deployments reach 45-65% of theoretical bandwidth after PagedAttention and scheduling overhead.
Your results
Planning estimates only. Real vLLM throughput depends on serving engine version, prefix caching, chunked prefill, quantization, and request mix. Benchmark your actual stack and prompts before setting an SLA.
Get your full vLLM sizing report
We will email you a personalized throughput and GPU count breakdown across model size and quantization options, and a Netray inference specialist will follow up with a benchmark plan.
No spam. Your results stay private. Unsubscribe anytime.
How continuous batching changes the math
A single isolated request is memory-bandwidth bound: the GPU reads every weight from memory for every generated token, so a 70B model on two H100s tops out around single-digit tens of tokens per second per stream. Continuous batching changes the unit economics by processing many sequences per forward pass, amortizing that memory read across dozens of concurrent decodes. The gain is not linear, it flattens once the workload shifts from memory-bound to compute-bound, which is why this estimator caps effective concurrency near 40 sequences and applies a scheduling overhead factor rather than assuming perfect scaling.
- Single-stream decode speed is dominated by memory bandwidth divided by model size, not GPU compute (FLOPS).
- Continuous batching (vLLM's core innovation) lets one GPU serve many users concurrently for a small latency cost per user.
- Throughput gains from batching saturate around 32-48 concurrent sequences on typical mid-size deployments.
- PagedAttention reduces KV cache fragmentation, which is what makes high concurrency practical in the first place.
Reading the result against your SLA
Aggregate tokens per second is a capacity number, not a latency number: it tells you how much total work the replica can absorb, not how fast any one user's response streams back. Convert it to requests per minute using your typical response length, then compare that against your peak concurrent user count. If requests per minute falls short of peak demand, you need more replicas, a smaller or more quantized model, or a faster GPU class, not just a bigger batch size, because batch size beyond the saturation point buys little additional throughput while degrading per-user latency.
Where teams get vLLM sizing wrong
The most common mistake is sizing off a vendor's peak-benchmark tokens-per-second figure, which is measured at maximum batch saturation with short prompts and rarely reflects production traffic mixed with long context and bursty load. The second is ignoring that quantization changes the model footprint input directly: an FP8 70B model reads roughly 70 GB instead of 140 GB, which can meaningfully change how many GPUs you need for the same throughput target. Always re-run sizing after a quantization decision, not before it.
- Benchmark with your actual prompt and response length distribution, not a synthetic short-prompt test.
- Re-check GPU count needed after any quantization change, since it directly shrinks the memory footprint input.
- Reserve headroom for traffic bursts; sizing at exactly average load produces visible latency spikes at peak.
How Netray sizes production vLLM deployments
Netray deploys open-weight models on vLLM, SGLang, and TensorRT-LLM inside customer networks for manufacturers and defense contractors who cannot send inference traffic to a public API. We benchmark candidate models and quantization levels against your actual production prompts, tune batching and scheduling parameters, and validate throughput against your real peak concurrency before you commit to hardware. Engagements typically begin with a two-week benchmarking sprint that produces a costed, right-sized serving architecture.
Frequently Asked Questions
Why does adding more GPUs not scale throughput linearly?
Tensor parallelism across GPUs adds interconnect communication overhead at every layer boundary, so doubling GPUs rarely doubles throughput. In practice you gain roughly 1.5 to 1.8 times per doubling on NVLink-connected systems and less over PCIe. Past four to eight GPUs for a single replica, it is usually more efficient to run additional independent replicas behind a load balancer than to keep growing one tensor-parallel group.
What is a realistic bandwidth efficiency to assume for vLLM?
Most production vLLM deployments achieve 45-65% of theoretical GPU memory bandwidth after accounting for kernel launch overhead, PagedAttention block management, and scheduling. Newer versions with chunked prefill and prefix caching push toward the higher end, especially for workloads with repeated system prompts. Treat anything above 70% as optimistic unless you have measured it on your own hardware and model.
Does batch size have a downside once throughput saturates?
Yes. Past the saturation point, adding more concurrent sequences no longer meaningfully raises aggregate throughput but does increase the time each individual user waits for their tokens, since the GPU is now dividing attention across more work than it can process in parallel. The practical target is the highest batch depth that still meets your per-user latency SLA, not the highest batch depth the GPU can technically accept.
How much does quantization change these throughput numbers?
Substantially. Moving a 70B model from FP16 to FP8 halves the memory footprint, which roughly doubles single-stream decode speed and lets more concurrent sequences fit in the same VRAM. Moving to INT4 can quadruple decode speed relative to FP16. Always re-run this estimator with the quantized model size before finalizing a hardware order.
Get a benchmarked vLLM sizing plan tuned to your models, your prompts, and your peak concurrency target.
Related Tools
KV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
On-Prem AIConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
On-Prem AIMulti-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
Go Deeper
vLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
SGLang vs vLLM: An Honest Serving Comparison
SGLang vs vLLM compared for production LLM serving: RadixAttention vs PagedAttention, structured output performance, ecosystem maturity, and which to pick.
LLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.