How to tune LLM inference batch size for the right throughput vs latency tradeoff
increasing max-num-seqs or batch size does not improve throughput as expected
Also appears as
- unsure what batch size to use for production llm serving
- higher batch size makes latency worse without helping throughput
Short answer
Batch size is a direct tradeoff between throughput and per-request latency: larger batches keep the GPU busier and raise aggregate tokens-per-second, but each additional concurrent sequence adds contention for the same compute and memory, increasing the latency of every individual request. The right batch size is not the largest one that fits in memory, it is the point on that curve, the knee, where added throughput per unit of batch size starts costing more latency than your SLO allows, and it should be derived from measurement against your actual latency target, not a fixed default.
Affects: Any continuous-batching LLM server (vLLM, TGI, SGLang) tuning max-num-seqs or equivalent batch limits
Find your knee, then set the limit
- 1Run a load test sweeping concurrency (batch size) from low to high, recording both aggregate throughput and p50/p99 per-request latency at each level.
- 2Plot or tabulate throughput and latency together; identify the point where throughput gains per added concurrent request flatten while latency keeps climbing, that is the knee.
- 3Set max-num-seqs (or equivalent) at or just below that knee if latency matters, or push further past it only if pure throughput is the priority and latency SLO allows it.
- 4Re-run the sweep whenever prompt length distribution, output length, or model changes meaningfully, since the knee shifts with all three.
- 5Validate the chosen setting under realistic mixed-length traffic, not just uniform short prompts, since long prompts consume disproportionate batch capacity.
How to confirm this is your problem
- Throughput keeps climbing as concurrency rises, but individual request latency also keeps climbing without a clear plateau
- A batch size that worked well in testing produces missed latency SLOs once real, more varied traffic arrives
- Increasing max-num-seqs past a certain point causes preemption or memory pressure instead of more throughput
- No clear answer for what batch size to configure, settings are copied from another deployment or a default
Root causes and fixes
Batch size set without measuring the actual throughput/latency tradeoff for this workload
The relationship between batch size and both throughput and latency is workload-specific, driven by model size, GPU, and prompt/output length distribution, so a value that works well for one deployment can be badly miscalibrated for another. Copying a default or a number from a blog post skips the one step that actually determines the right setting: measuring your own throughput-latency curve.
Fix: Run a concurrency sweep against your real model, hardware, and representative traffic, and choose max-num-seqs based on where your latency SLO is crossed, not a borrowed number.
Batch size set above the point where added concurrency still helps throughput meaningfully
Past a certain concurrency, additional in-flight sequences contend for the same fixed compute and memory bandwidth, so each additional sequence adds proportionally less new throughput while still adding queueing and compute contention that raises every request's latency, a classic diminishing-returns curve.
Fix: Identify the knee via a concurrency sweep, and cap max-num-seqs there rather than at the theoretical maximum the GPU's memory could support.
SLO-driven sizing not applied, batch size chosen for peak throughput instead of the latency budget
Maximizing aggregate throughput and meeting a p99 latency SLO are frequently in direct tension; optimizing purely for the former by maximizing batch size will often violate the latter, especially for interactive or streaming use cases where users notice per-token latency directly.
Fix: Start from your actual latency SLO (e.g. p99 time-per-output-token under X ms) and work backward to the maximum batch size that satisfies it, treating throughput as the outcome of that constraint rather than the primary target.
Batch size tuned only against uniform short prompts, not the real mixed-length distribution
A small number of long-prompt or long-generation requests within a batch can consume disproportionate compute and KV cache relative to short ones, so a batch size validated only against short, uniform test prompts can behave very differently, and worse, once real traffic with a long tail of prompt lengths arrives.
Fix: Include a realistic mix of short and long prompts/generations in the load test used to determine batch size, not a synthetic uniform benchmark.
Confusing batch size (max-num-seqs) with max-num-batched-tokens, leading to mismatched expectations
Some servers expose both a sequence-count limit and a token-count-per-step limit; tuning only one while leaving the other at a restrictive default can cap effective throughput in a way that looks like a batch size problem but is actually the other parameter constraining it.
Fix: Check both the max concurrent sequences setting and the max tokens processed per scheduling step, and tune them together rather than assuming a single parameter controls batching.
vllm serve MODEL_NAME --max-num-seqs 128 --max-num-batched-tokens 8192
Diagnostic commands
Sweep concurrency and record throughput plus p50/p99 latency
for c in 4 8 16 32 64 128; do echo concurrency=$c; done
Tabulating throughput and latency together across this sweep is the only reliable way to find the knee; eyeballing a single concurrency level tells you nothing about the tradeoff shape.
Check current server-side batching configuration
curl -s localhost:8000/v1/models | python -m json.tool
Confirm what max-num-seqs and max-num-batched-tokens are actually set to, since a mismatch between assumed and actual config is a common source of confusing benchmark results.
Monitor GPU utilization and memory alongside the sweep
nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1
If utilization is already near 100 percent well before the latency knee appears, you are compute-bound and further batch size increases will mostly add latency; if memory approaches its ceiling first, you are at risk of preemption before you are compute-bound.
Stopping it from happening again
- Treat batch size as a measured, workload-specific setting, not a value to copy from documentation or another deployment
- Define your latency SLO before tuning throughput, and size batch limits to satisfy it
- Re-run the concurrency sweep after any change to model, hardware, or typical prompt/output length
- Test with representative mixed-length traffic, not uniform synthetic prompts
When this becomes an architecture problem
If the measured knee of the throughput/latency curve still cannot deliver both your required throughput and your latency SLO on current hardware, that is a capacity problem, not a tuning problem: you need more GPUs (horizontal scaling or tensor parallelism), a smaller/faster model, or a relaxed SLO, all of which are architecture decisions beyond batch size configuration.
Frequently asked questions
Is there a universal 'good' batch size for LLM serving?
No. It depends on model size, GPU, prompt and output length distribution, and your specific latency SLO, all of which vary by deployment. Any specific number quoted online is only a starting point for your own measurement, not a value to adopt directly.
Should I always maximize batch size for the best cost per token?
Maximizing batch size generally improves throughput and therefore cost efficiency up to the knee of the curve, but pushing past it risks preemption from KV cache exhaustion, which actively hurts both cost and latency. The cost-optimal point is close to, but not necessarily at, the same knee that satisfies your latency SLO.
How often should batch size settings be revisited?
Any time the model, GPU, or the shape of production traffic (average prompt length, output length, concurrency pattern) changes meaningfully. A setting tuned for one traffic mix can be meaningfully wrong for another, so treat it as something to re-validate periodically, not a one-time decision.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
Free ToolLLM Latency Budget Planner
Break total response time into time-to-first-token, generation time, and network overhead, then see your exact margin or shortfall against a target SLA.
Free ToolKV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Related problems
LLM serving throughput collapses once load increases past a certain point
Throughput collapsing past a load threshold is almost always KV cache exhaustion: once in-flight requests' combined KV cache exceeds available GPU memory, the scheduler preempts some sequences, discarding their KV cache and forcing a full recompute when they resume, which burns GPU cycles on redundant work instead of new tokens. The fix is admission control that keeps the server below its true KV cache-limited concurrency, not just retrying harder or adding a bigger queue.
GPU utilization stays low during LLM inference even under load
Low GPU utilization during inference almost always means the GPU is waiting on something else: request concurrency is too low for the batching scheduler to fill, the client code is calling the server synchronously one request at a time, tokenization or network I/O is serialized in front of the GPU call, or max-num-seqs is set too low to admit enough concurrent sequences. Raising effective concurrency, either by fixing the client or the server's admission limits, is almost always the fix, not more GPU compute.
LLM inference is much slower in production than in benchmarks
Production inference is usually slower than a benchmark because real traffic exposes problems a single-request test never hits: full-precision weights instead of BF16/FP16, no continuous batching so requests queue one at a time, CPU-bound tokenization or post-processing sitting in front of the GPU call, or hardware whose memory bandwidth cannot keep up with the model size and concurrency you actually see. Fix the dtype and batching first, they account for most of the gap, then profile the request path for CPU-bound steps.
Inference gets much slower as context length grows toward 32k, 64k, or 128k tokens
Long-context slowness is not a bug, it is the fundamental cost structure of attention: self-attention compute scales roughly quadratically with sequence length in the prefill pass, and the KV cache that must be stored per token scales linearly with sequence length, multiplying memory pressure across every concurrent request. A 128k-token context is not the same cost as eight 16k-token contexts, it is dramatically more expensive per request in both compute and memory, which is why advertised max context length is rarely the practical operating point for concurrent production traffic.
GuideLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.