LLM Batching and Throughput Tuning: A Field Guide
Batching and throughput tuning for LLM inference is fundamentally a latency-versus-throughput tradeoff, and the mistake most teams make is optimizing one number in isolation without a stated latency budget to constrain it against. Continuous batching in vLLM and SGLang already handles the mechanics of efficient batching automatically; the tuning work that remains is setting concurrency limits, understanding how batch composition affects per-request latency, and load testing against a traffic pattern that resembles your production reality rather than a synthetic benchmark. This guide covers the tuning methodology, not just the flags.
The Core Tradeoff: max-num-seqs and Latency Budget
Raising --max-num-seqs allows more concurrent sequences into each batch iteration, which raises aggregate throughput because the GPU processes more work per forward pass, but it also raises per-request latency because each sequence's decode step now competes with more neighbors for the same compute. There is no universally correct value; the correct value is whatever satisfies your stated latency SLA at the highest throughput that SLA allows. Define the SLA first, in concrete terms (for example, p95 ITL under 50ms, p95 TTFT under 2 seconds), then binary-search --max-num-seqs against a realistic load test until you find the ceiling that still meets it. Revisit this number whenever your prompt-length distribution, model, or hardware changes, since the relationship is not stable across those variables.
- Define a concrete latency SLA before tuning (specific p95 TTFT and ITL targets, not a vague goal)
- Binary-search --max-num-seqs against a realistic load test to find the throughput ceiling within that SLA
- Revisit after any change to model, hardware, quantization, or prompt-length distribution
- Separate SLA tiers (interactive vs batch) into separate deployments or priority queues rather than one shared setting
Batch Composition Matters More Than Batch Size Alone
A batch of 32 short requests behaves very differently from a batch of 32 requests where a few are long documents, because compute cost per decode step scales with the mix, not just the count. Chunked prefill (covered in our KV cache optimization guide) addresses the specific problem of a long prompt's prefill blocking others' decode steps, but even with it enabled, a batch heavy with long-generation requests consumes more of your latency budget per iteration than a batch of short ones. If your workload mixes fundamentally different request types, a chat assistant serving both quick lookups and long document summarization, consider separate deployments or priority queues sized independently, rather than tuning one shared --max-num-seqs to compromise between two very different latency profiles.
Load Testing Methodology That Predicts Production Behavior
Synthetic load tests using uniform prompt length and a constant request rate systematically overstate achievable throughput and understate tail latency, because real traffic has bursts, a wide prompt-length distribution, and correlated request patterns (many users hitting the system right after a product announcement, for instance). Build a load test from a replay of real production logs where available, or a synthetic distribution that matches your measured or projected prompt-length percentiles and request-rate variance. Test at your projected peak, not your average, and specifically test the transition from moderate to peak load, since that is where queue depth and TTFT degrade nonlinearly and where a config that looked fine at steady-state can fail during a spike.
- Replay real production request logs when available, rather than synthetic uniform-length tests
- Match your measured prompt-length distribution's percentiles, not just its average
- Test the ramp from moderate to peak load specifically, since nonlinear degradation happens at that transition
- Include realistic request-rate burstiness, not a constant arrival rate, since bursts are what actually cause incidents
Scaling Out vs Scaling Up
When a single instance's tuned --max-num-seqs ceiling is not enough capacity for projected peak traffic, the choice is scaling up (larger GPU, more GPUs via tensor parallelism per instance) or scaling out (more replica instances behind a load balancer). Scaling out is generally the safer default for throughput growth, since it adds fault isolation and enables rolling deployment, while scaling up mainly helps when a single request needs more compute or memory than one GPU provides (a model too large for one card, requiring tensor parallelism regardless of throughput considerations). For most concurrency-driven throughput needs where the model already fits comfortably on your current hardware, add replicas rather than reaching for bigger or more tensor-parallel GPUs per instance.
How Netray Tunes Batching for Client Deployments
Netray establishes a concrete latency SLA with the client before touching any config flag, then builds a load test from real traffic logs or a matched synthetic distribution to find the actual --max-num-seqs ceiling within that SLA on the client's specific hardware and model. Where workloads mix fundamentally different request shapes, interactive chat alongside batch document processing, we typically recommend separate deployment tiers rather than a compromised shared configuration. This tuning methodology is a standard part of our vLLM production deployment and cost optimization engagements, and it is usually where a client's first production incident gets prevented rather than diagnosed after the fact.
Frequently Asked Questions
What is the right value for max-num-seqs in vLLM?
There is no universal correct value; it depends on your latency SLA. Define a concrete target (for example, p95 inter-token latency under 50ms) then binary-search --max-num-seqs against a realistic load test to find the highest throughput that still meets that target on your specific model and hardware. Revisit the value whenever your prompt-length distribution, model, hardware, or quantization changes, since the relationship between concurrency and latency is not stable across those variables.
Should I scale up or scale out for more LLM serving throughput?
Scale out (add replica instances behind a load balancer) is generally the safer default for growing throughput, since it adds fault isolation and enables rolling deployment without downtime. Scale up (bigger GPU or more GPUs per instance via tensor parallelism) mainly matters when a single request needs more compute or memory than one GPU provides, such as a model too large for one card. If your model already fits comfortably on current hardware, add replicas rather than reaching for larger per-instance hardware.
Why do synthetic load tests overstate real-world LLM serving throughput?
Synthetic tests typically use uniform prompt length and a constant request arrival rate, which does not match real production traffic that has bursty arrival patterns and a wide, often long-tailed prompt-length distribution. This systematically understates tail latency and overstates achievable throughput. Build load tests from replayed production logs where possible, or a synthetic distribution matched to your measured prompt-length percentiles and request-rate variance, and specifically test the ramp from moderate to peak load.
Key Takeaways
- 1The Core Tradeoff: max-num-seqs and Latency Budget: Raising --max-num-seqs allows more concurrent sequences into each batch iteration, which raises aggregate throughput because the GPU processes more work per forward pass, but it also raises per-request latency because each sequence's decode step now competes with more neighbors for the same compute. There is no universally correct value; the correct value is whatever satisfies your stated latency SLA at the highest throughput that SLA allows.
- 2Batch Composition Matters More Than Batch Size Alone: A batch of 32 short requests behaves very differently from a batch of 32 requests where a few are long documents, because compute cost per decode step scales with the mix, not just the count. Chunked prefill (covered in our KV cache optimization guide) addresses the specific problem of a long prompt's prefill blocking others' decode steps, but even with it enabled, a batch heavy with long-generation requests consumes more of your latency budget per iteration than a batch of short ones.
- 3Load Testing Methodology That Predicts Production Behavior: Synthetic load tests using uniform prompt length and a constant request rate systematically overstate achievable throughput and understate tail latency, because real traffic has bursts, a wide prompt-length distribution, and correlated request patterns (many users hitting the system right after a product announcement, for instance). Build a load test from a replay of real production logs where available, or a synthetic distribution that matches your measured or projected prompt-length percentiles and request-rate variance.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolAI Inference Latency Calculator
Estimate decode throughput, time to first token, and end-to-end response time for a self-hosted model from GPU memory bandwidth, parameter count, and quantization.
Free ToolConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
Terms used in this article
Need your inference deployment load tested against real traffic before it hits production? Netray will define the SLA, build the load test, and tune the config to match.
Related Resources
vLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
AI & AutomationKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
AI & AutomationLLM Observability: TTFT, ITL, Throughput, and GPU Dashboards
LLM inference observability: track TTFT, inter-token latency, throughput, and GPU utilization with dashboards that catch problems before users report them.