Why LLM inference is much slower in production than your benchmarks, and how to fix it
vllm serving is way slower in production than the benchmark numbers showed
Also appears as
- tokens per second drops once real traffic hits the endpoint
- inference is fast in a single curl test but slow under the load balancer
Short answer
Production inference is usually slower than a benchmark because real traffic exposes problems a single-request test never hits: full-precision weights instead of BF16/FP16, no continuous batching so requests queue one at a time, CPU-bound tokenization or post-processing sitting in front of the GPU call, or hardware whose memory bandwidth cannot keep up with the model size and concurrency you actually see. Fix the dtype and batching first, they account for most of the gap, then profile the request path for CPU-bound steps.
Affects: Any self-hosted serving stack (vLLM, TGI, TensorRT-LLM, plain transformers), most visible on 7B-70B parameter models running on 24GB-80GB GPUs
Find the bottleneck in one pass
- 1Confirm the model is loaded in bfloat16 or float16, not float32: check the serving command or model config for dtype/torch_dtype.
- 2Confirm you are running a continuous-batching server (vLLM, TGI, SGLang) and not a naive for-loop over requests.
- 3Run nvidia-smi during a burst of concurrent requests: if GPU utilization stays under 40-50 percent, the bottleneck is upstream (batching, network, or CPU), not the GPU.
- 4Time the request end to end versus GPU-only time (server logs usually separate these); a large gap points to tokenization, guardrails, or serialization overhead outside the model call.
- 5If GPU memory bandwidth is the limiter (large model, small GPU), apply quantization (AWQ, GPTQ, or FP8) or move to a GPU with more memory bandwidth for the concurrency you need.
How to confirm this is your problem
- Single test requests feel fine but latency climbs once concurrent users are added
- Tokens per second in production is a fraction of the number quoted in the model or serving framework's benchmarks
- nvidia-smi shows low utilization even while users report the service is slow
- Response times vary wildly between requests of similar length
Root causes and fixes
Model loaded in FP32 (full precision) instead of BF16/FP16
FP32 doubles the bytes that must move from GPU memory for every weight and activation, and it does not use the tensor core paths that modern GPUs are optimized for. Inference is memory-bandwidth-bound during decode, so doubling the bytes read per token roughly halves throughput, and you also lose the tensor-core speedup that BF16/FP16 unlock.
Fix: Load the model with bfloat16 (fp16 on older GPUs without bf16 support). In vLLM this is the default when the checkpoint supports it; verify with --dtype bfloat16 explicitly rather than relying on an unexamined default.
python -c "import torch; print(torch.cuda.is_bf16_supported())" vllm serve MODEL_NAME --dtype bfloat16
No continuous batching, requests are effectively served one at a time
Decoding one token at a time for one request leaves most of the GPU's compute idle because the workload is memory-bound, not compute-bound, at batch size 1. A continuous-batching scheduler interleaves the decode steps of many requests so the same memory read serves many sequences per step, which is where the real throughput gain comes from.
Fix: Serve with vLLM, TGI, or SGLang instead of a hand-rolled loop calling model.generate() per request. Confirm --max-num-seqs is set high enough to admit multiple concurrent requests rather than defaulting to a low value.
vllm serve MODEL_NAME --max-num-seqs 256
Serving on hardware whose memory bandwidth cannot support the model size and concurrency in use
Decode-phase throughput is bounded by how fast the GPU can stream weights (and KV cache) from HBM. A 70B model on a 24GB card forced into aggressive offload, or any model pushed far beyond the concurrency its memory bandwidth supports, will show low and inconsistent throughput regardless of software tuning.
Fix: Right-size the GPU to the model and target concurrency, or apply quantization to shrink the memory footprint and the bytes moved per token. Use a sizing calculator before committing to hardware.
CPU-bound pre/post-processing (tokenization, guardrails, JSON assembly) serialized around the GPU call
Python-level tokenization, regex-based guardrails, or synchronous logging/formatting steps run on the CPU and, if not overlapped with GPU work, add straight-line latency to every request. This shows up as high end-to-end latency with normal GPU utilization, because the GPU is idle while the CPU step runs.
Fix: Profile the request handler separately from the model call. Move tokenization to the batch scheduler (vLLM does this internally), and make guardrail/post-processing steps async or run them concurrently with the next request's prefill.
Model not quantized when VRAM is tight, forcing CPU offload or paging
When a model barely fits or does not fit in VRAM, frameworks may offload layers to CPU RAM or swap KV cache pages more aggressively, both of which are orders of magnitude slower than pure GPU execution because of the PCIe transfer cost.
Fix: Apply AWQ, GPTQ, or FP8 quantization to shrink the model until it comfortably fits with headroom for KV cache, instead of relying on offload.
Diagnostic commands
Check GPU utilization under concurrent load
nvidia-smi --query-gpu=utilization.gpu,utilization.memory,memory.used --format=csv -l 1
Utilization consistently under 40-50 percent during a burst of concurrent requests points to a bottleneck outside the GPU (batching config, network, or CPU preprocessing). Utilization near 90-100 percent means the GPU itself is the limiter and you need quantization or more/better hardware.
Confirm the serving dtype
curl -s localhost:8000/v1/models | python -m json.tool
Some serving frameworks report the active dtype in the models endpoint or startup logs. If it shows float32, that is the first fix regardless of anything else.
Compare single-request vs concurrent latency
for i in 1 2 4 8 16; do echo concurrency=$i; done
Run the same prompt at increasing concurrency levels and record p50/p99 latency. If per-request latency barely rises with concurrency, batching is working; if it rises linearly, requests are being serialized.
Stopping it from happening again
- Load test with realistic concurrent traffic before launch, not a single-request smoke test
- Pin the serving dtype explicitly in your deployment config instead of trusting a framework default
- Add a GPU utilization and p99 latency dashboard to your monitoring from day one
- Re-run the sizing exercise whenever you change model, concurrency target, or context length
When this becomes an architecture problem
If you have confirmed correct dtype, continuous batching, and reasonable GPU utilization but throughput still cannot meet your SLO at the concurrency you need, the problem is capacity, not configuration: you need a bigger or additional GPU, tensor parallelism across multiple GPUs, or a smaller/quantized model, which is an architecture and hardware sizing decision rather than a tuning fix.
Frequently asked questions
Why is my model fast in a Jupyter notebook but slow behind an API?
A notebook typically runs one request at a time with no network, serialization, or guardrail overhead, and often no continuous batching either. Production adds concurrent traffic, HTTP overhead, and pre/post-processing, all of which the notebook test never exercises. Benchmark with the actual serving stack and realistic concurrency, not a bare model.generate() call.
Does upgrading to a newer GPU always fix slow inference?
Only if the current GPU's memory bandwidth or capacity is actually the bottleneck. If the real issue is batch size 1, FP32 weights, or CPU-bound preprocessing, a faster GPU will still be underutilized. Fix software-level issues first, then re-measure before spending on hardware.
What is the single highest-leverage fix for slow production inference?
In practice, enabling continuous batching at a sensible max-num-seqs value combined with BF16/FP16 weights closes most of the gap between benchmark and production throughput. Both are configuration changes, not hardware changes, and should be verified before any capacity investment.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolConcurrent Users Per GPU Calculator
Estimate how many connected users one GPU can support, accounting for both VRAM limits and throughput limits, plus the fact that most users are not actively streaming at any given moment.
Free ToolLLM Latency Budget Planner
Break total response time into time-to-first-token, generation time, and network overhead, then see your exact margin or shortfall against a target SLA.
Related problems
GPU utilization stays low during LLM inference even under load
Low GPU utilization during inference almost always means the GPU is waiting on something else: request concurrency is too low for the batching scheduler to fill, the client code is calling the server synchronously one request at a time, tokenization or network I/O is serialized in front of the GPU call, or max-num-seqs is set too low to admit enough concurrent sequences. Raising effective concurrency, either by fixing the client or the server's admission limits, is almost always the fix, not more GPU compute.
Not sure how to tune batch size for LLM inference throughput vs latency
Batch size is a direct tradeoff between throughput and per-request latency: larger batches keep the GPU busier and raise aggregate tokens-per-second, but each additional concurrent sequence adds contention for the same compute and memory, increasing the latency of every individual request. The right batch size is not the largest one that fits in memory, it is the point on that curve, the knee, where added throughput per unit of batch size starts costing more latency than your SLO allows, and it should be derived from measurement against your actual latency target, not a fixed default.
High time to first token (TTFT) on LLM inference requests
High time to first token almost always comes from one of four sources: a long prompt makes the prefill pass compute-bound and simply takes time to process, the server has no prefix caching so a repeated system prompt or RAG context is recomputed on every request, the model or GPU had to cold-start (weights loading, CUDA graph capture, JIT warmup), or the request sat in a queue behind other requests before its prefill even began. Prefix caching and admission-aware queueing fix most production cases.
Cost per token for self-hosted LLM inference is higher than expected
Cost per token is dominated by GPU utilization far more than by hardware choice: an underutilized GPU serving one request at a time can cost more per token than a well-tuned smaller GPU serving at full continuous-batching concurrency. Before concluding self-hosting is not worth it, check utilization, whether the model is right-sized for the task, whether quantization and prefix/response caching are in use, and whether the comparison to an API is even apples-to-apples once amortization is accounted for.
GuideLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideThe LLM Inference Cost Optimization Playbook
Cut LLM inference costs with a practical playbook: quantization, batching, GPU right-sizing, caching, and the on-prem vs API breakeven math for 2026.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.