Why time to first token is high on your LLM endpoint, and how to fix it
time to first token is several seconds even for short prompts
Also appears as
- streaming response takes a long time before the first token appears
- ttft spikes under load even though total generation time looks normal
Short answer
High time to first token almost always comes from one of four sources: a long prompt makes the prefill pass compute-bound and simply takes time to process, the server has no prefix caching so a repeated system prompt or RAG context is recomputed on every request, the model or GPU had to cold-start (weights loading, CUDA graph capture, JIT warmup), or the request sat in a queue behind other requests before its prefill even began. Prefix caching and admission-aware queueing fix most production cases.
Affects: Any streaming LLM endpoint, especially RAG applications with long injected context and multi-tenant endpoints serving many concurrent users
Cut TTFT in the right order
- 1Measure prefill time separately from queue time and decode time; most serving frameworks expose this in metrics or logs.
- 2If the prompt (especially a shared system prompt or RAG context) is long and repeated across requests, enable prefix caching (vLLM's automatic prefix caching or SGLang's RadixAttention) so the shared prefix is computed once.
- 3If TTFT is high only on the first request after startup, that is a cold start: warm the server with a few dummy requests before accepting real traffic.
- 4If TTFT rises specifically under concurrent load, check queue depth and admission control; the request may be waiting for a batching slot, not actually computing.
- 5For very long prompts, consider chunked prefill so a single huge prefill does not block other requests' decode steps in the same batch.
How to confirm this is your problem
- First streamed token takes multiple seconds even on a lightly loaded server
- TTFT scales with prompt length far more than expected
- TTFT is fine on isolated tests but spikes as soon as several users hit the endpoint at once
- The very first request after a deploy is dramatically slower than subsequent ones
Root causes and fixes
Long prompt makes prefill compute-bound
Prefill processes the entire input prompt in one forward pass, and unlike decode, its cost scales with the square of sequence length for the attention computation (before optimizations) plus linearly for the feed-forward layers. A long system prompt, few-shot examples, or a large RAG context can make prefill take hundreds of milliseconds to seconds on its own, before a single output token is produced.
Fix: Shorten the prompt where possible, and cache the shared prefix (system prompt, RAG template) with prefix caching so only the unique tail of the prompt needs a fresh prefill.
No prefix caching for a shared system prompt or repeated context
Without prefix caching, every request recomputes the KV cache for the entire prompt from scratch, even if 90 percent of it (system prompt, tool definitions, RAG boilerplate) is identical to the previous request. That recomputation is pure wasted prefill time that directly adds to TTFT.
Fix: Enable automatic prefix caching in vLLM or use SGLang's RadixAttention, and structure prompts so the shared portion comes first and is byte-identical across requests.
vllm serve MODEL_NAME --enable-prefix-caching
Cold start: weight loading, CUDA graph capture, or kernel warmup on the first request
The first request after server startup (or after autoscaling spins up a new replica) can trigger CUDA graph capture, JIT compilation of fused kernels, or lazy weight loading, all of which add one-time latency that has nothing to do with the prompt itself.
Fix: Send a handful of warmup requests immediately after the server reports healthy, before routing real traffic to it, especially important for autoscaled replicas.
Queueing delay before prefill starts
If max-num-seqs or the scheduler's admitted-request limit is already at capacity, a new request waits in queue before its prefill even begins. From the client's perspective this looks identical to slow prefill, but the GPU has not touched the request yet.
Fix: Track queue wait time as a separate metric from compute time, and size max-num-seqs and replica count to your actual concurrency and latency SLO rather than raw throughput alone.
One large prefill blocking other requests' decode steps in the same batch
Without chunked prefill, a very long prompt's prefill can occupy an entire scheduling step, delaying the decode steps of other in-flight requests, which shows up as TTFT (and inter-token latency) spikes correlated with occasional long prompts from other users.
Fix: Enable chunked prefill so long prefills are split across multiple scheduling steps and interleaved with ongoing decodes.
vllm serve MODEL_NAME --enable-chunked-prefill
Diagnostic commands
Break down latency into queue, prefill, and decode
curl -s localhost:8000/metrics | grep -E 'time_to_first_token|queue_time|prefill'
If queue_time dominates, the fix is admission control or capacity, not prefill optimization. If prefill/TTFT dominates and correlates with prompt length, prefix caching or shortening the prompt is the fix.
Compare TTFT on the first request after startup vs. steady state
time curl -s localhost:8000/v1/completions -d '{"prompt":"hello","max_tokens":1}'A large gap between the first call and later calls confirms a cold-start problem that a warmup routine will fix.
Check whether prefix caching is active
curl -s localhost:8000/v1/models | python -m json.tool
Confirm the server was started with prefix caching enabled; a surprising number of slow-TTFT reports trace back to this flag simply being off.
Stopping it from happening again
- Structure prompts with the shared/static portion first so prefix caching can actually help
- Warm every new replica with dummy requests before it receives production traffic
- Set a TTFT-specific SLO and alert, separate from total latency, since it is driven by different causes
- Budget prefill cost explicitly when designing RAG context sizes rather than treating context as free
When this becomes an architecture problem
If TTFT stays high after enabling prefix caching and chunked prefill, and queue time is low, the prompts themselves are simply too long for your latency budget on the current hardware; that calls for a smaller/faster model for the latency-sensitive path, more GPUs to add parallel capacity, or a redesign of how much context you inject per request.
Frequently asked questions
Does a bigger GPU always reduce time to first token?
It helps for compute-bound long prompts, since prefill throughput scales with GPU compute, but it does not fix queueing delay or missing prefix caching. Diagnose which component (queue, prefill, cold start) dominates before assuming more hardware is the answer.
Is TTFT the same thing as latency?
No. TTFT is the time until the first output token streams back, driven mostly by prefill and queueing. Total latency also includes decode time for every subsequent token, which is driven by batching and throughput. A system can have low TTFT and still feel slow if decode is throttled, or the reverse.
How much does prefix caching actually save?
It scales with how much of the prompt is shared across requests. A RAG pipeline with a large fixed system prompt and tool schema can see the cached portion's prefill cost drop to near zero on cache hits, often cutting TTFT by more than half when the shared prefix is a large fraction of total prompt length.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
LLM Latency Budget Planner
Break total response time into time-to-first-token, generation time, and network overhead, then see your exact margin or shortfall against a target SLA.
Free ToolKV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolvLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolRAG Context Window Budget Calculator
Allocate your context window across system prompt, retrieved chunks, and conversation history, then see window utilization and the real cost of every RAG query.
Related problems
LLM inference is much slower in production than in benchmarks
Production inference is usually slower than a benchmark because real traffic exposes problems a single-request test never hits: full-precision weights instead of BF16/FP16, no continuous batching so requests queue one at a time, CPU-bound tokenization or post-processing sitting in front of the GPU call, or hardware whose memory bandwidth cannot keep up with the model size and concurrency you actually see. Fix the dtype and batching first, they account for most of the gap, then profile the request path for CPU-bound steps.
Inference gets much slower as context length grows toward 32k, 64k, or 128k tokens
Long-context slowness is not a bug, it is the fundamental cost structure of attention: self-attention compute scales roughly quadratically with sequence length in the prefill pass, and the KV cache that must be stored per token scales linearly with sequence length, multiplying memory pressure across every concurrent request. A 128k-token context is not the same cost as eight 16k-token contexts, it is dramatically more expensive per request in both compute and memory, which is why advertised max context length is rarely the practical operating point for concurrent production traffic.
LLM serving throughput collapses once load increases past a certain point
Throughput collapsing past a load threshold is almost always KV cache exhaustion: once in-flight requests' combined KV cache exceeds available GPU memory, the scheduler preempts some sequences, discarding their KV cache and forcing a full recompute when they resume, which burns GPU cycles on redundant work instead of new tokens. The fix is admission control that keeps the server below its true KV cache-limited concurrency, not just retrying harder or adding a bigger queue.
Not sure how to tune batch size for LLM inference throughput vs latency
Batch size is a direct tradeoff between throughput and per-request latency: larger batches keep the GPU busier and raise aggregate tokens-per-second, but each additional concurrent sequence adds contention for the same compute and memory, increasing the latency of every individual request. The right batch size is not the largest one that fits in memory, it is the point on that curve, the knee, where added throughput per unit of batch size starts costing more latency than your SLO allows, and it should be derived from measurement against your actual latency target, not a fixed default.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideSGLang vs vLLM: An Honest Serving Comparison
SGLang vs vLLM compared for production LLM serving: RadixAttention vs PagedAttention, structured output performance, ecosystem maturity, and which to pick.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.