Speculative Decoding in Production: What Actually Works
Speculative decoding speeds up autoregressive generation by having a small, fast draft model propose several tokens ahead, then having the full target model verify them in a single forward pass, accepting the tokens that match what the target model would have produced anyway. The catch that benchmark demos gloss over: the speedup shrinks and can disappear entirely under high concurrency, because speculative decoding trades extra compute (running two models, verifying multiple tokens per step) for reduced latency, and that tradeoff only pays off when the GPU has spare compute capacity to spend. At low concurrency with memory-bound generation, speculative decoding can deliver a genuine 1.5x to 2.5x speedup. At high concurrency where the GPU is already compute-saturated by continuous batching, the benefit compresses toward zero or can even regress throughput.
How Speculative Decoding Works
Standard autoregressive generation is memory-bandwidth bound, not compute bound, because generating one token requires reading the entire model's weights from GPU memory but only performs a small amount of compute per token. Speculative decoding exploits this by having a small draft model (often a distilled or smaller version of the target, or in n-gram methods, a lookup against recent context) propose k tokens ahead quickly, then the large target model verifies all k tokens in a single parallel forward pass, which uses roughly the same memory bandwidth as generating one token normally but validates several at once. Accepted tokens save real latency; rejected tokens fall back to standard generation from that point. The method is exact, not approximate: the output distribution matches what the target model alone would have produced, since rejected tokens are always resampled from the target model's own distribution.
Draft Models vs N-Gram and Lookahead Methods
There are three practical approaches to generating draft tokens. A separate smaller draft model (for example, using a 1B model to draft for a 70B target) gives the highest acceptance rates when the draft and target are well aligned, typically from the same model family, but adds deployment complexity: you now serve two models and must keep them in sync across updates. Self-speculative methods like Medusa or EAGLE add small prediction heads trained on top of the target model itself, avoiding a separate model but requiring model-specific training. N-gram or lookahead methods draft by matching against recent context or a static corpus, requiring no additional model at all, which works surprisingly well for tasks with high repetition such as code editing or structured extraction, but has a much lower acceptance rate on open-ended creative generation. vLLM supports multiple speculative decoding backends; pick based on whether you have a compatible draft model available and how much deployment complexity you can absorb.
- Separate draft model: highest acceptance rate, doubles the models you deploy and version together
- Self-speculative (Medusa, EAGLE-style heads): no second model to serve, requires target-model-specific training
- N-gram/lookahead: zero extra model, strong on repetitive tasks like code editing, weak on open-ended generation
- Acceptance rate is the metric to track; below roughly 50 percent, overhead usually exceeds benefit
Why Benchmarks Overstate the Real-World Speedup
Vendor and paper benchmarks for speculative decoding are typically run at low concurrency, often batch size 1, which is exactly where the memory-bandwidth-bound nature of generation makes speculative decoding shine. Production serving under continuous batching runs many concurrent sequences, which already keeps the GPU compute pipeline busy and shifts the bottleneck from memory bandwidth toward compute. Speculative decoding still runs the verification step's extra compute cost under this condition, but the memory-bandwidth savings that justified it at batch size 1 are less available to reclaim, since the GPU was not memory-bound to begin with. The practical result: speculative decoding delivers its best gains for low-concurrency, latency-sensitive single-user or few-user deployments, and its benefit compresses as concurrent load rises toward the levels a typical production multi-tenant deployment sees.
When Speculative Decoding Is Worth Deploying
It is worth deploying when your workload is genuinely latency-sensitive at low-to-moderate concurrency: an interactive coding assistant, a single power-user chat deployment, or any scenario where you are not running near GPU compute saturation. It is generally not worth the added complexity for high-concurrency multi-tenant API serving at scale, where continuous batching is already extracting most of the achievable GPU utilization and the compute overhead of drafting and verifying eats into the capacity you would otherwise sell as throughput. Measure before committing: run your actual production traffic replay with and without speculative decoding enabled and compare both latency and aggregate throughput, not just the headline per-request speedup number from a low-concurrency test.
- Good fit: latency-sensitive, low-to-moderate concurrency, single power user or small team deployments
- Poor fit: high-concurrency multi-tenant API serving already near compute saturation under continuous batching
- Always measure with your real traffic replay at your real concurrency, not a batch-size-1 benchmark
- Track acceptance rate in production; a drafting model that drifts from the target's distribution over time silently erodes the benefit
How Netray Evaluates Speculative Decoding for Clients
Netray tests speculative decoding against a client's actual concurrency profile before recommending it, because the gap between benchmark and production behavior is large enough to reverse the decision entirely. For latency-sensitive single-team deployments, such as an internal coding assistant, we have deployed draft-model and n-gram speculative decoding with measured 1.5x to 2x real-world speedups. For high-concurrency production API workloads we have equally often recommended against it after measurement showed the benefit compressed to noise once continuous batching was already saturating GPU compute. This kind of measure-first discipline is standard across our inference optimization engagements, covered further in our cost optimization playbook.
Frequently Asked Questions
Does speculative decoding actually work in production LLM serving?
It works well for low-to-moderate concurrency, latency-sensitive deployments, delivering genuine 1.5x to 2.5x speedups in those conditions. It works far less well at high concurrency, where continuous batching already keeps the GPU compute-saturated, since speculative decoding trades extra compute for reduced memory-bandwidth pressure, and that tradeoff has little room to pay off once compute is already the bottleneck. Always test against your actual production concurrency before committing.
What is the difference between draft-model and n-gram speculative decoding?
Draft-model speculative decoding uses a separate, smaller model from the same family to propose tokens, giving the highest acceptance rates but requiring you to deploy and version two models together. N-gram or lookahead methods draft by matching recent context or a static corpus with no separate model needed, working well on repetitive tasks like code editing but with lower acceptance on open-ended creative generation. Self-speculative methods like EAGLE add prediction heads to the target model itself as a middle ground.
What acceptance rate makes speculative decoding worthwhile?
Below roughly 50 percent acceptance, the overhead of running the draft-and-verify cycle usually exceeds the benefit, since rejected tokens fall back to standard generation and you have paid extra compute for no gain. Well-aligned draft and target models on suitable tasks often see 60 to 80 percent acceptance. Track acceptance rate as a production metric, since a draft model that drifts from the target's output distribution over time silently erodes the speedup without an obvious failure signal.
Key Takeaways
- 1How Speculative Decoding Works: Standard autoregressive generation is memory-bandwidth bound, not compute bound, because generating one token requires reading the entire model's weights from GPU memory but only performs a small amount of compute per token. Speculative decoding exploits this by having a small draft model (often a distilled or smaller version of the target, or in n-gram methods, a lookup against recent context) propose k tokens ahead quickly, then the large target model verifies all k tokens in a single parallel forward pass, which uses roughly the same memory bandwidth as generating one token normally but validates several at once.
- 2Draft Models vs N-Gram and Lookahead Methods: There are three practical approaches to generating draft tokens. A separate smaller draft model (for example, using a 1B model to draft for a 70B target) gives the highest acceptance rates when the draft and target are well aligned, typically from the same model family, but adds deployment complexity: you now serve two models and must keep them in sync across updates.
- 3Why Benchmarks Overstate the Real-World Speedup: Vendor and paper benchmarks for speculative decoding are typically run at low concurrency, often batch size 1, which is exactly where the memory-bandwidth-bound nature of generation makes speculative decoding shine. Production serving under continuous batching runs many concurrent sequences, which already keeps the GPU compute pipeline busy and shifts the bottleneck from memory bandwidth toward compute.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
Speculative Decoding Speedup Calculator
Model the real decode speedup speculative decoding delivers from your draft model's acceptance rate, proposal depth, and per-step overhead.
Free ToolAI Inference Latency Calculator
Estimate decode throughput, time to first token, and end-to-end response time for a self-hosted model from GPU memory bandwidth, parameter count, and quantization.
Free ToolvLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Terms used in this article
Considering speculative decoding for your inference deployment? Netray will benchmark it against your real concurrency profile before you build around an assumption.
Related Resources
vLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
AI & AutomationLLM Batching and Throughput Tuning: A Field Guide
Tune LLM inference batching and throughput: max-num-seqs, latency-throughput tradeoffs, load testing methodology, and scaling patterns that hold up.
AI & AutomationThe LLM Inference Cost Optimization Playbook
Cut LLM inference costs with a practical playbook: quantization, batching, GPU right-sizing, caching, and the on-prem vs API breakeven math for 2026.