Speculative Decoding Speedup Calculator
This free speculative decoding speedup calculator models the real throughput gain from pairing a fast draft model with your target model, using the standard expected-tokens-per-step formula from speculative decoding research. It is built for platform engineers evaluating whether speculative decoding is worth the added serving complexity for their workload. Enter baseline decode speed, how many tokens the draft model proposes per step, its acceptance rate, and its per-step overhead, and the tool returns effective decode speed and net speedup factor.
Your numbers
Target model decode speed without speculative decoding, per stream.
How many tokens the draft model speculates ahead before the target model verifies them, often called gamma.
Share of draft-proposed tokens the target model confirms. Higher means the draft model closely mimics the target.
Extra wall-clock time each verification step costs to run the draft model, relative to a plain decode step.
Your results
Modeling estimate using the standard expected-tokens-per-step formula for speculative decoding. Real speedup also depends on serving engine implementation and batch size interaction. Benchmark on your actual draft and target model pairing.
Get your full speculative decoding benchmark
We will email you a personalized speedup analysis across draft model options and acceptance rate scenarios, and a Netray inference specialist will follow up with an implementation plan.
No spam. Your results stay private. Unsubscribe anytime.
How speculative decoding actually works
A small, fast draft model proposes several tokens ahead in one step. The larger target model then verifies all of them in a single parallel forward pass, which costs roughly the same as generating one token normally, and accepts a prefix of the proposals up to the first rejection. When the draft model's predictions closely match what the target model would have generated, most speculated tokens get accepted, and you effectively get several tokens for the cost of one target-model step. When acceptance is low, you pay draft model overhead for little benefit, since rejected tokens still had to be verified.
- Draft model proposes multiple tokens; target model verifies them all in one parallel pass.
- Speedup comes from token-per-step gains outpacing the overhead of running the draft model.
- Acceptance rate is driven by how closely the draft model's output distribution matches the target model.
- Best-fit draft models are usually smaller versions or distilled versions of the exact target model, not an unrelated small model.
Why acceptance rate dominates the result
The expected-tokens-per-step formula is sharply nonlinear in acceptance rate: at 50% acceptance with four draft tokens per step, you gain roughly 1.9 tokens per step; at 85% acceptance with the same draft depth, you gain roughly 4.4 tokens per step, more than double the throughput gain for a 35-point acceptance improvement. This is why the single highest-leverage investment in a speculative decoding deployment is choosing or fine-tuning a draft model that closely tracks the target model's behavior on your actual traffic distribution, not just picking the smallest available draft model.
When speculative decoding is not worth the complexity
Low acceptance rates below roughly 40-50% frequently produce net speedups close to break-even once draft model overhead is accounted for, while adding real operational complexity: another model to serve, monitor, and keep in sync with target model updates. Speculative decoding pays off most clearly for latency-sensitive single-stream use cases; its benefit shrinks under heavy batched serving because the target model's verification pass competes with other concurrent requests for the same GPU resources.
- Below roughly 45-50% acceptance, overhead often eats most or all of the throughput gain.
- Benefit is largest for single-stream, latency-sensitive serving, not heavily batched high-concurrency workloads.
- Draft model must be kept synchronized with target model updates, adding an ongoing maintenance burden.
How Netray implements speculative decoding for on-prem serving
Netray tunes speculative decoding configurations for customers whose primary constraint is latency rather than raw throughput, common in interactive shop-floor and engineering assistant use cases. We benchmark candidate draft models against your actual target model and traffic distribution to measure real acceptance rate before committing to a production configuration, rather than trusting a published figure from an unrelated workload. Engagements typically include a speculative decoding feasibility benchmark as part of inference tuning.
Frequently Asked Questions
What acceptance rate should I expect in practice?
It depends heavily on how closely the draft model matches the target model's training and behavior. A draft model that is a smaller version of the exact same model family, ideally distilled specifically from the target, commonly achieves 60-85% acceptance. An unrelated small model from a different family typically achieves 30-50%, which often is not enough to overcome draft model overhead. Always measure on your own traffic rather than assuming a published figure transfers.
Does speculative decoding change model output quality?
No, correctly implemented speculative decoding is mathematically guaranteed to produce output statistically identical to the target model running alone, because rejected draft tokens are always replaced with a token sampled correctly from the target model's own distribution. This is fundamentally different from distillation or quantization, which do change output. Speculative decoding is a pure speed optimization with no accuracy tradeoff when implemented correctly.
How does batch size affect speculative decoding benefit?
Benefit shrinks as batch size grows. At low concurrency, the target model's verification pass has spare compute capacity to absorb essentially for free, so speculative decoding is close to pure upside. At high batch depth, that same verification pass now competes with genuine decode work from other concurrent requests for the same GPU resources, so the effective speedup at production concurrency is typically lower than what a single-stream benchmark suggests.
Is speculative decoding worth the added operational complexity?
For latency-sensitive interactive applications with a well-matched draft model achieving above roughly 65-70% acceptance, yes, the throughput and latency gains are usually worth maintaining an additional model. For high-throughput batch workloads already running near-saturated continuous batching, the benefit is smaller and quantization or a more aggressive model choice is often a simpler lever to pull first.
Get a benchmarked speculative decoding configuration measured on your actual model and traffic, not a published estimate.
Related Tools
LLM Latency Budget Planner
Break total response time into time-to-first-token, generation time, and network overhead, then see your exact margin or shortfall against a target SLA.
On-Prem AIvLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
On-Prem AILLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Go Deeper
Speculative Decoding in Production: What Actually Works
Speculative decoding in production: draft models, n-gram methods, and where speedups hold up under real concurrent traffic versus benchmark demos.
vLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
SGLang vs vLLM: An Honest Serving Comparison
SGLang vs vLLM compared for production LLM serving: RadixAttention vs PagedAttention, structured output performance, ecosystem maturity, and which to pick.