Why the same prompt produces different outputs, and how to make responses consistent
identical prompt returns different responses each time it is sent
Also appears as
- outputs differ between production replicas for the same input
- responses are not reproducible even with the same prompt and settings
Short answer
Inconsistent outputs for an apparently identical prompt usually come from one of three sources: sampling is not actually deterministic (temperature above zero and no fixed seed), continuous batching introduces small floating-point nondeterminism because the exact batch composition changes token-level numerics run to run, or different replicas behind a load balancer are quietly running different quantization or even different model revisions. Full bit-for-bit determinism is hard to guarantee in batched GPU inference, but the practical fix is to control sampling explicitly and make sure all replicas are provably running the same model artifact.
Affects: Any self-hosted LLM serving deployment, especially multi-replica production setups and anything using sampling-based generation
Isolate the source of variation
- 1Set temperature to 0 (or use greedy/deterministic decoding) and fix a random seed if the API supports one, then retest for consistency.
- 2If outputs are still inconsistent at temperature 0, send the same prompt directly to a single replica repeatedly (bypassing the load balancer) to rule out cross-replica differences.
- 3Compare the model checkpoint hash, quantization method, and serving framework version across all replicas to confirm they are provably identical.
- 4If a single replica is still nondeterministic at temperature 0, check whether continuous batching's variable batch composition is the source, since exact floating-point results can shift slightly depending on what else is in the batch.
- 5Decide how much determinism you actually need: for most production use cases, tight variance rather than bit-for-bit identical output across all conditions is a more realistic and sufficient target.
How to confirm this is your problem
- Sending the exact same prompt twice produces noticeably different text
- Automated evaluation scores fluctuate run to run without any code or data changes
- Some users report different answers to the same question depending on which server instance handled the request
- Outputs are consistent in a single-replica dev environment but vary in the multi-replica production deployment
Root causes and fixes
Sampling temperature above zero with no fixed seed
Any temperature greater than zero introduces randomness into token selection by design, sampling from a probability distribution rather than always picking the highest-probability token, so different runs will legitimately produce different text even with everything else held constant. This is intentional model behavior, not a bug, and is the most common reason for perceived inconsistency.
Fix: Set temperature to 0 for deterministic, repeatable output where consistency matters more than diversity, or set a fixed random seed per request if the framework supports it and diversity is still desired.
curl -s localhost:8000/v1/completions -d '{"prompt":"...","temperature":0}'Different replicas behind a load balancer are running different quantization or model revisions
If replicas were deployed at different times, from different build pipelines, or with slightly different configuration, one replica might be serving an FP16 checkpoint while another serves an AWQ or GPTQ quantized version, or replicas may be pinned to different model revisions entirely, producing genuinely different outputs for the same prompt depending purely on which replica handled the request.
Fix: Pin the exact model revision and quantization method in a single source of truth (a deployment manifest or container image tag), and add a startup check that verifies all replicas report the same checkpoint hash and quantization before joining the load balancer pool.
Batching nondeterminism from continuous batching's varying batch composition
GPU floating-point operations are not strictly associative, and the exact order and grouping of operations in a batched matrix multiplication can differ depending on what other sequences happen to be in the same batch at that moment, producing tiny numerical differences that can occasionally tip a token selection at the margin, even at temperature 0, especially near-tied logits.
Fix: Accept that bit-for-bit determinism under continuous batching is difficult to guarantee even at temperature 0, and instead validate consistency using semantic similarity or task-level correctness rather than exact string matching, unless the framework offers a specific deterministic batching mode.
No fixed seed passed when true reproducibility is required for compliance or debugging
Even with temperature 0 largely removing sampling randomness, some code paths (top-k/top-p filtering edge cases, or explicit sampling for creative use cases) still depend on a random seed, and if the API call does not pass one explicitly, the framework may use a different seed each time, reintroducing variability that could otherwise be eliminated.
Fix: Explicitly pass a fixed seed parameter on every request where reproducibility must be auditable, and document that this is required for any compliance-sensitive use case.
Different serving framework versions across replicas with slightly different default sampling or numeric behavior
Version drift between replicas (one running an older vLLM build, another a newer one) can change default numeric behavior, kernel implementations, or even default sampling parameters, producing outputs that differ across replicas even when the request and model checkpoint are otherwise identical.
Fix: Pin the exact serving framework version in the container image and treat any replica running a different version as a deployment bug to be fixed, not tolerated.
Diagnostic commands
Test the same prompt at temperature 0 against a single replica repeatedly
for i in 1 2 3; do curl -s localhost:8000/v1/completions -d '{"prompt":"test","temperature":0,"max_tokens":20}'; doneIf outputs still differ at temperature 0 on a single replica, the source is batching nondeterminism or a code path not respecting temperature 0, not cross-replica drift or sampling randomness in the usual sense.
Compare model and quantization metadata across all replicas
curl -s localhost:8000/v1/models | python -m json.tool
Any difference in reported model revision, dtype, or quantization method between replicas is a direct confirmation that cross-replica drift, not model behavior, explains the inconsistency users are seeing.
Check serving framework version on each replica
pip show vllm | grep -i version
Mismatched versions across replicas point to a deployment pipeline issue (inconsistent image tags or rollout) that should be fixed at the CI/CD level, not worked around at the application level.
Stopping it from happening again
- Pin exact model revision, quantization method, and serving framework version in a single deployment manifest used by every replica
- Add a startup health check that verifies checkpoint hash and config match across the replica pool before serving traffic
- Set temperature and seed explicitly on every request rather than relying on framework defaults
- Evaluate output consistency with semantic or task-level metrics rather than exact string equality where batching nondeterminism is expected
When this becomes an architecture problem
If strict bit-for-bit reproducibility is a hard compliance or audit requirement (e.g. regulated industries needing to reproduce exact historical outputs), that goes beyond configuration and into architecture: it typically requires disabling continuous batching for the affected workload, running single-sequence deterministic inference, or maintaining a separate reproducibility-mode deployment path, all of which trade throughput for guaranteed determinism.
Frequently asked questions
Can I get perfectly deterministic output from a GPU-batched LLM server?
Close to it at temperature 0 with a fixed seed and a single pinned model/quantization, but not perfectly guaranteed, because continuous batching's floating-point nondeterminism can still occasionally shift results at the margin. True bit-for-bit determinism generally requires giving up batching efficiency, which is a real tradeoff to make consciously rather than assume away.
Does quantization make outputs less consistent?
Quantization itself does not add run-to-run randomness, a correctly quantized model at temperature 0 is just as internally deterministic as full precision. The inconsistency problem arises when different replicas are quantized differently or run different quantization methods, producing systematically different (not randomly different) outputs between replicas.
Why do outputs vary even though our load balancer routes to identical replica images?
If the images are provably identical and temperature is 0, remaining variation is most likely continuous batching's floating-point nondeterminism, since exact numeric results can depend on what else is in the batch at inference time. This is a known limitation of high-throughput batched serving, not a deployment misconfiguration.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
LLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Free ToolvLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolLLM Serving Capacity Planner
Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.
Related problems
vLLM multi-LoRA serving fails to load or apply an adapter
Multi-LoRA serving in vLLM requires the server to be launched with --enable-lora plus explicit capacity flags such as max-lora-rank, max-loras, and max-cpu-loras sized to your actual adapters, and each adapter must be registered by name with --lora-modules so requests can reference it. Mismatches between an adapter's real rank or count and what the server was configured for produce hard failures rather than silent truncation.
GGUF quantization damages model output quality
GGUF quantization below roughly 4 bits per weight (Q2_K, Q3_K_S) trades accuracy aggressively for size and speed, and on smaller models or reasoning-heavy tasks this shows up as incoherent, repetitive, or factually unreliable output. Q4_K_M and Q5_K_M are the widely used sweet spots that keep most of the quality of the full-precision model while still cutting memory roughly in half or more, and Q2/Q3 should be reserved for cases where fitting in VRAM matters more than output quality.
Model output quality dropped noticeably after quantization
Quality degradation after quantization usually comes from choosing too aggressive a quantization level for the model size and task, quantizing layers that are unusually sensitive to precision loss (often attention output projections and the final layers), or from trusting perplexity as the only quality signal when perplexity can look nearly unchanged while task-specific accuracy drops meaningfully. FP8 is close to lossless for most models and tasks, while INT4 methods carry real risk that must be validated with a task-specific eval set before shipping, not assumed safe from a perplexity number alone.
LLM serving throughput collapses once load increases past a certain point
Throughput collapsing past a load threshold is almost always KV cache exhaustion: once in-flight requests' combined KV cache exceeds available GPU memory, the scheduler preempts some sequences, discarding their KV cache and forcing a full recompute when they resume, which burns GPU cycles on redundant work instead of new tokens. The fix is admission control that keeps the server below its true KV cache-limited concurrency, not just retrying harder or adding a bigger queue.
GuideLLM Quantization: AWQ vs GPTQ vs FP8 vs GGUF
AWQ, GPTQ, FP8, and GGUF compared for production LLM serving: memory savings, throughput impact, quality loss, and which format fits which deployment.
GuideLLM Observability: TTFT, ITL, Throughput, and GPU Dashboards
LLM inference observability: track TTFT, inter-token latency, throughput, and GPU utilization with dashboards that catch problems before users report them.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.