Why vLLM keeps generating past the end of turn, and how to fix stop tokens
Model output continues generating text, often repeating or hallucinating a new user turn, instead of stopping at the expected end of turn
Also appears as
- finish_reason: length instead of stop, with max_tokens hit unexpectedly
- Response includes trailing special tokens as literal text instead of stopping generation
Short answer
Runaway generation almost always means the token the model actually emits to end a turn doesn't match what vLLM is told to stop on, either because generation_config.json's eos_token_id is stale, a custom stop string wasn't passed in the request, or a fine-tune introduced a new end-of-turn token the base config doesn't know about. Fix it by explicitly passing the correct stop token id or stop strings rather than relying on defaults.
Affects: vLLM 0.4 and later, most common with newly released model families whose stop or eos tokens differ from what's configured in generation_config.json, or with custom fine-tunes
Stop the runaway generation
- 1Check the model's generation_config.json for eos_token_id and confirm it matches the token the model actually emits at the end of a turn.
- 2If the model uses a special end-of-turn token not captured by eos_token_id, pass it explicitly via the request's stop parameter or vLLM's generation config override.
- 3For chat completions, ensure the chat template's expected end-of-turn token is included in the sampling params' stop_token_ids.
- 4As an immediate mitigation, add a stop string, the literal end-of-turn marker, to the request body's stop list.
- 5Re-test and confirm finish_reason returns stop at the expected point, not length.
How to confirm this is your problem
- finish_reason in the response is length, meaning it hit max_tokens, when you expected stop.
- Model output continues past a natural answer, sometimes hallucinating a new question and answering that too.
- Special tokens appear as literal visible text in the output instead of terminating generation.
- Problem appears specifically after switching to a new model or a custom fine-tune, not on the previously deployed model.
Root causes and fixes
Model's actual end-of-turn token differs from the configured eos_token_id
vLLM stops generation by default when the model samples the token id(s) listed in eos_token_id from the model's generation_config.json. Many chat-formatted models signal end of turn with a dedicated special token distinct from the tokenizer's classic EOS token, and if that special token isn't included in eos_token_id, vLLM has no reason to stop when the model emits it.
Fix: Identify the model's actual end-of-turn token by checking its chat template and tokenizer special tokens, and pass it explicitly via stop_token_ids in the request or override generation_config.json.
python -c "from transformers import AutoTokenizer; t = AutoTokenizer.from_pretrained('MODEL_ID'); print(t.special_tokens_map)"Custom fine-tune changed the expected stop token without updating generation_config.json
When a model is fine-tuned to a new chat format, adding or changing special tokens for turn boundaries, the generation_config.json shipped alongside it may not have been regenerated to reflect the new eos_token_id, so the training format and the inference-time stop condition silently diverge.
Fix: Regenerate or manually edit generation_config.json for the fine-tuned checkpoint to include the correct eos_token_id matching the actual training format.
Client request omits a needed stop string for a model whose format requires an explicit stop list
Some deployments rely on the caller passing an explicit stop sequence in the API request, rather than relying purely on eos_token_id, because the serving stack's default configuration doesn't cover every valid end marker. Omitting it lets generation continue until max_tokens truncates it.
Fix: Add the model's known end-of-turn marker string explicitly to the request body's stop parameter as a defensive measure.
max_tokens set very high, masking that stop tokens are entirely ignored
A generous max_tokens value can make a true stop-token bug look like merely verbose output rather than an obvious runaway, since generation eventually halts at the token budget instead of visibly spiraling. This delays diagnosis because the failure mode is subtle rather than dramatic.
Fix: Temporarily lower max_tokens during debugging so a stop-token failure is immediately obvious rather than masked by a large budget.
vLLM version predates support for a model's newer stop-token convention
Support for correctly parsing a given model family's generation_config.json conventions, especially multiple eos_token_id values as a list, was added in specific vLLM releases. An older pinned version may only honor a single legacy eos_token_id and silently ignore additional stop tokens the model actually needs.
Fix: Upgrade vLLM to a version that documents support for the model family's generation config format.
pip install -U vllm
Diagnostic commands
Inspect the model's configured stop tokens
cat generation_config.json
Compare eos_token_id here against the actual special tokens the model was trained to emit; a mismatch is the most common root cause.
Check special tokens the tokenizer knows about
python -c "from transformers import AutoTokenizer; t = AutoTokenizer.from_pretrained('MODEL_ID'); print(t.special_tokens_map, t.additional_special_tokens)"Look for an end-of-turn style token that may not be reflected in eos_token_id.
Check finish_reason across several test requests
curl http://localhost:8000/v1/chat/completions -d ... | python -m json.tool
length instead of stop consistently across requests confirms the model isn't hitting a recognized stop condition, not just occasional verbosity.
Stopping it from happening again
- Validate finish_reason equals stop, not length, during smoke testing of every new model or fine-tune before production rollout.
- Keep an explicit stop_token_ids or stop list in your standard request template rather than relying solely on the model's shipped generation_config.json.
- Regenerate generation_config.json as part of your fine-tuning pipeline whenever chat format or special tokens change.
- Log and alert on an unusually high proportion of length finish reasons in production, which often indicates a stop-token regression after a model or vLLM upgrade.
When this becomes an architecture problem
If this keeps recurring across every new model or fine-tune your team ships, standardize a validation step in the fine-tuning and release pipeline that checks eos_token_id against the actual trained format, rather than discovering it in production after each release.
Frequently asked questions
Why doesn't vLLM just detect end of turn automatically for any model?
It relies on the model's own declared configuration, generation_config.json's eos_token_id plus any stop strings you pass, because there's no universal signal across different model families and fine-tunes for what counts as done; the model itself has to tell the server what its stop token is.
Is passing an explicit stop string in every request a reasonable permanent fix?
It's a reasonable defensive practice, especially across a fleet of different models, but the more robust fix is correcting eos_token_id at the source so every client benefits without having to remember to pass it manually.
Does a high max_tokens value cause this bug?
No, but it does hide it: with a generous token budget, runaway generation just looks like an unusually long, sometimes incoherent response instead of an obvious immediate failure, which is why testing with a modest max_tokens during debugging makes the problem easier to spot.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
On-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Free ToolOpen-Weight Model Selector
A 10-question assessment that matches your hardware budget, workload complexity, and operational maturity to the right open-weight model size class.
Related problems
vLLM error: no chat template found for this model
The /v1/chat/completions endpoint needs a Jinja chat template to turn a messages array into the model's expected prompt format, and base pretrained checkpoints plus some older fine-tunes simply don't ship one. Fix it by supplying --chat-template pointing at a template file matching the model family, or by switching to the raw /v1/completions endpoint with a manually formatted prompt.
vLLM ignores tool or function calls, or returns them as plain text
Unlike the hosted OpenAI API, vLLM does not enable tool or function calling by default. You must launch with --enable-auto-tool-choice plus a --tool-call-parser matching your specific model family, and the model itself must have been trained to emit tool-call syntax its parser recognizes. Without both pieces, requests either error out or the model just writes the function call as plain text in its response content.
Fine-tuned model scores worse than the base model
A fine-tuned model that scores worse than its own base model almost always means the evaluation is contaminated (test examples leaked into training) or unfair (a genuinely improved model getting compared under a broken harness), the inference prompt format doesn't match the exact format used during training, or the fine-tuning process optimized for surface style and tone rather than the underlying capability the benchmark actually measures. Check inference prompt formatting first, since it is the single most common cause.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideHow to Evaluate a Fine-Tuned Model Before Production
Evaluate a fine-tuned model before production: held-out eval sets, task-specific metrics, calibrated LLM-as-judge setups, and regression testing.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.