Why your vLLM server won't start, and how to diagnose the real cause
ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in useAlso appears as
- OSError: You are trying to access a gated repo... 401 Client Error
- ValueError: Model architecture 'XxxForCausalLM' is not supported for now
Short answer
vLLM server startup failures collapse into four buckets: the port is already bound by another process, Hugging Face auth is missing or expired for a gated repo, there isn't enough free VRAM for the requested model and context, or the installed vLLM version doesn't yet support the model's architecture. Read the last traceback line, not just the top, to tell them apart.
Affects: vLLM 0.4 and later, any OS/GPU combination; symptoms differ by root cause but all present as the process exiting before it reaches Uvicorn running
Diagnose in order, fastest first
- 1Check the last lines of the traceback for one of: address already in use, 401 Client Error or gated repo, CUDA out of memory, or architecture is not supported.
- 2Port conflict: pass --port 8001 or kill whatever is already listening on 8000.
- 3Auth: run huggingface-cli login or export HF_TOKEN, and confirm you accepted the model's license on its Hugging Face page.
- 4VRAM: lower --max-model-len, add --quantization, or move to a smaller model or more GPUs.
- 5Unsupported architecture: upgrade vLLM with pip install -U vllm, or check the model card for the minimum vLLM version it requires.
How to confirm this is your problem
- Process exits within seconds, never reaching Uvicorn running on http://0.0.0.0:8000.
- Traceback is long and includes framework internals, making it easy to miss the actual root-cause line at the bottom.
- Works for one model but not another on the same machine.
- Worked yesterday, fails today after a vLLM, driver, or model revision change.
Root causes and fixes
Port already bound by a previous vLLM process or another service
A prior vLLM instance that didn't shut down cleanly, for example killed with SIGKILL instead of SIGTERM, can leave the port held by a zombie process, or another service is already using port 8000. The OS refuses the bind and Python raises an OSError before the app object is even created.
Fix: Find and kill the process on that port, or start vLLM on a different --port.
lsof -i :8000 kill -9 PID vllm serve MODEL_ID --port 8001
Gated or private Hugging Face repo without a valid token
Meta, Google, and other model owners gate weights behind license acceptance. vLLM's download step hits Hugging Face's API, gets a 401 or 403 because no token is present or the token's account hasn't accepted the license, and the whole process aborts before serving starts.
Fix: Log in with huggingface-cli login or export HF_TOKEN, and accept the license on the model page using that same account.
huggingface-cli login export HF_TOKEN=hf_xxx
Insufficient VRAM for weights plus KV cache at the requested settings
If the model's weights alone don't fit in the visible GPU(s) at the requested dtype, or fit but leave no room for even a minimal KV cache pool, vLLM throws a CUDA OOM during weight loading rather than a graceful message. This is distinct from the KV-cache-sizing ValueError; it happens lower in the startup sequence.
Fix: Reduce --max-model-len, add --quantization awq/gptq/fp8, use --tensor-parallel-size across more GPUs, or pick a smaller model.
vllm serve MODEL_ID --quantization awq --max-model-len 4096
Model architecture not yet supported by the installed vLLM version
New model families ship with novel attention or MoE layouts; vLLM adds support for each architecture in a specific release. Running a brand-new model against an older pinned vLLM raises a clear ValueError naming the unsupported architecture class.
Fix: Upgrade vLLM to the version documented on the model card, or wait for a point release if support was only just merged.
pip install -U vllm python -c "import vllm; print(vllm.__version__)"
CUDA or driver mismatch with the installed vLLM wheel
vLLM wheels are built against specific CUDA toolkit versions; a driver too old for the compiled CUDA runtime causes low-level initialization failures that surface as import errors or crashes before the HTTP server starts.
Fix: Match your vLLM wheel's CUDA version to your driver, or reinstall using the CUDA-version-specific install instructions from vLLM's docs.
nvidia-smi pip show vllm
Diagnostic commands
Check what's holding the port
lsof -i :8000
A PID here means something is already listening; kill it or change --port.
Test Hugging Face auth independently of vLLM
python -c "from huggingface_hub import whoami; print(whoami())"
An error here confirms it's an auth problem before you even touch vLLM; fix login first.
Check free VRAM before serving
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
If used memory is near total before you start, another process or a zombie vLLM instance is consuming the VRAM you need.
Confirm vLLM version
python -c "import vllm; print(vllm.__version__)"
Cross-check this version number against the model card's minimum supported vLLM version.
Stopping it from happening again
- Use a process manager such as systemd or Kubernetes that ensures old vLLM processes are fully terminated before a new one starts on the same port.
- Bake HF_TOKEN into your deployment secrets rather than relying on an interactive login, especially for CI/CD restarts.
- Pin your vLLM version in requirements and test new model releases in staging before assuming compatibility.
- Add a startup health check that fails fast and loud in your orchestrator, rather than silently retrying a broken config.
When this becomes an architecture problem
If every model you try fails to load on VRAM grounds, or you're constantly chasing vLLM version bumps to support new architectures, that's a sign your hardware or your model-release process needs redesigning, not another flag tweak.
Frequently asked questions
Why does the traceback look so long when the actual problem is simple?
vLLM wraps Ray, PyTorch, and Hugging Face internals, so a single root cause like a 401 can print through several layers of framework code. Always read the last few lines of the traceback, where the actual exception type and message live, rather than the top.
Do I need a token for every model?
No, only for gated or private repos, which includes many Meta and Google releases requiring license acceptance. Fully open weights like most Qwen, DeepSeek, and Mistral community releases don't need a token.
Is it safe to just keep retrying with pip install -U vllm?
It's a reasonable first step for unsupported architecture errors, but pin the resulting version afterward and test it against your other models before rolling to production, since a vLLM upgrade can also change default behavior for models you already run.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolOn-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Free ToolSelf-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
Related problems
vLLM runs out of memory during startup, before serving any requests
vLLM's startup OOMs happen because it preallocates a KV cache pool sized against gpu_memory_utilization right after loading weights, so the failure point is engine initialization, not user traffic. Fix it by lowering gpu_memory_utilization if it's set too aggressively for actual free VRAM, lowering max_model_len, or reducing weight footprint with quantization or more GPUs.
401/403 Unauthorized pulling a gated model from HuggingFace
A 401 or 403 on a gated HuggingFace repo means the request reached the Hub but was rejected for authorization, not because the model does not exist. The three real causes are: you have not accepted the model's license on the web page with the account tied to your token, the token exists but was created without read access to gated repos, or the token is valid but was never actually passed to the download call (no HF_TOKEN in the environment, no login run). Fix by accepting the license, generating a token with the right scope, and exporting it where the client library will find it.
vLLM OpenAI-compatible API returns 404 Not Found
This 404 is almost always a client-side mismatch, not a server bug: either the request hit the wrong route, such as missing the /v1 prefix, or the model field in the request body doesn't match the exact served-model-name (or default model repo id) vLLM registered at startup. Fix the URL and model name to match what /v1/models actually reports.
CUDA version mismatch between PyTorch and the system driver
PyTorch ships its own bundled CUDA runtime inside the wheel, so it never uses your system's CUDA toolkit (the one nvcc reports). The only number that matters is the driver's maximum supported CUDA version, shown top right in nvidia-smi output. Fix the mismatch by installing a torch wheel built for a CUDA version at or below that number, not by touching nvcc or the toolkit.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideOn-Prem LLM Deployment Architecture: Reference Guide
Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.