Inference Servingvllmhuggingfacecuda

Why your vLLM server won't start, and how to diagnose the real cause

Error
ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use

Also appears as

  • OSError: You are trying to access a gated repo... 401 Client Error
  • ValueError: Model architecture 'XxxForCausalLM' is not supported for now

Short answer

vLLM server startup failures collapse into four buckets: the port is already bound by another process, Hugging Face auth is missing or expired for a gated repo, there isn't enough free VRAM for the requested model and context, or the installed vLLM version doesn't yet support the model's architecture. Read the last traceback line, not just the top, to tell them apart.

Affects: vLLM 0.4 and later, any OS/GPU combination; symptoms differ by root cause but all present as the process exiting before it reaches Uvicorn running

Diagnose in order, fastest first

  1. 1Check the last lines of the traceback for one of: address already in use, 401 Client Error or gated repo, CUDA out of memory, or architecture is not supported.
  2. 2Port conflict: pass --port 8001 or kill whatever is already listening on 8000.
  3. 3Auth: run huggingface-cli login or export HF_TOKEN, and confirm you accepted the model's license on its Hugging Face page.
  4. 4VRAM: lower --max-model-len, add --quantization, or move to a smaller model or more GPUs.
  5. 5Unsupported architecture: upgrade vLLM with pip install -U vllm, or check the model card for the minimum vLLM version it requires.

How to confirm this is your problem

  • Process exits within seconds, never reaching Uvicorn running on http://0.0.0.0:8000.
  • Traceback is long and includes framework internals, making it easy to miss the actual root-cause line at the bottom.
  • Works for one model but not another on the same machine.
  • Worked yesterday, fails today after a vLLM, driver, or model revision change.

Root causes and fixes

Most common

Port already bound by a previous vLLM process or another service

A prior vLLM instance that didn't shut down cleanly, for example killed with SIGKILL instead of SIGTERM, can leave the port held by a zombie process, or another service is already using port 8000. The OS refuses the bind and Python raises an OSError before the app object is even created.

Fix: Find and kill the process on that port, or start vLLM on a different --port.

Commands
lsof -i :8000
kill -9 PID
vllm serve MODEL_ID --port 8001
Common

Gated or private Hugging Face repo without a valid token

Meta, Google, and other model owners gate weights behind license acceptance. vLLM's download step hits Hugging Face's API, gets a 401 or 403 because no token is present or the token's account hasn't accepted the license, and the whole process aborts before serving starts.

Fix: Log in with huggingface-cli login or export HF_TOKEN, and accept the license on the model page using that same account.

Commands
huggingface-cli login
export HF_TOKEN=hf_xxx
Common

Insufficient VRAM for weights plus KV cache at the requested settings

If the model's weights alone don't fit in the visible GPU(s) at the requested dtype, or fit but leave no room for even a minimal KV cache pool, vLLM throws a CUDA OOM during weight loading rather than a graceful message. This is distinct from the KV-cache-sizing ValueError; it happens lower in the startup sequence.

Fix: Reduce --max-model-len, add --quantization awq/gptq/fp8, use --tensor-parallel-size across more GPUs, or pick a smaller model.

Commands
vllm serve MODEL_ID --quantization awq --max-model-len 4096
Occasional

Model architecture not yet supported by the installed vLLM version

New model families ship with novel attention or MoE layouts; vLLM adds support for each architecture in a specific release. Running a brand-new model against an older pinned vLLM raises a clear ValueError naming the unsupported architecture class.

Fix: Upgrade vLLM to the version documented on the model card, or wait for a point release if support was only just merged.

Commands
pip install -U vllm
python -c "import vllm; print(vllm.__version__)"
Rare

CUDA or driver mismatch with the installed vLLM wheel

vLLM wheels are built against specific CUDA toolkit versions; a driver too old for the compiled CUDA runtime causes low-level initialization failures that surface as import errors or crashes before the HTTP server starts.

Fix: Match your vLLM wheel's CUDA version to your driver, or reinstall using the CUDA-version-specific install instructions from vLLM's docs.

Commands
nvidia-smi
pip show vllm

Diagnostic commands

Check what's holding the port

lsof -i :8000

A PID here means something is already listening; kill it or change --port.

Test Hugging Face auth independently of vLLM

python -c "from huggingface_hub import whoami; print(whoami())"

An error here confirms it's an auth problem before you even touch vLLM; fix login first.

Check free VRAM before serving

nvidia-smi --query-gpu=memory.used,memory.total --format=csv

If used memory is near total before you start, another process or a zombie vLLM instance is consuming the VRAM you need.

Confirm vLLM version

python -c "import vllm; print(vllm.__version__)"

Cross-check this version number against the model card's minimum supported vLLM version.

Stopping it from happening again

  • Use a process manager such as systemd or Kubernetes that ensures old vLLM processes are fully terminated before a new one starts on the same port.
  • Bake HF_TOKEN into your deployment secrets rather than relying on an interactive login, especially for CI/CD restarts.
  • Pin your vLLM version in requirements and test new model releases in staging before assuming compatibility.
  • Add a startup health check that fails fast and loud in your orchestrator, rather than silently retrying a broken config.

When this becomes an architecture problem

If every model you try fails to load on VRAM grounds, or you're constantly chasing vLLM version bumps to support new architectures, that's a sign your hardware or your model-release process needs redesigning, not another flag tweak.

Frequently asked questions

Why does the traceback look so long when the actual problem is simple?

vLLM wraps Ray, PyTorch, and Hugging Face internals, so a single root cause like a 401 can print through several layers of framework code. Always read the last few lines of the traceback, where the actual exception type and message live, rather than the top.

Do I need a token for every model?

No, only for gated or private repos, which includes many Meta and Google releases requiring license acceptance. Fully open weights like most Qwen, DeepSeek, and Mistral community releases don't need a token.

Is it safe to just keep retrying with pip install -U vllm?

It's a reasonable first step for unsupported architecture errors, but pin the resulting version afterward and test it against your other models before rolling to production, since a vLLM upgrade can also change default behavior for models you already run.

Related problems

vLLM runs out of memory during startup, before serving any requests

vLLM's startup OOMs happen because it preallocates a KV cache pool sized against gpu_memory_utilization right after loading weights, so the failure point is engine initialization, not user traffic. Fix it by lowering gpu_memory_utilization if it's set too aggressively for actual free VRAM, lowering max_model_len, or reducing weight footprint with quantization or more GPUs.

401/403 Unauthorized pulling a gated model from HuggingFace

A 401 or 403 on a gated HuggingFace repo means the request reached the Hub but was rejected for authorization, not because the model does not exist. The three real causes are: you have not accepted the model's license on the web page with the account tied to your token, the token exists but was created without read access to gated repos, or the token is valid but was never actually passed to the download call (no HF_TOKEN in the environment, no login run). Fix by accepting the license, generating a token with the right scope, and exporting it where the client library will find it.

vLLM OpenAI-compatible API returns 404 Not Found

This 404 is almost always a client-side mismatch, not a server bug: either the request hit the wrong route, such as missing the /v1 prefix, or the model field in the request body doesn't match the exact served-model-name (or default model repo id) vLLM registered at startup. Fix the URL and model name to match what /v1/models actually reports.

CUDA version mismatch between PyTorch and the system driver

PyTorch ships its own bundled CUDA runtime inside the wheel, so it never uses your system's CUDA toolkit (the one nvcc reports). The only number that matters is the driver's maximum supported CUDA version, shown top right in nvidia-smi output. Fix the mismatch by installing a torch wheel built for a CUDA version at or below that number, not by touching nvcc or the toolkit.

Guide

vLLM Production Deployment: A Practitioner's Guide

Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.

Guide

On-Prem LLM Deployment Architecture: Reference Guide

Reference architecture for on-prem LLM deployment: inference servers, GPU sizing, RAG pipelines, and security zones for regulated manufacturers.

Guide

On-Prem LLM Inference Hardware in 2026: A Roundup

On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.

Still stuck, or tired of fighting your own infrastructure?

Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.