Why vLLM installation fails, and how to get a clean working install
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. vllm 0.6.3 requires torch==2.4.0, but you have torch 2.5.1 which is incompatible
Also appears as
- ImportError: libcudart.so.12: cannot open shared object file: No such file or directory
- RuntimeError: vLLM is not compatible with your CUDA version
Short answer
vLLM ships prebuilt wheels compiled against a specific, exact torch and CUDA version, and it uses custom CUDA kernels that only work with that pairing. Most install failures happen because torch was already installed separately, or an existing environment has an incompatible CUDA toolkit, so the fix is almost always a fresh virtual environment where pip resolves vLLM and its exact torch dependency together in one pass.
Affects: vLLM across recent versions, especially when torch or another CUDA-dependent package is pre-installed in the environment before vLLM.
Fastest path to a clean vLLM install
- 1Create a fresh virtual environment: python -m venv vllm-env && source vllm-env/bin/activate.
- 2Do not pre-install torch. Run pip install vllm and let it pull the exact torch version it requires.
- 3If you must use a specific CUDA build, check vLLM's release notes for the supported CUDA version and install using the matching prebuilt wheel or index.
- 4Verify the install with python -c "import vllm; print(vllm.__version__)".
- 5Run a minimal offline inference smoke test before deploying a server.
How to confirm this is your problem
- pip reports a dependency conflict naming an exact torch version vLLM requires
- import vllm raises an ImportError about a missing shared CUDA library
- vLLM installs cleanly but crashes on first model load with a CUDA kernel error
- Installation works on one machine but fails on another with the same OS
Root causes and fixes
torch was installed separately before vLLM, creating a version conflict
vLLM pins an exact torch version per release because its custom CUDA kernels (for paged attention and continuous batching) are compiled against that specific torch build's ABI. If a different torch is already present, pip either refuses the install or silently leaves an incompatible pairing that crashes at runtime instead of at install time.
Fix: Uninstall any existing torch first, or better, start from a fresh environment and let pip install vllm resolve the correct torch version itself rather than pre-installing torch manually.
pip uninstall -y torch torchvision torchaudio pip install vllm
Generic pip install pulled a wheel built for the wrong CUDA version
vLLM publishes wheels tied to specific CUDA major versions. On a system whose driver only supports an older CUDA generation, the default wheel from PyPI may target a newer CUDA than the driver allows, producing the same driver-too-old failure seen with plain PyTorch installs.
Fix: Check the vLLM documentation for the CUDA version matrix for your target release, and install using the CUDA-specific instructions (a matching prebuilt wheel or explicit index) rather than the bare pip install vllm command.
Building vLLM from source without a matching CUDA toolkit
Installing from source instead of a prebuilt wheel requires compiling custom CUDA kernels with nvcc, which means the installed CUDA toolkit version must align with what vLLM's build system expects. A mismatched or missing toolkit causes the compilation step itself to fail with cryptic nvcc errors.
Fix: Prefer the prebuilt wheel whenever your platform is supported; only build from source when you need an unreleased feature, and match the CUDA toolkit version to vLLM's documented build requirements first.
Unsupported Python version forces a slow source build with its own failures
vLLM publishes prebuilt wheels for specific Python versions. A very new Python release (for example, right after a Python minor version launches) often lacks a prebuilt wheel for weeks, so pip silently falls back to building from source, which then hits unrelated compiler and toolchain errors.
Fix: Check vLLM's supported Python versions in its documentation and use one with published wheels, typically by creating the virtual environment with an explicit, slightly older Python version such as 3.10 or 3.11.
Conflicting xformers or flash-attn versions pulled in as transitive dependencies
vLLM depends on attention kernel libraries like xformers or FlashAttention for some backends. If an incompatible version is already installed, or a later manual pip install upgrades one of them independently, the mismatch surfaces as an import error deep inside vLLM's attention backend selection.
Fix: Let vLLM's own dependency resolution manage xformers and flash-attn versions; avoid manually pinning or upgrading them unless you have confirmed compatibility in vLLM's release notes.
Diagnostic commands
Check what versions actually got installed
pip show vllm torch
Compare the reported torch version against what vLLM's release notes say it requires. Any mismatch here explains most import-time and runtime failures.
Reproduce the import failure with a full traceback
python -c "import vllm"
The traceback usually names the exact missing shared library or incompatible symbol, pointing directly to whether the problem is torch, CUDA, or an attention kernel library.
Confirm the CUDA toolkit matches what vLLM's build expects
nvcc --version
Only relevant if you are building from source; irrelevant for prebuilt wheel installs where torch and vLLM bundle their own CUDA runtime.
Confirm torch itself works before blaming vLLM
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
If this fails, the problem is your base torch/CUDA/driver setup, not vLLM specifically; fix that layer first.
Stopping it from happening again
- Always install vLLM into a fresh, dedicated virtual environment rather than an environment with pre-existing ML packages.
- Pin the exact vLLM version in requirements.txt or a lockfile so upgrades are deliberate, tested changes.
- Check the CUDA and Python compatibility matrix in vLLM's release notes before every upgrade, not just on first install.
- Bake a tested vLLM environment into a container image for production so individual machine drift cannot break the install.
When this becomes an architecture problem
If you are standardizing vLLM across many GPU node types with different drivers and CUDA generations, or need an air-gapped install with no PyPI access, this becomes an image and artifact pipeline problem rather than a one-off pip troubleshooting session.
Frequently asked questions
Should I install torch before or after vLLM?
After, or not at all. Let pip install vllm resolve and install its required torch version automatically in a fresh environment. Pre-installing torch yourself is the single most common cause of vLLM's dependency conflict errors, because it locks in a version that may not match what vLLM's kernels were compiled against.
Can I use vLLM with any CUDA version I want?
No. vLLM's prebuilt wheels target specific CUDA major versions, and its custom kernels only work correctly with the CUDA runtime torch itself bundles. Check the release notes for your target vLLM version and match your driver and CUDA choice to what that release documents as supported.
Why does building vLLM from source take so long and then fail?
Source builds compile custom CUDA kernels with nvcc, which is slow and memory-intensive, and depends on having a matching CUDA toolkit and compatible host compiler installed. Most users should use the prebuilt wheel; only build from source for unreleased features and expect to debug compiler-level errors if you do.
I get a working install on one server but not another. Why?
This almost always means the two servers have different driver versions, CUDA toolkits, or pre-existing torch installs. Compare nvidia-smi's CUDA Version and pip show torch output across both machines to find the divergence, then standardize on a container image instead of per-machine pip installs.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
Free ToolSelf-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
Free ToolOn-Prem AI Deployment Checklist
A 30-point pre-deployment checklist covering use cases, hardware, security, model operations, and rollout for self-hosted enterprise LLMs.
Related problems
CUDA version mismatch between PyTorch and the system driver
PyTorch ships its own bundled CUDA runtime inside the wheel, so it never uses your system's CUDA toolkit (the one nvcc reports). The only number that matters is the driver's maximum supported CUDA version, shown top right in nvidia-smi output. Fix the mismatch by installing a torch wheel built for a CUDA version at or below that number, not by touching nvcc or the toolkit.
FlashAttention install fails during compilation or gets killed
FlashAttention's pip install compiles CUDA kernels from source unless an exact prebuilt wheel exists for your torch, CUDA, Python, and C++ ABI combination, and that compilation is extremely RAM-hungry per parallel job. The build gets silently OOM-killed on machines without enough memory unless you limit MAX_JOBS, and separately fails if your CUDA toolkit does not match the version torch itself was built against.
Python dependency conflicts across transformers, tokenizers, and numpy
The Hugging Face stack (transformers, tokenizers, accelerate) is released in tightly coupled lockstep versions, so upgrading one package independently over time leaves combinations that were never tested together and break silently at import or runtime. The fix is resolving the entire stack in a single pip install pass from a fresh virtual environment, guided by a pinned requirements.txt, rather than incrementally patching individual packages.
vLLM server won't start (port in use, auth, VRAM, or unsupported architecture)
vLLM server startup failures collapse into four buckets: the port is already bound by another process, Hugging Face auth is missing or expired for a gated repo, there isn't enough free VRAM for the requested model and context, or the installed vLLM version doesn't yet support the model's architecture. Read the last traceback line, not just the top, to tell them apart.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideSGLang vs vLLM: An Honest Serving Comparison
SGLang vs vLLM compared for production LLM serving: RadixAttention vs PagedAttention, structured output performance, ecosystem maturity, and which to pick.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.