Why FlashAttention fails to install, and how to get it building successfully
ninja: build stopped: subcommand failed
Also appears as
- RuntimeError: Error building extension 'flash_attn_2_cuda'
- Killed
- c++: fatal error: Killed signal terminated program cc1plus
Short answer
FlashAttention's pip install compiles CUDA kernels from source unless an exact prebuilt wheel exists for your torch, CUDA, Python, and C++ ABI combination, and that compilation is extremely RAM-hungry per parallel job. The build gets silently OOM-killed on machines without enough memory unless you limit MAX_JOBS, and separately fails if your CUDA toolkit does not match the version torch itself was built against.
Affects: flash-attn package installs via pip on any machine, especially those with less than 32GB of system RAM or a CUDA toolkit version that does not match the installed torch build.
Fastest path to a successful build or install
- 1First try to find a matching prebuilt wheel for your exact torch, CUDA, and Python version from the flash-attn GitHub releases page, which skips compilation entirely.
- 2If you must build from source, limit parallel compile jobs to control memory use: MAX_JOBS=4 pip install flash-attn --no-build-isolation.
- 3Confirm you have enough free RAM (roughly 4 to 8GB per job) with free -h before starting the build.
- 4Confirm nvcc --version matches the CUDA version torch was built against (python -c "import torch; print(torch.version.cuda)").
- 5If the build still gets killed, run it on a machine with more RAM, then copy the resulting wheel to the target machine.
How to confirm this is your problem
- pip install flash-attn runs for a long time then prints Killed with no further explanation
- ninja build error referencing a failed subcommand with no clear root cause
- Build succeeds on a workstation with lots of RAM but fails on a smaller cloud instance
- Error building extension flash_attn_2_cuda during setup.py execution
Root causes and fixes
Compiling from source runs out of RAM because MAX_JOBS was not limited
FlashAttention's setup.py compiles multiple large CUDA kernel files in parallel using all available CPU cores by default. Each parallel nvcc compile job can use several gigabytes of RAM, so on machines with limited memory (common on cloud GPU instances that are GPU-heavy but RAM-light), the OS OOM killer terminates the compiler mid-build, producing an unhelpful 'Killed' message.
Fix: Set the MAX_JOBS environment variable to a small number like 2 or 4 before running pip install flash-attn, which reduces peak memory use at the cost of a longer build time.
MAX_JOBS=4 pip install flash-attn --no-build-isolation
CUDA toolkit version does not match what torch was compiled against
FlashAttention's build system compiles against the CUDA toolkit visible on PATH via nvcc, but the resulting kernels must be binary compatible with the torch build actually running the process. A toolkit that diverges from torch's own bundled CUDA version produces subtle compile-time or load-time failures.
Fix: Check python -c "import torch; print(torch.version.cuda)" and install a matching CUDA toolkit version, or better, use a prebuilt flash-attn wheel that already matches your torch build.
No prebuilt wheel exists for your exact torch, CUDA, Python, and ABI combination
FlashAttention publishes prebuilt wheels for common combinations of torch version, CUDA version, Python version, and the C++11 ABI flag. Any deviation, such as a very recent torch release or an uncommon Python version, means pip falls back to a from-source build automatically, which is where most of these failures originate.
Fix: Check the flash-attn GitHub releases page for a wheel matching your exact environment before attempting to build; if none exists, consider using a slightly older torch version that does have a matching wheel.
gcc or g++ version outside the range the CUDA toolkit supports
Compiling CUDA extensions requires nvcc to invoke a host C++ compiler, and each CUDA toolkit release only supports a specific range of gcc/g++ major versions. A system gcc that is too new (common on recently released Linux distributions) causes nvcc to reject the compiler outright before any FlashAttention-specific code even compiles.
Fix: Install an older, supported gcc/g++ version alongside the system default and point CC and CXX environment variables at it for the duration of the build.
Insufficient disk space in /tmp during the build
The compilation process generates large intermediate object files in a temporary build directory. On machines with a small root partition or a /tmp mounted on limited space, the build fails partway through with a disk-full error that can be mistaken for a memory or compiler problem.
Fix: Check available space with df -h /tmp, and if low, point the build's temporary directory (TMPDIR environment variable) to a partition with more free space before retrying.
Diagnostic commands
Check available system RAM before starting a build
free -h
Less than roughly 16GB of free memory strongly suggests you should limit MAX_JOBS to 2 or fewer, or move the build to a larger machine.
Check whether the installed CUDA toolkit matches torch's bundled version
nvcc --version
Compare this output against python -c "import torch; print(torch.version.cuda)"; a mismatch is a common cause of both build failures and runtime crashes after a successful build.
Confirm the compiler is within the CUDA toolkit's supported range
gcc --version
Check this against your CUDA toolkit's documented supported host compiler versions; a gcc that is too new is a frequent, easy-to-miss cause of compile failures.
Watch memory usage live during the build to confirm an OOM condition
watch -n 1 free -h
If available memory drops to near zero right before the process is Killed, that confirms an OOM kill and validates lowering MAX_JOBS as the fix.
Stopping it from happening again
- Always check for a prebuilt wheel matching your exact environment before attempting a source build.
- Default to setting MAX_JOBS explicitly in your build scripts and Dockerfiles rather than relying on the compiler's automatic parallelism.
- Build FlashAttention once on a machine with ample RAM and bake the resulting wheel into your container image, rather than rebuilding on every deploy.
- Pin flash-attn, torch, and CUDA toolkit versions together in your environment lockfile so a routine upgrade does not silently break the build again.
When this becomes an architecture problem
If you need FlashAttention across a heterogeneous fleet of GPU nodes with varying RAM and CUDA toolkit versions, or in an air-gapped environment where source builds are the only option, standardize on a single build machine that produces wheels for the whole fleet rather than troubleshooting each node individually.
Frequently asked questions
What does MAX_JOBS actually control in a FlashAttention build?
MAX_JOBS limits how many CUDA kernel files the compiler builds in parallel. Each parallel job can consume several gigabytes of RAM, so on memory-constrained machines, an unlimited MAX_JOBS setting causes the operating system to kill the build process partway through. Setting MAX_JOBS to 2 or 4 trades build speed for a much lower peak memory footprint.
Why does the build just say 'Killed' with no error message?
'Killed' with no further detail is the classic signature of the Linux OOM killer terminating a process that exceeded available memory. It is not a bug in FlashAttention itself; it means the compilation step used more RAM than the machine had available, and the fix is to reduce parallelism or use a machine with more memory.
Should I always try to avoid building FlashAttention from source?
Yes, whenever possible. Prebuilt wheels published for common torch, CUDA, and Python combinations skip compilation entirely and avoid all of these failure modes. Only build from source when no matching wheel exists for your specific environment, and expect it to take significantly longer and require more careful RAM and compiler version management.
Do I need the CUDA toolkit installed if I only use prebuilt wheels?
No. If you install a prebuilt FlashAttention wheel that matches your torch and CUDA version exactly, you do not need a separately installed CUDA toolkit or nvcc; the necessary CUDA runtime pieces are already handled by your torch installation. The toolkit is only required for compiling from source.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolSelf-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
Free ToolKV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Related problems
gcc or g++ version rejected while building CUDA extensions
Every CUDA toolkit release only supports compiling with a specific range of gcc and g++ major versions, and nvcc explicitly rejects anything outside that range rather than risk generating broken code. This most often surfaces on recently released Linux distributions whose default gcc is newer than what an older, already-installed CUDA toolkit supports, and the fix is installing a supported older gcc/g++ version alongside the default and pointing CC and CXX at it for the build.
vLLM fails to install or import due to torch and CUDA mismatches
vLLM ships prebuilt wheels compiled against a specific, exact torch and CUDA version, and it uses custom CUDA kernels that only work with that pairing. Most install failures happen because torch was already installed separately, or an existing environment has an incompatible CUDA toolkit, so the fix is almost always a fresh virtual environment where pip resolves vLLM and its exact torch dependency together in one pass.
CUDA version mismatch between PyTorch and the system driver
PyTorch ships its own bundled CUDA runtime inside the wheel, so it never uses your system's CUDA toolkit (the one nvcc reports). The only number that matters is the driver's maximum supported CUDA version, shown top right in nvidia-smi output. Fix the mismatch by installing a torch wheel built for a CUDA version at or below that number, not by touching nvcc or the toolkit.
bitsandbytes cannot find or detect a working CUDA setup
bitsandbytes needs to locate the exact CUDA runtime shared library at import time and load a matching precompiled binary for that version. The setup fails most often because LD_LIBRARY_PATH points at a different CUDA installation than the one it detected, or because an older bitsandbytes version could not auto-detect the GPU correctly. Upgrading to the latest bitsandbytes and running its built-in diagnostic resolves the majority of cases.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.