Why llama.cpp fails to build, and how to fix common CMake and CUDA errors
CMake Error: CUDA_TOOLKIT_ROOT_DIR not found
Also appears as
- nvcc fatal : Unsupported gpu architecture
- error: 'CUDA_R_16F' was not declared in this scope
- make: *** [Makefile] Error 1
Short answer
Most llama.cpp build failures come from missing or mismatched CUDA toolkit installs, wrong cmake backend flags (forgetting to enable GGML_CUDA or GGML_METAL), or a host compiler version newer than the CUDA toolkit supports. Confirm the toolkit is installed and on PATH, pass the correct backend flag for your hardware, and match your compiler version to what your CUDA version officially supports.
Affects: llama.cpp built from source on Linux, macOS, and Windows, with CUDA, Metal, or CPU-only backends
Fastest path to a successful build
- 1Confirm the CUDA toolkit is installed and visible: nvcc --version should print a version, not a command-not-found error.
- 2Clean any previous build directory to avoid stale cmake cache conflicts: rm -rf build.
- 3Reconfigure with the correct backend flag for your hardware, for example cmake -B build -DGGML_CUDA=ON for NVIDIA GPUs or -DGGML_METAL=ON on Apple Silicon.
- 4Build with cmake --build build --config Release -j.
- 5If the compiler is rejected, install or switch to a gcc/clang version that your CUDA toolkit's release notes list as supported.
How to confirm this is your problem
- cmake configuration fails before compilation even starts, citing a missing CUDA toolkit path
- nvcc reports an unsupported GPU architecture during compilation
- The build succeeds but the resulting binary runs on CPU only, with no GPU acceleration
- A compiler error references a CUDA header symbol that seems to not exist
Root causes and fixes
CUDA toolkit not installed or not on PATH
Having an NVIDIA driver installed, which nvidia-smi confirms, is not the same as having the CUDA toolkit and its compiler nvcc installed; cmake's CUDA-enabled configuration step specifically needs nvcc and the toolkit's headers and libraries, and fails immediately if it cannot locate them, which is a different and more basic requirement than just having a working GPU driver.
Fix: Install the CUDA toolkit matching your driver's supported CUDA version, and ensure nvcc is on PATH before running cmake.
nvcc --version export PATH=/usr/local/cuda/bin:$PATH
Missing or wrong backend flag passed to cmake
llama.cpp supports multiple compute backends (CUDA, Metal, Vulkan, CPU-only), and each requires an explicit cmake flag to enable; building without the flag for your hardware silently produces a CPU-only binary or, in stricter configurations, a configuration error because required libraries were never searched for.
Fix: Pass the correct backend flag explicitly: GGML_CUDA=ON for NVIDIA GPUs, GGML_METAL=ON for Apple Silicon, and confirm after building that the binary reports GPU offload capability at runtime.
cmake -B build -DGGML_CUDA=ON cmake -B build -DGGML_METAL=ON
GPU compute capability not included in the CMAKE_CUDA_ARCHITECTURES list
CUDA code is compiled for specific compute capability targets, and if your GPU's architecture is not in the list cmake was configured with, either because it defaulted to an older list or you set it manually and missed one, nvcc reports an unsupported architecture error for that GPU at build or runtime.
Fix: Set CMAKE_CUDA_ARCHITECTURES to match your GPU's actual compute capability, which you can look up from nvidia-smi's compute_cap query or NVIDIA's published list for your GPU model.
nvidia-smi --query-gpu=compute_cap --format=csv cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89
Host compiler version incompatible with the installed CUDA toolkit
Each CUDA toolkit release officially supports a specific range of gcc, g++, or MSVC versions as the host compiler; a system with a newer default compiler than that range, common after an OS upgrade, causes nvcc to reject headers or produce cryptic template and declaration errors during compilation.
Fix: Install an older, supported gcc/g++ version alongside the system default and point CUDA at it explicitly via the CMAKE_CUDA_HOST_COMPILER cmake variable or update-alternatives, rather than fighting the default compiler.
sudo update-alternatives --config gcc cmake -B build -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-11
Stale cmake cache from a previous configuration
Reconfiguring cmake with new flags in an existing build directory sometimes leaves stale cached variables from the previous run, which can produce confusing errors that look like a fresh misconfiguration but are actually leftover state from before you fixed the toolkit or flags.
Fix: Delete the build directory entirely and reconfigure from scratch after any change to toolchain, compiler, or major cmake flags.
rm -rf build cmake -B build -DGGML_CUDA=ON
Diagnostic commands
Confirm the CUDA compiler is available
nvcc --version
Command not found means the toolkit is missing or not on PATH, independent of whether the driver works; install the toolkit before touching cmake flags.
Check your GPU's compute capability
nvidia-smi --query-gpu=compute_cap --format=csv
Use this value to set CMAKE_CUDA_ARCHITECTURES explicitly if the default architecture list does not include your card.
Check the installed compiler version
gcc --version
Compare this against your CUDA toolkit's documented supported host compiler range; a mismatch is a common source of obscure template and header errors.
Confirm the built binary actually reports GPU support
./build/bin/llama-cli --version
Modern llama.cpp builds report which backends were compiled in; if CUDA or Metal is absent from that output, the build succeeded but without the acceleration you intended.
Stopping it from happening again
- Pin a specific CUDA toolkit and compiler version pair known to work together, and document it in your build scripts rather than relying on whatever the OS currently defaults to.
- Always do a clean build (rm -rf build) after changing backend flags, toolkit versions, or compiler versions.
- Keep a short reference in your deployment repo that reproduces the exact cmake invocation for each target platform (CUDA Linux, Metal macOS, CPU-only).
- Prefer prebuilt Ollama binaries or official Docker images when you do not specifically need a custom llama.cpp build, since they ship pre-validated toolchains.
When this becomes an architecture problem
If you are maintaining custom llama.cpp builds across many different GPU generations and driver versions, standardizing on a small number of validated Docker build images, rather than building from source on every target machine, is worth the upfront investment and removes an entire class of toolchain drift problems.
Frequently asked questions
Do I need the full CUDA toolkit, or is the NVIDIA driver enough to build llama.cpp with GPU support?
You need the full CUDA toolkit, which includes nvcc and CUDA headers and libraries, in addition to the driver. The driver alone lets a prebuilt binary run on the GPU; building CUDA code from source requires the toolkit's compiler.
Why does my llama.cpp build succeed but still run on CPU only?
Most likely the backend flag (GGML_CUDA=ON or GGML_METAL=ON) was not passed during cmake configuration, so the default CPU-only backend was built instead. Reconfigure explicitly with the correct flag and rebuild from a clean directory.
Can I use a newer compiler than my CUDA toolkit officially supports?
It sometimes works but is unsupported and can produce confusing compilation errors that are hard to diagnose. If you hit strange template or header errors during a CUDA build, installing an older, officially supported compiler version alongside the system default is the more reliable fix.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
NVIDIA GPU Selector for LLM Workloads
Score your workload across model size, concurrency, latency, budget, and facility power to get a recommended GPU tier from RTX-class to multi-node B200 clusters.
Free ToolSelf-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
Related problems
llama.cpp fails to load a GGUF model file
A GGUF load failure in llama.cpp is almost always one of three things: the file was truncated or corrupted during download, the file uses a quantization or metadata format newer than your llama.cpp build supports, or the model was split into multiple GGUF shards and only some of them were downloaded. Verify the file size and checksum first, then check your llama.cpp version against the GGUF version the file requires.
gcc or g++ version rejected while building CUDA extensions
Every CUDA toolkit release only supports compiling with a specific range of gcc and g++ major versions, and nvcc explicitly rejects anything outside that range rather than risk generating broken code. This most often surfaces on recently released Linux distributions whose default gcc is newer than what an older, already-installed CUDA toolkit supports, and the fix is installing a supported older gcc/g++ version alongside the default and pointing CC and CXX at it for the build.
CUDA version mismatch between PyTorch and the system driver
PyTorch ships its own bundled CUDA runtime inside the wheel, so it never uses your system's CUDA toolkit (the one nvcc reports). The only number that matters is the driver's maximum supported CUDA version, shown top right in nvidia-smi output. Fix the mismatch by installing a torch wheel built for a CUDA version at or below that number, not by touching nvcc or the toolkit.
nvidia-smi command not found or fails to communicate with the driver
nvidia-smi not found or unable to communicate almost always means the NVIDIA kernel module never loaded, and the two most common reasons are that the driver was never installed, or the driver is installed but Secure Boot is blocking the unsigned kernel module from loading. WSL2 users hit a different variant: the driver must be installed on the Windows host, never inside the Linux guest.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
GuideApple Silicon for Local LLM Development: A Practical Guide
Apple Silicon for local LLM development: M-series unified memory, MLX vs llama.cpp performance, and where Mac fits versus GPU servers for enterprise teams.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.