Why Ollama runs on CPU instead of your GPU, and how to fix it
level=WARN source=gpu.go msg="no compatible GPUs were discovered"
Also appears as
- Error: no NVIDIA/AMD GPU detected, falling back to CPU mode
- ollama ps shows 100% CPU, 0% GPU for a loaded model
Short answer
Ollama falls back to CPU silently, without an obvious error, most often because the NVIDIA driver is missing inside a container, the model does not fit in available VRAM so Ollama offloads some or all layers to system RAM, or the GPU simply is not visible to the process. Check ollama ps for the CPU/GPU split and nvidia-smi for driver visibility before assuming the model itself is slow.
Affects: Ollama on Linux and Windows with NVIDIA or AMD GPUs, and Ollama running inside Docker or Kubernetes
Fix it in 60 seconds
- 1Run nvidia-smi on the host. If it fails or shows no GPU, the driver or GPU is not visible at the OS level and nothing Ollama does will fix that.
- 2If running in Docker, confirm you started the container with --gpus all and the NVIDIA Container Toolkit is installed.
- 3Run ollama ps while a model is loaded and check the PROCESSOR column for the CPU/GPU percentage split.
- 4If the split shows partial GPU usage, the model is larger than free VRAM; try a smaller quantization or a smaller model size.
- 5Restart the ollama service after fixing driver or container GPU access, since it detects GPUs at startup.
How to confirm this is your problem
- Generation is far slower than expected and CPU usage sits near 100% while GPU utilization stays at 0%
- ollama ps shows a percentage split like 100% CPU / 0% GPU next to the loaded model
- The same model was fast on bare metal but became slow after moving to a Docker container
- nvidia-smi shows the GPU as idle even while a prompt is actively generating
Root causes and fixes
NVIDIA driver or CUDA libraries not visible inside the container
Ollama running inside a Docker container only sees the GPU if the NVIDIA Container Toolkit is installed on the host and the container was launched with GPU passthrough enabled; without it, the container has no path to the driver and Ollama correctly falls back to CPU rather than failing outright.
Fix: Install the NVIDIA Container Toolkit on the host, then run the container with --gpus all (Docker) or the correct device plugin resource request (Kubernetes), and verify with nvidia-smi run inside the container.
docker run --gpus all -v ollama:/root/.ollama -p 11434:11434 ollama/ollama nvidia-smi
Model is larger than available VRAM
Ollama estimates how many transformer layers fit in free VRAM and offloads the rest to system RAM automatically; if the quantized model plus its KV cache exceeds available VRAM, some or all layers run on CPU, which produces much slower tokens per second without any explicit error.
Fix: Check ollama ps for the CPU/GPU percentage split, then move to a smaller quantization (for example Q4_K_M instead of Q8_0) or a smaller parameter count model that fits fully in VRAM.
ollama ps nvidia-smi --query-gpu=memory.used,memory.total --format=csv
GPU driver installed but the wrong CUDA compute capability or an unsupported GPU
Very old GPUs, or systems with a driver version too old for the CUDA runtime Ollama bundles, are not recognized during GPU discovery, so Ollama logs a warning and falls back to the CPU backend even though the card is physically present.
Fix: Update the NVIDIA driver to a version that matches or exceeds the CUDA toolkit version Ollama ships with, and confirm the GPU is on Ollama's supported compute capability list.
nvidia-smi --query-gpu=driver_version,compute_cap --format=csv
AMD GPU without ROCm support configured
AMD GPU support in Ollama depends on ROCm being correctly installed and the specific GPU being on the supported HSA override list; unsupported or unlisted AMD cards fall back to CPU even when the ROCm stack is present.
Fix: Verify your AMD GPU is on Ollama's supported list and set the HSA_OVERRIDE_GFX_VERSION environment variable if needed for a close-but-unlisted GPU architecture.
export HSA_OVERRIDE_GFX_VERSION=10.3.0
Multiple GPU processes competing for VRAM
If another process (a training job, a second Ollama instance, or another inference server) already holds most of the VRAM, Ollama's own allocation attempt fails or is truncated, and it silently reduces GPU layer offload to fit in what remains.
Fix: Check nvidia-smi for other processes holding VRAM and stop or reschedule them, or dedicate a specific GPU to Ollama with CUDA_VISIBLE_DEVICES.
nvidia-smi export CUDA_VISIBLE_DEVICES=0
Diagnostic commands
Confirm the GPU is visible at all
nvidia-smi
If this command fails or lists no devices, the problem is at the driver/OS level, not Ollama; fix the driver install before touching Ollama config.
Check Ollama's own CPU/GPU split
ollama ps
The PROCESSOR column shows a percentage like 100% GPU, 50%/50%, or 100% CPU for the currently loaded model, telling you directly whether offload is partial or total.
Check GPU memory headroom
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
If used is close to total before you even load a model, another process is holding VRAM that Ollama needs.
Tail the Ollama service logs at startup
journalctl -u ollama -n 100 --no-pager
Look for lines mentioning gpu.go or CUDA driver discovery; a no compatible GPUs were discovered warning confirms Ollama itself never found the card.
Stopping it from happening again
- Always verify nvidia-smi works inside the exact container image you deploy, not just on the host, before shipping to production.
- Size models against actual free VRAM, not total VRAM, accounting for the KV cache and any other GPU workloads on the box.
- Pin the NVIDIA driver and CUDA toolkit versions in your base image so a routine OS update cannot silently break GPU discovery.
- Add a startup health check that asserts ollama ps reports GPU usage before marking a deployment healthy.
When this becomes an architecture problem
If you are repeatedly fighting driver and VRAM sizing issues across multiple machines, that points to a hardware and capacity planning gap rather than a one-off config bug, and it is worth sizing GPUs against your real model and context length requirements before buying more hardware.
Frequently asked questions
Does Ollama tell me explicitly when it falls back to CPU?
Not prominently. It logs a warning during GPU discovery and shows the actual split in ollama ps, but there is no loud error, which is why silent CPU fallback is such a common source of unexplained slowness.
Why is GPU usage 0% even though nvidia-smi sees the card?
Nvidia-smi being able to see the card only confirms the driver is installed at the OS level; Ollama needs its own CUDA discovery to succeed too, and inside containers this additionally requires the NVIDIA Container Toolkit and GPU passthrough flags.
Can I force Ollama to only use the GPU and fail instead of falling back to CPU?
Ollama does not currently expose a strict GPU-only flag that errors out; the practical approach is to monitor ollama ps in your deployment pipeline and treat a CPU-heavy split as a failed health check.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolSelf-Hosted LLM Hardware Estimator
Estimate the VRAM footprint, GPU count, and hardware budget required to self-host an open-weight LLM with your concurrency and context needs.
Free ToolNVIDIA GPU Selector for LLM Workloads
Score your workload across model size, concurrency, latency, budget, and facility power to get a recommended GPU tier from RTX-class to multi-node B200 clusters.
Related problems
Ollama generates tokens very slowly
Slow Ollama generation almost always traces back to the model running partly or fully on CPU instead of GPU, either because it does not fit in VRAM, the GPU was never detected, or n_gpu_layers is set too low in a llama.cpp-based config. Check the eval rate in verbose output and the CPU/GPU split in ollama ps before tuning anything else.
torch.cuda.is_available() returns False even though a GPU is present
torch.cuda.is_available() returning False almost always means either the installed torch wheel is a CPU-only build, or the process cannot see the GPU due to a driver, container, or environment variable problem. Checking torch.version.cuda for None immediately tells you whether you have a CPU-only wheel, which is the single most common cause and the fastest thing to rule out.
GPU not visible inside a Docker container
Docker containers cannot see a host GPU unless the NVIDIA Container Toolkit is installed and the nvidia runtime is registered with the daemon, since containers are isolated from host devices by default. The fix is almost always to install nvidia-container-toolkit, run nvidia-ctk runtime configure, restart Docker, and launch with --gpus all. If nvidia-smi already fails on the host itself, the problem is the driver, not Docker.
NVIDIA GPU Operator pods stuck installing or crashlooping
GPU Operator installation problems almost always come from a preinstalled host driver conflicting with the Operator's own driver container, or from Node Feature Discovery never labeling GPU nodes so downstream components stay unscheduled. Check kubectl get pods -n gpu-operator first to see which subsystem is failing, then confirm whether the node has a preexisting driver and whether NFD applied the expected NVIDIA labels.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
GuideCPU Inference for Small Language Models: When It Works
CPU inference for small language models explained: Intel AMX, llama.cpp, realistic throughput numbers, and when skipping the GPU actually makes sense.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.