Why Ollama generates tokens slowly, and how to speed it up
total duration: 45.2s, eval rate: 1.84 tokens/s
Also appears as
- ollama run is extremely slow compared to before
- prompt eval rate: 12 tokens/s eval rate: 2 tokens/s
Short answer
Slow Ollama generation almost always traces back to the model running partly or fully on CPU instead of GPU, either because it does not fit in VRAM, the GPU was never detected, or n_gpu_layers is set too low in a llama.cpp-based config. Check the eval rate in verbose output and the CPU/GPU split in ollama ps before tuning anything else.
Affects: Ollama and llama.cpp on any GPU or CPU-only setup, most noticeable on 7B+ models on consumer GPUs
Fastest path to normal speed
- 1Run the same prompt with ollama run <model> --verbose and note the eval rate in tokens per second.
- 2Run ollama ps while the model is loaded and check whether the PROCESSOR column shows a GPU percentage or 100% CPU.
- 3If it shows CPU, work through the ollama-not-using-gpu causes: driver visibility, VRAM headroom, and container GPU passthrough.
- 4If GPU usage is already high but still slow, check nvidia-smi -q -d TEMPERATURE for thermal throttling.
- 5For llama.cpp directly, confirm n_gpu_layers is set high enough to offload the full model rather than a low default.
How to confirm this is your problem
- Verbose output reports an eval rate far below what the GPU should be capable of for that model size
- The same model was fast previously but slowed down after a driver update, a model swap, or a longer prompt
- GPU utilization in nvidia-smi bounces near 0% during generation despite the model being loaded
- Generation speed drops noticeably as conversation context grows longer
Root causes and fixes
Model running on CPU instead of GPU
CPU inference for a multi-billion parameter model is an order of magnitude slower than GPU inference because it lacks the massive parallelism of a GPU's thousands of cores; this happens whenever the GPU is not detected, VRAM is insufficient, or the container lacks GPU passthrough, and it is the single biggest driver of complaints about slow Ollama.
Fix: Confirm GPU usage with ollama ps and nvidia-smi, then fix driver visibility, container GPU passthrough, or reduce model/quantization size so it fits fully in VRAM.
ollama ps nvidia-smi
Model larger than VRAM causing partial layer offload
When a model almost but not quite fits in VRAM, Ollama offloads the layers that do not fit to system RAM and CPU, so every forward pass bounces between GPU and CPU compute for different layers, which is dramatically slower than running fully on either device alone.
Fix: Move to a smaller quantization of the same model (Q4_K_M instead of Q8_0 or FP16) or reduce num_ctx, since the KV cache also consumes VRAM and competes with model weights for the same budget.
ollama ps ollama run <model>:q4_K_M
Thermal throttling under sustained load
Consumer GPUs and laptops with inadequate cooling reduce their clock speed automatically once they hit a thermal limit during long generation runs or back-to-back requests, so the first few tokens are fast and throughput degrades progressively the longer the GPU stays busy.
Fix: Check GPU temperature during a long generation with nvidia-smi, improve case airflow or reapply thermal paste on workstation GPUs, and consider a power limit adjustment if throttling is severe.
nvidia-smi -q -d TEMPERATURE
n_gpu_layers set too low in a llama.cpp-backed config
When llama.cpp is invoked directly, or through a wrapper that exposes the setting, n_gpu_layers controls how many transformer layers get offloaded to GPU; a low or default value that does not match your actual VRAM budget leaves most layers on CPU even though the GPU has room to spare.
Fix: Raise n_gpu_layers to a value that fills available VRAM, or set it to a very high number to offload as many layers as possible and let the runtime cap it at what fits.
./main -m model.gguf -ngl 999 -p "test"
Long context growing the KV cache faster than expected
Every additional token of context adds to the KV cache, and once that cache plus model weights exceed VRAM, the runtime starts trading off cache size or offloading more aggressively, so a conversation that started fast noticeably slows down as it grows longer.
Fix: Reduce num_ctx to the smallest value your use case actually needs, or summarize/trim older turns instead of keeping full history in every request.
ollama run <model> --verbose
Diagnostic commands
Get exact tokens per second
ollama run <model> --verbose
Compare the reported eval rate against published benchmarks for that model size and quantization on similar hardware; a rate an order of magnitude lower points to CPU fallback.
Check the CPU/GPU processing split
ollama ps
A GPU percentage below 100% for a model that should fit fully in VRAM confirms partial CPU offload is the bottleneck.
Watch GPU utilization live during generation
nvidia-smi dmon -s u
Utilization near 0% while tokens are streaming means the GPU is essentially idle and CPU is doing the work; near 100% utilization with still-slow output points to a genuinely undersized GPU for that model.
Check GPU temperature under load
nvidia-smi -q -d TEMPERATURE
Temperatures near the throttle limit during a long run confirm thermal throttling is reducing clocks and therefore tokens per second.
Stopping it from happening again
- Benchmark expected tokens per second for your model and quantization on your specific GPU before deploying, so a regression is obvious.
- Size models to fit fully in VRAM with headroom for the KV cache at your target context length, not just the raw weight size.
- Ensure adequate cooling and airflow for GPUs running sustained inference workloads, especially in dense workstation or edge deployments.
- Cap num_ctx to a realistic value for your application rather than leaving it at a large default that grows the KV cache unnecessarily.
When this becomes an architecture problem
If you have confirmed full GPU offload, adequate cooling, and a right-sized quantization and are still not hitting acceptable throughput, the model or context length is simply too large for the GPU you have, and the right move is proper capacity planning across a bigger card, multiple GPUs, or a purpose-built serving stack like vLLM instead of continuing to tune a single-box Ollama setup.
Frequently asked questions
Is Ollama slower than vLLM for the same model?
For single-user, single-request workloads on one GPU, per-token speed is often similar since both ultimately run the same model. Ollama is not designed for high-concurrency serving though, and throughput under multiple simultaneous requests will be much lower than a batching-aware server like vLLM.
Does a longer conversation history slow down every response?
Yes, because Ollama has to account for the growing context on each turn, and the KV cache for that context consumes more VRAM as it grows, which can push a model from fully GPU-resident to partially offloaded partway through a long conversation.
What is a normal tokens per second for a 7B model on a consumer GPU?
On a modern 24GB consumer GPU with the model fully offloaded, tens of tokens per second is typical for a 7B class model at a reasonable quantization; single-digit tokens per second on that hardware is a strong signal something is falling back to CPU.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolKV Cache Memory Calculator
Calculate KV cache memory per sequence and per batch from model architecture and context length, then see how many concurrent sequences your GPU can hold.
Free ToolLLM Quantization Memory Savings Calculator
Compare FP16, FP8, and INT4 memory footprints for any model size and see how many fewer GPUs quantization requires to serve it.
Related problems
Ollama not using the GPU, falls back to CPU
Ollama falls back to CPU silently, without an obvious error, most often because the NVIDIA driver is missing inside a container, the model does not fit in available VRAM so Ollama offloads some or all layers to system RAM, or the GPU simply is not visible to the process. Check ollama ps for the CPU/GPU split and nvidia-smi for driver visibility before assuming the model itself is slow.
Ollama silently truncates earlier conversation turns
Ollama silently truncates conversation history once the total tokens exceed num_ctx, which defaults to a relatively small value in many client configurations, dropping the oldest turns without any error or warning to the user or the calling application. The fix is to explicitly set num_ctx to a value that matches both your actual conversation length needs and the model's supported maximum, and to monitor token counts rather than assuming the full history is always sent.
GPU utilization stays low during LLM inference even under load
Low GPU utilization during inference almost always means the GPU is waiting on something else: request concurrency is too low for the batching scheduler to fill, the client code is calling the server synchronously one request at a time, tokenization or network I/O is serialized in front of the GPU call, or max-num-seqs is set too low to admit enough concurrent sequences. Raising effective concurrency, either by fixing the client or the server's admission limits, is almost always the fix, not more GPU compute.
High time to first token (TTFT) on LLM inference requests
High time to first token almost always comes from one of four sources: a long prompt makes the prefill pass compute-bound and simply takes time to process, the server has no prefix caching so a repeated system prompt or RAG context is recomputed on every request, the model or GPU had to cold-start (weights loading, CUDA graph capture, JIT warmup), or the request sat in a queue behind other requests before its prefill even began. Prefix caching and admission-aware queueing fix most production cases.
GuideKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.