Why CPU offloading makes LLM inference so slow, and when it is still worth it
Model runs without an out-of-memory error after enabling device_map="auto" with CPU offload, but generation throughput drops to a fraction of a token per second
Also appears as
- Inference works but is 10x-50x slower with cpu offload enabled
- accelerate offload_folder makes generation unusably slow
Short answer
CPU offload trades memory capacity for speed because every offloaded layer's weights must cross the PCIe bus (typically 16-64 GB/s) on every forward pass, versus terabytes-per-second on-GPU HBM bandwidth; this is not a bug, it is the fundamental cost of running more model than your GPU can hold. The real fix is usually to reduce how much needs to be offloaded (quantize first) rather than trying to make offloading itself faster.
Affects: Any deployment using CPU or disk offload (accelerate device_map="auto" with offload, llama.cpp partial GPU layers, DeepSpeed ZeRO-Infinity) to fit a model that does not fit entirely in VRAM
Recover throughput or make an informed tradeoff
- 1Quantize the model to 4-bit or 8-bit first, since this often shrinks it enough to fit entirely in VRAM and eliminates the need for offload altogether, which is almost always faster than any offload configuration.
- 2If offload is still necessary, offload the minimum number of layers possible rather than a large fraction, since throughput degrades roughly in proportion to how much of the model must cross PCIe per token.
- 3Prefer GPU-resident KV cache with only weights offloaded, rather than offloading KV cache itself, since KV cache is read on every single generation step and is far more sensitive to bandwidth than weights.
- 4If your workload can tolerate batch processing instead of interactive latency, batch requests to amortize the PCIe transfer cost across more tokens per transfer.
- 5If throughput remains unacceptable, treat this as confirmation that the workload needs more or bigger GPUs rather than continuing to tune an offload configuration that is fundamentally bandwidth-bound.
How to confirm this is your problem
- Model loads and generates correct output with no OOM error, but tokens per second drops by 10x or more compared to full-GPU serving
- GPU utilization (nvidia-smi) is low or spiky, alternating between idle and brief bursts, rather than sustained high utilization
- Increasing batch size does not proportionally improve throughput the way it would on a fully GPU-resident model
- The slowdown is proportional to how many layers or what fraction of weights are offloaded to CPU or disk
Root causes and fixes
Offloaded layers must be transferred over PCIe on every forward pass, and PCIe bandwidth is orders of magnitude lower than GPU HBM bandwidth
GPU HBM (on-device memory) delivers roughly 2-3 TB/s of bandwidth on modern data center GPUs, while PCIe Gen4/Gen5 delivers roughly 32-64 GB/s between CPU and GPU, a gap of 40-90x. Every offloaded layer's weights must cross that much slower link for every single token generated, so the more of the model lives off-GPU, the more the per-token latency is dominated by data transfer rather than compute.
Fix: Minimize the fraction of the model that needs to be offloaded, ideally by quantizing first so more (or all) of the model fits natively in VRAM, since offload cost scales directly with how many bytes must cross PCIe per forward pass.
python -c "print(2e12/50e9, 'x faster HBM is than typical PCIe Gen4')"
device_map="auto" offloaded more layers to CPU than strictly necessary because of a conservative memory margin
Accelerate's automatic device mapping reserves some GPU memory headroom by default when deciding how many layers to place on GPU versus CPU, which can be more conservative than your actual available memory allows, pushing more layers to the slow path than a tighter manual configuration would.
Fix: Manually specify a max_memory dict per device to allow accelerate to place more layers on GPU, verifying with nvidia-smi that you are not leaving significant unused VRAM headroom while offloading unnecessarily.
from accelerate import infer_auto_device_map
device_map = infer_auto_device_map(model, max_memory={0: "22GiB", "cpu": "64GiB"})Disk offload (not just CPU RAM offload) is active, adding storage I/O latency on top of PCIe transfer
When CPU RAM itself is insufficient and an offload_folder is configured, weights spill to disk, adding storage read latency (even on NVMe, meaningfully slower than RAM) on top of the PCIe transfer cost, compounding the slowdown further than RAM-only offload.
Fix: Ensure enough CPU RAM is available that offload stays in RAM rather than spilling to disk, since RAM-to-GPU offload, while still slow relative to native VRAM, is meaningfully faster than adding a disk round trip on every forward pass.
free -h
KV cache, not just weights, is being kept partly on CPU
KV cache must be read and updated on every single decoding step for every token generated, making it far more latency-sensitive to offload than weights, which are read once per layer per forward pass; offloading KV cache to CPU multiplies the PCIe traffic per token far more than offloading static weights does.
Fix: Keep KV cache entirely GPU-resident even if some weight layers must be offloaded, prioritizing KV cache placement on GPU over weight placement when memory is tight.
PCIe link is running at a lower generation or lane width than the hardware supports, for example due to a BIOS setting or a shared riser
A GPU installed in a slot negotiated at PCIe Gen3 x8 instead of its supported Gen4/Gen5 x16 halves or quarters available bandwidth without any error being raised, silently making offload performance far worse than the hardware's real capability.
Fix: Check the negotiated PCIe link speed and width for the GPU and correct BIOS, riser, or slot configuration issues if it is running below the card's rated specification.
nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv
Diagnostic commands
Measure actual tokens per second with and without offload
python -c "import time; t0=time.time(); model.generate(**inputs, max_new_tokens=100); print(100/(time.time()-t0), 'tokens/sec')"
Compare this against a fully GPU-resident configuration of a similarly sized model; a 10-50x gap confirms PCIe-bound offload behavior rather than a misconfiguration elsewhere.
Watch GPU utilization pattern during generation
nvidia-smi dmon -s u -c 20
Low, spiky utilization with idle gaps between bursts indicates the GPU is frequently waiting on data transfer from CPU rather than continuously computing, the classic signature of offload-bound throughput.
Confirm negotiated PCIe link speed matches hardware capability
nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.gen.max,pcie.link.width.current,pcie.link.width.max --format=csv
If current is lower than max, you are losing bandwidth to a configuration issue on top of the inherent offload cost, and fixing that recovers some throughput before any software-level change.
Stopping it from happening again
- Treat CPU/disk offload as a capacity-of-last-resort fallback in architecture decisions, not a routine serving configuration for production latency-sensitive workloads.
- Quantize before considering offload, since the memory saved by quantization often eliminates the need for offload entirely.
- Size GPU memory for your model at deployment planning time so offload is never required in the production path.
- If offload is used for occasional large-model access (not production serving), set throughput expectations accordingly rather than being surprised by them.
When this becomes an architecture problem
If your production workload requires interactive latency and the model that meets your accuracy bar only fits with CPU offload even after quantization, that is a clear signal you need more or larger GPUs rather than continuing to tune offload settings, since the PCIe bandwidth ceiling is a hardware property, not a software one.
Frequently asked questions
Why is CPU offload so much slower than just running a smaller model?
Offload does not reduce the amount of computation; it adds a PCIe data transfer step before the GPU can use each offloaded layer's weights, on every single forward pass. PCIe bandwidth (tens of GB/s) is roughly 40-90x slower than GPU HBM bandwidth (terabytes/s), so any meaningful fraction of the model living off-GPU dominates total latency, often making offload far slower in practice than simply serving a smaller model that fits entirely in VRAM.
Is there a way to make CPU offload fast?
Not fundamentally; PCIe bandwidth is a hardware ceiling that no software configuration bypasses. You can minimize the damage by offloading as few layers as possible, keeping KV cache GPU-resident, avoiding disk-level offload, and ensuring the PCIe link is negotiated at its full rated speed, but offload will always be substantially slower than fully GPU-resident serving for the same model.
Does NVLink change this calculation?
Only between GPUs, not between CPU and GPU; NVLink connects GPU-to-GPU at far higher bandwidth than PCIe (hundreds of GB/s to a few TB/s depending on generation), which is why multi-GPU tensor parallelism does not suffer the same bottleneck as CPU offload. CPU-to-GPU transfer still goes through PCIe (or a CPU-GPU interconnect like NVLink-C2C on select platforms) regardless of how the GPUs are connected to each other.
Should I use CPU offload for a batch processing job instead of interactive serving?
It is a more reasonable fit there, since batch jobs can amortize the fixed PCIe transfer cost across more tokens and are typically not latency-sensitive on a per-request basis. It is still meaningfully slower than fully GPU-resident inference, so validate that the total job completion time meets your batch window before committing to this architecture.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
CPU-Only LLM Inference Feasibility Calculator
Estimate realistic tokens per second for CPU-only inference from model size, quantization, and system RAM bandwidth, and see whether it clears your target throughput.
Free ToolGPU Server Buy vs Rent Calculator
Turn GPU price, power cost, and cloud hourly rates into an annual owned-versus-rented comparison, plus the utilization rate at which buying starts to win.
Free ToolNVIDIA GPU Selector for LLM Workloads
Score your workload across model size, concurrency, latency, budget, and facility power to get a recommended GPU tier from RTX-class to multi-node B200 clusters.
Related problems
The model is too large to fit on a single GPU
This is a hard arithmetic ceiling, not a bug: a 70B parameter model needs about 140 GB in bf16, which exceeds every single current GPU. There are exactly three valid fixes: quantize the weights to reduce bytes-per-parameter, shard the model across multiple GPUs with tensor or pipeline parallelism, or choose a smaller model that fits your single-GPU budget at the precision you need.
How to reduce VRAM usage for LLM inference
VRAM usage during inference comes from three independent budgets: model weights (params x bytes/param), KV cache (scales with batch x context length), and activation/workspace memory. Reducing usage means attacking whichever budget dominates: quantize weights to cut the largest fixed cost, cap max context length and concurrency to cut the largest variable cost, and use an efficient attention/serving backend to minimize workspace overhead.
GPU utilization stays low during LLM inference even under load
Low GPU utilization during inference almost always means the GPU is waiting on something else: request concurrency is too low for the batching scheduler to fill, the client code is calling the server synchronously one request at a time, tokenization or network I/O is serialized in front of the GPU call, or max-num-seqs is set too low to admit enough concurrent sequences. Raising effective concurrency, either by fixing the client or the server's admission limits, is almost always the fix, not more GPU compute.
GuideOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
GuideCPU Inference for Small Language Models: When It Works
CPU inference for small language models explained: Intel AMX, llama.cpp, realistic throughput numbers, and when skipping the GPU actually makes sense.
GuideThe LLM Inference Cost Optimization Playbook
Cut LLM inference costs with a practical playbook: quantization, batching, GPU right-sizing, caching, and the on-prem vs API breakeven math for 2026.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.