Why you get CUDA out of memory when nvidia-smi shows free VRAM
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB. GPU 0 has a total capacity of 24.00 GiB of which 3.21 GiB is free
Also appears as
- RuntimeError: CUDA out of memory, but nvidia-smi shows several GB free
- OutOfMemoryError despite low reported memory.used
Short answer
This almost always means memory fragmentation: the allocator has enough total free memory but no single contiguous block large enough for the requested allocation, because the address space is broken into many small free-and-used segments from prior allocations of different sizes. The fix is enabling PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, reducing allocation size variability, or restarting the process to reset the address space.
Affects: Any CUDA GPU running PyTorch-based inference or training with long-running processes and variable tensor shapes
Resolve a fragmentation-driven OOM
- 1Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True as an environment variable before starting the process; this lets PyTorch grow existing memory segments instead of creating new, differently-sized ones that fragment the address space.
- 2If the error persists, restart the process: fragmentation is a property of the current CUDA context's address space and does not survive a fresh process start.
- 3Reduce variability in tensor shapes where possible, for example by padding batches to a fixed sequence length bucket instead of exact-length batching, so the allocator reuses same-sized blocks more often.
- 4Lower the largest single allocation you are requesting (batch size, sequence length, or KV cache block size) so it can fit in the largest available contiguous free block even under current fragmentation.
- 5Call torch.cuda.empty_cache() to release cached free blocks back to the driver, which can sometimes allow the driver to coalesce free space, though this is not guaranteed to fix fragmentation.
How to confirm this is your problem
- nvidia-smi memory.free reports several GB available, larger than the allocation size the error mentions
- Error mentions a modest allocation size (hundreds of MB to a few GB) failing despite apparently ample free memory
- The failure appears intermittently, often correlated with runs that have been serving variable-length requests for a while
- A fresh process restart with an identical workload succeeds where the long-running process failed
Root causes and fixes
Repeated allocation and deallocation of differently sized tensors fragments the CUDA address space into many small non-contiguous free blocks
PyTorch's caching allocator manages memory in segments; when tensors of many different sizes are allocated and freed over a long-running process (varying batch sizes, sequence lengths, or KV cache blocks), the free memory ends up scattered across many small gaps rather than one large contiguous region, so a new large allocation can fail even though the sum of free memory would be enough.
Fix: Enable expandable segments so the allocator grows existing segments in place instead of carving out new fixed-size ones, which substantially reduces this class of fragmentation in PyTorch 2.0 and later.
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python your_script.py
Variable-length sequences in inference or training create allocations of many different sizes back to back
Serving requests of wildly different sequence lengths (say 50 tokens then 4000 tokens) forces the allocator to repeatedly find or create blocks of very different sizes, which is a primary driver of fragmentation compared to a workload with consistent batch and sequence shapes.
Fix: Bucket requests into a small number of fixed sequence-length tiers (pad to 512, 1024, 2048, and so on) so the allocator reuses a limited set of block sizes instead of a continuous spread.
max_length = min(2048, next_bucket_size(actual_length))
A long-running server process accumulates fragmentation over hours or days without ever restarting
Fragmentation is cumulative within a single CUDA context; a process that has been serving thousands of varied requests without restart has had far more opportunity to fragment its address space than one that just started, even at the same instantaneous memory_allocated() level.
Fix: Schedule periodic, low-traffic-window restarts of long-lived inference servers as a standing operational practice, treating fragmentation as a slow-accumulating resource similar to memory leaks in other long-lived services.
systemctl restart vllm-server
Multiple models or workloads share one process and interleave differently sized allocations
Running more than one model, or a model plus an embedding service, in the same Python process means their allocation patterns interleave in the same address space, increasing the variety of block sizes requested and therefore the fragmentation rate compared to a dedicated single-model process.
Fix: Isolate distinct workloads (different models, different batch profiles) into separate processes or containers, each with its own CUDA context and address space.
An older PyTorch version predates the expandable_segments allocator improvement
expandable_segments was introduced in PyTorch 2.0 as a specific fix for this fragmentation pattern; versions before that lack the option entirely, so the same workload can be meaningfully more fragmentation-prone purely due to PyTorch version.
Fix: Upgrade to a current PyTorch release that supports expandable_segments, since this is a targeted allocator improvement rather than a general version bump.
pip install --upgrade torch
Diagnostic commands
Compare free reported by nvidia-smi against the largest contiguous free block PyTorch can find
python -c "import torch; print(torch.cuda.memory_summary())"
memory_summary() breaks down allocated, reserved, and free memory by size class; if total free is large but no size class matches your allocation, fragmentation is confirmed as the cause.
Check whether expandable_segments is currently enabled
python -c "import os; print(os.environ.get('PYTORCH_CUDA_ALLOC_CONF'))"If this is unset or does not include expandable_segments:True, that is the first and highest-value change to make before investigating further.
Confirm the failure disappears after a clean process restart
kill <pid>; nvidia-smi
If the exact same allocation succeeds immediately after a restart with no other changes, this strongly confirms accumulated fragmentation rather than a genuine capacity shortfall.
Stopping it from happening again
- Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True as a standing default in your deployment environment, not a reactive fix.
- Bucket or pad variable-length inputs into a small number of fixed shapes rather than allowing unbounded shape variability.
- Schedule periodic restarts of long-running inference processes as routine maintenance, especially for services handling highly variable request sizes.
- Keep PyTorch and CUDA toolkit versions current to benefit from ongoing allocator improvements.
When this becomes an architecture problem
If fragmentation-driven OOMs recur even with expandable_segments enabled, shape bucketing in place, and regular restarts scheduled, the workload's shape variability may genuinely exceed what a single-process allocator can manage gracefully, which is a signal to redesign request batching (for example routing very long and very short requests to separately sized worker pools) rather than continuing to tune allocator flags.
Frequently asked questions
Why does PyTorch say there is not enough memory when nvidia-smi shows free VRAM?
nvidia-smi reports total free memory on the device, but PyTorch's allocator needs a single contiguous block large enough for the requested tensor. If free memory is scattered across many small gaps from prior allocations of different sizes, a request larger than any individual gap fails even though the sum of free memory would be plenty. This is memory fragmentation, and it is one of the most common causes of confusing OOM errors.
What does PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True actually change?
It changes how PyTorch's caching allocator grows memory segments: instead of allocating new fixed-size blocks that end up scattered across the address space, it allows existing segments to expand contiguously as needed. This substantially reduces the class of fragmentation caused by variable allocation sizes over a long-running process, and is safe to enable by default on PyTorch 2.0 and later.
Does restarting the process really fix fragmentation, or just hide the symptom?
It genuinely fixes it, at least until fragmentation accumulates again. A process restart destroys the CUDA context entirely, so the new context starts with a clean, unfragmented address space. It is not a permanent architectural fix, but it is a legitimate and commonly used operational mitigation, especially combined with expandable_segments to slow how quickly fragmentation reaccumulates.
Does fragmentation affect inference and training equally?
Both are affected, but training workloads with highly variable sequence lengths and frequent allocation and deallocation cycles (each training step allocates fresh activation memory) tend to fragment faster than steady-state inference serving with a stable batch shape. Long-running inference servers handling highly variable request lengths, however, can fragment just as severely over time.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolLLM Serving Capacity Planner
Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.
Related problems
PyTorch GPU memory fragmentation causing intermittent OOM
PyTorch explicitly detects and reports fragmentation in this error, pointing you at PYTORCH_CUDA_ALLOC_CONF for a reason: the caching allocator's memory is split into segments sized for past allocations, and a new allocation that does not match any free segment's size fails even with adequate total free memory. Setting expandable_segments:True and normalizing input shapes are the two highest-leverage fixes.
GPU memory stays full after inference finishes
This is expected PyTorch behavior, not a leak: the caching allocator keeps freed GPU memory reserved for future allocations instead of returning it to the driver, so nvidia-smi shows the process's total reserved memory rather than what is actually in use. The real leak to check for is a growing number across requests (Python references keeping tensors alive), not a single high plateau after one inference call.
CUDA out of memory when loading an LLM
This happens because model weights alone require roughly 2 bytes per parameter in fp16/bf16 (a 70B model needs about 140 GB before you even run inference), and that number does not fit your GPU. The fix is to either quantize the weights (AWQ, GPTQ, FP8, or GGUF), split the model across multiple GPUs with tensor parallelism, or pick a GPU with enough VRAM for the parameter count you are loading.
GuideLLM Observability: TTFT, ITL, Throughput, and GPU Dashboards
LLM inference observability: track TTFT, inter-token latency, throughput, and GPU utilization with dashboards that catch problems before users report them.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.