Why PyTorch GPU memory fragmentation causes intermittent CUDA out of memory errors
RuntimeError: CUDA out of memory. Tried to allocate X MiB (GPU 0; ...; Y MiB free; ...); see documentation for PYTORCH_CUDA_ALLOC_CONF
Also appears as
- torch.cuda.OutOfMemoryError: ... this may be caused by fragmentation. See PYTORCH_CUDA_ALLOC_CONF documentation
- CUDA memory allocation failed even though total free memory should be sufficient
Short answer
PyTorch explicitly detects and reports fragmentation in this error, pointing you at PYTORCH_CUDA_ALLOC_CONF for a reason: the caching allocator's memory is split into segments sized for past allocations, and a new allocation that does not match any free segment's size fails even with adequate total free memory. Setting expandable_segments:True and normalizing input shapes are the two highest-leverage fixes.
Affects: Long-running PyTorch processes with variable tensor shapes, including inference servers, training loops with variable-length batches, and multi-tenant model hosting
Stop fragmentation-driven OOMs
- 1Set the environment variable before process start: export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which allows the allocator to grow segments contiguously instead of fragmenting the address space with new fixed-size ones.
- 2Restart the affected process; expandable_segments changes allocator behavior going forward, it does not defragment memory already reserved under the old behavior.
- 3Normalize input shapes by padding or bucketing batches into a small number of fixed sizes, reducing the variety of allocation sizes the allocator has to manage.
- 4If using an older PyTorch, also try PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128 (or a value tuned to your typical allocation size) to limit how large a single cached block can be before it is split, reducing certain fragmentation patterns.
- 5For serving workloads, schedule periodic process restarts as a standing operational practice, since fragmentation accumulates over the lifetime of a process regardless of these settings.
How to confirm this is your problem
- Error message explicitly references PYTORCH_CUDA_ALLOC_CONF or fragmentation in its text
- Failures are intermittent and correlate with process uptime rather than a fixed workload size
- torch.cuda.memory_summary() shows many small free blocks rather than one large contiguous region
- The identical allocation succeeds immediately after a process restart
Root causes and fixes
Highly variable allocation sizes over a long process lifetime scatter free memory into many differently-sized, non-adjacent blocks
PyTorch's allocator manages memory as segments it requests from CUDA and then subdivides for individual tensors. When allocation sizes vary a lot (different batch sizes, sequence lengths, or intermediate tensor shapes across different requests), freed blocks end up sized for their original tensor rather than for whatever comes next, so new large requests may not find a matching free block even when total free memory is ample.
Fix: Enable expandable_segments so segments can grow to accommodate new allocation sizes in place instead of forcing the allocator to carve out new differently-sized segments.
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
max_split_size_mb is left at its default, allowing large blocks to be split into many small pieces that cannot easily recombine
By default, PyTorch may split a large cached block to satisfy a smaller allocation request, and those split pieces are not automatically recombined when freed, gradually reducing the largest available contiguous block over time even as total free memory stays constant.
Fix: Set max_split_size_mb to a value close to your largest typical allocation size, preventing the allocator from creating excessively small split fragments that never recombine into something useful for future large allocations.
export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:256
The process has been running for an extended period without restart, accumulating fragmentation
Fragmentation is a property of the CUDA context's address space and accumulates over the lifetime of a process; a server that has served requests continuously for days has had far more opportunity to fragment than a freshly started one, even at an identical instantaneous memory footprint.
Fix: Build periodic restarts into your operational runbook for long-lived inference servers, treating fragmentation similarly to a slow memory leak that resets cleanly on restart.
Multiple distinct workloads with different tensor shape profiles share the same process and CUDA context
Running an embedding model alongside a generation model, or batching requests from very different client applications with different sequence length distributions, in the same process multiplies the variety of allocation sizes competing for the same address space, accelerating fragmentation compared to a single, uniform workload.
Fix: Isolate workloads with meaningfully different shape profiles into separate processes, each with its own CUDA context and address space, even if it means running more small processes instead of one large one.
An outdated PyTorch version lacks the expandable_segments allocator option entirely
expandable_segments was added as a targeted fix for this fragmentation class starting with PyTorch 2.0; workloads on older versions cannot use it and are more exposed to fragmentation regardless of shape normalization efforts.
Fix: Upgrade PyTorch to a current release to gain access to the improved allocator behavior, checking that your CUDA toolkit version remains compatible with the new PyTorch build.
pip install --upgrade torch
Diagnostic commands
Inspect free block sizes directly
python -c "import torch; print(torch.cuda.memory_summary())"
The summary table breaks down memory by size class (blocks under 1MB, 1-10MB, and so on); many small free blocks with no large ones present is a direct signature of fragmentation.
Check whether expandable_segments is active
python -c "import os; print(os.environ.get('PYTORCH_CUDA_ALLOC_CONF', 'not set'))"If unset, this is very likely the single highest-value change available before deeper investigation.
Reproduce with a controlled shape sweep
python -c "import torch; a=[torch.randn(i*1000000, device='cuda') for i in range(1,50)]; del a; torch.cuda.empty_cache(); print(torch.cuda.memory_summary())"
This synthetic test deliberately fragments memory with varying sizes; comparing memory_summary() before and after with expandable_segments toggled demonstrates the effect directly on your own PyTorch version.
Stopping it from happening again
- Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True as a default environment setting across all training and serving deployments, not an incident response.
- Standardize input shapes through padding or bucketing wherever request-level variability allows it.
- Build scheduled restarts into operational runbooks for any long-running inference process.
- Keep PyTorch versions current to benefit from ongoing allocator improvements around this exact problem.
When this becomes an architecture problem
If fragmentation persists as a recurring production issue even with expandable_segments enabled, shapes normalized, and restarts scheduled, consider whether the workload's inherent shape diversity (for example, one service handling both short chat completions and long document summarization) should be split into separately provisioned serving pools rather than continuing to fight the allocator.
Frequently asked questions
What exactly does PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True do?
It changes PyTorch's CUDA memory allocator so that memory segments can grow contiguously to accommodate new allocation sizes, instead of the allocator creating new, separately sized segments each time a request does not match an existing free block. This directly targets the scenario where varying tensor sizes over a long-running process leave free memory scattered into unusable small gaps.
Does this setting have any downside?
The main tradeoff is a small amount of additional bookkeeping overhead and it requires PyTorch 2.0 or later plus a compatible CUDA driver. For the large majority of workloads experiencing fragmentation-related OOMs, the memory efficiency gain substantially outweighs this cost, which is why it is increasingly recommended as a default rather than a special-case fix.
Why does the OOM error specifically mention fragmentation and point at this setting?
PyTorch's allocator can detect the specific pattern of failing to satisfy an allocation despite sufficient total free memory, which is the signature of fragmentation rather than genuine memory exhaustion, and it surfaces this in the error message with a pointer to the relevant configuration documentation so you do not have to guess at the root cause.
Will expandable_segments fix an OOM caused by the model genuinely being too large for the GPU?
No. If your total free memory is genuinely insufficient for the requested allocation, no allocator setting changes the underlying math; expandable_segments only helps when adequate free memory exists but is unusable due to fragmentation. Confirm with torch.cuda.memory_summary() that total free memory is actually sufficient before expecting this setting to help.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolLLM Serving Capacity Planner
Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.
Related problems
CUDA out of memory even though nvidia-smi shows free VRAM
This almost always means memory fragmentation: the allocator has enough total free memory but no single contiguous block large enough for the requested allocation, because the address space is broken into many small free-and-used segments from prior allocations of different sizes. The fix is enabling PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, reducing allocation size variability, or restarting the process to reset the address space.
GPU memory stays full after inference finishes
This is expected PyTorch behavior, not a leak: the caching allocator keeps freed GPU memory reserved for future allocations instead of returning it to the driver, so nvidia-smi shows the process's total reserved memory rather than what is actually in use. The real leak to check for is a growing number across requests (Python references keeping tensors alive), not a single high plateau after one inference call.
How to reduce VRAM usage for LLM inference
VRAM usage during inference comes from three independent budgets: model weights (params x bytes/param), KV cache (scales with batch x context length), and activation/workspace memory. Reducing usage means attacking whichever budget dominates: quantize weights to cut the largest fixed cost, cap max context length and concurrency to cut the largest variable cost, and use an efficient attention/serving backend to minimize workspace overhead.
GuideLLM Observability: TTFT, ITL, Throughput, and GPU Dashboards
LLM inference observability: track TTFT, inter-token latency, throughput, and GPU utilization with dashboards that catch problems before users report them.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.