Why GPU memory stays full after inference finishes, and whether it is actually a leak
nvidia-smi still shows the process holding X GiB of GPU memory after the script has finished running or the request has completed
Also appears as
- CUDA memory not freed after del model
- GPU memory usage does not decrease after torch.cuda.empty_cache()
Short answer
This is expected PyTorch behavior, not a leak: the caching allocator keeps freed GPU memory reserved for future allocations instead of returning it to the driver, so nvidia-smi shows the process's total reserved memory rather than what is actually in use. The real leak to check for is a growing number across requests (Python references keeping tensors alive), not a single high plateau after one inference call.
Affects: Any long-running Python process using PyTorch or a serving framework built on it, including vLLM, transformers pipelines, and custom inference servers
Confirm whether this is normal caching or a real leak
- 1Compare torch.cuda.memory_allocated() (memory actually in use) against torch.cuda.memory_reserved() (what nvidia-smi shows); a large gap between them is the caching allocator working as designed, not a leak.
- 2If you need the memory back for another process on the same GPU, call torch.cuda.empty_cache() to release the unused cached blocks back to the driver.
- 3If memory_allocated() itself grows across repeated inference calls (not just memory_reserved()), you have a real leak: check for tensors kept alive by Python references such as accumulating lists, un-detached outputs, or a growing KV cache in a custom serving loop.
- 4For a hard reset between workloads on the same process, del the model object, call gc.collect(), then torch.cuda.empty_cache(), in that order.
- 5If nothing but ending the process itself frees the memory, the process is the correct unit of isolation: run separate inference jobs in separate processes rather than reusing one long-lived process for unrelated models.
How to confirm this is your problem
- nvidia-smi memory.used for the process stays constant and high well after the last request finished
- torch.cuda.empty_cache() reduces the nvidia-smi number somewhat but not fully
- Restarting the process (not just the model) reliably frees the memory, confirming it is process-scoped not model-scoped
- memory_allocated() is much lower than memory_reserved() when checked interactively
Root causes and fixes
PyTorch's caching allocator intentionally retains freed GPU memory for reuse rather than returning it to the CUDA driver
Requesting memory from the CUDA driver (cudaMalloc) is expensive relative to typical tensor lifetimes, so PyTorch's allocator caches previously freed blocks and reuses them for future allocations of similar size, without a driver round trip. nvidia-smi reports memory reserved by the process's CUDA context, which includes this cache, not just currently-live tensors.
Fix: This is not something to fix; it is the intended design. If you specifically need the memory available to another process on the same GPU, call torch.cuda.empty_cache() to return the unused cached blocks to the driver.
python -c "import torch; print(torch.cuda.memory_allocated()/1e9, torch.cuda.memory_reserved()/1e9)"
A reference to output tensors, hidden states, or a KV cache is kept alive somewhere in the serving loop
If a list, a global variable, or a logging call retains a reference to a GPU tensor from each request, Python's garbage collector cannot free the underlying CUDA memory even after the request completes, so memory_allocated() itself climbs request over request rather than plateauing.
Fix: Audit the serving loop for accumulating structures (metrics buffers, debug logs, caches) that hold onto tensors, and explicitly .detach().cpu() or del any tensor you do not need to keep on GPU past the request.
output = output.detach().cpu() del hidden_states; import gc; gc.collect()
Autograd graphs are retained because inference code was not run inside torch.no_grad()
Without torch.no_grad(), every forward pass builds a graph of intermediate activations needed for a hypothetical backward pass, and those activations are kept alive as long as anything references the output, multiplying real memory use for inference workloads that will never call backward().
Fix: Wrap all inference-only forward passes in torch.no_grad() or use model.eval() combined with torch.inference_mode(), which additionally disables autograd bookkeeping entirely.
with torch.inference_mode():
output = model.generate(**inputs)A CUDA stream or context from a crashed or killed thread is not cleaned up
If a worker thread or subprocess is killed (not gracefully exited) while holding CUDA allocations, the parent process's CUDA context can retain those allocations until the process itself terminates, since the driver associates memory with the process, not the thread.
Fix: Ensure worker processes shut down cleanly (catch signals, call cleanup handlers) rather than being SIGKILLed, and treat a killed worker's memory as unrecoverable until the whole process restarts.
nvidia-smi --query-compute-apps=pid,used_memory --format=csv
Memory fragmentation makes the allocator hold more reserved memory than the live tensor set would otherwise require
Repeated allocation and deallocation of differently-sized tensors (variable batch sizes or sequence lengths) can leave the allocator's free blocks too fragmented to satisfy new requests from cache, forcing it to request and retain additional driver memory even though total live tensor memory is unchanged.
Fix: Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to reduce fragmentation-driven over-reservation, or standardize batch/sequence shapes where practical.
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
Diagnostic commands
Separate real usage from cached reservation
python -c "import torch; print('allocated', torch.cuda.memory_allocated()/1e9); print('reserved', torch.cuda.memory_reserved()/1e9)"allocated is live tensor memory; reserved is what nvidia-smi shows. If reserved is much higher than allocated, this is normal caching. If allocated itself is high and climbing across requests, you have a genuine reference leak.
Track allocated memory across repeated identical requests
python -c "import torch; [print(i, torch.cuda.memory_allocated()/1e9) for i in range(5)]" # run inside your request loop
A flat or oscillating number confirms normal steady-state behavior; a monotonically increasing number confirms a leak that needs code-level investigation, not an allocator setting.
Get a full memory snapshot with allocation history
python -c "import torch; torch.cuda.memory._record_memory_history(); torch.cuda.memory._dump_snapshot('snapshot.pickle')"The snapshot, viewable with PyTorch's memory visualizer, shows exactly which allocation call sites are holding memory, pinpointing the leaking reference instead of guessing.
Stopping it from happening again
- Always wrap inference-only code in torch.inference_mode() rather than relying on model.eval() alone.
- Add a periodic assertion in long-running services that memory_allocated() has not grown beyond an expected ceiling across N requests.
- Avoid global accumulator lists or logging calls that capture raw GPU tensors instead of detached, CPU-side summaries.
- Treat memory_reserved() as an operational metric for capacity planning and memory_allocated() as the correctness metric for leak detection; do not conflate the two.
When this becomes an architecture problem
If memory_allocated() itself is provably climbing after code review rules out obvious reference leaks, and the growth correlates with a specific library (a custom KV cache implementation, a third-party inference wrapper) rather than your own code, this becomes a library-level investigation or a case for isolating that component in its own restart-able process rather than a quick fix.
Frequently asked questions
Is it normal for nvidia-smi to show high memory usage even when no requests are running?
Yes. PyTorch's caching allocator keeps freed memory reserved for fast reuse rather than returning it to the CUDA driver after every tensor is freed. This shows up in nvidia-smi as memory.used for your process even during idle periods, and is expected behavior, not a bug or leak.
What is the difference between torch.cuda.memory_allocated() and memory_reserved()?
memory_allocated() is the memory currently occupied by live tensors your code is actually using. memory_reserved() is the total memory PyTorch's allocator has claimed from the CUDA driver, including cached blocks kept around for future allocations. nvidia-smi's per-process figure corresponds to memory_reserved(), not memory_allocated().
Does torch.cuda.empty_cache() hurt performance if I call it after every request?
Yes, calling it frequently defeats the purpose of the caching allocator and forces expensive driver-level memory operations on the next allocation, typically slowing throughput. Only call it when you specifically need to free memory for another process sharing the GPU, not as a routine part of a request handler.
Why does restarting the process fix memory issues that empty_cache() does not?
A process restart tears down the entire CUDA context, unconditionally releasing every allocation including ones the caching allocator or a stuck reference would otherwise keep alive. This is why process-level isolation (one model per process, restartable independently) is a more reliable operational pattern than trying to fully reset state within a single long-lived process.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
GPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolLLM Serving Capacity Planner
Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.
Related problems
PyTorch GPU memory fragmentation causing intermittent OOM
PyTorch explicitly detects and reports fragmentation in this error, pointing you at PYTORCH_CUDA_ALLOC_CONF for a reason: the caching allocator's memory is split into segments sized for past allocations, and a new allocation that does not match any free segment's size fails even with adequate total free memory. Setting expandable_segments:True and normalizing input shapes are the two highest-leverage fixes.
CUDA out of memory even though nvidia-smi shows free VRAM
This almost always means memory fragmentation: the allocator has enough total free memory but no single contiguous block large enough for the requested allocation, because the address space is broken into many small free-and-used segments from prior allocations of different sizes. The fix is enabling PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, reducing allocation size variability, or restarting the process to reset the address space.
GPU or host memory usage keeps growing in a long-running LLM service
Slow memory growth over hours or days in an LLM service is rarely a leak in the model itself, it is almost always one of: a KV cache pool that grows because completed sequences are not being freed correctly, client sessions or connections that are opened but never closed, LoRA adapters that accumulate in memory across many fine-tuned variants without eviction, or memory fragmentation that reduces effectively usable memory even though nothing is technically leaked. Isolating which of these it is requires tracking memory over time correlated with request volume, adapter count, and connection count separately.
GuideLLM Observability: TTFT, ITL, Throughput, and GPU Dashboards
LLM inference observability: track TTFT, inter-token latency, throughput, and GPU utilization with dashboards that catch problems before users report them.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.