Why CUDA runs out of memory during fine-tuning even when inference works fine
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.24 GiB. GPU 0 has a total capacity of 79.15 GiB of which 623.12 MiB is free
Also appears as
- RuntimeError: CUDA error: out of memory during backward pass
- OutOfMemoryError: CUDA out of memory when calling optimizer.step()
Short answer
Training needs far more memory than inference for the same model because it must hold weights, gradients, optimizer states, and activations simultaneously. AdamW alone adds about 8 bytes per parameter for its two fp32 moment buffers, so full fine-tuning of a 7B model can need 60-70+ GB versus about 14 GB for inference of the same weights. The fix is LoRA/QLoRA to shrink trainable parameters, gradient checkpointing to shrink activation memory, or a paged/8-bit optimizer to shrink optimizer state.
Affects: Full fine-tuning and LoRA/QLoRA training with HuggingFace Trainer, TRL, or Axolotl on any CUDA GPU, most common with full-parameter fine-tuning of 7B+ models on a single 40-80 GB GPU
Cut training memory without changing your dataset or model
- 1Switch from full fine-tuning to LoRA or QLoRA so only adapter parameters (often under 1 percent of the model) carry gradients and optimizer state.
- 2Enable gradient checkpointing (gradient_checkpointing=True in TrainingArguments) to trade compute time for a large activation memory reduction.
- 3Use a memory-efficient optimizer: bitsandbytes paged_adamw_8bit instead of the default AdamW, which cuts optimizer state from 8 bytes/param to about 2 bytes/param.
- 4Reduce per_device_train_batch_size and raise gradient_accumulation_steps to keep the same effective batch size at a fraction of the activation memory.
- 5If still short, add DeepSpeed ZeRO Stage 2 or 3 to shard optimizer state and gradients across GPUs instead of replicating them on each device.
How to confirm this is your problem
- Model loads fine and the first forward pass succeeds, but OOM hits on the backward pass or optimizer.step()
- OOM appears a few steps or a few epochs into training, not immediately, as memory creeps up
- Reducing batch size to 1 still fails, pointing at optimizer state or weight memory rather than activations
- Works with LoRA but fails with full fine-tuning on the identical hardware and dataset
Root causes and fixes
Full fine-tuning with AdamW keeps four copies of the trainable parameters in memory at once
Mixed-precision full fine-tuning needs fp16/bf16 weights (2 bytes), an fp32 master copy (4 bytes), and AdamW's two fp32 moment buffers, m and v (4 bytes each, 8 bytes total). That is roughly 14-18 bytes per trainable parameter before activations, versus 2 bytes for pure inference, which is why the identical GPU that serves a model comfortably cannot fine-tune it in full precision.
Fix: Switch to LoRA or QLoRA so the frozen base weights need no optimizer state at all, and only the small adapter matrices carry the 8 bytes/param AdamW overhead.
pip install peft from peft import LoraConfig, get_peft_model model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"]))
Activation memory grows with batch size, sequence length, and number of layers, and is not checkpointed
Every layer's forward-pass activations must be kept in memory until the backward pass consumes them, and that memory scales roughly linearly with batch_size x sequence_length x hidden_size x num_layers. Long context fine-tuning or large batches multiply this quickly, and without checkpointing it often dwarfs the weight and optimizer memory combined.
Fix: Enable gradient checkpointing so only a subset of activations are kept and the rest are recomputed during backward, typically cutting activation memory by 60-80 percent at the cost of extra compute.
training_args = TrainingArguments(gradient_checkpointing=True, per_device_train_batch_size=1, gradient_accumulation_steps=16)
Default AdamW keeps optimizer states in fp32 even under mixed-precision training
Mixed precision only affects the forward/backward compute dtype; by default the optimizer's internal moment buffers remain fp32 for numerical stability, which is where the 8 bytes/param figure comes from. This is invisible in the training script but shows up directly in memory profiles.
Fix: Use bitsandbytes 8-bit or paged AdamW variants, which quantize optimizer states to roughly 2 bytes/param with minimal convergence impact for most fine-tuning runs.
from transformers import TrainingArguments TrainingArguments(optim="paged_adamw_8bit")
Evaluation runs during training accumulate memory because eval batches are not detached from the autograd graph
If an eval loop calls the model without wrapping it in torch.no_grad() or model.eval() context correctly, PyTorch keeps building the autograd graph for eval batches too, silently adding to memory that never gets freed until the run ends or crashes.
Fix: Wrap all evaluation and generation calls in torch.no_grad() and confirm model.eval() is set, so no gradient graph is retained for inference-only forward passes.
with torch.no_grad():
outputs = model(**eval_batch)Dataset examples with unusually long sequences spike activation memory on specific batches
If sequence lengths are not capped or sorted, a rare very-long example in an otherwise short-sequence dataset can produce a batch whose activation memory is many times the typical batch, causing an OOM that appears random and step-dependent rather than consistent.
Fix: Set a hard max_seq_length truncation and consider length-based bucketing/sorting so batches contain similarly sized sequences instead of one outlier forcing worst-case memory on every step.
tokenizer(text, truncation=True, max_length=2048)
Diagnostic commands
Profile memory by training phase
python -c "import torch; print(torch.cuda.memory_allocated()/1e9, torch.cuda.memory_reserved()/1e9)" # call after forward, after backward, after optimizer.step()
A large jump specifically at optimizer.step() confirms optimizer state is the bottleneck; a jump during backward with a huge batch points to activations; a jump during model load points to base weights.
Check trainable parameter count versus total
python -c "print(sum(p.numel() for p in model.parameters() if p.requires_grad), sum(p.numel() for p in model.parameters()))"
If trainable count equals total count, you are doing full fine-tuning and paying the full 8 bytes/param optimizer cost; if it is a small fraction, LoRA is already active and the OOM is more likely activation-driven.
Watch memory over the first few training steps
nvidia-smi --query-gpu=memory.used --format=csv -l 1
Memory that climbs steadily rather than plateauing after step 2-3 suggests a leak (often ungathered eval graphs or growing logging buffers) rather than a fixed-size configuration problem.
Stopping it from happening again
- Default new fine-tuning projects to LoRA/QLoRA and only justify full fine-tuning when adapter-based training demonstrably underperforms on your task.
- Always enable gradient checkpointing for anything above a 7B base model as a standing default, not an emergency fix.
- Calculate expected memory (weights plus optimizer plus activations) before a training run using your batch size and sequence length, not after it crashes.
- Cap max_seq_length explicitly in the tokenizer rather than relying on the dataset to be well-behaved.
When this becomes an architecture problem
When even LoRA plus gradient checkpointing plus 8-bit optimizer still cannot fit your target batch size and sequence length on the largest single GPU you have, that is a signal to move to multi-GPU training with DeepSpeed ZeRO or FSDP, which is an infrastructure decision worth planning rather than patching around.
Frequently asked questions
Why does full fine-tuning need so much more memory than inference?
Inference only needs the weights resident, about 2 bytes per parameter in bf16. Training additionally needs gradients (another 2 bytes/param), an fp32 master weight copy (4 bytes/param), and AdamW's two fp32 moment buffers (8 bytes/param combined), plus activation memory that scales with batch size and sequence length. The combined multiplier is commonly 12-20x the inference footprint.
Does LoRA really avoid all that overhead?
Yes, because LoRA freezes the base model weights entirely; only the small low-rank adapter matrices, typically well under 1 percent of total parameters, require gradients and optimizer state. The base weights still need to be resident in memory (or in 4-bit with QLoRA), but the optimizer state that dominates full fine-tuning memory shrinks by roughly two orders of magnitude.
Is gradient checkpointing safe to leave on permanently?
Yes for most workloads. It recomputes activations during the backward pass instead of storing them, typically adding 20-30 percent to wall-clock training time in exchange for a large activation memory reduction. Leave it on by default for any model where memory is a binding constraint, and only disable it if you have confirmed spare VRAM and want the throughput back.
Can I fine-tune a 70B model on a single 80 GB GPU?
Not with full-parameter fine-tuning; the optimizer state and gradients alone exceed 80 GB before activations. QLoRA (4-bit base weights plus LoRA adapters) makes this feasible on a single 80 GB GPU for moderate batch sizes and sequence lengths, which is exactly the technique that made single-GPU fine-tuning of large models practical.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
LoRA Fine-Tuning Cost Calculator
Turn model size, dataset tokens, epochs, and rank into a GPU-hour and dollar estimate for a LoRA fine-tuning run on rented or owned hardware.
Free ToolQLoRA vs Full Fine-Tuning Cost Calculator
See the GPU memory footprint, GPU-hour requirement, and dollar cost gap between QLoRA and full fine-tuning for the same model size and dataset.
Free ToolFine-Tuning GPU-Hours Estimator
Estimate GPU-hours for LoRA, QLoRA, and full fine-tuning on the same model size and dataset, so you can compare method tradeoffs before choosing.
Related problems
Out of memory when merging a LoRA adapter into the base model
Merging a LoRA adapter mathematically requires the base weights in a real-valued (not 4-bit quantized) format so the adapter delta can be added in, which means a QLoRA workflow that trained happily in 4-bit suddenly needs a full fp16/bf16 copy of the base model just to merge, often doubling or more the memory footprint versus either training or inference alone. The fix is to merge on CPU, merge in a lower-footprint dtype, or skip merging entirely by serving the adapter unmerged.
Gradient checkpointing errors during fine-tuning
Gradient checkpointing errors during fine-tuning almost always come from three sources: the use_reentrant parameter left unset (it now must be explicit and False is usually correct for transformer models), an attention implementation that isn't fully compatible with checkpointing's re-computation approach, or leaving use_cache=True enabled while checkpointing is on, which conflicts because checkpointing recomputes the forward pass and a live KV cache assumes it won't be recomputed. Set use_reentrant=False and use_cache=False together.
Loss becomes NaN during fine-tuning
NaN loss during training is most often caused by fp16 numeric overflow in gradients or activations, which bf16 avoids because of its wider exponent range. Other common causes are a learning rate spike (especially right after warmup), a small number of corrupt or malformed training samples, and unsafe division or log operations in a custom loss function. Switch to bf16 first if your hardware supports it, then check for corrupt samples and unstable LR.
CUDA out of memory when loading an LLM
This happens because model weights alone require roughly 2 bytes per parameter in fp16/bf16 (a 70B model needs about 140 GB before you even run inference), and that number does not fit your GPU. The fix is to either quantize the weights (AWQ, GPTQ, FP8, or GGUF), split the model across multiple GPUs with tensor parallelism, or pick a GPU with enough VRAM for the parameter count you are loading.
GuideLoRA vs QLoRA: Choosing the Right Fine-Tuning Method
LoRA vs QLoRA for enterprise fine-tuning: rank and alpha choices, real VRAM math by model size, and when each method actually wins.
GuideFine-Tuning LLMs On-Prem with Enterprise Data
Fine-tune LLMs on-prem with enterprise data: LoRA vs full fine-tuning, dataset prep, GPU requirements, eval, and when RAG beats tuning altogether.
GuideFine-Tuning Failure Modes: What Actually Goes Wrong
Fine-tuning failure modes that actually derail enterprise projects: catastrophic forgetting, eval overfitting, data leakage, and how to catch each one.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.