Why your training container crashes with a shared memory error, and how to fix it
RuntimeError: DataLoader worker (pid 1234) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory
Also appears as
- OSError: [Errno 28] No space left on device (writing to /dev/shm)
- torch.multiprocessing.shared_memory ... Bus error
Short answer
PyTorch DataLoader workers crash with a bus error inside Docker or Kubernetes because the default container shared memory allocation is only 64MB, far too small for multi-worker data loading. Set --shm-size explicitly in Docker, or add a Memory-medium emptyDir volume at /dev/shm in Kubernetes, sized to your batch size and worker count, and the crash disappears without touching your training code.
Affects: PyTorch DataLoader with num_workers greater than 0 running inside Docker or Kubernetes, most common during fine-tuning and large batch training
Fix it in 60 seconds
- 1Confirm the cause: set num_workers=0 temporarily; if the crash disappears, shared memory is confirmed as the root cause.
- 2In Docker, rerun with --shm-size=8g added to your existing docker run command.
- 3In Kubernetes, add an emptyDir volume with medium Memory and a sizeLimit mounted at /dev/shm in the pod spec.
- 4Restore your original num_workers setting and rerun training.
- 5If it still crashes, increase shm-size further or check df -h /dev/shm during the run to see actual usage.
How to confirm this is your problem
- DataLoader worker killed by signal Bus error
- training crashes only when num_workers is greater than 0
- same code works with num_workers=0 but crashes with num_workers=4 or higher
- OSError no space left on device referencing /dev/shm
Root causes and fixes
Docker's default 64MB /dev/shm is too small for multi-worker dataloaders
Docker allocates a default shared memory size of 64MB for /dev/shm inside a container, which PyTorch's DataLoader workers use to pass tensors between worker processes and the main process; with several workers and reasonably sized batches, this default fills up almost instantly, and the kernel kills the worker process with a bus error rather than letting it silently corrupt memory.
Fix: Increase the container's shared memory allocation explicitly at run time, since the 64MB default is a Docker safety default, not a hard system limit.
docker run --shm-size=8g --gpus all myimage docker run --ipc=host --gpus all myimage
Kubernetes emptyDir /dev/shm is not backed by memory medium, or has no size limit set
Kubernetes pods get their own tiny default /dev/shm similar to Docker unless you explicitly mount an emptyDir volume at /dev/shm with medium Memory and a sizeLimit; without this, the pod inherits the same small shared memory ceiling as a default Docker container, and any dataloader with multiple workers hits the same bus error inside Kubernetes.
Fix: Add an emptyDir volume with medium Memory and an explicit sizeLimit mounted at /dev/shm in the pod spec, sized to comfortably exceed your batch size times worker count.
Too many DataLoader workers relative to batch size and available shared memory
Each DataLoader worker holds its own share of in-flight batches in shared memory simultaneously; setting num_workers very high without a corresponding increase in shm-size multiplies memory pressure linearly, so a shm-size that was fine at 4 workers can still be exhausted at 16 workers even on the same batch size.
Fix: Either raise shm-size further or reduce num_workers, and treat the two as a joint tuning knob rather than adjusting only one.
python -c "import torch; print(torch.utils.data.DataLoader)"
Using pin_memory with very large batches multiplies shared memory pressure further
pin_memory=True causes the DataLoader to allocate page-locked host memory for faster GPU transfer, and combined with shared memory tensors from multiple workers, very large batch or sequence lengths for LLM training data can push total shared memory usage well beyond what a modestly sized shm allocation anticipated.
Fix: When training with very large batches or long sequences, size /dev/shm generously (16GB or more) or reduce pin_memory usage if shared memory remains constrained.
Diagnostic commands
Check the container's current shared memory size
df -h /dev/shm
The default Docker allocation shows as 64M; if your workload needs more than what is shown as available, this confirms the exhaustion.
Reproduce with a minimal worker count
python train.py --num_workers 0
If the crash disappears with zero workers, it confirms shared memory, not a data bug, is the root cause, since num_workers=0 does not use inter-process shared memory.
Check Kubernetes pod's shm mount
kubectl exec <pod> -- df -h /dev/shm
Confirms whether the pod actually has an emptyDir Memory volume mounted at /dev/shm or is stuck with the tiny default.
Watch shared memory usage live during training
watch -n1 df -h /dev/shm
Rising usage that plateaus near the limit right before the crash confirms shm-size, not a code bug, is the bottleneck.
Stopping it from happening again
- Set --shm-size explicitly on every training container, never rely on Docker's 64MB default
- Add an explicit Memory-medium emptyDir for /dev/shm in every Kubernetes training pod spec
- Size shm relative to batch_size times num_workers times sample size, not a fixed guess
- Add a startup check that logs df -h /dev/shm so shm exhaustion is visible in logs before a crash, not just after
When this becomes an architecture problem
If you are running many concurrent training jobs across a shared cluster and each needs a different shm-size tuned to its batch size and worker count, standardizing a training pod template with a generous, monitored shm allocation is worth doing once at the platform level rather than tuning per job.
Frequently asked questions
Why does Docker default to only 64MB for /dev/shm?
It is a conservative default set by Docker to avoid one container consuming excessive host memory through shared memory allocation by default; most simple containerized applications never need more than this. Machine learning workloads that use multiprocessing to parallelize data loading are a common exception, since PyTorch's DataLoader workers pass batches between processes via shared memory, and that usage pattern was not what the small default was designed around.
What's the difference between --shm-size and --ipc=host?
--shm-size=8g sets an explicit, isolated size for the container's own /dev/shm, keeping it contained and predictable. --ipc=host instead shares the host's entire IPC namespace and shared memory with the container, effectively removing any practical size limit but also removing the isolation between the container and host processes. For production training jobs, an explicit --shm-size is usually preferred over --ipc=host for isolation reasons.
How do I set shared memory size for a training pod in Kubernetes?
Kubernetes has no shm-size flag equivalent; instead you mount an emptyDir volume with medium Memory and a sizeLimit at the /dev/shm path in both the pod's volumes and the container's volumeMounts sections. Without this explicit volume, the pod's /dev/shm defaults to the same small size as an unconfigured Docker container, and multi-worker DataLoaders will hit the same bus error.
What size should I set for /dev/shm during LLM fine-tuning?
There is no universal number since it scales with your batch size, sequence length, and num_workers, but a common practical starting point for fine-tuning workloads is somewhere between 4GB and 16GB, then watching df -h /dev/shm during an actual run to see how close usage gets to the limit. It is safer to allocate generously, since unused shared memory does not consume host RAM until it is actually written to.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Fine-Tuning GPU-Hours Estimator
Estimate GPU-hours for LoRA, QLoRA, and full fine-tuning on the same model size and dataset, so you can compare method tradeoffs before choosing.
Free ToolLoRA Fine-Tuning Cost Calculator
Turn model size, dataset tokens, epochs, and rank into a GPU-hour and dollar estimate for a LoRA fine-tuning run on rented or owned hardware.
Free ToolKubernetes Cluster Cost Calculator
Estimate the true monthly and annual cost of a Kubernetes cluster, including compute, control plane fees, managed service premiums, and utilization waste.
Related problems
PyTorch GPU memory fragmentation causing intermittent OOM
PyTorch explicitly detects and reports fragmentation in this error, pointing you at PYTORCH_CUDA_ALLOC_CONF for a reason: the caching allocator's memory is split into segments sized for past allocations, and a new allocation that does not match any free segment's size fails even with adequate total free memory. Setting expandable_segments:True and normalizing input shapes are the two highest-leverage fixes.
LLM container image is tens of gigabytes and slow to pull
LLM container images balloon past ten or twenty gigabytes almost always because model weights were copied directly into a layer instead of mounted at runtime, or because a devel CUDA base image and unstaged build tools shipped into production by mistake. Remove weights from the Dockerfile, switch to a runtime base image and a multi-stage build, and image size typically drops by an order of magnitude without any change to the serving code.
Kubernetes PersistentVolumeClaim errors when serving model weights
PersistentVolumeClaim errors serving model weights almost always come from using a ReadWriteOnce volume with more than one replica, since that access mode only allows a single node to mount it at a time. Switch to a ReadOnlyMany-capable storage class, mount weights read-only, and set volumeBindingMode to WaitForFirstConsumer to avoid zone mismatches; if the volume mounts fine but loading is still slow, the real problem is storage throughput, not access mode.
CUDA out of memory even though nvidia-smi shows free VRAM
This almost always means memory fragmentation: the allocator has enough total free memory but no single contiguous block large enough for the requested allocation, because the address space is broken into many small free-and-used segments from prior allocations of different sizes. The fix is enabling PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, reducing allocation size variability, or restarting the process to reset the address space.
GuideFine-Tuning LLMs On-Prem with Enterprise Data
Fine-tune LLMs on-prem with enterprise data: LoRA vs full fine-tuning, dataset prep, GPU requirements, eval, and when RAG beats tuning altogether.
GuideBuilding Fine-Tuning Datasets From Enterprise Data
Build fine-tuning datasets from enterprise data: instruction formats, deduplication methods, PII scrubbing, and quality filtering that actually works.
GuideOn-Prem GPU Cluster Design: Node Sizing, Networking, and Storage
Design an on-prem GPU cluster: node sizing for H100/H200/B200, InfiniBand vs RoCE networking, storage throughput, and rack power for enterprise AI workloads.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.