Why you get an NCCL error on multi-GPU training or inference, and how to actually fix it
RuntimeError: NCCL error: unhandled system error, NCCL version 2.18.1
Also appears as
- NCCL error: invalid usage, condition: ncclInvalidUsage
- NCCL WARN Cuda failure 'invalid device ordinal'
- ncclInternalError: Internal check failed
Short answer
An NCCL error during multi-GPU training or inference is almost always a symptom of a rank that crashed, a version mismatch across processes, or bad GPU topology, not a bug in NCCL itself. Enable NCCL_DEBUG=INFO first and read the per-rank logs before touching timeouts or retry logic.
Affects: PyTorch 2.x with NCCL 2.14 and later, any multi-GPU node or multi-node cluster, most common above 4 GPUs per job
Find the real failure before changing NCCL settings
- 1Re-run the job with NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=ALL set on every rank so you get per-rank transport and error detail instead of a single opaque exception.
- 2Check every rank's log, not just rank 0, for a Python traceback, CUDA error, or OOM message that appears before the NCCL error; that is usually the real root cause.
- 3Confirm every node and container is running the exact same PyTorch, CUDA, and NCCL version with python -c "import torch; print(torch.__version__, torch.version.cuda)" on each host.
- 4Run nvidia-smi on every node during the failure window to rule out a crashed or ECC-faulted GPU.
- 5If ranks are dying from OOM, fix memory pressure (batch size, gradient checkpointing, ZeRO stage) rather than retrying, since the NCCL error will keep recurring.
How to confirm this is your problem
- Training or serving crashes with a generic 'NCCL error' or 'unhandled system error' with no clear Python stack trace
- The failure only happens with more than one GPU or more than one node, never on a single GPU
- Some ranks print the error while others just hang or get killed
- The error appears intermittently, more often under load or after a long-running job
Root causes and fixes
One or more GPU processes crashed or exited before the collective completed
NCCL collectives are synchronous across ranks; when one rank segfaults, OOMs, or is killed, the surviving ranks remain blocked inside an all-reduce or broadcast until a watchdog eventually throws 'unhandled system error' or aborts the whole process group. The NCCL error you see is usually a downstream symptom, not the original fault.
Fix: Check per-rank logs for a Python traceback, CUDA error, or OOM killer message that appears before the NCCL error, since fixing that underlying crash removes the NCCL error entirely.
CUDA and NCCL library version mismatch between the container or host, or across nodes
NCCL is tightly coupled to the CUDA driver and the NCCL build bundled with PyTorch. Mixing a host driver too old for the installed CUDA runtime, or running different NCCL or PyTorch versions on different nodes, produces invalid usage or internal errors when ranks negotiate a transport during initialization.
Fix: Pin identical PyTorch, CUDA, and NCCL versions in one container image and deploy that same image to every node instead of installing packages per node.
python -c "import torch; print(torch.__version__, torch.version.cuda)"
GPU device ordinal mismatch from CUDA_VISIBLE_DEVICES conflicting with local_rank assignment
When CUDA_VISIBLE_DEVICES is set inconsistently across ranks (or an orchestrator sets it differently than your launch script expects), a rank's local_rank can point at a GPU index that does not exist or is already claimed by another process, producing an invalid device ordinal error at NCCL init time.
Fix: Print torch.cuda.device_count() and the resolved CUDA_VISIBLE_DEVICES on every rank at startup, and make sure your launcher sets local_rank consistently with the actual visible device list.
Insufficient shared memory (/dev/shm) inside a Docker container for NCCL's SHM transport
NCCL uses shared memory for intra-node communication between GPUs. Docker's default /dev/shm size (often 64MB) is far smaller than large collectives need, so NCCL can fail or silently perform poorly when it cannot allocate the shared memory segment it wants.
Fix: Launch containers with a larger shared memory allocation, for example docker run --shm-size=8g, or mount a tmpfs volume sized for your largest expected collective.
Faulty GPU, PCIe or NVLink hardware fault, or an ECC error corrupting a collective
A degrading GPU, a marginal PCIe or NVLink connection, or an uncorrected ECC memory error can corrupt data mid-transfer or cause a GPU to drop out of a collective, which NCCL reports as an internal or system error without indicating the physical cause.
Fix: Check dmesg for Xid errors and nvidia-smi for uncorrected ECC error counts on every GPU involved, and reseat or replace any GPU that shows recurring hardware faults.
Diagnostic commands
Rerun with verbose NCCL logging
NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=ALL torchrun --nproc_per_node=8 train.py
Look for the last successful collective op and which rank stops responding; a transport-level warning points to networking, while a rank simply going silent points to a crash on that rank.
Check for OS-level GPU faults
dmesg | grep -i xid
Any Xid error logged around the failure time indicates a GPU hardware or driver fault (a failing GPU, ECC error, or thermal issue), not a bug in your training code.
Confirm identical library versions across nodes
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.nccl.version())"
If this prints different NCCL or CUDA versions on different nodes or containers, that mismatch is a common cause of invalid usage and internal errors during rank negotiation.
Check GPU health and utilization at failure time
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,ecc.errors.uncorrected.volatile.total --format=csv
A GPU showing 0 percent utilization while others are busy, or a nonzero uncorrected ECC error count, points to that specific GPU or its host as the failing rank.
Stopping it from happening again
- Pin exact PyTorch, CUDA, and NCCL versions in your container image and rebuild the same image for every node instead of installing per-node.
- Add a lightweight health check (nvidia-smi, a small all-reduce smoke test) to node provisioning before a training job is scheduled onto it.
- Log per-rank memory usage during training so an OOM shows up as a clear cause instead of an opaque NCCL abort.
- Keep NCCL_DEBUG=INFO enabled (or logged to a file) for production distributed jobs so failures are diagnosable after the fact.
When this becomes an architecture problem
If the same job fails intermittently across different nodes with no consistent rank or GPU, and version pinning and health checks do not resolve it, the issue is likely in your cluster's interconnect or GPU topology and needs an infrastructure-level review rather than another code change.
Frequently asked questions
Does raising NCCL_TIMEOUT fix an NCCL error?
Raising the timeout only helps if the underlying operation genuinely needs more time, such as a very large collective over a slow network. It does not fix an error caused by a crashed process, a version mismatch, or bad topology, and can turn a fast, diagnosable failure into a slow, confusing hang instead.
Why does the same code work on one GPU but fail with NCCL on multiple GPUs?
Single-GPU training never touches NCCL, so any bug in your distributed setup, such as mismatched versions, a rank that crashes only under the extra memory or process load, or unsupported peer-to-peer topology, stays invisible until you add a second GPU or node.
Is NCCL_DEBUG=INFO safe to leave on in production?
Yes. It adds modest log volume but no meaningful performance overhead, and having those logs available is often the difference between diagnosing a production distributed failure in minutes versus hours of blind guessing.
Can a bad network cable or switch cause an NCCL error instead of just a timeout?
Yes. A flaky NIC, cable, or switch port can produce transport-level errors, not just slowness, which surface as NCCL internal or system errors. Checking dmesg and switch-side error counters alongside NCCL logs helps distinguish this from a software bug in your training code.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Multi-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
Free ToolGPU Cluster Buildout Cost Calculator
Turn GPU count and class into a full cluster budget covering server chassis, networking fabric, power and cooling capex, and install, with a true cost per GPU.
Free ToolNVIDIA GPU Selector for LLM Workloads
Score your workload across model size, concurrency, latency, budget, and facility power to get a recommended GPU tier from RTX-class to multi-node B200 clusters.
Related problems
NCCL collective operation timeout during distributed training
An NCCL timeout means one or more ranks did not reach a collective operation (all-reduce, broadcast, all-gather) within the configured window, almost always because a straggler rank is slow or stuck, not because NCCL is malfunctioning. Raising NCCL_TIMEOUT can mask the symptom, but the durable fix is finding and removing the straggler: a data loading stall, an OOM-crashed rank, or a checkpoint write blocking one process.
Multi-node training hangs with no error after rendezvous
A multi-node job that hangs with no error almost always means not every rank actually joined the process group: a mismatched world size, a wrong MASTER_ADDR or MASTER_PORT, a firewall blocking the ephemeral ports NCCL negotiates after rendezvous, or a node that silently OOMed are the four most common causes. Because NCCL blocks silently while waiting for missing ranks, there is often no error at all until you manually intervene or hit a long default timeout.
GPU peer-to-peer (P2P) access not working between GPUs on the same node
GPU peer-to-peer (P2P) access fails when the PCIe topology, IOMMU or ACS settings, or the GPU model itself does not support a direct memory path between two devices, forcing all transfers through the CPU and host memory instead of GPU-to-GPU. Setting NCCL_P2P_DISABLE=1 is a useful diagnostic to confirm P2P is the problem, but it only removes the crash by falling back to a slower path; it does not restore the P2P bandwidth you actually need for good multi-GPU performance.
InfiniBand not detected, NCCL falls back to slow TCP sockets
NCCL falls back to slow TCP sockets when it cannot find a usable InfiniBand device, most often because the IB kernel modules or rdma-core drivers are not installed or loaded, the fabric's subnet manager is not running so ports stay down, or NCCL environment variables point at the wrong network interface. Checking ibstat to confirm the hardware and fabric are actually up is the first step, before touching any NCCL environment variables.
GuideMulti-GPU LLM Serving: Tensor vs Pipeline Parallelism
Multi-GPU LLM serving explained: tensor parallelism vs pipeline parallelism, NCCL interconnect requirements, and when to split a model across GPUs.
GuideMulti-Node LLM Training Infrastructure: Networking and Storage
Multi-node LLM training infrastructure explained: InfiniBand vs RoCE tradeoffs, storage throughput needs, and cluster topology for enterprise fine-tuning.
GuideOn-Prem GPU Cluster Design: Node Sizing, Networking, and Storage
Design an on-prem GPU cluster: node sizing for H100/H200/B200, InfiniBand vs RoCE networking, storage throughput, and rack power for enterprise AI workloads.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.