Why GPU peer-to-peer access does not work between GPUs, and why disabling it is a diagnostic step, not a fix
RuntimeError: NCCL WARN Cuda failure 'peer access is not supported between these two devices'
Also appears as
- cudaErrorPeerAccessNotEnabled: peer access is not supported between these two devices
- Training or serving is dramatically slower with 4 or 8 GPUs than expected, only when P2P is disabled
Short answer
GPU peer-to-peer (P2P) access fails when the PCIe topology, IOMMU or ACS settings, or the GPU model itself does not support a direct memory path between two devices, forcing all transfers through the CPU and host memory instead of GPU-to-GPU. Setting NCCL_P2P_DISABLE=1 is a useful diagnostic to confirm P2P is the problem, but it only removes the crash by falling back to a slower path; it does not restore the P2P bandwidth you actually need for good multi-GPU performance.
Affects: Any multi-GPU server without NVLink, most common on workstation motherboards, servers with IOMMU or VFIO enabled, and consumer GPUs like the RTX 4090
Diagnose the topology before deciding P2P cannot be fixed
- 1Run nvidia-smi topo -m to see the actual P2P and NVLink matrix between every GPU pair in the system.
- 2If P2P shows as unsupported and IOMMU is enabled in BIOS or the kernel command line, try disabling IOMMU (or setting iommu=pt) and re-check the topology matrix.
- 3As a diagnostic only, set NCCL_P2P_DISABLE=1 and confirm the error goes away; this tells you P2P is the cause but is not the fix, since it forces slower host-routed transfers.
- 4Check for PCIe ACS redirection with lspci, since some server BIOSes enable it by default and it silently defeats P2P even on capable hardware.
- 5If the GPUs are consumer cards outside an NVLink-capable pairing, treat the lack of P2P as expected behavior and plan tensor or data parallelism around slower interconnect rather than expecting a fix.
How to confirm this is your problem
- Multi-GPU training or serving is dramatically slower than expected relative to single-GPU throughput times GPU count
- nvidia-smi topo -m shows PHB or SOC rather than PIX or NV between GPUs you expect to communicate directly
- NCCL logs mention falling back to sockets or failing to open a CUDA IPC handle
- The problem appears on one specific server or motherboard but not another with the same GPU model
Root causes and fixes
GPUs are installed in PCIe slots without a direct P2P-capable path, such as behind different CPU sockets or on a consumer board without a PCIe switch
P2P DMA requires a topology where the PCIe root complex or a shared switch can route memory transfers directly between two GPUs' address spaces. On dual-socket servers, GPUs attached to different CPU sockets typically cannot P2P at all without a specific interconnect; on consumer boards, GPUs may share a root complex but lack a switch that supports P2P forwarding.
Fix: Check nvidia-smi topo -m and, where possible, place GPUs that must communicate frequently in slots under the same PCIe switch or CPU socket; for a from-scratch build, choose a server board and GPU count specifically validated for P2P or NVLink.
nvidia-smi topo -m lspci -tv
IOMMU is enabled in BIOS or the kernel, which blocks the DMA remapping that P2P transfers rely on
IOMMU (VT-d on Intel, AMD-Vi on AMD) is often enabled by default on server boards for virtualization or security isolation. It intercepts and remaps device DMA, and unless explicitly configured to allow P2P DMA between devices, it silently forces all transfers through the CPU, disabling the fast path even though the PCIe topology would otherwise support it.
Fix: If virtualization or passthrough is not required, disable IOMMU in BIOS or set iommu=off (or iommu=pt for a middle ground) in the kernel boot parameters, then reboot and re-check the topology matrix.
cat /proc/cmdline | grep -i iommu
The GPU model itself does not support P2P outside of NVLink, which is common for consumer cards
NVIDIA restricts full PCIe P2P support on several consumer GPU lines regardless of topology, reserving guaranteed high-bandwidth peer access for NVLink-connected or datacenter-class GPUs. On these cards, even a perfect PCIe topology with IOMMU disabled will not enable P2P.
Fix: Accept that consumer GPUs will route through the host for cross-GPU transfers and plan accordingly (smaller tensor-parallel groups, more reliance on data parallelism), or move to datacenter GPUs (A100, H100, or NVLink-equipped cards) if P2P bandwidth is required.
PCIe ACS (Access Control Services) is enabled on the root complex or a switch, forcing peer traffic through the CPU for isolation
ACS is a PCIe feature that isolates devices from each other for security and IOMMU groupings, commonly enabled by default in server BIOS or by virtualization-focused kernel defaults. It intercepts P2P transactions and routes them back through the root complex, defeating the purpose of P2P without producing an obvious error, just reduced bandwidth or an outright block.
Fix: Check ACS status with lspci -vvv and disable ACS in BIOS if security requirements allow, or use a documented ACS override kernel parameter if your distribution supports it.
lspci -vvv | grep -A 5 'Access Control Services'
The system is a virtualized or GPU-passthrough (VFIO) environment where P2P is disabled by design for isolation between virtual machines
VFIO-based GPU passthrough intentionally isolates each GPU into its own IOMMU group to maintain VM security boundaries, which is fundamentally incompatible with the shared-memory assumptions P2P relies on. This is a deliberate tradeoff of the virtualization layer, not a misconfiguration.
Fix: If P2P performance is required, run training directly on bare metal rather than through VFIO passthrough VMs, or accept the P2P-disabled performance profile as the cost of the virtualization architecture.
Diagnostic commands
Check the P2P and NVLink topology matrix
nvidia-smi topo -m
Look for PIX (same PCIe switch, good) or NV (NVLink, best) versus PHB or SOC (different root complex or socket, P2P unlikely or unavailable) between the GPU pairs you care about.
Check whether IOMMU is enabled
cat /proc/cmdline | grep -i iommu
Presence of intel_iommu=on or amd_iommu=on without a corresponding passthrough allowance suggests IOMMU is actively remapping DMA and may be blocking P2P.
Check PCIe ACS settings
lspci -vvv | grep -B 5 -A 10 'Access Control Services'
An ACSCtl line showing SrcValid or RequestRedir enabled on the relevant bridge indicates ACS is actively redirecting peer traffic through the root complex.
Confirm the impact of disabling P2P as a controlled test
NCCL_P2P_DISABLE=1 python your_multi_gpu_script.py
If the error disappears and throughput drops noticeably compared to expected NVLink or P2P bandwidth, this confirms P2P was both broken and needed; it is not itself a solution, just proof of the diagnosis.
Stopping it from happening again
- When procuring multi-GPU servers, choose boards and chassis explicitly validated for P2P or NVLink at your intended GPU count, not just a generic multi-GPU claim.
- Standardize BIOS settings (IOMMU, ACS) across your fleet and document them, so a new node does not silently regress on P2P support.
- Benchmark P2P bandwidth as part of new node acceptance testing, not after a production job underperforms.
- For workloads that truly need P2P bandwidth at scale, budget for NVLink or NVSwitch-equipped hardware rather than relying on PCIe topology alone.
When this becomes an architecture problem
If your current server topology or GPU model fundamentally cannot support P2P, consumer GPUs, cross-socket placement, or a board without a P2P-capable switch, and the workload genuinely needs that bandwidth, this is a hardware procurement decision, not something BIOS or driver settings can fix.
Frequently asked questions
Does NCCL_P2P_DISABLE=1 fix the peer access error?
It stops the crash by forcing NCCL to route through the host instead of attempting direct GPU-to-GPU transfers, so in that narrow sense it makes the job run. But it does not restore P2P bandwidth, so multi-GPU throughput will be noticeably lower; treat it as a way to confirm the diagnosis and keep a job running, not as the actual fix.
Why does nvidia-smi topo -m show PHB instead of PIX for my GPUs?
PHB means the GPUs are connected through a PCIe host bridge rather than a shared switch, which usually means they sit on different root complexes, often tied to different CPU sockets on a dual-socket server. That topology typically does not support fast P2P, regardless of driver or IOMMU settings.
Do consumer GPUs like the RTX 4090 support P2P at all?
NVIDIA restricts full P2P support on most current consumer GPU lines outside of specific NVLink-bridged configurations, so you should generally expect P2P to be unavailable or limited on these cards even with ideal PCIe topology and IOMMU disabled. Datacenter GPUs such as the A100 and H100 support P2P and NVLink much more broadly.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
NVIDIA GPU Selector for LLM Workloads
Score your workload across model size, concurrency, latency, budget, and facility power to get a recommended GPU tier from RTX-class to multi-node B200 clusters.
Free ToolGPU Cluster Buildout Cost Calculator
Turn GPU count and class into a full cluster budget covering server chassis, networking fabric, power and cooling capex, and install, with a true cost per GPU.
Free ToolAI Server Rack Power Budget Calculator
Convert GPUs per rack, TDP, host overhead, and facility PUE into a real rack power budget with redundancy, then check it against your available circuit capacity.
Related problems
NCCL error during multi-GPU training or inference
An NCCL error during multi-GPU training or inference is almost always a symptom of a rank that crashed, a version mismatch across processes, or bad GPU topology, not a bug in NCCL itself. Enable NCCL_DEBUG=INFO first and read the per-rank logs before touching timeouts or retry logic.
InfiniBand not detected, NCCL falls back to slow TCP sockets
NCCL falls back to slow TCP sockets when it cannot find a usable InfiniBand device, most often because the IB kernel modules or rdma-core drivers are not installed or loaded, the fabric's subnet manager is not running so ports stay down, or NCCL environment variables point at the wrong network interface. Checking ibstat to confirm the hardware and fabric are actually up is the first step, before touching any NCCL environment variables.
Tensor parallelism fails because the model does not split evenly across GPUs
Tensor parallelism fails when the chosen degree does not evenly divide the model's attention heads, and often its key/value heads and hidden size, so the framework cannot split the projection weights across ranks. It also fails in practice, without an assertion, when the GPUs assigned to the TP group differ in VRAM or compute, since the even weight split then fits some ranks and not others.
NCCL collective operation timeout during distributed training
An NCCL timeout means one or more ranks did not reach a collective operation (all-reduce, broadcast, all-gather) within the configured window, almost always because a straggler rank is slow or stuck, not because NCCL is malfunctioning. Raising NCCL_TIMEOUT can mask the symptom, but the durable fix is finding and removing the straggler: a data loading stall, an OOM-crashed rank, or a checkpoint write blocking one process.
GuideOn-Prem GPU Cluster Design: Node Sizing, Networking, and Storage
Design an on-prem GPU cluster: node sizing for H100/H200/B200, InfiniBand vs RoCE networking, storage throughput, and rack power for enterprise AI workloads.
GuideNVIDIA H100 vs H200 vs B200 for Enterprise AI in 2026
Compare NVIDIA H100, H200, and B200 GPUs on specs, price, availability, and performance per dollar for enterprise LLM inference and training in 2026.
GuideMulti-GPU LLM Serving: Tensor vs Pipeline Parallelism
Multi-GPU LLM serving explained: tensor parallelism vs pipeline parallelism, NCCL interconnect requirements, and when to split a model across GPUs.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.