Why accelerate distributed launches fail, and how its config actually needs to match your hardware
ValueError: You can't train a model that has been loaded with `device_map='auto'` in any distributed mode
Also appears as
- ValueError: DistributedType.MULTI_GPU does not match the number of processes launched
- RuntimeError: Please run `accelerate config` to set up your environment before launching training
Short answer
Accelerate errors during launch almost always mean the saved configuration does not match the current machine's actual GPU count, node count, or distributed type, or that the model was loaded with device_map='auto' inference-style sharding and then also handed to accelerate's training-mode preparation, which are two incompatible placement strategies. Regenerating the config for the current machine, or passing explicit CLI overrides, resolves most cases.
Affects: HuggingFace accelerate 0.20 and later, used standalone or underneath Trainer, DeepSpeed, or PEFT, on single-node multi-GPU and multi-node clusters
Regenerate or explicitly override the accelerate config for this machine
- 1Run accelerate env to print what accelerate currently believes about your environment and compare it against the actual GPU count and node setup.
- 2If the config is stale, rerun accelerate config interactively, or delete the cached default_config.yaml and let it regenerate.
- 3For quick fixes without regenerating the whole config, pass explicit overrides directly on the CLI: accelerate launch --num_processes=N --num_machines=M --multi_gpu train.py.
- 4If you are loading the model with device_map='auto' for inference-style sharding, do not also call accelerator.prepare() on that model for training; pick one placement strategy.
- 5For multi-node runs, regenerate machine_rank and main_process_ip per node rather than copying one node's config file to the others unchanged.
How to confirm this is your problem
- Job launches but immediately errors about distributed type, device_map, or process count mismatch before any training step runs
- The same code and config worked on a different machine but fails here after a change in GPU count or node topology
- Mixed precision behavior (fp16 or bf16) does not match what you expect, or a scaler or dtype error appears
- Copying a working accelerate config to a new node causes that node to hang or error at launch
Root causes and fixes
The saved accelerate config does not match the current machine's actual GPU count, node count, or distributed type
accelerate config writes a YAML file capturing the exact hardware topology you answered questions about at the time: number of GPUs, number of machines, and distributed backend. If that config is later reused on a machine with a different GPU count, or after moving from a single-GPU dev box to a multi-GPU or multi-node cluster, accelerate launches with the wrong topology and errors or misbehaves immediately.
Fix: Rerun accelerate config on the actual machine you are launching from, or bypass the cached config entirely by passing explicit flags like --num_processes and --num_machines directly to accelerate launch.
accelerate env accelerate config
A model loaded with device_map='auto' is then also passed to accelerator.prepare() for distributed training preparation
device_map='auto' is designed for splitting a model across devices for inference or generation, computing its own device assignment based on available memory. Accelerator.prepare() assumes it is responsible for placement (via DDP, FSDP, or DeepSpeed) and wrapping the model accordingly. Combining both means two different systems both believe they control device placement, which raises an explicit error to prevent silent corruption of the model's state.
Fix: Choose one strategy: use device_map='auto' for inference-only workflows without wrapping in accelerator.prepare(), or load the model normally and let accelerator.prepare() handle placement entirely for training.
Mixed precision setting in the accelerate config does not match what the training script or DeepSpeed config expects
accelerate's mixed_precision setting controls whether it wraps the optimizer step with a gradient scaler and how it casts activations. If your training script or an underlying DeepSpeed config independently sets a different precision, the two layers can disagree about whether a loss-scaling step is needed, producing dtype errors or silently degraded numerical behavior.
Fix: Set mixed precision in exactly one place, preferably the accelerate config, and verify with accelerate env that the effective setting matches what your training script assumes.
An accelerate config generated on a single-GPU machine was copied unchanged to a multi-node cluster without updating machine_rank or main_process_ip
Multi-node accelerate configs need a unique machine_rank per node and a shared main_process_ip that every node can reach, values that are meaningless or wrong when copied verbatim from a single-machine config. Every node ending up with the same machine_rank causes a rendezvous conflict instead of a clean multi-node launch.
Fix: Generate the config per node, or template it from a single source with the correct machine_rank and IP substituted for each node, rather than distributing one static file to the whole cluster.
A stale accelerate cache uses deprecated keys from an older accelerate version that the current version does not recognize
accelerate has occasionally renamed or removed config keys across major versions; a config file written by an old version and never regenerated can silently be missing required fields or contain fields the current version ignores, producing confusing startup errors unrelated to your actual hardware.
Fix: After upgrading accelerate, delete and regenerate default_config.yaml rather than assuming an old config remains compatible.
Diagnostic commands
Print accelerate's current understanding of the environment
accelerate env
Compare the reported distributed_type, num_processes, and num_machines against what you actually intend to launch with; any mismatch here is the direct cause of most launch failures.
Inspect the raw saved config file
cat ~/.cache/huggingface/accelerate/default_config.yaml
Check machine_rank, main_process_ip, num_machines, and num_processes values directly, especially on multi-node setups where a copied config is a common mistake.
Test the launch configuration with accelerate's built-in check
accelerate test
Runs a minimal distributed smoke test using the current config; a failure here isolates the problem to accelerate's setup itself, before you spend time debugging your actual training script.
Stopping it from happening again
- Regenerate or explicitly template the accelerate config as part of your deployment pipeline for every new machine, rather than reusing a config across different hardware.
- Keep mixed precision, batch size, and distributed type settings in a single source of truth and document which one owns them.
- Run accelerate test as a pre-flight check in CI or on new cluster nodes before launching a real training job.
- Version-control your accelerate config templates per environment instead of relying on the interactive prompt each time.
When this becomes an architecture problem
If your team is regularly hand-editing accelerate configs per machine and still hitting mismatches, standardize on a templated, version-controlled config generation step in your deployment tooling, or move fully to a DeepSpeed or Kubernetes-native launch mechanism, since manual per-node config management does not scale reliably past a handful of machines.
Frequently asked questions
Do I need to run accelerate config on every machine separately?
You need a config whose values, such as num_processes, num_machines, machine_rank, and main_process_ip, are correct for each machine's role in the job. You can generate it interactively per machine, or template one config and substitute the per-node values programmatically. Copying one machine's finished config unchanged to another with different hardware or role is what typically breaks.
Can I skip accelerate config entirely and just use CLI flags?
Yes. Passing --num_processes, --num_machines, --multi_gpu, and similar flags directly to accelerate launch overrides the cached config for that run, which is often simpler and less error-prone than maintaining config files, especially for straightforward single-node multi-GPU setups.
Why does device_map='auto' conflict with accelerate training specifically?
device_map='auto' is accelerate's own inference-oriented sharding utility, and it computes a device placement it expects to remain unchanged. Accelerator.prepare(), used for training, applies its own placement and wrapping logic. Using both means two parts of the same library disagree about who owns device placement, which accelerate detects and blocks rather than risk incorrect results.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Fine-Tuning GPU-Hours Estimator
Estimate GPU-hours for LoRA, QLoRA, and full fine-tuning on the same model size and dataset, so you can compare method tradeoffs before choosing.
Free ToolMulti-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
Free ToolLoRA Fine-Tuning Cost Calculator
Turn model size, dataset tokens, epochs, and rank into a GPU-hour and dollar estimate for a LoRA fine-tuning run on rented or owned hardware.
Related problems
Expected all tensors to be on the same device error in multi-GPU code
This error means a tensor was created or moved to a specific device, often cuda:0 or cpu, that does not match the device another tensor in the same operation lives on, which happens most often when leftover manual .to(device) calls from single-GPU code collide with automatic sharding from device_map='auto' or DistributedDataParallel. The fix is to trace exactly which tensor has the wrong device with a quick print of tensor.device, not to guess and add more .to() calls.
DeepSpeed ZeRO configuration errors at training startup
DeepSpeed refuses to start when its config's batch-size fields do not agree with each other, since train_batch_size must equal train_micro_batch_size_per_gpu times gradient_accumulation_steps times world_size exactly. A second common failure comes from enabling ZeRO stage 3 with CPU or NVMe offload on hardware that lacks enough system RAM or fast enough storage, which surfaces as tensor or contiguity errors rather than a clear resource message.
Multi-node training hangs with no error after rendezvous
A multi-node job that hangs with no error almost always means not every rank actually joined the process group: a mismatched world size, a wrong MASTER_ADDR or MASTER_PORT, a firewall blocking the ephemeral ports NCCL negotiates after rendezvous, or a node that silently OOMed are the four most common causes. Because NCCL blocks silently while waiting for missing ranks, there is often no error at all until you manually intervene or hit a long default timeout.
NCCL error during multi-GPU training or inference
An NCCL error during multi-GPU training or inference is almost always a symptom of a rank that crashed, a version mismatch across processes, or bad GPU topology, not a bug in NCCL itself. Enable NCCL_DEBUG=INFO first and read the per-rank logs before touching timeouts or retry logic.
GuideFine-Tuning Failure Modes: What Actually Goes Wrong
Fine-tuning failure modes that actually derail enterprise projects: catastrophic forgetting, eval overfitting, data leakage, and how to catch each one.
GuideMulti-Node LLM Training Infrastructure: Networking and Storage
Multi-node LLM training infrastructure explained: InfiniBand vs RoCE tradeoffs, storage throughput needs, and cluster topology for enterprise fine-tuning.
GuideLoRA vs QLoRA: Choosing the Right Fine-Tuning Method
LoRA vs QLoRA for enterprise fine-tuning: rank and alpha choices, real VRAM math by model size, and when each method actually wins.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.