Why vLLM's tensor-parallel-size must divide attention heads evenly, and how to fix the error
ValueError: Total number of attention heads (32) must be divisible by tensor parallel size (3)
Also appears as
- AssertionError: tensor_parallel_size (X) must be less than or equal to the number of GPUs (Y)
- RuntimeError: CUDA error: invalid device ordinal
Short answer
vLLM shards attention heads and KV heads evenly across GPUs for tensor parallelism, so --tensor-parallel-size must be a divisor of the model's head count and must equal the number of GPUs vLLM can actually see. Pick a TP size from the model's valid divisors, commonly 1, 2, 4, or 8, and make sure CUDA_VISIBLE_DEVICES exposes exactly that many GPUs.
Affects: vLLM 0.3 and later, any multi-GPU host; occurs whenever --tensor-parallel-size doesn't evenly divide the model's num_attention_heads or num_key_value_heads, or doesn't match visible GPU count
Fix it in a few minutes
- 1Find the model's num_attention_heads and num_key_value_heads in its config.json.
- 2Pick a --tensor-parallel-size that evenly divides both, commonly 1, 2, 4, or 8 for standard architectures.
- 3Confirm CUDA_VISIBLE_DEVICES exposes exactly that many physical GPUs, for example 0,1,2,3 for TP=4.
- 4Re-launch with vllm serve MODEL_ID --tensor-parallel-size 4.
- 5If the head count only divides evenly by an odd or prime number, use fewer GPUs than you own for tensor parallelism and consider pipeline parallelism for the rest.
How to confirm this is your problem
- Server crashes at initialization with a ValueError naming attention heads and your requested TP size.
- Works with --tensor-parallel-size 2 or 4 but fails at 3, 5, 6, or other non-divisor values.
- Different models on the same host need different TP sizes even with the same GPU count.
- Error mentions invalid device ordinal when TP size exceeds visible GPU count.
Root causes and fixes
Requested TP size is not a divisor of the model's attention head count
Tensor parallelism splits each attention layer's heads evenly across ranks so each GPU computes an equal, non-overlapping share of the heads. If the head count doesn't divide evenly, vLLM has no valid way to shard the weight matrices and refuses to start rather than silently pad or drop heads.
Fix: Look up num_attention_heads, and num_key_value_heads for GQA models, in the model's config.json, and choose a TP size from its integer divisors.
python -c "from transformers import AutoConfig; c = AutoConfig.from_pretrained('MODEL_ID'); print(c.num_attention_heads, getattr(c, 'num_key_value_heads', None))"TP size exceeds the number of GPUs actually visible to the process
vLLM spawns one worker process per TP rank and assigns it a GPU ordinal. If you ask for --tensor-parallel-size 4 but CUDA_VISIBLE_DEVICES only exposes 2 devices, or the container only has 2 GPUs passed through, the later ranks fail to acquire a device.
Fix: Set --tensor-parallel-size to match the length of CUDA_VISIBLE_DEVICES exactly, and verify with nvidia-smi -L inside the same environment vLLM runs in.
nvidia-smi -L echo $CUDA_VISIBLE_DEVICES
GQA models have fewer KV heads than attention heads, and only KV heads divide cleanly
Grouped-query attention models such as Llama 3, Mistral, and Qwen have far fewer key/value heads than query heads. Both must be shardable by TP size, so a TP size that divides the larger attention head count but not the smaller KV head count will still fail.
Fix: Check num_key_value_heads specifically, not just num_attention_heads; TP size must divide both.
Container or scheduler only allocated a subset of the requested GPUs
In Kubernetes or Docker, a resource request for N GPUs that only gets M less than N allocated, due to a misconfigured device plugin or resource quota, will present fewer devices than vLLM's launch flag assumes, producing the same divisibility or ordinal errors even though your manifest looks correct.
Fix: Verify the actual GPU count inside the running container, not just the requested count in the deployment spec.
kubectl exec POD_NAME -- nvidia-smi -L
Mixing tensor-parallel-size with pipeline-parallel-size incorrectly
When both --tensor-parallel-size and --pipeline-parallel-size are set, the total GPU count must equal TP times PP. A mismatch between that product and visible devices produces the same class of device-ordinal errors even when TP alone would have divided cleanly.
Fix: Confirm tensor_parallel_size multiplied by pipeline_parallel_size equals your total GPU count before launch.
Diagnostic commands
Get the model's head counts
python -c "from transformers import AutoConfig; c = AutoConfig.from_pretrained('MODEL_ID'); print('heads:', c.num_attention_heads, 'kv_heads:', getattr(c, 'num_key_value_heads', 'n/a'))"Both numbers must be evenly divisible by your chosen TP size; if only one is, TP size has to divide the smaller KV head count.
List visible GPUs in the exact runtime environment
nvidia-smi -L
Count the lines; --tensor-parallel-size must equal this count, or TP times PP if using pipeline parallelism too.
Check CUDA_VISIBLE_DEVICES scoping
echo $CUDA_VISIBLE_DEVICES
If unset, all GPUs on the host are visible; if set, only the listed ordinals are, and vLLM's TP size must match that list's length.
Stopping it from happening again
- Standardize on GPU counts per node, such as 2, 4, or 8, that are known divisors for the model families you deploy.
- Script a pre-flight check that reads config.json head counts and validates --tensor-parallel-size before launching in CI/CD.
- Document per-model TP size in your deployment manifests so on-call engineers don't have to rederive it during an incident.
- Test new model releases' head-count divisibility against your standard GPU counts before promoting to production.
When this becomes an architecture problem
If a model's head count only divides evenly by GPU counts you don't have available, or by 1, that's an architecture-hardware mismatch: consider pipeline parallelism instead of tensor parallelism, a differently-shaped model variant, or provisioning nodes with a GPU count that matches the model's valid divisors.
Frequently asked questions
Why must tensor-parallel-size divide the head count evenly?
Each GPU in a TP group computes a disjoint, equal-sized slice of the attention heads for every layer. If the head count can't be split into equal integer groups, vLLM has no correct way to partition the weight matrices, so it fails fast instead of producing subtly wrong outputs.
What TP sizes are safe for common models?
Most modern open-weight models such as Llama, Qwen, and Mistral use head counts that are powers of two or have small prime factors, so TP of 1, 2, 4, or 8 usually works. Always confirm against the specific model's config.json rather than assuming.
Can I use 3 or 5 GPUs for tensor parallelism?
Only if the model's attention and KV head counts are divisible by that number, which is uncommon for mainstream architectures. If you only have 3 or 5 GPUs and the model doesn't support that split, use fewer GPUs for TP and consider pipeline parallelism for the remainder.
Size it properly next time
Free calculators that prevent this class of failure before you provision hardware.
Multi-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolNVIDIA GPU Selector for LLM Workloads
Score your workload across model size, concurrency, latency, budget, and facility power to get a recommended GPU tier from RTX-class to multi-node B200 clusters.
Related problems
vLLM runs out of memory during startup, before serving any requests
vLLM's startup OOMs happen because it preallocates a KV cache pool sized against gpu_memory_utilization right after loading weights, so the failure point is engine initialization, not user traffic. Fix it by lowering gpu_memory_utilization if it's set too aggressively for actual free VRAM, lowering max_model_len, or reducing weight footprint with quantization or more GPUs.
vLLM server won't start (port in use, auth, VRAM, or unsupported architecture)
vLLM server startup failures collapse into four buckets: the port is already bound by another process, Hugging Face auth is missing or expired for a gated repo, there isn't enough free VRAM for the requested model and context, or the installed vLLM version doesn't yet support the model's architecture. Read the last traceback line, not just the top, to tell them apart.
Tensor parallelism fails because the model does not split evenly across GPUs
Tensor parallelism fails when the chosen degree does not evenly divide the model's attention heads, and often its key/value heads and hidden size, so the framework cannot split the projection weights across ranks. It also fails in practice, without an assertion, when the GPUs assigned to the TP group differ in VRAM or compute, since the even weight split then fits some ranks and not others.
NCCL error during multi-GPU training or inference
An NCCL error during multi-GPU training or inference is almost always a symptom of a rank that crashed, a version mismatch across processes, or bad GPU topology, not a bug in NCCL itself. Enable NCCL_DEBUG=INFO first and read the per-rank logs before touching timeouts or retry logic.
GuideMulti-GPU LLM Serving: Tensor vs Pipeline Parallelism
Multi-GPU LLM serving explained: tensor parallelism vs pipeline parallelism, NCCL interconnect requirements, and when to split a model across GPUs.
GuidevLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
GuideOn-Prem GPU Cluster Design: Node Sizing, Networking, and Storage
Design an on-prem GPU cluster: node sizing for H100/H200/B200, InfiniBand vs RoCE networking, storage throughput, and rack power for enterprise AI workloads.
Still stuck, or tired of fighting your own infrastructure?
Netray deploys and operates on-prem AI for regulated manufacturers and defense suppliers. We have debugged this stack in production, on air-gapped networks, at scale.