Multi-GPU Tensor Parallelism Calculator: Model Real Scaling Efficiency
This free multi-GPU tensor parallelism calculator models how throughput actually scales as you split a model across more GPUs, and it is built for infrastructure engineers deciding between a larger tensor-parallel group and more independent model replicas. Enter your tensor-parallel degree, single-GPU baseline throughput, and interconnect class, and the tool returns effective speedup, scaling efficiency percentage, and exactly how much throughput interconnect overhead costs you compared to theoretical linear scaling.
Your numbers
How many GPUs the model is split across within one replica.
Measured or estimated tokens per second if this model somehow ran on one GPU alone.
Represents how well throughput holds up as GPUs are added. Higher means scaling stays closer to linear.
Your results
Modeling estimate using a power-law approximation of scaling efficiency. Real scaling depends on model architecture, sequence length, and serving engine communication kernels. Benchmark your actual configuration before finalizing a cluster topology.
Get your full GPU topology benchmark
We will email you a personalized scaling efficiency comparison across interconnect and topology options, and a Netray infrastructure specialist will follow up with a procurement recommendation.
No spam. Your results stay private. Unsubscribe anytime.
Why tensor parallelism scales sub-linearly
Splitting a model's layers across multiple GPUs requires every GPU to exchange activations with its peers at nearly every layer boundary, an all-reduce communication step that adds latency the GPUs would not incur working alone. On fast NVLink-connected systems within one server, that overhead is small and scaling stays close to linear up to four or eight GPUs. Over PCIe or across nodes via network fabric, communication latency grows and efficiency drops meaningfully faster as GPU count rises, which is exactly what the interconnect efficiency exponent in this calculator represents.
- NVLink and NVSwitch within one server deliver the best scaling, often 85-95% efficiency at 4-8 GPUs.
- PCIe-only interconnects lose meaningfully more efficiency per doubling of GPU count, often 65-80%.
- Multi-node scaling over InfiniBand sits between these, better than PCIe but behind single-node NVLink.
- Efficiency loss compounds: doubling GPU count rarely doubles throughput regardless of interconnect quality.
When to grow the tensor-parallel group versus add replicas
Growing tensor-parallel degree only makes sense when a model genuinely does not fit, or does not perform acceptably, on fewer GPUs. Once a model comfortably fits and hits acceptable single-stream latency at a given GPU count, additional capacity is almost always better spent on independent replicas behind a load balancer than on a larger tensor-parallel group, because replica scaling is close to perfectly linear while tensor-parallel scaling is not. The common mistake is defaulting to the largest tensor-parallel group available rather than the smallest one that meets latency requirements.
Reading throughput lost to overhead
This figure quantifies the real cost of interconnect limitations in tokens per second, which is the number worth bringing to a hardware procurement conversation. A large gap between theoretical linear scaling and effective throughput at a given GPU count is a strong signal that either the interconnect needs upgrading (moving from PCIe to NVLink-connected systems), or that the deployment should shift toward more independent replicas at a smaller tensor-parallel degree rather than continuing to add GPUs to one group with diminishing returns.
- A large throughput gap at high GPU count suggests replicas would use hardware more efficiently than a bigger tensor-parallel group.
- Upgrading interconnect (PCIe to NVLink-connected hardware) directly raises the efficiency ceiling for a given GPU count.
- Track this number over time as you scale, since the inflection point where replicas beat further tensor-parallel growth is workload-specific.
How Netray designs GPU cluster topology
Netray plans on-prem GPU clusters for manufacturers and defense contractors, and topology decisions, how many GPUs per tensor-parallel group versus how many independent replicas, are where a large share of wasted hardware spend actually hides. We benchmark real scaling efficiency on your target hardware and model combination rather than assuming vendor marketing numbers, and design cluster topology around the crossover point where replicas beat further tensor-parallel scaling. Engagements typically include a topology benchmark before final hardware procurement.
Frequently Asked Questions
Why not just always maximize the tensor-parallel degree for the fastest model?
Because interconnect overhead compounds with every additional GPU in the group, so throughput per dollar declines past a certain point even as raw single-stream latency keeps improving slightly. If your goal is maximum total cluster throughput across many users rather than the lowest possible latency for one user, more independent replicas at a smaller tensor-parallel degree almost always deliver more aggregate tokens per second for the same GPU budget.
How much does NVLink actually matter versus PCIe for LLM serving?
Substantially, especially past four GPUs. Within a single NVLink-connected server, scaling efficiency commonly stays above 85% at 4-8 GPUs. The same model split over PCIe-only interconnects can drop to 65-75% efficiency at the same GPU count, meaning you pay for GPUs that deliver meaningfully less than their nameplate contribution. This is a primary reason purpose-built AI servers with NVSwitch cost more than commodity multi-GPU configurations.
Does tensor parallelism affect latency or just throughput?
Both, in different directions. Tensor parallelism can reduce single-stream latency by splitting the memory-bandwidth-bound decode work across more GPUs, letting each request's tokens generate faster. But it does so at sub-linear efficiency due to communication overhead, and beyond four to eight GPUs the added communication cost can start eroding the latency benefit rather than continuing to improve it, which is why most production deployments cap tensor-parallel degree at four or eight even when more GPUs are available.
What interconnect efficiency should I assume for multi-node deployments?
Multi-node tensor parallelism over InfiniBand NDR typically lands around 0.80-0.85 on the efficiency exponent used here, noticeably better than PCIe-only single-node setups but behind NVLink-connected single-node systems. Multi-node tensor parallelism is generally avoided when possible in favor of keeping one replica within a single NVLink-connected node and scaling replica count across nodes instead, precisely because cross-node communication overhead is harder to control.
Get a benchmarked GPU cluster topology that avoids paying for interconnect overhead you do not need.
Related Tools
vLLM Throughput Estimator
Estimate aggregate tokens-per-second throughput for a vLLM deployment from model size, GPU class, and batch depth, accounting for continuous batching gains.
On-Prem AILLM Serving Capacity Planner
Convert a peak concurrent user target directly into a required GPU count with redundancy, then see the daily token and response capacity that hardware delivers.
On-Prem AIGPU Cluster Utilization Calculator
Turn GPU capital, amortization, and operating cost into an effective cost per productive GPU hour, and find the utilization threshold where owning beats renting.
Go Deeper
Multi-GPU LLM Serving: Tensor vs Pipeline Parallelism
Multi-GPU LLM serving explained: tensor parallelism vs pipeline parallelism, NCCL interconnect requirements, and when to split a model across GPUs.
On-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
Enterprise GPU Cluster Planning for AI Workloads
Plan an enterprise GPU cluster for AI workloads: H100 vs L40S sizing, networking, power, cooling, and cost models for on-prem LLM inference and training.