Multi-GPU LLM Serving: Tensor Parallelism vs Pipeline Parallelism
A model that does not fit on one GPU forces a choice between tensor parallelism and pipeline parallelism, and the wrong choice on the wrong interconnect turns a serving deployment into a latency problem. Tensor parallelism splits individual layers across GPUs and requires fast, low-latency interconnect (NVLink, not just PCIe) because every forward pass needs an all-reduce communication step between GPUs. Pipeline parallelism splits the model by layer ranges across GPUs, tolerates slower interconnect better, but adds pipeline bubble overhead and complicates request scheduling. For most single-node multi-GPU inference serving, tensor parallelism is the right default when NVLink is available; pipeline parallelism becomes relevant mainly for multi-node deployments or very large MoE models where communication topology matters more.
Tensor Parallelism: Splitting Layers, Sharing Every Forward Pass
Tensor parallelism (TP) splits weight matrices within each layer across GPUs, so every GPU computes a portion of every layer and results are combined via an all-reduce operation after each layer. This means every single token generation step requires GPU-to-GPU communication, which is why TP demands high-bandwidth, low-latency interconnect: NVLink (900 GB/s on H100 class systems) makes this nearly free, while falling back to PCIe (even PCIe 5.0 at roughly 128 GB/s) introduces communication overhead that can dominate compute time and tank throughput. In vLLM, set --tensor-parallel-size to the number of GPUs; the framework handles sharding automatically for supported architectures. TP scales well up to 8 GPUs on a single NVLink-connected node (a DGX or HGX-class server) and is the standard choice for serving a single large dense model, such as a 70B or 405B parameter model, that exceeds one GPU's memory.
- Requires NVLink or equivalent high-bandwidth interconnect; PCIe-only multi-GPU setups suffer badly under TP
- Every layer's forward pass requires an all-reduce, so TP degree is bounded by interconnect topology, not just GPU count
- vLLM's --tensor-parallel-size handles sharding automatically for supported model architectures
- Best fit: single large dense model on one NVLink-connected node, up to 8-way TP on current DGX/HGX hardware
Pipeline Parallelism: Splitting by Layer Range
Pipeline parallelism (PP) assigns contiguous layer ranges to different GPUs or nodes, so a request flows sequentially through GPU 1's layers, then GPU 2's layers, and so on, with far less frequent communication than TP (only activations passed at stage boundaries, not every layer). This makes PP tolerant of slower interconnect and the natural choice for spanning multiple nodes over InfiniBand or even Ethernet where NVLink is not available. The cost is pipeline bubbles: GPUs later in the pipeline sit idle while waiting for the first request to arrive through earlier stages, which continuous batching mitigates but does not eliminate. PP also complicates request scheduling and generally serves fewer requests per GPU than an equivalent TP setup when NVLink is available, so it is mainly justified when the model does not fit within a single NVLink domain even with TP, common with the largest MoE models like DeepSeek V3 or Kimi K2.
- Only passes activations at stage boundaries, tolerating slower interconnect than TP requires
- Necessary when a model exceeds one NVLink domain, common for very large MoE models spanning multiple nodes
- Pipeline bubble overhead reduces GPU utilization versus TP on the same hardware, mitigated but not eliminated by continuous batching
- vLLM supports combining TP within a node and PP across nodes for the largest models
NCCL, Interconnect, and the Overhead You Do Not See in Benchmarks
Both TP and PP rely on NCCL (NVIDIA Collective Communications Library) for GPU-to-GPU communication, and NCCL performance is entirely bound by the physical interconnect topology. Verify actual NVLink connectivity with nvidia-smi topo -m before assuming a server's GPUs are fully NVLink-connected; some cloud instances and older server configurations have partial NVLink topology where only some GPU pairs get full bandwidth. For multi-node setups, InfiniBand (typically 200-400 Gb/s per link on current hardware) is close to mandatory for TP across nodes; standard Ethernet introduces enough latency that PP or a fully sharded single-node approach usually wins. Benchmark your actual topology before committing to a TP degree, since a benchmark run on a fully NVLink-connected reference system will not predict performance on hardware with a different or partial interconnect.
Practical Guidance for Choosing a Parallelism Strategy
For most enterprise deployments serving a single dense model up to 70B or so parameters on an 8-GPU NVLink node, tensor parallelism alone is the right answer, no pipeline parallelism needed. For models that exceed a single node's memory even with TP, such as the largest MoE models, combine TP within each node with PP across nodes. If your hardware lacks NVLink entirely, either quantize aggressively enough to fit within fewer GPUs and avoid multi-GPU serving altogether, or accept PP's lower per-GPU utilization as the cost of the interconnect you have. Do not default to PP because it seems simpler to reason about; on NVLink hardware it is almost always the wrong choice for throughput.
How Netray Designs Multi-GPU Serving Topologies
Netray sizes multi-GPU serving topology against the client's actual GPU fleet interconnect, verified with real topology checks rather than assumed NVLink connectivity, before recommending a tensor or pipeline parallelism strategy. For clients deploying frontier-scale MoE models across multiple nodes, we design the combined TP-within-node, PP-across-node topology and validate it against real request load, not synthetic benchmarks. This is core to our on-prem GPU cluster design work, where getting the interconnect and parallelism strategy wrong is the single most common cause of a client's expensive GPU cluster underperforming its theoretical throughput.
Frequently Asked Questions
When should I use tensor parallelism versus pipeline parallelism for LLM serving?
Use tensor parallelism when your GPUs are connected via NVLink within a single node and the model exceeds one GPU's memory; it is the default for serving a large dense model up to roughly 70B parameters on an 8-GPU NVLink server. Use pipeline parallelism when spanning multiple nodes or when NVLink is unavailable, since PP tolerates slower interconnect at the cost of pipeline bubble overhead. Very large MoE models often combine both: TP within each node, PP across nodes.
Does tensor parallelism work well without NVLink?
Not well. Tensor parallelism requires an all-reduce communication step after every layer's forward pass, which demands the high bandwidth and low latency NVLink provides (roughly 900 GB/s on H100-class hardware). Falling back to PCIe interconnect, even PCIe 5.0, introduces communication overhead that can dominate compute time and significantly reduce throughput. Verify actual NVLink topology with nvidia-smi topo -m before committing to a tensor parallelism degree.
How many GPUs can I use for tensor parallelism?
Tensor parallelism scales well up to 8 GPUs on a single NVLink-connected node (typical DGX or HGX-class server topology), which is the practical ceiling for most single-node deployments. Beyond that, you need to span multiple nodes, which requires combining tensor parallelism within each node with pipeline parallelism across nodes over InfiniBand, since standard Ethernet interconnect introduces too much latency for cross-node tensor parallelism to perform well.
Key Takeaways
- 1Tensor Parallelism: Splitting Layers, Sharing Every Forward Pass: Tensor parallelism (TP) splits weight matrices within each layer across GPUs, so every GPU computes a portion of every layer and results are combined via an all-reduce operation after each layer. This means every single token generation step requires GPU-to-GPU communication, which is why TP demands high-bandwidth, low-latency interconnect: NVLink (900 GB/s on H100 class systems) makes this nearly free, while falling back to PCIe (even PCIe 5.0 at roughly 128 GB/s) introduces communication overhead that can dominate compute time and tank throughput.
- 2Pipeline Parallelism: Splitting by Layer Range: Pipeline parallelism (PP) assigns contiguous layer ranges to different GPUs or nodes, so a request flows sequentially through GPU 1's layers, then GPU 2's layers, and so on, with far less frequent communication than TP (only activations passed at stage boundaries, not every layer). This makes PP tolerant of slower interconnect and the natural choice for spanning multiple nodes over InfiniBand or even Ethernet where NVLink is not available.
- 3NCCL, Interconnect, and the Overhead You Do Not See in Benchmarks: Both TP and PP rely on NCCL (NVIDIA Collective Communications Library) for GPU-to-GPU communication, and NCCL performance is entirely bound by the physical interconnect topology. Verify actual NVLink connectivity with nvidia-smi topo -m before assuming a server's GPUs are fully NVLink-connected; some cloud instances and older server configurations have partial NVLink topology where only some GPU pairs get full bandwidth.
Put this into numbers
Free interactive tools for exactly this problem. No signup to use them.
Multi-GPU Tensor Parallelism Calculator
Model how tensor-parallel throughput actually scales across multiple GPUs, accounting for interconnect overhead that keeps scaling sub-linear.
Free ToolGPU Sizing Calculator for LLM Inference
Work out how many GPUs you need to serve a given open-weight model to your user base, based on memory footprint and token throughput.
Free ToolSmall Language Model Fit Assessment
Answer eight questions about task complexity, volume, latency, data sensitivity, and cost to see whether a small language model can replace your frontier model spend.
Terms used in this article
Planning a multi-GPU LLM serving deployment? Netray will validate your interconnect topology and recommend the right parallelism strategy before you provision hardware.
Related Resources
vLLM Production Deployment: A Practitioner's Guide
Deploy vLLM in production: continuous batching, PagedAttention, config flags that matter, and the metrics to watch before you trust it with real traffic.
AI & AutomationOn-Prem LLM Inference Hardware in 2026: A Roundup
On-prem LLM inference hardware for 2026: H100 vs H200 vs B200 pricing, when A100 fleets still work, and how to size GPUs against real serving needs.
AI & AutomationKV Cache Optimization: Prefix Caching and Chunked Prefill
KV cache optimization techniques for production LLM serving: prefix caching, chunked prefill, PagedAttention, and sizing memory for concurrent users.