Why pipeline parallelism scales poorly, and why the bubble, not the code, is usually the problem

Pipeline parallelism only overlaps stages once the pipeline is full: at the start of every step, stage 2 must wait for stage 1 to finish its first microbatch, stage 3 waits for stage 2, and so on, with a mirrored drain at the end. The fraction of step time spent in this unavoidable bubble is roughly the stage count minus one divided by the number of microbatches, so with only as many microbatches as stages, more than half the step can be idle bubble time, and adding stages without adding microbatches makes the ratio worse.

Fix: Increase the number of microbatches, via gradient accumulation or your framework's num_microbatches setting, to several times your pipeline stage count; a common rule of thumb is 4x or more, so the bubble becomes a small fraction of total step time.

Pipeline parallel frameworks typically split a model into stages by layer count, not actual compute cost per layer. Embedding and output layers can be disproportionately expensive, especially with large vocabularies, or disproportionately cheap compared to a typical transformer block, so an even layer count split can still produce a badly imbalanced compute split, and every stage's throughput is capped by the slowest one.

Fix: Manually rebalance the partition points based on measured per-layer compute time rather than layer count alone, giving the embedding and output stages fewer transformer blocks to compensate for their extra cost.

Smaller microbatches use less activation memory per stage but require more of them to reach a target global batch size; if activation memory is already tight, you may be forced into fewer, larger microbatches purely to fit in VRAM, which is exactly the configuration that maximizes bubble overhead.

Fix: Add or increase activation checkpointing, recomputing activations during backward instead of storing them, to reduce per-microbatch memory footprint and free headroom for more, smaller microbatches.

In 3D-parallel setups, tensor-parallel all-reduce operations happen within each pipeline stage while pipeline stages themselves communicate activations forward and gradients backward between GPUs; if these are not scheduled to overlap properly, tensor-parallel communication can serialize with pipeline handoffs, adding latency on top of the pipeline bubble rather than hiding it.

Fix: Use a framework with built-in support for combined 3D parallelism scheduling rather than hand-rolling the combination, since correct overlap of tensor-parallel and pipeline-parallel communication is a nontrivial scheduling problem.

Every microbatch's activations and gradients must physically move between the GPUs assigned to consecutive pipeline stages; if that link is a slow PCIe path or an oversubscribed network rather than NVLink or InfiniBand, the transfer time itself adds to the bubble on top of the theoretical fill and drain overhead, and this effect is invisible in bubble-fraction math that assumes instant handoffs.

Fix: Place adjacent pipeline stages on GPUs with the fastest available interconnect between them, same node or NVLink-connected where possible, and reserve cross-node links for the outermost or least-frequent communication boundaries.