Pipeline Parallelism for LLM Training: GPipe, 1F1B, Interleaved, Zero-Bubble and DualPipe Schedules (2026)
Every GPU in a pipeline-parallel training job spends part of each step doing nothing. With 8 stages and 8 micro-batches, 7 of every 15 time slots on each device are idle, a 46.7% loss that no amount of kernel tuning will recover. That idle time is the pipeline bubble, and the history of pipeline parallelism since 2018 is largely a history of attacking it: GPipe exposed it, 1F1B tamed its memory cost, interleaving shrank it, zero-bubble schedules split the backward pass to fill it, and DualPipe overlapped what remained with expert-parallel communication.
This matters in 2026 because frontier models no longer fit on one node, and the inter-node links that pipeline stages cross are an order of magnitude slower than NVLink. Choosing the wrong schedule costs double-digit percentages of a cluster that bills by the GPU-hour.
This guide derives the bubble formula, works through activation memory per schedule, explains why each successor schedule exists, and ends with a Python simulator you can run to reproduce the numbers.
What this covers: why we pipeline at all, the bubble derivation with worked numbers, GPipe versus 1F1B versus interleaved schedules, the zero-bubble backward split, DualPipe, composition with tensor/data/expert parallelism, stage partitioning, and the failure modes that bite in production.
Context and Background: Why Pipeline at All?
Training a large language model requires holding weights, gradients, optimizer state, and activations. For mixed-precision Adam, a common rule of thumb is roughly 16 bytes per parameter for weights, gradients and optimizer state combined, before any activation memory. At that rate a 70-billion-parameter model needs on the order of 1.1 TB of model state, far beyond one 80 GB accelerator. Something must split the model.
There are three classical axes. Data parallelism replicates the model and splits the batch; it does not reduce per-device model state unless combined with sharding such as ZeRO or FSDP, which our guide to data, tensor, pipeline, FSDP and ZeRO parallelism compares in detail. Tensor parallelism splits individual matrix multiplications across devices, but it needs an all-reduce (or reduce-scatter and all-gather pair) twice per transformer layer in the forward pass and twice in the backward, so it only makes sense on the fastest links, normally inside one NVLink domain.
Pipeline parallelism splits the model by depth. Stage 0 holds the first layers, stage 1 the next block, and so on. The only traffic between stages is the activation tensor at a layer boundary in the forward direction and its gradient in the backward direction, point-to-point. For a micro-batch of size b, sequence length s and hidden size h, that is b·s·h elements per hop, which is small compared with the per-layer all-reduce volume of tensor parallelism. That is the core appeal: pipeline stages tolerate slow inter-node links, so they are the natural axis to cut across nodes.
The price is the bubble. Stage k cannot start micro-batch 0 until stage k-1 has finished it, so the pipeline must fill at the start and drain at the end of every optimizer step. GPipe, from Huang et al. at Google (arXiv:1811.06965), introduced the batch-splitting scheme that made this practical. The authors reported near-linear speedup when partitioning across accelerators, demonstrated on a 557-million-parameter AmoebaNet image model and a 6-billion-parameter, 128-layer multilingual Transformer. PipeDream and its variants then explored asynchronous and weight-versioned alternatives; PipeDream-2BW claims up to 20x faster training for large GPT and BERT models while keeping update semantics close to data parallelism, per its abstract.
The industrial default for synchronous LLM training became the Megatron-LM combination of tensor, pipeline and data parallelism described by Narayanan et al. (arXiv:2104.04473). Their headline result: a 1-trillion-parameter model sustained 502 petaFLOP/s on 3,072 GPUs, 52% of theoretical peak per GPU, with an interleaved schedule adding more than 10% throughput at similar memory. This post is about the schedule layer of that stack, and a related long-context technique, ring attention and context parallelism, composes with it as another orthogonal axis.
One framing note before the mechanics. “Synchronous” here means every micro-batch in a step uses the same weights, and the optimizer runs once after all gradients are accumulated. Everything below preserves that semantics unless stated, because it keeps loss curves reproducible and comparable with data-parallel baselines. The cost of that purity is the flush, and the flush is the bubble.
The Core Mechanism: Micro-Batches, the Bubble, and Activation Memory
Direct answer. In synchronous pipeline parallelism, the global batch is split into m micro-batches that flow through p stages. Because the first and last stages must wait for the pipeline to fill and drain, a fraction (p-1)/(m+p-1) of every device’s time is idle. Raising m shrinks the bubble; the schedule (GPipe, 1F1B, interleaved, zero-bubble) decides how much activation memory that costs.
Figure 1: Layer blocks assigned to p pipeline stages. Activations move forward point-to-point and gradients move backward; the first and last stages wait while the pipeline fills and drains, producing the pipeline bubble.
Figure 1 shows the topology. Four stages hold consecutive layer blocks; micro-batches enter at stage 0, each forward pass emits an activation tensor to the next stage, the last stage computes the loss, and gradients retrace the path in reverse. The long-description version: the dependency graph is a grid with p rows and m columns, forward edges go down-right, backward edges go up-right, and any schedule is just a choice of execution order on each row that respects those edges.
Deriving the bubble fraction
Let t_f be the forward time of one micro-batch on one stage and t_b the backward time. Assume balanced stages and ignore communication. The ideal time per device, with perfect overlap, is m·(t_f + t_b), because each device must process all m micro-batches forward and backward.
Now trace the critical path. The last micro-batch’s forward reaches the final stage only after traversing p stages, and its backward must then return through p stages. In the standard flush schedule, the first device cannot begin the backward of micro-batch 0 until p-1 forward hops and p-1 backward hops have elapsed, so each device is idle for (p-1)·t_f at the start and (p-1)·t_b at the end. Total time is therefore (m + p – 1)·(t_f + t_b), and the bubble fraction is:
bubble = (p-1)(t_f+t_b) / ((m+p-1)(t_f+t_b)) = (p-1) / (m+p-1)
The ratio of bubble time to ideal time is (p-1)/m, which is the form Narayanan et al. use. Note the result is independent of t_f and t_b, as long as they are uniform across stages.
Worked numbers for p = 8 stages:
| Micro-batches m | Bubble fraction (p-1)/(m+p-1) | Efficiency |
|---|---|---|
| 8 | 7/15 = 46.7% | 53.3% |
| 16 | 7/23 = 30.4% | 69.6% |
| 32 | 7/39 = 17.9% | 82.1% |
| 64 | 7/71 = 9.9% | 90.1% |
| 128 | 7/135 = 5.2% | 94.8% |
The rule of thumb that falls out: you want m at least 4 to 8 times p to keep the bubble under roughly 15 to 20%. That constraint has a real cost, because m·b·d_dp equals the global batch size, where b is the micro-batch size and d_dp the data-parallel degree. Pushing m up means either a bigger global batch (which may exceed the critical batch size and hurt sample efficiency) or a smaller micro-batch (which lowers GPU arithmetic intensity and kernel efficiency). This tension, not the formula, is why scheduling research continues.
Activation memory is the real constraint
A stage must keep the forward activations of a micro-batch until that micro-batch’s backward pass consumes them. Call the per-micro-batch activation footprint on one stage A. The schedule determines how many micro-batches are in flight at once.
In GPipe, every stage runs all m forwards before any backward, so each stage stores m·A. In the 1F1B schedule, stage i (0-indexed) has at most p – i micro-batches in flight, so the worst case is stage 0 with p·A, independent of m. The DeepSeek-V3 report’s Table 2 expresses the same fact as an activation cost of PP for 1F1B.
Illustrative arithmetic, with A = 2 GB (a made-up but plausible figure for a mid-sized layer block with moderate recomputation): at p = 8 and m = 32, GPipe needs 64 GB of activations on every stage, while 1F1B needs 16 GB on stage 0 and less on later stages. On an 80 GB card that already holds several gigabytes of weights and optimizer shards, that difference is the difference between running and out-of-memory. This is why 1F1B displaced GPipe for LLM training even though the two have identical bubble fractions.
Activation checkpointing (recomputation) changes A rather than the schedule. Storing only stage-boundary inputs and recomputing inside the stage drops A sharply, at the price of roughly one extra forward pass, around a 33% increase in compute when t_b = 2·t_f. It interacts with every schedule below, and we return to it in the trade-offs section.
The 1F1B schedule
The 1F1B schedule, popularized by PipeDream and used in its synchronous “flush” form in Megatron-LM, has three phases per stage. In warm-up, stage i runs p – i – 1 extra forwards (bounded by m). In steady state it alternates strictly one forward, one backward. In cool-down it drains the remaining backwards. The alternation is what bounds the in-flight count: each new forward is admitted only when a backward frees memory.
Figure 2: The 1F1B schedule per stage. Warm-up depth shrinks with stage index; steady state alternates forward and backward; cool-down drains backward passes. The bubble equals GPipe’s, but in-flight activations are capped at p.
The first stage has the deepest warm-up (p-1 forwards before its first backward arrives), the last stage has none, and that asymmetry is why memory is imbalanced: stage 0 stores up to p micro-batches while the last stage stores one. Real pipeline parallelism deployments exploit this by giving early stages fewer layers or more aggressive recomputation, a point developed in the partitioning section.
A subtle property worth stating: 1F1B does not reduce the bubble at all. Its total time is the same (m+p-1)(t_f+t_b). It is a pure memory optimization that makes a large m affordable, and large m is what actually shrinks the bubble. Treat it as an enabler, not a cure.
Shrinking the Pipeline Bubble: Interleaving, Zero-Bubble, and DualPipe
The three successors to plain 1F1B in modern pipeline parallelism attack different terms. Interleaving divides the bubble by shrinking the unit of work. Zero-bubble scheduling fills the bubble by splitting the backward pass into pieces with different dependencies. DualPipe halves the effective pipeline depth by running two directions at once and hides communication behind computation. Each pays for its gain in a different currency: communication volume, memory, or implementation complexity.
Interleaved 1F1B and virtual stages
Narayanan et al. proposed assigning each device v non-contiguous model chunks instead of one contiguous block. With 16 layers on 4 devices, plain pipelining gives device 0 layers 1 to 4; with v = 2, device 0 holds layers 1, 2 and 9, 10, device 1 holds 3, 4 and 11, 12, and so on. A micro-batch now traverses the device ring twice per forward pass.
Each chunk is 1/v the size, so the forward and backward time for one chunk are t_f/v and t_b/v. The fill and drain phases are measured in chunk-steps across p devices, so the bubble time shrinks to (p-1)(t_f+t_b)/v. The bubble fraction becomes:
bubble_interleaved = (p-1) / (v·m + p – 1)
For p = 8, m = 16, v = 2 this is 7/39 = 17.9%, versus 30.4% for plain 1F1B at the same m, and the simulator at the end of this post prints the same analytic values. With v = 4 it is 7/71 = 9.9%, equal to plain 1F1B at m = 64.
The catch is communication. A micro-batch crosses v times as many stage boundaries, so point-to-point traffic grows by a factor of v. On a cluster where inter-node bandwidth is the scarce resource, that can eat the gain. The Megatron-LM paper reports the interleaved schedule improving throughput by more than 10% at similar memory use on its large-batch configurations, and the benefit naturally shrinks as the batch grows, because the bubble is already small (my inference from the formula, not a quoted result). Memory is slightly higher than plain 1F1B because warm-up admits more chunk-forwards before the first backward; the exact in-flight count follows the Megatron implementation’s warm-up rule, which I do not reproduce here. Interleaving also requires m to be a multiple of p in the common implementation, a practical constraint on batch-size choices.
Zero-bubble pipeline parallelism: splitting the backward pass
The key observation behind zero bubble pipeline parallelism, from Qi, Wan, Huang and Lin (arXiv:2401.10241) is that the backward pass of a linear layer is two independent computations. Given an output gradient dY, the input gradient dX = dY·Wᵀ is on the critical path, because the previous stage needs it. The weight gradient dW = Xᵀ·dY is not; nothing downstream in the pipeline waits for it, only the optimizer does. Conventional schedules compute both together as one block B. Zero-bubble schedules separate them into B (input gradient) and W (weight gradient) and let W float.
Figure 3: Splitting the backward pass. B (input gradient) feeds the previous stage immediately; W (weight gradient) depends only on the local activations and output gradient, so a scheduler can slot it into otherwise idle time before the optimizer step.
With W free to move, a scheduler can drop W blocks into the idle slots of the bubble. Using the notation of the DeepSeek-V3 report (F forward chunk, B full backward chunk, W the weight-gradient part of backward), the bubble of the ZB1P variant is (PP-1)(F+B-2W), against (PP-1)(F+B) for 1F1B. Take F = 1, B = 2, W = 1 in arbitrary units. Then 1F1B has a bubble of 7 × 3 = 21 units at PP = 8, while ZB1P has 7 × (3 – 2) = 7 units, one third. That is the same ratio the zero-bubble authors describe for their memory-matched handcrafted schedule, and it appears in this derivation without any tuning.
The paper goes further. It reports up to 23% higher throughput than 1F1B under a similar memory limit, rising to 31% when the memory constraint is relaxed, and it claims to be the first to reach zero bubbles under synchronous training semantics. Two additional ideas make that possible. First, an automatic scheduler searches for a schedule given measured F, B, W and communication times plus a memory limit, rather than relying on a hand-drawn pattern. Second, the optimizer step normally forces a global synchronization (checking for NaN or Inf values, computing the global gradient norm for clipping) that serializes all stages; the authors add a mechanism to bypass this synchronization while keeping results equivalent. I have not independently verified the details of that bypass beyond the abstract, so check the paper before relying on it for a gradient-clipping-sensitive run.
The memory trade-off is the honest cost. The zero-bubble schedule that approaches zero bubbles holds more micro-batches in flight, and activations must live until W runs, not merely until B runs. The simpler ZB1P keeps 1F1B’s PP activation cost per the DeepSeek-V3 table, which is why it is a drop-in improvement; the near-zero-bubble variants need roughly twice the activation memory of 1F1B, as I read the paper, and I label that figure as approximate.
There is also an engineering catch. Splitting B and W requires the framework to expose the two gradients separately. Autograd normally fuses them, so you need custom backward implementations for linear layers (or hooks that defer the weight-gradient GEMM). Attention and normalization layers have more tangled gradient dependencies. Teams that adopt zero-bubble schedules typically restrict the split to the big GEMMs, which hold most of the FLOPs anyway.
DualPipe: bidirectional pipelines with overlapped communication
DeepSeek-V3 (arXiv:2412.19437) is a 671-billion-parameter mixture-of-experts model with about 37 billion parameters activated per token, trained on a cluster of 2,048 H800 GPUs for a reported 2.788 million H800 GPU hours. Its MoE layers send tokens to experts on other nodes via all-to-all, and the report states that cross-node expert parallelism produces a computation-to-communication ratio of roughly 1:1. Ordinary pipeline schedules leave that communication exposed. DualPipe was designed so it is hidden.
The report’s training setup uses 16-way pipeline parallelism, 64-way expert parallelism spanning 8 nodes, and no tensor parallelism, avoiding what it calls costly tensor parallelism. DualPipe feeds micro-batches from both ends of the pipeline simultaneously, so each device holds both a “forward-direction” stage and a “reverse-direction” stage. That is why parameters are held twice (2x in the table below), and the report argues the cost is small because the large EP degree already shards expert weights thinly.
Its core trick is chunk-level overlap. A forward chunk and a backward chunk, belonging to different micro-batches, are executed together, with each chunk subdivided into attention, all-to-all dispatch, MLP, and all-to-all combine pieces. While one chunk’s all-to-all is on the wire, the other chunk’s attention or MLP computation runs, and the backward chunks are themselves split B/W in the zero-bubble style. The report notes that only 20 of the 132 streaming multiprocessors on an H800 are dedicated to communication kernels, so the overlap costs some compute capacity.
| Method | Bubble | Parameters per device | Peak activations |
|---|---|---|---|
| 1F1B | (PP-1)(F+B) | 1x | PP |
| ZB1P | (PP-1)(F+B-2W) | 1x | PP |
| DualPipe | (PP/2-1)(F&B+B-3W) | 2x | 2xPP+1 (paper) |
Table: Bubble and memory comparison as given in the DeepSeek-V3 report, Table 2. F&B is an overlapped forward and backward chunk pair. The public DualPipe repository’s README table lists the DualPipe activation cost as PP+1; I report the paper’s 2xPP+1 and flag the discrepancy between the two sources as unresolved.
Plug in PP = 8 and, as a conservative assumption, treat the overlapped F&B pair as taking the sum F+B = 3 (the real overlapped time can be lower, which would shrink the bubble further). With F = 1, B = 2, W = 1, DualPipe’s bubble is (4-1) × (3 + 2 – 3) = 6 units, against 21 for 1F1B and 7 for ZB1P. The structural gain is the PP/2 factor: with two directions, a micro-batch only needs to traverse half the pipeline before meeting its counterpart, so the fill and drain latency halves. The report also states that, with DualPipe, neither the bubbles nor the activation memory increase as the number of micro-batches grows, which is the property that lets large-m training scale.
Figure 4: DualPipe feeds micro-batches from both pipeline ends. Forward and backward chunks from different micro-batches run together so MoE all-to-all dispatch and combine overlap attention and MLP computation. The price is two parameter copies per device.
DualPipe’s code is open source under the MIT license in the deepseek-ai/DualPipe repository. Two cautions: it is tuned for a specific stack (MoE, large EP, InfiniBand plus NVLink), and for a dense model with no all-to-all, most of the overlap benefit disappears, leaving only the halved fill and drain at the price of duplicated weights.
Runnable Simulator: Bubble Ratio and Peak In-Flight Micro-Batches
The following script is a small list scheduler. It models p stages, m micro-batches, forward time 1 and backward time 2, with the zero-bubble variant splitting backward into B = 1 and W = 1. It ignores communication, so it measures schedule structure, not wall-clock performance. I ran it to verify the numbers quoted in this post. The interleaved result uses the analytic Megatron formula, because my greedy scheduler does not reproduce Megatron’s fixed interleaved ordering and I did not want to present a misleading simulated number.
"""Pipeline-schedule simulator: GPipe, 1F1B and a zero-bubble-style B/W split."""
def simulate(p, m, mode, tf=1.0, tb=2.0):
f_t = tf
b_t, w_t = (tb / 2, tb / 2) if mode == "zb" else (tb, 0.0)
kinds = ("F", "B", "W") if mode == "zb" else ("F", "B")
remaining = {(k, s, mb) for s in range(p) for mb in range(m) for k in kinds}
done, free = {}, [0.0] * p
inflight, peak = [0] * p, [0] * p
cap = (lambda d: m) if mode == "gpipe" else (lambda d: p - d)
dur = {"F": f_t, "B": b_t, "W": w_t}
def ready(k, s, mb):
if k == "F":
return 0.0 if s == 0 else done.get(("F", s - 1, mb))
if k == "B":
if ("F", s, mb) not in done: return None
if s == p - 1: return done[("F", s, mb)]
nxt = done.get(("B", s + 1, mb))
return None if nxt is None else max(nxt, done[("F", s, mb)])
return done.get(("B", s, mb))
while remaining:
best = None
for (k, s, mb) in remaining:
if k == "F" and inflight[s] >= cap(s): continue
r = ready(k, s, mb)
if r is None: continue
start = max(free[s], r)
pri = {"F": 0, "B": 1}[k] if mode == "gpipe" else {"B": 0, "F": 1, "W": 2}[k]
key = (start, pri, mb, s)
if best is None or key < best[0]: best = (key, (k, s, mb))
(start, _, _, _), (k, s, mb) = best
done[(k, s, mb)] = free[s] = start + dur[k]
remaining.discard((k, s, mb))
if k == "F":
inflight[s] += 1; peak[s] = max(peak[s], inflight[s])
if k == "B": inflight[s] -= 1
T = max(done.values())
return T, 1 - m * (tf + tb) / T, max(peak)
def interleaved_bubble(p, m, v):
return (p - 1) / (v * m + p - 1)
if __name__ == "__main__":
p = 8
for m in (8, 16, 32, 64):
row = []
for mode in ("gpipe", "1f1b", "zb"):
T, b, pk = simulate(p, m, mode)
row.append(f"{mode}: bubble={b:.3f} peak={pk}")
print(m, " | ".join(row), f"| formula={(p-1)/(m+p-1):.3f}",
f"| interleaved v=2: {interleaved_bubble(p, m, 2):.3f}")
The output at p = 8 (bubble ratio, with peak in-flight micro-batches on the busiest stage in parentheses):
| m | GPipe | 1F1B | ZB-style | Formula (p-1)/(m+p-1) | Interleaved v=2 (analytic) |
|---|---|---|---|---|---|
| 8 | 0.467 (8) | 0.467 (8) | 0.226 (8) | 0.467 | 0.304 |
| 16 | 0.304 (16) | 0.304 (8) | 0.127 (8) | 0.304 | 0.179 |
| 32 | 0.179 (32) | 0.179 (8) | 0.068 (8) | 0.179 | 0.099 |
| 64 | 0.099 (64) | 0.099 (8) | 0.035 (8) | 0.099 | 0.052 |
Three readings. First, GPipe and 1F1B produce identical bubble ratios that match the closed form exactly, which validates the derivation. Second, peak in-flight micro-batches for GPipe grows linearly with m (64 at m = 64) while 1F1B stays flat at p = 8, which is the entire memory argument in one column. Third, the zero-bubble-style split, even with a simple greedy policy, cuts the bubble roughly in half to one third; the exact factor depends on m and policy, and a searched schedule such as the paper’s automatic scheduler would do better. A caveat on the in-flight column for the ZB-style schedule: the simulator frees an activation when B completes, but in reality part of the activation must persist until W runs, so real zero-bubble memory is higher than the table suggests.
Composing Pipeline Parallelism with Tensor, Data, and Expert Parallelism
No production run uses pipeline parallelism alone. The standard recipe for pipeline parallelism at scale maps parallelism axes onto the network hierarchy: tensor parallelism inside the NVLink domain (8 GPUs in a typical server, larger in rack-scale NVLink systems), pipeline parallelism across nodes, and data parallelism across pipeline replicas. The total GPU count is the product t × p × d. For a 3,072-GPU job like the Megatron-LM trillion-parameter run, one configuration is t = 8, p = 64, d = 6; I use that decomposition purely as an arithmetic illustration of the product, not as the paper’s reported configuration.
The reasoning for the mapping is bandwidth per byte moved. Tensor parallelism moves large activations through collectives on the critical path of every layer, so it needs the highest bandwidth available. Pipeline parallelism moves one activation tensor per micro-batch per boundary. Data parallelism moves gradients once per step and can overlap with the backward pass. For the network side of this story, see our comparison of NVLink, InfiniBand and RoCE for NCCL collectives.
Mixture-of-experts models add expert parallelism (EP), where experts are sharded across devices and tokens are routed with all-to-all. DeepSeek-V3’s choice, 16-way PP and 64-way EP across 8 nodes with no tensor parallelism, shows the design space has moved: when the model is sparse, per-token compute is modest, tensor parallelism’s fine-grained collectives are poor value, and the expensive resource is the all-to-all. That is why DualPipe is built around hiding it.
Data parallelism interacts with pipeline parallelism through the gradient all-reduce. In a flush schedule, gradients for early stages are ready only at the very end of the step, so the data-parallel reduction cannot start until the cool-down phase. Frameworks overlap the reduction of stage k with the backward passes still running on other stages, but the last stage to finish defines the tail. With ZeRO-1 sharding, as DeepSeek-V3 uses according to its report, optimizer state is partitioned across data-parallel ranks, which cuts memory without changing the pipeline schedule. ZeRO-2 and ZeRO-3 are awkward with pipelining because they add reduce-scatters or parameter gathers per micro-batch, multiplying communication by m.
Stage partitioning and load balance
The bubble formula assumed identical stages. Real transformers are not identical across depth. The first stage carries the embedding table, the last carries the output projection and the loss, and with a large vocabulary (100k to 150k tokens) the output GEMM and cross-entropy can cost as much as several transformer layers. If you give every stage the same layer count, the last stage becomes the straggler and every other stage idles behind it.
The bubble generalizes: the step time is governed by the slowest stage’s per-micro-batch time multiplied by roughly m + p – 1, plus whatever fill and drain skew exists. A stage that is 10% slower than the rest stretches the whole pipeline by about 10%, which is why balancing matters as much as schedule choice. Practical fixes include assigning fewer transformer layers to the first and last stages, profiling real per-stage forward and backward times rather than counting layers, and in MoE models accounting for router and all-to-all cost. Heterogeneous layers, such as alternating dense and MoE blocks or sliding-window versus global attention, make layer count a poor proxy for time.
Memory imbalance under 1F1B pulls the opposite way. Stage 0 holds p micro-batches of activations while the last stage holds one, so equalizing time loads stage 0 with the highest memory pressure. Teams commonly apply stronger activation recomputation only on early stages, trading compute where memory is tight. Zero-bubble tooling frames this as a joint search over schedule, time and memory, which is the reason its automated scheduler takes measured costs and a memory limit as inputs.
Deeper Analysis: Choosing a Schedule
The pipeline parallelism schedules differ along four axes: bubble, activation memory, communication volume, and implementation burden. The table below uses the same p and m for each, with the bubble expressed relative to plain 1F1B and memory in units of per-micro-batch activations A. Entries marked “reported” come from the cited papers; others are derived from the formulas in this post.
| Schedule | Bubble fraction | Peak activation per stage | Extra communication | Main requirement |
|---|---|---|---|---|
| GPipe | (p-1)/(m+p-1) | m x A | none | simple; any framework |
| 1F1B (flush) | (p-1)/(m+p-1) | up to p x A | none | schedule engine |
| Interleaved 1F1B, v chunks | (p-1)/(v·m+p-1) | slightly above 1F1B | v times point-to-point | m multiple of p, layers divisible by p·v |
| ZB1P-style | about one third of 1F1B in the F=1,B=2,W=1 example | about p x A (reported by DeepSeek-V3 Table 2) | none | separate B and W kernels |
| Zero-bubble near-zero | approaches 0 (reported) | roughly twice 1F1B (my reading, approximate) | none | automatic scheduler, optimizer-sync bypass |
| DualPipe | (p/2-1)(F&B+B-3W) | 2xp+1 (paper table) | overlapped with compute | 2x parameters, bidirectional feed, MoE-style all-to-all to hide |
A decision procedure follows from the table. If the bubble is already under about 10% because m is at least 8p, stop optimizing the schedule and look at stage balance and communication overlap; the maximum remaining gain from zero-bubble is under 10 points of utilization and costs real engineering. If m is limited by the global batch size, so m is only 2p to 4p, the bubble is 20 to 33%, and interleaving or zero-bubble splitting pays handsomely. If you train a large MoE with cross-node all-to-all exposed on the critical path, DualPipe-style overlap addresses a cost the other schedules do not even model.
Consider a worked scenario. A dense 70B model on 512 GPUs uses t = 8, p = 8, d = 8. The global batch is 1,024 sequences, micro-batch size 2. Then m = 1024 / (2 × 8) = 64 micro-batches per pipeline, and the 1F1B bubble is 7/71 = 9.9%. Zero-bubble work might claw back perhaps 6 to 8 points of that, meaningful but modest. Now halve the global batch to 256 for a fine-tuning regime: m = 16, bubble 30.4%, and interleaving with v = 2 brings it to 17.9% at the cost of doubling pipeline point-to-point traffic. Here the schedule matters far more. The same cluster and model can sit on either side of the threshold depending on the batch regime, which is why the question “which schedule is best” has no universal answer.
Communication cost deserves a number. For micro-batch b = 2, sequence s = 8,192, hidden h = 8,192 and bf16, one boundary activation is 2 × 8192 × 8192 × 2 bytes = 268 MB. Sent over a 400 Gb/s (50 GB/s) NIC, it takes about 5.4 ms, ignoring latency and protocol overhead. Compared with per-stage layer compute for a block of ~10 transformer layers at that size, which runs in the tens to hundreds of milliseconds, this is hideable, but interleaving with v = 4 quadruples the hops, and a stage that serializes its sends with compute loses the benefit. Overlapping sends with the next chunk’s compute is mandatory, not optional.
Trade-offs, Gotchas, and What Goes Wrong
Batch-size coupling. In pipeline parallelism the bubble is only cheap when m is large, and m is the global batch divided by micro-batch size and DP degree. Scaling out data parallelism at a fixed global batch reduces m and inflates the bubble, an easily missed interaction. When someone doubles the cluster by adding replicas, the pipeline efficiency can drop even though per-GPU work looks unchanged.
Straggler amplification. Pipelines are tightly coupled. One slow GPU, a thermal-throttled node, or a flaky link delays every downstream and upstream stage in lockstep, and with 1F1B there is no slack to absorb it. Detecting a straggler requires per-stage timing telemetry; a global tokens-per-second number hides which stage is at fault.
Deadlocks and ordering bugs. Point-to-point sends and receives between stages must be posted in a consistent order. A mismatch between sender and receiver ordering, especially with interleaving or custom zero-bubble schedules, produces hangs rather than errors. NCCL send/recv pairs deadlock silently, and the timeout often surfaces as a collective watchdog failure far from the cause.
Checkpointing interplay. Activation recomputation shifts compute from memory to FLOPs: roughly one extra forward per backward, about a 33% compute increase if t_b = 2·t_f. In zero-bubble schedules the recomputation interacts with the B/W split, because recomputed activations must exist when W eventually runs. Mixing selective recomputation (only attention scores, for example) with a deferred W is often the best compromise but needs careful memory accounting. Separately, saving a training checkpoint for fault tolerance requires a consistent snapshot across stages, which a flush at step boundaries provides naturally; asynchronous pipelines such as PipeDream-style weight stashing make this harder.
Numerical gotchas. Gradient clipping needs the global norm, which in a pipeline requires a cross-stage reduction after the last backward. Loss scaling and NaN detection have the same structure. This is exactly the synchronization the zero-bubble paper targets, and bypassing it requires verifying that the skipped check does not change behavior, for instance by rolling back if a late check finds a bad value. Test this on your own workload.
Low-fidelity bubble models. The (p-1)/(m+p-1) formula assumes uniform stages and free communication. Real bubbles are larger. A good internal practice is to compare measured step time against m·(t_f+t_b) per stage, using profiled values, and attribute the gap to bubble, imbalance, and exposed communication separately.
When pipeline parallelism is the wrong tool. For models that fit across one NVLink domain with FSDP or ZeRO-3, adding pipeline stages adds a bubble and complexity for no memory benefit you need. Pipeline parallelism earns its place when inter-node bandwidth, not memory alone, is the binding constraint.
Practical Recommendations
Start from the constraint, not the schedule. Compute m from your real global batch, micro-batch size and DP degree, then evaluate (p-1)/(m+p-1). If it exceeds 15%, you have a schedule problem; if it is below 10%, you likely have a balance or communication problem.
For pipeline parallelism, default to the 1F1B schedule with a flush, because it is the best-supported option and caps activation memory at about p micro-batches. Move to interleaved 1F1B when the bubble is above 15% and you have inter-node bandwidth to spare, starting with v = 2 and measuring before trying v = 4. Consider zero-bubble schedules when you control the framework, can split B and W for your large GEMMs, and have memory headroom. Consider DualPipe-style designs only for large MoE models where all-to-all is exposed, and remember it doubles resident parameters per device.
Profile real per-stage times before fixing a partition. Put fewer layers on the embedding and output stages. Add per-stage timers and alert on stage skew. Test deadlock freedom at small scale with a simulator like the one above before burning cluster hours.
Checklist before a long run:
- Compute the bubble from real m and p; record it as an expected-efficiency baseline.
- Measure per-stage forward, backward and (if split) weight-gradient time; rebalance until the spread is under about 5%.
- Confirm activation memory on stage 0 with headroom of at least 10% for fragmentation.
- Overlap pipeline sends with compute and confirm in a trace.
- Verify the global-norm clipping path and NaN handling under your schedule.
- Run a short soak test with fault injection to confirm stage failures surface as errors, not hangs.
Frequently Asked Questions
What is pipeline parallelism in LLM training?
Pipeline parallelism splits a model by depth across devices: each device (a stage) holds consecutive layers. A batch is divided into micro-batches that stream through the stages, with activations passed forward and gradients backward point-to-point. It reduces per-device memory and tolerates slower inter-node links, but introduces idle time called the pipeline bubble, which schedules like 1F1B, interleaving and zero-bubble try to reduce.
How do you calculate the pipeline bubble?
For a synchronous flush schedule with p stages and m micro-batches, the bubble fraction is (p-1)/(m+p-1), independent of forward and backward times when stages are balanced. With p = 8 and m = 32 it is 7/39, about 17.9%. Interleaving with v chunks per device changes it to (p-1)/(v·m+p-1), while zero-bubble and DualPipe schedules reduce it further by filling or overlapping idle time.
Is 1F1B better than GPipe?
For the bubble, no: both give the same (p-1)/(m+p-1) fraction. For memory, yes. GPipe stores activations for all m micro-batches on each stage, while 1F1B caps in-flight micro-batches at roughly p by starting backward passes early. That lets you raise m to shrink the bubble without running out of memory, which is why 1F1B is the standard in LLM training frameworks.
What is zero-bubble pipeline parallelism?
It is a family of schedules, introduced by Qi et al. (arXiv:2401.10241), that splits the backward pass into an input-gradient part (B) on the critical path and a weight-gradient part (W) that can be deferred. Scheduling W into idle slots, plus bypassing the optimizer-step synchronization, removes the bubble under synchronous semantics. The authors report up to 23% higher throughput than 1F1B at similar memory, and 31% with relaxed memory.
What is DualPipe and when should I use it?
DualPipe is DeepSeek’s bidirectional pipeline schedule from the DeepSeek-V3 report. It feeds micro-batches from both pipeline ends and overlaps forward and backward chunks with MoE all-to-all communication, shrinking the bubble to (PP/2-1)(F&B+B-3W) at the cost of two parameter copies per device. Use it for large mixture-of-experts models with exposed cross-node all-to-all, not typically for small dense models.
How many micro-batches do I need?
As a rule, use m of at least 4 to 8 times the number of stages to keep the bubble under roughly 15 to 20%. With p = 8, m = 32 gives 17.9% and m = 64 gives 9.9%. The real limit is the global batch: m times micro-batch size times data-parallel degree must stay within the batch size your optimization recipe tolerates, which is why interleaving or zero-bubble schedules matter when m is small.
Further Reading
- Data, tensor, pipeline, FSDP and ZeRO parallelism compared, the parent guide to every parallelism axis mentioned here.
- Ring attention and context parallelism for long-context training, a sequence-dimension axis that composes with pipeline stages.
- NVLink vs InfiniBand vs RoCE for NCCL collectives, the network layer that determines where pipeline boundaries should sit.
- Huang et al., GPipe, arXiv:1811.06965.
- Narayanan et al., Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM, arXiv:2104.04473.
- Qi et al., Zero Bubble Pipeline Parallelism, arXiv:2401.10241.
- DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv:2412.19437, and the DualPipe repository.
By Riju — about
