NCCL All-Reduce and GPU Cluster Network Design: NVLink vs InfiniBand vs RoCE

A training job on thousands of GPUs does not fail because the matrix multiplications are slow. It stalls because every few hundred milliseconds all of those GPUs must agree on a sum, and the slowest link in the building sets the pace. The operation that performs that agreement is the NCCL all-reduce, and the network underneath it, NVLink inside the rack and InfiniBand or RoCE between racks, decides whether your expensive accelerators spend their time computing or waiting.

This matters more in 2026 than it did two years ago. Scale-up domains have grown from eight GPUs to seventy-two, Ethernet is mounting a credible challenge to InfiniBand for scale-out, and a single link flap or misconfigured priority class can idle a cluster that costs more per hour than most engineering teams cost per year. You will leave this post with a cost model you can compute by hand, a mental map of which NCCL algorithm runs where, a decision framework for fabric choice, and a debugging playbook you can run tonight.

What this covers: the alpha-beta cost model for ring, tree and NVLS collectives with worked numbers, NVLink and NVSwitch domains, scale-up versus scale-out, InfiniBand versus RoCEv2 congestion control, rail-optimized topologies, Ultra Ethernet status, NCCL tuning, and the failure modes that actually take clusters down.

Context and Background

Data-parallel and sharded training both reduce to a handful of collective operations. In classic data parallelism every GPU holds a full model replica, computes gradients on its own micro-batch, and then runs an all-reduce so that every replica ends up with the same averaged gradient. In sharded schemes such as FSDP and ZeRO, the same traffic is split into a reduce-scatter on the backward pass and an all-gather on the forward pass. Tensor parallelism adds all-reduces inside every transformer layer, and mixture-of-experts models add all-to-all exchanges for token routing, a pattern we covered in our mixture-of-experts architecture deep dive.

The NVIDIA Collective Communications Library (NCCL) is the de facto implementation of these collectives for NVIDIA GPUs. PyTorch’s torch.distributed uses it as the default GPU backend, so most practitioners meet NCCL through dist.all_reduce without ever reading its source. Underneath, NCCL discovers the hardware topology, builds rings and trees across GPUs and NICs, picks an algorithm and a protocol per message size, and launches CUDA kernels that move data over NVLink, PCIe, InfiniBand verbs, or plain sockets. The NCCL 2.19-era behavior is analyzed in the academic paper Demystifying NCCL, which is the best public walk-through of the protocols and algorithms and which we cite throughout.

Three physical layers matter. The first is the scale-up fabric: NVLink and NVSwitch, which connect GPUs with memory-semantics load/store access at hundreds of gigabytes per second per GPU. The second is the scale-out fabric: one NIC per GPU (commonly 400 Gb/s class) connecting through leaf and spine switches using InfiniBand or RDMA over Converged Ethernet version 2 (RoCEv2). The third is the host path: PCIe, GPUDirect RDMA, and CPU NUMA placement, which quietly caps performance whenever it is misconfigured. If you want the protocol-level comparison of the scale-out options, our earlier piece on RDMA, RoCE v2, InfiniBand and Ultra Ethernet is the companion to this one; here we focus on how NCCL actually uses those fabrics.

The thesis of this post is simple and slightly contrarian: network design for training is collective design. Teams buy fabric by headline bisection bandwidth, but NCCL does not consume bisection bandwidth. It consumes a specific pattern of neighbor-to-neighbor flows, and those flows are either well matched to the topology or they are not. Matching the pattern, with rails, with hierarchical algorithms, and with congestion control that tolerates synchronized bursts, is worth more than another tier of switches.

How NCCL All-Reduce Really Works

An all-reduce takes one tensor of n bytes on each of p ranks and leaves every rank with the elementwise sum. The straightforward implementation, send everything to one rank, sum, and broadcast back, funnels all traffic through a single link and scales terribly. NCCL instead uses bandwidth-optimal schedules in which every link carries roughly the same amount of data at the same time.

Direct answer: NCCL all-reduce chooses between ring, tree and in-network (NVLS or CollNet) algorithms per message size. Ring minimizes bytes moved per link and wins for large messages. Tree minimizes the number of sequential hops and wins for small, latency-bound messages. NVLS offloads the reduction into NVSwitch hardware where available.

Figure 1: Ring all-reduce decomposes into a reduce-scatter and an all-gather; each phase takes p minus 1 steps, which produces both the latency and bandwidth terms of the cost model.

The diagram shows the decomposition. In the reduce-scatter phase the tensor is cut into p chunks and each rank repeatedly sends one chunk to its ring neighbor while receiving and accumulating another. After p minus 1 steps each rank owns the fully reduced version of exactly one chunk. The all-gather phase then circulates those reduced chunks for another p minus 1 steps until every rank holds all of them. The Demystifying NCCL paper gives the exact NCCL step count for the fused kernel: a ring all-reduce loop iteration takes 2k minus 1 steps for k GPUs, built from send, recvReduceSend, recvReduceCopySend, recvCopySend and recv primitives.

The alpha-beta cost model

The standard way to reason about a collective is the alpha-beta model. Sending m bytes across one link costs alpha (a fixed per-message latency) plus m divided by B, where B is the link bandwidth. Alpha captures software launch cost, NIC processing, and switch hops; B captures the wire.

For a ring all-reduce over p ranks with n bytes each, there are 2(p minus 1) steps, and each step moves n/p bytes. The total time is therefore:

T_ring = 2(p − 1)·α + 2·((p − 1)/p)·(n/B)

Two properties follow immediately. First, the bandwidth term approaches 2n/B as p grows: each GPU sends and receives roughly twice its tensor size regardless of cluster size, which is why ring is called bandwidth-optimal. Second, the latency term grows linearly with p. At a thousand ranks that term is two thousand alpha values, and it is the reason a pure ring collapses on small messages at scale.

A tree has the opposite profile. In a tree reduce followed by a tree broadcast, data climbs log2(p) levels and descends log2(p) levels, so the latency term is roughly 2·log2(p)·α. A naive single tree, however, leaves leaf links idle during half the operation and concentrates traffic on interior nodes. NCCL therefore uses a double binary tree: two complementary trees in which, per the paper, no node is a non-leaf in both trees, so every link is busy in both directions. With pipelining the bandwidth term approaches that of the ring, giving approximately:

T_tree ≈ 2·log2(p)·α + 2·(n/B)

The constants differ in practice because pipelining is imperfect, but the shape is reliable: logarithmic latency, near-ring bandwidth for large messages.

Worked numbers

The following figures are illustrative, derived from the formulas above with assumed inputs, not measurements. Assume p = 1,024 ranks, B = 50 GB/s per rank (a 400 Gb/s NIC), and α = 5 µs per step as a round figure for a multi-hop fabric.

For a 1 GiB gradient bucket, the ring bandwidth term is about 2 × 0.999 × 1.07 GB / 50 GB/s ≈ 43 ms, and the latency term is 2 × 1,023 × 5 µs ≈ 10.2 ms. Total: roughly 53 ms, and 81 percent of that is the wire. The tree gives about 100 µs of latency plus roughly 43 ms of bandwidth, so the two are within about 20 percent of each other and bandwidth dominates either way.

For a 1 MiB message the picture inverts. The bandwidth term is only about 42 µs. The ring latency term is still 10.2 ms, which is roughly 240 times the wire time, while the tree’s latency term is about 100 µs. Here the tree is around 70 times faster. This is exactly why NCCL’s tuner switches algorithms by message size, and why a model with thousands of tiny all-reduces (many small parameter tensors, no bucketing) performs far worse than the same bytes in a few large buckets.

A caveat on the example: real NCCL rings are hierarchical. Hops inside a node over NVLink have an alpha closer to a microsecond than five, and only the inter-node hops pay the network cost, so the effective latency term is smaller than 2(p − 1) × 5 µs. The qualitative conclusion, that rings lose on small messages at scale, holds.

Protocols: Simple, LL and LL128

Orthogonal to the algorithm is the wire protocol. According to the Demystifying NCCL paper, NCCL has three protocols that matter. Simple moves large chunks and synchronizes with memory fences; it reaches near-peak bandwidth but pays about 6 µs of per-hop latency in the paper’s table. LL (low latency) packs 4 bytes of data with a 4-byte flag into 8-byte atomic stores so the receiver can poll for arrival without a fence; it has about 1 µs per hop but delivers only 25 to 50 percent of peak bandwidth, and it stages through host memory, which rules out GPUDirect RDMA. LL128 sends 120 bytes of data plus an 8-byte flag in 128-byte units, reaching roughly 95 percent of peak at about 2 µs per hop, but it requires hardware that guarantees atomic 128-byte writes and NCCL disables it elsewhere.

The paper’s own measurements on 16 GH200 nodes (150 GB/s intra-node, 25 GB/s per direction inter-node) found LL and LL128 best below about 64 KiB, Simple best for multi-gigabyte all-reduce, ring best for large messages and tree best for small ones. The latency values in its table are quoted without a stated measurement method, so treat them as orders of magnitude.

NVLS and in-network reduction

On NVSwitch-based systems NCCL can use NVLS (NVLink SHARP), which moves the summation into the switch: each GPU writes its data once to a multicast address, the switch reduces, and results come back, so each GPU sends about n rather than 2n bytes within the domain. NVIDIA’s multi-node tuning guide notes that the NVLink Switch generation used in GB200 systems includes SHARP support including FP8. The paper lists NVLS and NVLS Tree, together with CollNet Direct and CollNet Chain, as supporting only the Simple protocol for all-reduce in the 2.19 era. CollNet is the analogous idea for the scale-out fabric, offloading to InfiniBand SHARP-capable switches. I could not verify exact current NVLS bandwidth multipliers for 2026 software releases, so check the NCCL release notes for your version before you quote a number to your finance team.

NVLink, NVSwitch and the Scale-Up Domain

NVLink is not a network in the Ethernet sense. It is a memory fabric: a GPU can load from or store to a peer’s HBM across the link, with the peer-to-peer addressing handled by the hardware. That is why NCCL kernels inside a node look like CUDA kernels copying between buffers rather than like NIC verbs. NVSwitch chips make the topology fully connected, so any GPU reaches any other at full per-GPU bandwidth.

Generation numbers worth knowing

NVIDIA’s documentation states that fifth-generation NVLink provides 1.8 TB/s per GPU, double the 900 GB/s of the previous generation on an HGX H200 baseboard. More importantly for architecture, the NVLink domain grew from at most eight GPUs on an HGX baseboard to up to 72 Blackwell GPUs in a GB200 NVL72 rack, with the NVLink Switch delivering a stated 130 TB/s of aggregate GPU bandwidth in that 72-GPU domain. The GB200 Superchip itself joins two Blackwell GPUs and one Grace CPU over 900 GB/s NVLink-C2C. I am quoting these as NVIDIA marketing-document figures: they are aggregate bidirectional numbers, and real all-reduce bus bandwidth will land below them.

The practical consequence is the ratio. A 400 Gb/s NIC delivers 50 GB/s per direction. Against 900 GB/s per direction from NVLink 5 (half of the 1.8 TB/s total, assuming the figure is bidirectional aggregate, which is NVIDIA’s usual convention), the scale-up fabric is roughly 18 times faster per GPU than the scale-out link. Any parallelism strategy that puts chatty collectives on the scale-out fabric pays for that ratio.

Scale-up versus scale-out as a design axis

Figure 2: A scale-up NVLink domain behind NVSwitch, with one NIC per GPU attached to a rail-aligned leaf switch and a shared spine layer.

The diagram shows the two tiers and how a flow crosses them. Inside the domain, GPUs reach each other through NVSwitch. To leave the domain, traffic exits through a NIC, travels the rail-aligned leaf and spine fabric, and re-enters a remote domain through the matching NIC. Because crossing the boundary costs an order of magnitude in bandwidth, the single most valuable placement rule in large-scale training is: put the communication-heavy parallelism inside the scale-up domain.

Tensor parallelism issues several all-reduces per layer, on the critical path, with no opportunity to overlap. It belongs inside NVLink. Expert parallelism’s all-to-all is similarly bandwidth hungry and benefits from the domain size growth that NVL72 brought. Data-parallel gradient reduction, by contrast, is large, infrequent, and overlappable with the backward pass, so it tolerates the scale-out fabric well. Pipeline parallelism sends point-to-point activations once per micro-batch and is the most network-tolerant of all.

The NVL72 generation therefore changes the calculus: with 72 GPUs per domain, tensor and expert parallel degrees that previously forced traffic onto the network can stay on NVLink. It also changes the failure domain. A rack is now one scheduling and fault unit, and losing one GPU or one switch tray can remove a seventy-two-GPU block from a job.

How NCCL maps a job onto the hardware

At init, NCCL probes the topology (NVLink connectivity, PCIe tree, NIC locations, NUMA affinity) and builds a graph. It then searches for rings and trees that maximize bandwidth, replicating them across multiple channels. Per the paper, each channel maps to one CUDA thread block, and the kernel grid is sized by the number of channels, so more channels means more SMs spent on communication. This is the hidden cost of collective bandwidth: NCCL kernels compete with your compute kernels for streaming multiprocessors, and aggressive overlap trades a few percent of compute for hiding the communication.

Across nodes, NCCL prefers to keep GPU i on node A talking to GPU i on node B through the same NIC. This is the “rail” behavior, and it is controlled by NCCL_CROSS_NIC; the documentation describes value 0 as keeping the same NIC (suited to rail-optimized networks), 1 as allowing different NICs (suited to networks where all NICs share a switch), and 2 as the default that prefers the same NIC but allows others when faster. If your fabric is rail-optimized and your NCCL is not honoring it, intra-rail traffic gets pushed through the spine and your effective bandwidth drops.

InfiniBand vs RoCEv2 for Scale-Out Training

Once traffic leaves the NVLink domain, two fabrics dominate: InfiniBand and RoCEv2. Both give NCCL the same verbs-level interface (reliable connected queue pairs, RDMA writes, GPUDirect RDMA), so NCCL’s algorithms are identical. What differs is how the fabric behaves when many ranks transmit at once, which is exactly what an all-reduce does.

Why collectives are hard on networks

An all-reduce is a synchronized, low-entropy workload. Thousands of NICs start transmitting at nearly the same instant, each with a small number of long-lived flows, and all of them are bulk transfers of identical size. Traditional Ethernet load balancing hashes flows to paths using the five-tuple; with few flows and identical sizes, hash collisions are not rare statistical noise, they are the common case. Two large flows landing on one uplink halve their bandwidth, and because the collective finishes only when the slowest flow finishes, the whole step slows to the speed of the unluckiest path.

This is the central reason scale-out AI fabrics differ from general data center networks. They need either per-packet load balancing, as in InfiniBand adaptive routing and in the packet spraying that Ultra Ethernet specifies, or a topology and flow-placement discipline that avoids collisions by construction, as rail-optimized designs do.

InfiniBand: lossless by construction

InfiniBand is credit-based at the link layer: a sender transmits only when the receiver has advertised buffer credits, so packets are not dropped for congestion. This gives the transport simple, predictable behavior, and RC (reliable connection) retransmission rarely fires. InfiniBand switches also support adaptive routing, in which a packet can take a different output port based on queue state, and in-network reduction via SHARP, which NCCL exposes as CollNet. A centralized subnet manager configures forwarding and can program a topology-aware routing function.

The trade-offs are commercial and operational rather than technical. InfiniBand is effectively a single-vendor ecosystem for switches and NICs in the AI market, its management plane (the subnet manager) is a separate system from your IP network tooling, and your network team likely knows Ethernet better. Credit-based flow control can also propagate back-pressure: a slow receiver can slow upstream links, a behavior known as congestion spreading, although the effect is generally better contained than Ethernet PFC storms.

RoCEv2: Ethernet with RDMA and a congestion-control burden

RoCEv2 carries InfiniBand transport semantics inside UDP/IP over standard Ethernet, so it routes across layer 3 and runs on commodity switch silicon. The cost is that Ethernet is lossy by default, and the RC transport handles loss poorly: a dropped packet typically triggers retransmission at a coarse granularity, which stalls the flow for far longer than a TCP-style recovery would. Production RoCE deployments therefore make the network nearly lossless using three mechanisms working together.

Priority Flow Control (PFC), IEEE 802.1Qbb, pauses a single traffic class on a link when a receive buffer fills, preventing drops. Explicit Congestion Notification (ECN), RFC 3168, lets switches mark packets with the Congestion Experienced bit before queues overflow. DCQCN, the congestion-control algorithm introduced by Microsoft and Mellanox researchers in a 2015 SIGCOMM paper, runs on the NIC: the receiver reflects ECN marks back to the sender in congestion notification packets and the sender cuts its rate, then recovers in stages. The design intent is that ECN throttles senders early so that PFC, the blunt instrument, almost never triggers.

Figure 3: RoCEv2 congestion path. ECN marks drive DCQCN rate cuts at the sender; if queues keep growing, PFC pauses the upstream hop and can spread into a PFC storm.

The figure traces the two feedback loops. The ECN loop is end to end and takes a round trip to act. The PFC loop is hop by hop and acts in microseconds, which is why it saves you from drops and also why it is dangerous: a pause on one hop stalls the upstream hop, which fills its own buffers and pauses the next, and the stall can spread across the fabric. If ECN thresholds are set too high relative to the PFC threshold, PFC fires before ECN has any effect, and you get a lossless but unstable network.

A decision matrix

Concern	InfiniBand (NDR class)	RoCEv2 (400G Ethernet)	Ultra Ethernet (emerging)
Loss handling	Credit-based, lossless by design	PFC + ECN, must be tuned	Designed not to need a lossless network
Load balancing	Adaptive routing in switch	ECMP hashing, optional per-packet features by vendor	Packet spraying with NIC-side reordering
In-network reduction	SHARP (CollNet)	Generally not available	Not in the 1.0 transport scope per sources I found
Operations skill set	Subnet manager, IB tooling	Standard IP and Ethernet tooling	Standard Ethernet plus new NICs
Vendor choice	Narrow	Broad	Broad, in principle
Maturity at 10k+ GPUs	Very high	High at hyperscalers, needs expertise	Early

Two notes on this matrix. The ECMP and packet-spraying entries describe the general design of each technology; the specific switch features available depend on the vendor and silicon generation, so verify against the datasheet. And the SHARP row reflects that SHARP is an InfiniBand switch feature: Ethernet fabrics generally lack an equivalent that NCCL can use, which matters for very large small-message all-reduces.

My view, labeled as opinion: for a team building its first cluster under roughly a thousand GPUs, InfiniBand is the lower-risk choice because the fabric works well out of the box. For hyperscale operators with deep network engineering teams, a mature Ethernet RoCE fabric avoids lock-in and reuses operations tooling. The middle ground is where RoCE projects fail, because the tuning effort is real and underestimated.

Ultra Ethernet Consortium status

The Ultra Ethernet Consortium published its 1.0 specification in June 2025, according to contemporaneous reports such as Network World’s coverage. The reported scope is a new Ultra Ethernet Transport (UET) protocol, a modern RDMA approach, congestion control that does not rely on a lossless network, a receiver-driven mode where the receiver can limit senders, and switch-level packet spraying with reordering handled at the NIC. HPE was quoted as saying that approximately 75 percent of UET draws from the HPE Slingshot transport. That last figure is a vendor statement and I have not independently verified it.

What I could not verify for this post is the state of shipping UEC-compliant NICs and switches in late 2026, or any independent benchmark comparing UET to RoCEv2 at scale. Treat UEC as a strong architectural direction: it directly addresses the two weaknesses above, hash collisions through spraying and PFC fragility through lossless-optional congestion control. It is not yet something to bet a delivery schedule on without vendor evidence in hand. For the protocol-level history, see our RDMA and Ultra Ethernet comparison.

Topology: Rail-Optimized Fat-Trees and Oversubscription

A fat-tree (Clos) network built from identical switches gives full bisection bandwidth in principle. A rail-optimized variant exploits the fact that each server has one NIC per GPU. NIC k of every server in a pod connects to the same leaf switch (rail k), so GPU k on any server reaches GPU k on any other server in one leaf hop, without touching the spine.

Why does this matter for NCCL? Because the hierarchical algorithms NCCL builds map neatly onto rails. In a typical multi-node all-reduce, GPUs first reduce inside each node over NVLink; then GPU k from every node participates in an inter-node ring or tree over rail k; finally results fan back out over NVLink. Each rail carries 1/8th of the data (for an eight-GPU node) in parallel, and the rail’s traffic stays within one leaf switch for same-pod communication. This is the traffic-pattern/topology match mentioned in the introduction: the fabric is built for the exact shape of the flows.

Sizing and oversubscription

A two-tier leaf-spine fabric built with 64-port switches supports at most 64 × 32 = 2,048 downlinks at 1:1 subscription (32 down, 32 up per leaf, 64 leaves), a standard fat-tree identity. Beyond that you need a third tier or fewer ports per endpoint, which adds hops and therefore latency alpha. This is simple arithmetic from the switch radix, not a vendor claim, though real switch port counts and breakout configurations vary by product.

Oversubscription is tempting for cost reasons and corrosive for collectives. Because all ranks in a ring-based all-reduce transmit simultaneously, a 2:1 oversubscribed spine does not average out the way mixed web traffic does; during the synchronized burst it delivers half the bandwidth for exactly the phase you care about. A defensible compromise is 1:1 at the leaf-to-spine tier for the training pod and oversubscription only toward storage or the datacenter edge, where traffic is statistical rather than synchronized.

Beyond rails: job placement matters

Even with a perfect fabric, scheduler placement can destroy rail alignment. If a job’s 64 nodes are scattered across leaf groups, ring neighbors may need to traverse the spine for every hop. Topology-aware scheduling (Slurm’s topology plugin, Kubernetes topology-spread constraints and the DRA-based GPU allocation described in our Dynamic Resource Allocation guide) keeps a job within as few leaf groups as possible. A fabric diagram does not buy you anything if the scheduler ignores it.

Tuning and Debugging NCCL in Practice

The first rule of NCCL tuning is to measure with the official benchmark before touching variables. The nccl-tests repository from NVIDIA builds all_reduce_perf, which reports both algorithm bandwidth and bus bandwidth. Bus bandwidth normalizes by the collective’s data-movement factor (for all-reduce, 2(p − 1)/p) so that the number is comparable to hardware link speed regardless of rank count. Compare it to your expected per-GPU NIC or NVLink rate; if it reaches a large fraction of that rate for big messages, the fabric and library are healthy.

# Build once
git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests && make MPI=1 MPI_HOME=/usr/lib/x86_64-linux-gnu/openmpi

# Sweep 8 B to 8 GiB, doubling, across 2 nodes x 8 GPUs, one rank per GPU
mpirun -np 16 -H nodeA:8,nodeB:8 \
  -x NCCL_DEBUG=INFO -x NCCL_DEBUG_SUBSYS=INIT,NET \
  ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

Read two things from the output. The bus bandwidth column at large sizes tells you whether you reach line rate. The NCCL INFO lines tell you which transport NCCL chose: the lines naming NET/IB mean RDMA verbs, while NET/Socket means NCCL fell back to TCP, which is the classic cause of “my 400G cluster performs like 10G”.

The environment variables that matter

The NCCL documentation defines a large set of variables; a small subset solves most real problems. NCCL_DEBUG accepts VERSION, WARN, INFO and TRACE (newer releases add ATTN). NCCL_SOCKET_IFNAME filters which IP interfaces are used for bootstrap and socket transport, accepting comma-separated prefixes with ^ to exclude and = for exact matches. NCCL_IB_HCA selects which RDMA devices and ports to use, in the form hca[:port[:rail[:plane]]], with ^ to exclude and = for exact match. For RoCE, NCCL_IB_GID_INDEX selects the GID (the documentation states the default became AUTO in recent releases, and older releases required you to set it explicitly to the RoCEv2 GID), and NCCL_IB_TC sets the traffic class so RDMA traffic lands in the lossless priority your switches expect. NCCL_CROSS_NIC controls rail behavior as discussed above.

For algorithm experiments, NCCL_ALGO and NCCL_PROTO restrict the tuner’s choices (for example forcing Tree or Ring), which is useful to prove that a regression is algorithm-specific. I did not verify the full current list of accepted values for each on the version you run, so confirm against your installed version’s documentation. Do not set them permanently in production unless a benchmark proves a gain; the built-in tuner is usually right, and pinned choices go stale when hardware changes.

A minimal torch.distributed check

import os, time, torch, torch.distributed as dist

def main():
    dist.init_process_group(backend="nccl")
    rank = dist.get_rank()
    torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
    n_bytes = 1 << 30                       # 1 GiB payload
    x = torch.ones(n_bytes // 4, device="cuda", dtype=torch.float32)

    for _ in range(5):                      # warm-up builds NCCL comms
        dist.all_reduce(x)
    torch.cuda.synchronize()

    iters = 20
    t0 = time.time()
    for _ in range(iters):
        dist.all_reduce(x)
    torch.cuda.synchronize()
    dt = (time.time() - t0) / iters

    p = dist.get_world_size()
    algbw = n_bytes / dt / 1e9
    busbw = algbw * 2 * (p - 1) / p         # same normalization as nccl-tests
    if rank == 0:
        print(f"ranks={p} time={dt*1e3:.1f} ms algbw={algbw:.1f} GB/s busbw={busbw:.1f} GB/s")

if __name__ == "__main__":
    main()
# Launch: torchrun --nnodes=2 --nproc_per_node=8 --rdzv_backend=c10d --rdzv_endpoint=nodeA:29500 check.py

This reproduces the nccl-tests normalization inside your actual training environment, which catches problems that the benchmark does not, such as a container image with a mismatched NCCL build or a missing RDMA device in the pod spec. The warm-up matters: the first call builds communicators and exchanges topology, and including it in the timing misleads.

Figure 4: A debugging flow. Establish a bus bandwidth baseline, confirm the transport, then bisect across nodes to isolate a slow rank.

Reading the INFO log

When the log prints the channel and ring or tree construction, look at three things. Confirm the chosen transport per connection. Confirm the number of channels and that the ring order follows the physical rails (a ring that zigzags across NICs suggests topology detection failed, often because of a virtualized PCIe topology in a VM or container without the right NCCL topology file). And confirm GPUDirect RDMA is in use rather than staging through host memory, which shows up as a GDR indicator in the connection lines on supported systems. Any one of these being wrong can cost you half of the achievable bandwidth without a single error message.

Trade-offs, Gotchas, and What Goes Wrong

The headline failures in large training clusters are less exotic than the technology suggests. Most are one of five patterns.

Stragglers. A collective completes at the speed of its slowest participant. One GPU thermally throttling, one NIC negotiated to a lower speed, or one node running an unrelated background process delays every rank at every step. The symptom is a healthy-looking cluster with mysteriously low utilization and a step time that varies between runs. The fix is per-node isolation: run nccl-tests on pairs of nodes, compare bus bandwidth, and drain the outlier. Continuous health checks before job launch are cheaper than a lost day.

Link flaps and optics. At 400G and above, optical transceivers and cables fail often enough, and degrade gracefully enough, that a flapping link is a normal operational event in a large cluster. On InfiniBand the subnet manager reroutes, causing a pause. On RoCE, a flap can trigger RC retransmission timeouts that stall a QP for a long time; NCCL_IB_TIMEOUT (documented range 0 to 31, default 20) controls the verbs timeout, and raising it trades slower failure detection for fewer spurious aborts. Know your failure budget: at tens of thousands of links, daily link events are statistically expected, which is why checkpointing cadence and fast job restart are part of network design, as covered in our post on spot GPU training and fault-tolerant checkpointing.

PFC storms and head-of-line blocking. As Figure 3 showed, pause frames propagate. A misbehaving NIC that stops draining its receive buffer, a PFC watchdog that is disabled, or a lossless priority shared with a non-RDMA workload can freeze large parts of the fabric. Mitigations are to isolate RDMA onto a dedicated priority, enable PFC watchdogs, set ECN thresholds well below PFC thresholds so DCQCN acts first, and keep the lossless domain small.

Silent fallback to sockets. If the RDMA device is not visible in the container, or NCCL_IB_HCA matches nothing, NCCL does not crash; it quietly uses TCP. Always grep the INFO log for the transport in CI for your training image.

Over-tuning. Copying a long list of NCCL_* variables from a blog post written for another fabric is a reliable way to lose performance. Variables that help on one topology (disabling LL128, pinning algorithms, changing QP counts) hurt on another. Change one variable, re-run the benchmark, keep the change only if bus bandwidth improves.

There is also a conceptual trap: chasing peak all-reduce bandwidth when the real issue is overlap. If your framework does not launch gradient all-reduces during the backward pass, or buckets are too small or too large, the best fabric in the world sits idle while compute runs and then becomes the bottleneck at the end of the step. Profile the timeline before you buy bandwidth.

Practical Recommendations

Start with placement, not hardware. Keep tensor and expert parallelism inside the NVLink domain, let data parallelism and pipeline stages use the scale-out fabric, and use a topology-aware scheduler so a job’s nodes sit in as few leaf groups as possible. These choices are free and often worth more than a fabric upgrade.

Choose the scale-out fabric by team capability. If your network engineers have not run lossless RoCE before, buy InfiniBand or accept a long tuning period. If you do go Ethernet, budget for PFC watchdogs, ECN tuning, per-priority isolation and a monitoring stack that exposes pause counts, ECN marks and per-queue depth. Evaluate Ultra Ethernet gear against measured collective performance on your own workload before committing.

Build the measurement habit early. Run nccl-tests as an acceptance test on every new rack, as a pre-flight check on each job allocation, and as a regression test when you upgrade drivers, NCCL, or switch firmware. A bus-bandwidth number from last month is the fastest way to tell a hardware problem from a software one.

Pin the NCCL version in the training image and record it with every run.
Verify NET/IB (or the intended RDMA transport) in the INFO log in CI.
Keep 1:1 subscription between leaf and spine for the training pod.
Align NCCL channels to rails; test NCCL_CROSS_NIC only if topology detection looks wrong.
Enable PFC watchdog and set ECN thresholds below PFC thresholds on RoCE fabrics.
Bucket gradients so all-reduces are large enough to be bandwidth-bound.
Drain nodes whose pairwise bus bandwidth is an outlier before launching.
Checkpoint on a cadence matched to your measured mean time between link events.

Frequently Asked Questions

What is NCCL all-reduce?

NCCL all-reduce is the NVIDIA Collective Communications Library operation that sums (or otherwise reduces) a tensor held by every GPU and leaves the identical result on all of them. It is the core gradient-synchronization step in data-parallel training. NCCL selects an algorithm (ring, tree, NVLS or CollNet) and protocol (Simple, LL, LL128) based on message size and topology, and runs it as CUDA kernels over NVLink, PCIe, or the network.

Is ring all-reduce or tree all-reduce faster?

It depends on message size and scale. Ring moves the minimum bytes per link, about twice the tensor size per GPU, so it wins for large messages. Its latency grows linearly with rank count, so at scale it loses on small messages to tree, whose latency grows logarithmically. NCCL’s tuner picks between them automatically; the Demystifying NCCL measurements found ring best for large messages and tree best for small ones.

What is the difference between NVLink and InfiniBand?

NVLink is a scale-up, memory-semantics fabric connecting GPUs inside a server or rack at up to 1.8 TB/s per GPU on fifth generation (NVIDIA’s figure), within domains of up to 72 GPUs. InfiniBand is a scale-out network connecting servers through NICs at roughly 400 Gb/s per port. NVLink is far faster per GPU but physically limited in reach; InfiniBand or Ethernet extends the cluster beyond the domain.

Is RoCE as good as InfiniBand for AI training?

It can be, at a cost in tuning. RoCEv2 runs RDMA over Ethernet and needs PFC, ECN and DCQCN configured carefully to avoid drops without causing pause storms. Large operators run it successfully at scale. InfiniBand is lossless by design and has adaptive routing and SHARP, so it works well with less effort. The decision is largely about your team’s network expertise and vendor strategy.

What is bus bandwidth in nccl-tests?

Bus bandwidth is the measured algorithm bandwidth multiplied by a collective-specific factor, 2(p − 1)/p for all-reduce, so that it reflects the data actually crossing each link. It lets you compare results across different rank counts and against the hardware limit directly. If large-message bus bandwidth is a healthy fraction of your NIC or NVLink rate, the fabric and library are working properly.

Why is my multi-node NCCL job slow?

The usual causes are silent fallback to TCP sockets, a straggler node, mismatched rail or NIC selection, oversubscribed or hash-colliding paths, and RoCE congestion settings. Run nccl-tests to get a baseline, set NCCL_DEBUG=INFO to confirm the transport and channel layout, then test node pairs to isolate outliers. Only after those checks should you experiment with algorithm or protocol overrides.

NVLink vs InfiniBand vs RoCE: NCCL All-Reduce, Collectives and GPU Cluster Network Design for AI Training (2026)