RDMA RoCE InfiniBand for AI Clusters: PFC, ECN, DCQCN and Ultra Ethernet

A training job on a thousand GPUs is a synchronized program. Every few hundred milliseconds all ranks stop computing and exchange gradients, and the step does not finish until the slowest transfer lands. That makes the network a tail-latency machine: one congested link, one dropped packet that triggers a retransmission timeout, or one pause storm can idle hundreds of accelerators that cost several dollars an hour each. The transport that carries those bytes is almost always RDMA RoCE InfiniBand, a family of remote direct memory access technologies that bypass the kernel and move memory between hosts with almost no CPU involvement.

The hard question in 2026 is not whether to use RDMA but which fabric to run it on: native InfiniBand with credit-based flow control, RoCE v2 on a carefully tuned lossless Ethernet, or the new Ultra Ethernet transport that abandons the lossless assumption altogether. This post explains how each works at the mechanism level, where each one fails, how to do the bandwidth and sizing arithmetic, and how to choose.

What this covers: verbs and queue pairs, InfiniBand credits versus Ethernet PFC, ECN and DCQCN, PFC storms and deadlock, rail-optimized topologies, NCCL bandwidth math, Ultra Ethernet 1.0 features, a sizing example, and a decision matrix.

Context and Background

Datacenter TCP was designed for a world where the kernel, the NIC and the application each copy data and each take interrupts. At 400 Gb/s a core has roughly 0.16 microseconds per 8 KB of payload if it is to keep up, which no kernel stack can sustain per flow. RDMA solves this by letting the NIC read and write registered application memory directly, with the transport state machine implemented in hardware. The 2015 SIGCOMM paper on DCQCN from Microsoft and Mellanox framed the motivation precisely: applications want 40 Gb/s-class throughput, very low per-hop latency and little CPU overhead, and TCP/IP stacks cannot deliver all three, while RDMA can (Zhu et al., SIGCOMM 2015).

Three families coexist today. InfiniBand (IB) is a complete, purpose-built network: its own link layer with credit-based flow control, its own routing (subnet manager, LIDs), and a transport layer with reliable connections. RoCE v2 (RDMA over Converged Ethernet, version 2) carries the InfiniBand transport inside UDP over IP, using destination port 4791, so it routes across Layer 3 fabrics and rides on commodity Ethernet switches. iWARP layers RDMA on TCP and has largely faded from AI clusters. A fourth entrant, Ultra Ethernet from the Ultra Ethernet Consortium (UEC), defines a new transport called UET that keeps Ethernet and IP but redesigns the layers above them for AI and HPC.

Why the fabric matters more for AI than for general cloud traffic comes down to flow character. Collective operations create few, very large, long-lived flows with synchronized start times. Classic ECMP load balancing hashes a flow’s five-tuple to one path, and with a handful of elephant flows the hash collides: two flows share a link while another sits idle. We covered the downstream effect on serving in NVIDIA Dynamo and LMCache disaggregated serving, where KV-cache transfers across nodes ride the same RDMA fabrics, and the training-side cost of stalls in spot GPU training and checkpointing.

The thesis of this post is simple: the lossless-versus-lossy decision is the central design choice, and everything else, from buffer headroom to topology to collective algorithm, follows from it. InfiniBand makes the network lossless by construction. RoCE v2 makes it lossless by configuration, which is where the operational pain lives. Ultra Ethernet makes loss cheap enough that you stop needing lossless.

RDMA Fundamentals: Verbs, Queue Pairs and Why the Kernel Steps Aside

Before comparing fabrics, we need the common programming model, because it explains both the speed and the fragility of RDMA.

Verbs, memory registration and zero copy

The RDMA API is called verbs (libibverbs on Linux). An application first registers a memory region, which pins the pages and gives the NIC a translation plus two keys: a local key (lkey) and a remote key (rkey). The rkey is what a remote peer presents to read or write that memory. Registration is expensive, so frameworks register large buffers once and reuse them. This is why NCCL allocates and registers staging buffers at communicator creation rather than per collective.

Once memory is registered, the NIC performs zero copy: data moves from application memory on host A, across the wire, directly into application memory on host B, with no intermediate kernel buffers. With GPUDirect RDMA, the NIC can DMA directly to and from GPU memory over PCIe, skipping host memory as well. That removes a bounce copy that would otherwise consume host memory bandwidth and add latency.

Queue pairs, work requests and completions

Communication happens through queue pairs (QPs): a send queue and a receive queue, plus completion queues (CQs) for results. The application posts work requests (WRs) to a queue, rings a doorbell register in the NIC from user space, and later polls a CQ for completions. No system call sits on the data path. This is kernel bypass: the kernel sets up resources, then steps out.

Operations come in two flavors. Two-sided operations, send and receive, require the receiver to pre-post a buffer. One-sided operations, RDMA write, RDMA read and atomics, name the remote address and rkey and complete with no involvement from the remote CPU. NCCL’s network plugin uses RDMA write with immediate data heavily, because the immediate value notifies the receiver without an extra message.

IB defines several QP transport types. Reliable Connection (RC) provides in-order, acknowledged delivery with hardware retransmission and is what NCCL uses by default over IB and RoCE. Unreliable Datagram (UD) is connectionless and unacknowledged. Reliable Connection is also the root of a scaling and failure issue: each QP carries per-connection state on the NIC, and the RC protocol assumes in-order delivery.

The go-back-N consequence

Classic RC recovery is go-back-N: when a packet is lost or arrives out of order, the receiver NAKs and the sender retransmits from the lost packet onward. This is cheap if loss is essentially zero and ruinous if it is not. A 1 percent loss rate can collapse throughput because each loss resends a window of in-flight data. That one property, an RC transport that tolerates neither loss nor reordering, is the reason both IB and RoCE v2 go to great lengths to be lossless, and the reason Ultra Ethernet had to redesign the transport rather than merely tune the network.

It also means packet spraying, the obvious cure for ECMP collisions, is hard on classic RC: spraying one flow across all paths reorders packets, and reordering looks like loss. Modern NICs add limited out-of-order handling and adaptive routing support, which we return to below.

InfiniBand versus RoCE v2: Two Ways to Be Lossless

The sections that follow are the heart of the comparison. The direct answer: InfiniBand prevents buffer overflow by never sending a packet unless the receiver has advertised buffer space, using per-hop credits. RoCE v2 on Ethernet prevents overflow reactively, with Priority Flow Control pausing the upstream sender, and then uses ECN-driven end-to-end congestion control (DCQCN) so that PFC stays a rarely used safety net.

Figure 1: Protocol stacks side by side. Verbs and the reliable transport are shared; the layers below them differ in flow control and routing.

The figure shows what is common (verbs and the IB transport headers) and what differs (link layer, flow control, routing, congestion signaling). Reading it bottom-up: InfiniBand has a native link layer with credits and a subnet manager that computes forwarding tables; RoCE v2 uses Ethernet framing, IP and UDP, with routing handled by BGP or another IP routing protocol and flow control bolted on through PFC.

InfiniBand: credit-based flow control

On an IB link, each virtual lane (VL) at the receiver advertises how many buffer credits it has. The sender decrements its credit count for every packet transmitted and may transmit only while credits remain. The receiver returns credits as it frees buffers. Because a sender can never overrun the receiver’s buffer, there is no congestion loss in the fabric: the network is lossless by construction, per hop, with no pause frames and no thresholds to tune.

Credits do not eliminate congestion; they convert it into backpressure. If an output port is oversubscribed, its input buffers fill, credits stop flowing, and upstream senders stall. This produces congestion spreading (head-of-line blocking up the tree), which is why IB also has an end-to-end mechanism. IB congestion control marks packets with FECN when queues pass a threshold, the destination reflects a BECN to the source, and the source throttles its injection rate using a table of inter-packet delays. This is conceptually the ancestor of ECN-based schemes on Ethernet.

Routing is centralized: a subnet manager discovers the topology and programs forwarding tables. Static routing can hash flows onto the same links, so modern IB switches support adaptive routing, where a switch picks among equal-cost output ports based on local queue occupancy and the fabric tolerates the resulting reordering through hardware support at the endpoints. Speeds are expressed by generation: NDR runs 400 Gb/s per port and XDR 800 Gb/s (verify exact availability with your vendor; rollout status for XDR varies).

RoCE v2: lossless Ethernet by configuration

Standard Ethernet drops packets when buffers fill. To hide that from an RC transport, operators enable Priority Flow Control (IEEE 802.1Qbb). PFC splits a link into eight priority classes; when the ingress buffer for one priority crosses a threshold (XOFF), the switch sends a pause frame for that priority only, and the upstream device stops sending it until buffer drains (XON). RDMA traffic is mapped to one or two lossless priorities, usually via DSCP-to-priority mapping, while other traffic stays lossy.

The catch is that PFC is a blunt, hop-by-hop, reactive mechanism. A pause is issued only after a buffer is already filling, so each port needs headroom: enough spare buffer to absorb everything in flight during the time it takes the pause to take effect. That time includes propagation on the cable in both directions, the transmission of any frame already started, and the reaction time of the upstream device.

Illustrative arithmetic, not a vendor figure: on a 400 Gb/s link with a 100 m cable, propagation is about 5 ns per meter, so 500 ns each way and about 1 microsecond round trip. In 1 microsecond, 400 Gb/s delivers 400,000 bits, or 50 KB. Add roughly two maximum-size frames (a 4,096-byte RDMA MTU plus headers, ×2 is about 8.3 KB) and some device reaction latency, and per-port, per-lossless-priority headroom lands in the 60 to 70 KB range. Multiply by 64 ports and several priorities and you are committing several megabytes of shared packet buffer to headroom alone. Faster links and longer cables grow it linearly, which is why lossless RoCE gets harder as speeds rise.

ECN and DCQCN: keeping PFC rare

If PFC fires constantly, you get head-of-line blocking and pause propagation. The fix is to slow senders before buffers hit XOFF. Switches mark packets with ECN (Explicit Congestion Notification, the two ECN bits in the IP header) when the queue exceeds a threshold. The receiving NIC sees the Congestion Experienced mark and returns a Congestion Notification Packet (CNP) to the sender. The sender’s NIC cuts its rate.

DCQCN (Data Center Quantized Congestion Notification) is the algorithm that defines these three roles: the congestion point (switch) marks with RED-style probabilistic marking governed by Kmin, Kmax and Pmax; the notification point (receiver NIC) generates CNPs; and the reaction point (sender NIC) adjusts the rate. The paper describes it as an end-to-end scheme implemented in Mellanox NICs and being deployed in Microsoft datacenters, with a fluid model and guidelines for setting switch buffer thresholds (DCQCN, SIGCOMM 2015).

The sender logic, in outline: on each CNP, it records a target rate equal to the current rate, updates a congestion estimate alpha as an exponentially weighted average, and multiplies its current rate by (1 − alpha/2). Absent CNPs, it recovers in stages: fast recovery (binary-search-like moves back toward the target rate), then additive increase, then hyper-increase. The exact gain constants and timer values differ between NIC firmware generations and I have not verified current defaults, so treat any specific numbers you see in blog posts, including ones you recall from the paper, as starting points to confirm on your hardware.

The key tuning rule from the paper’s guidance is ordering: ECN marking thresholds must trigger well below the PFC XOFF threshold, so that congestion control reacts first and PFC remains a backstop. If Kmin sits above XOFF, ECN never gets a chance and the fabric lives on pause frames.

Deeper Analysis: Topology, Collectives and Bandwidth Math

Rail-optimized fat-tree topologies

An 8-GPU server typically has eight NICs, one per GPU, connected to eight different leaf switches or groups of leaves. This is a rail-optimized design: GPU k on every server connects to the same “rail k” set of leaf switches. The payoff is that NCCL’s rings and trees, which communicate between same-index GPUs on different nodes after reducing inside the node over NVLink, stay within one rail and traverse only one leaf hop, never touching the spine unless the job spans multiple leaf domains.

Figure 2: Rail-optimized leaf-spine topology. Same-index GPUs across servers share a rail, so most collective traffic is one leaf hop.

In a non-blocking two-tier Clos built from radix-R switches, each leaf uses R/2 ports down and R/2 up. Maximum endpoints are R²/2. With 64-port 400G switches, that is 32 × 64 = 2,048 NICs. Beyond that you need a third tier or a larger radix, and every added tier adds hops, latency and congestion points.

Fat-tree gives full bisection bandwidth, but only if load balancing uses it. This is the practical weakness of hash-based ECMP for AI. With N elephant flows hashed over K equal paths, the expected number of collisions is substantial, and a single doubled-up link halves the throughput of two ranks and, through synchronization, slows the entire collective. Remedies: adaptive routing in the switch (IB, and some Ethernet silicon), per-packet or flowlet spraying, NIC-side multipath using multiple QPs with different source ports (NCCL can open several QPs per connection to improve entropy), or topology-aware job placement that keeps traffic within a rail.

Collective operations and the bandwidth formulas

AllReduce is the dominant collective in data parallel training. The NCCL tests define two metrics: algorithm bandwidth, algbw = S / t (data size over time), and bus bandwidth, busbw, which corrects for the algorithm so numbers are comparable to hardware peak. For AllReduce, busbw = algbw × 2(n−1)/n; for AllGather and ReduceScatter the factor is (n−1)/n, where n is the rank count (NCCL tests performance notes).

The factor reflects ring AllReduce: each rank sends and receives 2(n−1)/n of the buffer, a reduce-scatter phase followed by an allgather phase. As n grows, the factor approaches 2 and per-rank traffic becomes nearly independent of n. That is why ring bandwidth scales so well and why latency, which grows linearly with ring length, becomes the problem at large n. NCCL therefore also implements tree algorithms with logarithmic depth for small and medium messages, and both choose algorithms by message size.

Worked example, illustrative. Suppose 1,024 ranks AllReduce a 10 GB gradient buffer and each NIC delivers 400 Gb/s, that is 50 GB/s per direction at line rate. Traffic per rank is 10 GB × 2 × 1023/1024, about 19.98 GB. At 100 percent efficiency that takes 19.98 / 50 = 0.40 seconds. At a realistic 85 percent, 0.47 seconds. A single collision that halves one link’s bandwidth during the ring makes the bottleneck hop run at 25 GB/s and pushes the step toward 0.8 seconds, double the time, because a ring is only as fast as its slowest hop.

In practice, hierarchical algorithms first reduce inside each 8-GPU node over NVLink, so the inter-node volume per NIC is closer to the buffer split across eight rails: each rail carries one eighth of the data. That makes the fabric’s contribution much smaller than the naive number, and is why per-GPU NIC bandwidth, not the total, is the sizing quantity. Treat these numbers as arithmetic to calibrate intuition, then measure with all_reduce_perf.

The sequence of a lossless congestion event

Figure 3: An incast event. ECN marks drive CNPs and rate cuts; PFC only fires if queues still reach XOFF.

Walk through an incast: many senders target one receiver, as in the final step of a reduce. Queues build on the receiver’s leaf downlink. Packets crossing Kmin are probabilistically ECN marked; the receiver NIC emits CNPs, at most one per flow per interval; the senders cut rates; queues drain. If the burst outruns the control loop, which takes at least one round trip plus CNP pacing interval, the queue reaches XOFF and the leaf pauses the spines, which pause upstream. That pause is the failure mode we cover next.

Adaptive routing, packet spraying and what RC tolerates

Because flow-level hashing leaves capacity stranded, vendors moved to finer granularity. Adaptive routing in IB switches picks an output per packet or per small burst based on queue depth. On Ethernet, several switch ASIC vendors offer similar dynamic load balancing, sometimes called flowlet switching: a flow may change path when it has been idle long enough that the earlier packets have certainly arrived. Per-packet spraying is the extreme, with the best balance and the worst reordering. It works only when the endpoint can place out-of-order packets directly into memory, which is exactly what RDMA write semantics allow (the target address is in the packet) and what a lossless go-back-N RC cannot cope with unless the NIC adds reorder buffering.

Trade-offs, Gotchas, and What Goes Wrong

PFC storms and pause propagation

A PFC pause is hop by hop. If one receiver stalls, because of a hung host, a faulty NIC that stops draining its receive buffer, or a firmware bug, its leaf port pauses its upstream, which fills that switch’s buffers, which pause their upstream, and so on. A single broken NIC can freeze a large part of a fabric. This is a PFC storm. Mitigations include a PFC watchdog on switches (detect a queue paused beyond a timeout, then drop or ignore pause on that queue to break the storm) and the same watchdog on NICs, so that a host that cannot drain traffic stops emitting pauses.

PFC deadlock

PFC deadlock is subtler. If pause dependencies form a cycle in the buffer-dependency graph, every switch in the cycle is waiting for another to drain, and none can. Cycles need cyclic buffer dependency, which classically arises when routing is non-up-down: a flow takes an unexpected path (for example after a link failure, or because of misrouted packets that flood) going down to a leaf and back up. Strict up-down routing in a Clos prevents cycles in steady state, but failures, routing loops and ARP/MAC-table timing issues have produced real deadlocks in published operator reports, including those that motivated the Microsoft work. The standard defenses: keep routing strictly up-down, keep lossless priorities isolated, use watchdogs, and monitor pause counters per port as first-class SLO metrics.

Figure 4: Failure modes in a lossless RoCE v2 fabric, with the signal that reveals each and its usual mitigation.

Head-of-line blocking and victim flows

PFC pauses a whole priority on a link, not one flow. A congested flow sharing the priority with innocent flows stalls them too. The paper that introduced DCQCN argues PFC causes head-of-line blocking and unfairness, which is its stated motivation. Using separate priorities for separate traffic classes (storage, collectives, control) helps, but you have only a handful of lossless classes and each needs its own headroom.

Misconfiguration is the most common failure

In practice, most RoCE incidents are configuration drift. DSCP-to-priority mappings disagree between NIC and switch, so RDMA traffic lands in a lossy queue and gets dropped. ECN thresholds are copied from a 100G design to a 400G fabric. The MTU is 9,000 on switches but 1,500 on one host. A QoS trust mode is wrong on one firmware. RoCE rewards automated config validation and per-link counters, and punishes hand-configured fleets.

Where InfiniBand hurts

IB is not free of problems. It costs more per port in most procurement, ties you to essentially one vendor for switches, NICs and the subnet manager software, and treats the fabric as a separate silo from your Ethernet operations, tooling and staff skills. Credit-based backpressure still spreads congestion, and a large IB fabric needs careful subnet manager tuning, routing-engine choice and failure-recovery planning. Its upside is that the common case just works with far fewer knobs.

Ultra Ethernet: Making Loss Cheap Instead of Preventing It

The UEC published its 1.0 specification in June 2025; the consortium’s own year-in-review letter confirms the June 2025 release and says 27 new member companies joined during 2025 (UEC 2025 review). A secondary report dated 12 October 2026 says a 1.1 revision adds scale-up networking support, in-network collectives and programmable congestion management, including a Unified Forwarding Header said to save 20 bytes per packet; I could not confirm this on the consortium’s own page, so treat 1.1 details as unverified (WindowsMode report).

What UET changes

According to summaries of the 1.0 specification, UET runs on standard Ethernet and IP and does not change the core Ethernet layer (Viavi technical summary). It adds:

Multipath and packet spraying. Packets from one transfer are spread across all viable paths so link utilization is even and large flows do not create hot spots. Juniper describes spraying at the switch with reordering at the NIC (Network World).
Relaxed ordering and both ordered and unordered delivery. The transport no longer insists on in-order arrival, so spraying is safe.
Rapid loss recovery and packet trimming. A congested switch truncates a packet instead of dropping it and forwards the header in a higher-priority queue. The receiver learns exactly which packet was hit and can request a precise retransmission, so loss becomes a fast, targeted signal rather than a timeout.
New congestion control. Sender-based and receiver-based algorithms. The receiver-based mode lets the receiver limit what senders transmit, which directly targets incast. It does not depend on a lossless network.
Fast startup and teardown. Data can be sent before the handshake completes, helping short transfers, and idle connections can be torn down without a restart penalty.
Security. End-to-end encryption and authentication with key derivation, replay protection, and a group-keying scheme designed for collective-style group communication.

HPE has said that about 75 percent of UET draws on its Slingshot transport, per the Network World report; that is a vendor statement and I did not independently verify it.

Why this matters for the lossless debate

Trimming plus spraying plus selective retransmission means a drop costs one packet’s worth of retransmission, not a go-back-N window or a timer. With loss cheap, PFC can in principle be disabled, which removes storms, deadlock, and headroom accounting in one move. That is the real argument for UEC: not raw speed (IB and RoCE already run at 400 Gb/s) but operational simplicity and multi-vendor interoperability at scale. UEC’s compliance process was still maturing: the Viavi summary notes that the packet delivery sublayer checklist was released while trimming and semantic sublayer checklists were expected soon, so interoperability claims should be tested, not assumed.

Caveats on Ultra Ethernet

UET needs NIC support, so existing RoCE NICs cannot just be firmware-upgraded to it in general (check each vendor’s statement). Packet trimming needs switch ASIC support. And a spraying-based fabric makes tail behavior depend on path-length variance and the NIC’s reorder capacity, parameters you must benchmark. Real-world production reports at scale were sparse when I wrote this; I have not verified any large published deployment numbers.

Sizing Worked Example (Illustrative)

Take a 1,024-GPU training cluster: 128 servers with 8 GPUs, one 400 Gb/s NIC per GPU, and 64-port 400G switches.

Rail-optimized, non-blocking two-tier: each leaf uses 32 ports down, 32 up. Each rail has 128 NICs, requiring 128 / 32 = 4 leaves per rail; with 8 rails that is 32 leaves. Each rail’s 4 leaves present 4 × 32 = 128 uplinks; spines with 64 ports can absorb 64 leaf-downlinks each, so 2 spines per rail, 16 spines in total. The fabric needs 48 switches and 1,024 + 1,024 = 2,048 switch-to-switch and NIC links, most of them optical. At 2,048 endpoints the same design would simply double, which is the two-tier ceiling.

Buffer check: 64 ports × about 65 KB headroom per lossless priority is about 4.2 MB for one lossless class. If you run two, plan around 8.4 MB before any operating buffer for ECN marking. Compare that against the shared buffer of your actual ASIC, which I deliberately do not quote because it varies by silicon and I have not verified current parts.

Cost sketch with explicit assumptions: if optics and cables dominate and each 400G link needs two transceivers, 2,048 links need about 4,096 transceivers. Even before switches, this is why a topology decision (for example using copper DAC within a rack for the downlinks, or reducing tiers) shows up on the bill of materials. Do not read these as prices; they are counts.

Decision Matrix: Which Fabric When

Criterion	InfiniBand	RoCE v2 (lossless)	Ultra Ethernet
Loss handling	Credits, lossless by construction	PFC plus ECN, tuned	Lossy tolerant, trimming and retransmit
Congestion control	FECN/BECN, vendor	DCQCN and variants	UET sender and receiver based
Load balancing	Adaptive routing	ECMP, flowlet, vendor AR	Native packet spraying
Ops complexity	Lower, siloed	Highest, many knobs	Lower once mature
Vendor choice	Narrow	Broad	Broad by design
Maturity at scale	Highest	High	Early, verify
Best fit	Max performance, fast bring-up	Ethernet-centric shops with strong automation	Greenfield multi-vendor, long horizon

Practical Recommendations

If you run a small to mid-size training cluster and want it working next quarter, InfiniBand or a vendor-validated, turnkey RoCE design is the lowest-risk path. If your organization already runs large Ethernet fabrics and has network automation, RoCE v2 is viable, but budget for continuous config validation. If you are planning hardware three or more years out, evaluate Ultra Ethernet-capable NICs and switches, and insist on interoperability test evidence.

Whatever you choose, treat the network like a distributed system under test, not a cable plant.

Validate end to end with ib_write_bw, ib_write_lat and NCCL all_reduce_perf; compare busbw against the NIC line rate.
Keep ECN marking thresholds below PFC XOFF; verify with counters that ECN marks occur before pauses.
Enable PFC watchdogs on switches and NICs; alert on pause duration, not only on counts.
Enforce strict up-down routing; test link-failure scenarios for deadlock.
Automate DSCP, MTU and QoS validation across every host and switch port.
Increase entropy: multiple QPs per peer, adaptive routing where available, topology-aware placement.
Track tail latency per collective, not just average throughput.
Keep job checkpointing frequent; fabric events will occasionally kill steps.

For cross-site and hybrid connectivity, where routing policy rather than RDMA matters, see BGP for cloud engineers; the same IP routing fundamentals underpin the L3 side of RoCE v2.

Frequently Asked Questions

What is the difference between RoCE v2 and InfiniBand?

Both use the same verbs API and reliable transport semantics. InfiniBand is a complete network with its own link layer, credit-based flow control and subnet-managed routing. RoCE v2 encapsulates the IB transport in UDP and IP over Ethernet, so it routes over Layer 3 and uses commodity switches, but it needs PFC and ECN configured to behave losslessly. IB is simpler to run; RoCE is more flexible and multi-vendor.

Why does RDMA need a lossless network?

Classic reliable-connection RDMA recovers from loss with go-back-N retransmission, which resends everything after the lost packet. Even small loss rates therefore crater throughput and inflate tail latency. Fabrics avoid loss with credits on InfiniBand or with PFC plus ECN on Ethernet. Ultra Ethernet removes the dependency by redesigning the transport for selective retransmission and out-of-order delivery.

What is DCQCN and how does it work with PFC?

DCQCN is the congestion control for RoCE v2 introduced by Microsoft and Mellanox. Switches ECN-mark packets above a threshold, receivers send CNPs, and senders cut their rate and then recover in stages. PFC acts as the last-resort safety net. ECN thresholds must sit below PFC thresholds, so rate control reacts first and pause frames stay rare.

What is a PFC storm or PFC deadlock?

A PFC storm occurs when a stalled receiver triggers pauses that propagate hop by hop and freeze a large portion of the fabric. Deadlock occurs when pause dependencies form a cycle among switches so none can drain. Prevent them with strict up-down routing, PFC watchdogs on switches and NICs, isolated lossless priorities, and monitoring of pause counters.

Does Ultra Ethernet replace RoCE v2?

Not immediately. Ultra Ethernet 1.0, released in June 2025, defines a new transport, UET, with packet spraying, trimming and new congestion control, and it requires compatible NICs and switches. RoCE v2 remains widely deployed and will coexist for years. The UEC aims for multi-vendor interoperability, so evaluate it for new builds and confirm compliance evidence before committing.

How do I calculate NCCL AllReduce bandwidth?

Measure algorithm bandwidth as data size divided by time, then multiply by 2(n−1)/n for AllReduce to get bus bandwidth, where n is the number of ranks. Compare bus bandwidth with the NIC line rate. For AllGather and ReduceScatter the factor is (n−1)/n. Run NCCL’s all_reduce_perf across message sizes to see where you saturate.