Spectrum-X Ethernet AI Cluster Fabrics: How They Work and What Changed by 2026
The first version of this post asked a simple question: can Ethernet carry a 100,000-GPU training job as well as InfiniBand? That question has an answer now. xAI’s Colossus in Memphis ran on Ethernet from the start, Meta and Oracle publicly adopted the same platform in October 2025, and the argument has shifted from “whether” to “which Ethernet, and how far can it stretch.”
The stakes are practical. At this scale a training step is gated by the slowest flow in the fabric, so a network that delivers 60 percent of its line rate leaves roughly 40 percent of a very expensive GPU estate idle during communication phases. Every percentage point of effective bandwidth is worth more than almost any software tuning you can do above it.
This update explains how a Spectrum-X Ethernet AI cluster is actually built: the switch and the NIC as one co-designed system, how packet spraying and rate control work, how multiplane topologies reach half a million GPUs, what co-packaged optics change, how Spectrum-XGS links sites, and how the open Ultra Ethernet standard relates to all of it. You leave with a mental model, the failure modes, and a checklist for evaluating a fabric.
What this covers: why plain RoCEv2 struggles with AI traffic, the end-to-end Spectrum-X design, multiplane and photonics scaling, scale-across, Ultra Ethernet, operational trade-offs, recommendations and an FAQ.
Context and Background
AI training traffic differs from the enterprise traffic Ethernet grew up with. A data-parallel or mixture-of-experts job runs collectives, such as all-reduce, reduce-scatter and all-to-all, in lockstep. Every GPU sends large messages at roughly the same instant, and the step cannot finish until the slowest transfer lands. This produces a small number of very fat flows, called elephant flows, with almost no entropy in the packet headers that switches normally hash on.
Classic data-center Ethernet balances load with equal-cost multipath (ECMP). ECMP hashes the five-tuple of each flow and pins the flow to one path for its lifetime. With thousands of flows that works. With a handful of elephant flows it does not: two flows can hash onto the same uplink while a neighbouring uplink sits idle, and the collision persists for the whole transfer. Because the job waits for the slowest flow, one unlucky hash slows every GPU.
RoCEv2 (RDMA over Converged Ethernet version 2) adds another difficulty. RDMA transports were designed for lossless fabrics, so deployments leaned on Priority Flow Control (PFC) and ECN-based schemes such as DCQCN to avoid drops. PFC pauses an upstream port when a buffer fills, and at scale a pause can spread backwards through the topology, producing head-of-line blocking and, in bad cases, deadlock. Tuning thresholds across hundreds of switches is hard, and the symptoms show up as unexplained slow steps rather than clear errors.
InfiniBand sidestepped most of this with credit-based link-level flow control, centralised routing through its subnet manager, and in-network features such as SHARP. That is why it dominated early large training clusters. For a deeper protocol-level comparison of the three transports, see our guide to RoCE v2 vs InfiniBand vs Ultra Ethernet congestion control, and for the intra-rack side of the picture, the post on NVLink vs InfiniBand vs RoCE for NCCL collectives.
NVIDIA’s answer, Spectrum-X, keeps the Ethernet frame format and operations model but replaces the behaviours that hurt AI traffic. It does this by treating the switch and the NIC as one system. The ideas are not exotic: per-packet load balancing, receiver-side reordering and telemetry-driven rate control are all published techniques. The contribution is shipping them as a tested, co-designed product.
A note on terminology that confused the original post: “Spectrum-X” is a platform, not a chip. It combines Spectrum switch ASICs (Spectrum-4 at 51.2 Tb/s in the SN5600 generation, and newer photonics-based systems), SuperNICs (BlueField-3 and ConnectX-8 and later) and software. Treat any claim about “Spectrum-X” as a claim about the pair, never the switch alone.
The Spectrum-X Reference Architecture
Short answer: Spectrum-X is an Ethernet platform in which the switch sprays packets across all available paths using real-time load information, and the SuperNIC at the receiving end restores packet order and adjusts send rates from switch telemetry. The loop between the two is what lifts effective bandwidth above ECMP-based RoCE.

Figure 1: A Spectrum-X Ethernet AI cluster. SuperNICs attach GPUs to leaf switches, packets are sprayed across the spine, and telemetry flows back to the sending SuperNIC.
Figure 1 shows the closed loop. Each GPU is paired with a SuperNIC, which connects to a leaf switch. Leaves connect to a spine layer, and on larger clusters to a third tier. Data takes whichever path is least loaded at that instant. Congestion information travels back from the switch to the sender, which modulates its rate. Neither half works as well alone: spraying without a reordering receiver breaks RDMA ordering, and rate control without spraying cannot fix a hash collision.
Adaptive routing and packet spraying
In Spectrum-X, the switch makes a forwarding decision per packet or per small group of packets, choosing among equal-cost next hops according to current queue occupancy. This is the opposite of ECMP, which decides once per flow. The effect on elephant flows is large: a single 400 or 800 Gb/s flow can use every uplink at once instead of being confined to one.
The cost of spraying is out-of-order delivery. Packets of one message take different paths with different queueing delays and arrive shuffled. NVIDIA’s developer documentation describes the receiving SuperNIC handling this by placing arriving packets directly into the correct position in the receive buffer, a capability it calls Direct Data Placement. The application and NCCL see an ordered stream. This division of labour, a simple fast switch plus a smart endpoint, is the architectural core.
Be careful about what is known. NVIDIA states that adaptive routing is controlled by the switch, with the NIC handling reordering. Exact decision granularity, the load metric and the failure semantics are not fully documented publicly, so I treat them as implementation details rather than specification.
Congestion control driven by telemetry
Spraying removes collisions inside the fabric, but it does not remove incast, where many senders target one receiver and its last-hop port overflows. For that, Spectrum-X uses a congestion control scheme where the Spectrum switch reports in-band, high-frequency telemetry and the SuperNIC adjusts its transmission rate accordingly. NVIDIA describes the reaction as happening with microsecond-scale precision, though the vendor article gives no measured latency, so I flag that as a claim rather than a benchmark.
Contrast this with classic DCQCN, where the switch marks packets with ECN, the receiver reflects marks to the sender, and the sender backs off in coarse steps. The telemetry-based approach shortens the feedback loop and gives the NIC a quantity more informative than a single bit. The SuperNIC runs this logic on programmable hardware: NVIDIA documents a data path accelerator with 16 hyperthreaded cores used for congestion control and traffic management, programmable through its DOCA framework.
What the SuperNIC contributes
NVIDIA introduced the SuperNIC as a distinct product class: a NIC optimised for east-west GPU traffic, not for the general-purpose offload that a DPU performs. BlueField-3 SuperNICs run at 400 Gb/s, and ConnectX-8 doubles that to 800 Gb/s, with hardware RoCE, GPUDirect RDMA, line-rate programmable packet processing and inline crypto acceleration supporting IPsec and PSP, all per NVIDIA’s documentation. Later in this article we will see 1.6 Tb/s SuperNICs entering the picture with the Rubin generation.
Performance isolation
A shared AI factory runs many jobs at once. Performance isolation means one tenant’s burst should not degrade another’s. In Spectrum-X, load-aware spraying keeps links evenly utilised and the congestion control keeps queues short, which together reduce the interference that ECMP fabrics show. At Hot Chips 2026, NVIDIA claimed 2.2 times better multi-tenancy and 1.3 times lower bandwidth jitter than off-the-shelf Ethernet on 102.4 Tb/s switch systems and 1.6 Tb/s SuperNICs. These are vendor-reported comparisons against an unspecified baseline, so read them as direction, not as a number you will reproduce.
Evidence From Production: Colossus and the Hyperscalers
The most cited production reference is xAI’s Colossus cluster in Memphis. NVIDIA’s October 2024 announcement said the cluster used 100,000 Hopper GPUs on a Spectrum-X RDMA network, and that xAI was doubling it to 200,000 GPUs. The press release also reported that Colossus sustained 95 percent data throughput, whereas standard Ethernet at that scale reportedly delivers about 60 percent, and that there was zero application latency degradation or packet loss from flow collisions across the three fabric tiers. It also reported that the facility was built in 122 days and training started 19 days after the first rack arrived.
Two cautions apply. First, these are NVIDIA’s own statements, not an independent audit, and the 60 percent figure is described as typical for standard Ethernet rather than measured on the same hardware. Second, “zero packet loss from flow collisions” is narrower than “zero packet loss.” Still, the existence proof matters: a 100,000-GPU training cluster ran on Ethernet in a very short build window.
The October 2025 Open Compute Project Summit added two named adopters. Meta announced it is integrating Spectrum-X into its Facebook Open Switching System (FBOSS) and Minipack3N switch platform, and Oracle said it would use Spectrum-X to build giga-scale AI factories on the upcoming Vera Rubin architecture. Meta’s choice is notable because it runs FBOSS on its own hardware, so the integration is software-defined rather than a turnkey appliance.
These deployments share a lesson. Buyers did not adopt Ethernet for lower raw performance. They adopted it for supply chain breadth, operational familiarity, mixed-vendor tooling and the ability to run the same fabric for storage, front-end and back-end networks. Performance had to reach parity first, and spraying plus telemetry-driven rate control is what delivered it.
Scaling Out: Multi-Rail, Multiplane and the Half-Million-GPU Target

Figure 2: Multiplane Spectrum-X. Each SuperNIC divides its bandwidth across several switch planes, which multiplies the reachable GPU count for a given switch radix.
A switch has a fixed number of ports, and that radix limits how many GPUs a given number of tiers can reach. The traditional response is to add tiers, but each tier adds latency, power, cost and congestion points. Spectrum-X pursues a different lever: split each NIC’s bandwidth across several smaller links, each going to a different switch.
According to ServeTheHome’s live coverage of NVIDIA’s Hot Chips 2026 talk (August 2026), NVIDIA contrasted three layouts. A traditional top-of-rack design sits at around 2,000-GPU scale. A multi-rail design built from 100 Tb/s switches with 64 ports of 1.6 Tb/s across four rails reaches about 8,000 Rubin GPUs at 1.6 Tb/s each. A multiplane design with 512 ports of 200 Gb/s per switch, eight planes across four rails, reaches about 512,000 Rubin GPUs at 1.6 Tb/s per GPU. NVIDIA presented this as 64 times the scale of multi-rail, and as needing 1.7 times fewer scale-out switches than a traditional multi-tier single-rail design.
The arithmetic is intuitive. A switch with 64 ports of 1.6T and the same silicon exposing 512 ports of 200G have the same aggregate bandwidth, but the second has eight times the radix. Higher radix means fewer tiers for the same GPU count. The price is that every message is now carried over eight lower-speed links, and the NIC must spread and reassemble traffic across planes.
Why planes help with failures
Planes also change the failure profile. In a conventional fabric a failed link or switch removes a large slice of one GPU’s connectivity. In a multiplane fabric, losing one plane removes one eighth of each NIC’s bandwidth, and the other seven planes continue. NVIDIA’s Hot Chips material claimed that under partial bandwidth loss Spectrum-X multiplane retains 90 percent of bandwidth with a detection time of 2.68 milliseconds, while an off-the-shelf multiplane Ethernet design lost full bandwidth with 1,080 milliseconds for detection and recovery. It reported roughly 400 times faster detection and about 11 times faster recovery, around 100 milliseconds. These are vendor slide claims, relayed by a third-party live blog, and I have not seen them independently reproduced.
The reason failure detection speed matters is that collective libraries and training frameworks tolerate short stalls but not long ones. A recovery in 100 ms is typically absorbed within a step; a second-scale outage can trip timeouts and force checkpoint restarts, which at large scale cost far more than the network failure itself.
Optics: Where Photonics Changes the Equation

Figure 3: Pluggable transceivers versus co-packaged optics. Moving the optical engine next to the ASIC shortens the electrical path and removes a separate powered module.
At 800 Gb/s and 1.6 Tb/s per port, electrical signalling between the switch ASIC and a pluggable transceiver at the faceplate becomes a power and signal-integrity problem. Optics also dominate the power budget: NVIDIA estimated at Hot Chips 2026 that optics are about 10 percent of AI factory compute power, above a typical cloud data center. Our companion piece on the memory wall and optical interconnect explains the physics in more depth.
Co-packaged optics (CPO) moves the optical engine from a faceplate module to the same package as the switch ASIC. NVIDIA announced Spectrum-X Photonics and Quantum-X Photonics at GTC on 18 March 2025. The Ethernet configurations were listed as 128 ports at 800 Gb/s (or 512 at 200 Gb/s) for 100 Tb/s, and 512 ports at 800 Gb/s (or 2,048 at 200 Gb/s) for 400 Tb/s. The press release claimed four times fewer lasers, 3.5 times better power efficiency, 63 times greater signal integrity, and 10 times better network resiliency, with Spectrum-X Photonics switches expected in 2026.
At CES 2026 NVIDIA launched the Spectrum-X Ethernet Photonics systems, the SN6800 and SN6810. Coverage describes the SN6800 as a 409.6 Tb/s system, with 512 ports at 800 Gb/s or 2,048 ports at 200 Gb/s, using a quad-ASIC design with an integrated fiber shuffle. NVIDIA claimed about five times lower power per 1.6 Tb/s port than pluggable interconnects and about five times longer flap-free uptime than off-the-shelf Ethernet. At Hot Chips 2026 it described a CPO chip with micro ring modulators in production and a 3D-stacked silicon photonics engine built on TSMC’s COUPE process, claiming four times fewer lasers and ten times longer mean time between interruptions.
Note the naming trap. Press coverage of this generation uses “Spectrum-X Ethernet Photonics” and sometimes “Spectrum-6.” I could not confirm from the sources I fetched that the underlying ASIC is officially branded Spectrum-6 across all documents, so I use the system names. Also, the original press release headline mentions 1.6 Tb/s per port while the body lists 800 Gb/s ports; the 1.6T figure refers to the aggregate or to SuperNIC and per-lane-group configurations depending on the source. Always check the datasheet for the exact SKU.
What CPO does not fix
CPO changes serviceability. A faceplate transceiver can be swapped by a technician in minutes; a co-packaged engine is part of the switch. NVIDIA’s counter is that its engine can be tested before attachment, that detachable fiber connectors are added at final assembly and that the reliability gains outweigh serviceability losses. That argument is plausible but fleet data over several years is the real test, and it does not exist yet publicly.
Scale-Across: Spectrum-XGS
Scale-up (NVLink inside a rack) and scale-out (the Ethernet fabric between racks) are the two networks most readers know. NVIDIA now describes five: scale-up, scale-out, scale-across, the new scale-in, and AI context scale for key-value cache and storage. The point of the taxonomy is that no single fabric serves all of them well.
Spectrum-XGS is the scale-across piece, designed to link separate data centers into one logical AI system. Long-distance links introduce latency measured in milliseconds, not microseconds, and conventional congestion control tuned for in-building round trips behaves badly over them. NVIDIA positions XGS as adding distance-aware congestion control and traffic management so that multi-site training or inference behaves more predictably. At Hot Chips 2026 NVIDIA gave XGS specifications of 800 Gb/s per port on 102.4 Tb/s switches paired with ConnectX-9 SuperNICs, claiming up to 1.9 times lower multi-site latency and doubled scale-across performance. Oracle was cited as planning to use the capability for distributed AI factories.

Figure 4: Spectrum-XGS scale-across. Two sites each run a Spectrum-X fabric, and long-haul links between them carry traffic under distance-aware congestion control.
Physics still rules. Light travels roughly 200 kilometres per millisecond in fibre, so a 1,000 km span adds about 5 ms each way before any equipment delay. No congestion-control algorithm removes that. What XGS can do is keep the long link full and stable and avoid synchronized collapse. The workloads that benefit are those that tolerate latency: pipeline-parallel stages placed across sites, asynchronous or hierarchical gradient exchange, and inference replication. Tightly coupled tensor-parallel layers must remain inside a site.
I did not retrieve NVIDIA’s original XGS launch material in this research pass, so the date and customer details of the first announcement are not restated here beyond what the Hot Chips and OCP coverage above reports.
How Ultra Ethernet Relates
The Ultra Ethernet Consortium (UEC) published its 1.0 specification in June 2025. Founders include Cisco, Arista, HPE and Intel, with many more members since. The specification defines a new transport, Ultra Ethernet Transport (UET), aimed at AI and HPC. Per coverage of the release, it enables packet spraying at the switch with reordering at the NIC, introduces congestion control that does not depend on a lossless network and lets the receiver limit sender transmissions. HPE said about 75 percent of UET draws from its Slingshot transport.
The overlap with Spectrum-X is obvious. The same three ideas, spraying, endpoint reordering and receiver-aware rate control, appear in both. The difference is governance. UEC is a multi-vendor standard that lets different vendors’ NICs and switches interoperate. Spectrum-X is a single vendor’s integrated platform, where the pair is tested together. NVIDIA is also a UEC participant and has said that its roadmap supports the direction, but I could not verify in this pass the exact degree of UEC 1.0 compliance of shipping Spectrum-X hardware, so I make no claim there.
Here is the practical interpretation. If you want an optimised, validated stack now, Spectrum-X is the mature option. If your priority is multi-vendor sourcing, UET-capable NICs from other suppliers are the path to watch. Several 2026 NIC announcements from other vendors target UEC, but I do not cite specifics here because I did not verify them. For the transport-level detail see our earlier RoCE vs InfiniBand vs Ultra Ethernet comparison.
Worked Analysis: What Effective Bandwidth Is Worth
Marketing numbers are easier to judge with a small model. The figures below are illustrative arithmetic, not measurements, and exist to show how fabric efficiency converts into cluster economics.
Consider a cluster of 100,000 GPUs where, in each training step, communication that cannot be overlapped with compute takes 20 percent of step time on a fabric delivering 100 percent of its nominal bandwidth. If the fabric instead delivers 60 percent effective throughput, that exposed communication stretches to roughly 33 percent of the original step time (20 divided by 0.6), so the step grows from 100 to about 113 units. At 95 percent effective throughput the exposed communication is about 21 units and the step is about 101. Compared with the 60 percent case, the better fabric returns about 11 percent of cluster time. On a cluster whose cost is dominated by accelerators, 11 percent is a very large sum, which explains why buyers pay a premium for fabric efficiency.
Two caveats keep this honest. Overlap changes everything: well-tuned training overlaps most communication with compute, so only the exposed portion matters, and that portion varies widely by model and parallelism strategy. And the 60 versus 95 percent figures come from NVIDIA’s description of the Colossus deployment, which compares against a generic Ethernet baseline rather than a carefully tuned one.
Reading the vendor comparisons
| Claim | Source | Baseline | How to treat it |
|---|---|---|---|
| 95 percent throughput vs 60 percent | NVIDIA, Oct 2024 | Standard Ethernet at scale | Vendor figure for one deployment |
| 1.6x RDMA bandwidth, 2.2x multi-tenancy, 1.3x lower jitter | NVIDIA at Hot Chips 2026 | Off-the-shelf Ethernet | Directional, baseline unspecified |
| 14x NCCL all-reduce gain in Nemotron Ultra pre-training | NVIDIA at Hot Chips 2026 | Off-the-shelf Ethernet | Likely a pathological baseline, ask for config |
| 90 percent bandwidth kept under fault | NVIDIA at Hot Chips 2026 | Multi-tier topology dropping to zero | Topology-dependent, vendor slide |
The 14 times all-reduce number illustrates the pattern. A gain that large normally means the baseline suffered severe collisions or congestion collapse, which is plausible for default ECMP RoCE on elephant flows but is not what a tuned alternative would show. When you evaluate a fabric, demand the baseline configuration, the message sizes and the topology, then reproduce with your own NCCL tests.
How to test it yourself
Run NCCL tests, specifically all_reduce_perf and alltoall_perf, across message sizes from a few megabytes to several gigabytes, at the full cluster scale you intend to deploy and also with several concurrent jobs to measure interference. Record bus bandwidth as a fraction of the NIC line rate, plus the run-to-run variance. Variance is the early warning of collision-prone fabrics: an ECMP network often shows a good mean with a bad tail, because the unlucky hash only occurs on some runs. Then inject faults, pulling a link or draining a switch, and time how long collective throughput takes to recover. The ratio of tail to median matters more than the best result.
Spectrum-X, InfiniBand and Other Ethernet Options
The honest comparison has more than two columns. The table summarises qualitative differences; it avoids invented performance numbers.
| Dimension | Spectrum-X Ethernet | NVIDIA InfiniBand (Quantum-X) | Generic RoCEv2 Ethernet | UEC-based multi-vendor |
|---|---|---|---|---|
| Load balancing | Adaptive per-packet spraying | Adaptive routing in fabric | Per-flow ECMP | Packet spraying by spec |
| Loss handling | Telemetry-driven rate control, endpoint reordering | Credit-based lossless links | PFC plus ECN, tuning heavy | Not dependent on lossless network |
| Management model | Ethernet tooling, vendor telemetry | Subnet manager | Ethernet tooling | Ethernet tooling, evolving |
| Vendor choice | Single vendor pair | Single vendor | Many vendors | Many vendors, maturing |
| Maturity at 100K scale | Public production references | Long track record | Hard to tune | Early |
| In-network compute | Limited in public material | SHARP aggregation | None | Not in 1.0 |
The practical reading: InfiniBand remains a strong choice when the cluster is a single-purpose training machine and the operations team already knows it. Spectrum-X makes sense when you want Ethernet operations, shared fabric for storage and front-end traffic, or an integration path such as Meta’s FBOSS. Generic RoCEv2 can work at modest scale with disciplined tuning but is the option most likely to disappoint at 10,000 GPUs and above. UEC fabrics are the multi-vendor future, but production references are thinner.
Storage and the fourth fabric
Training clusters also move checkpoints and datasets, and a stalled checkpoint write wastes GPU time too. Spectrum-X has been marketed for storage traffic as well, on the basis that adaptive routing helps the bursty, large-block patterns of checkpointing. I treat the claimed storage bandwidth gains as unverified here. The architectural point stands regardless: running storage on the same Ethernet operations model is an argument for Ethernet that InfiniBand cannot match as easily.
Trade-offs, Gotchas, and What Goes Wrong
Single-vendor coupling. The headline gains come from the switch and NIC working as a pair. Mixing a third-party switch with a Spectrum-X SuperNIC, or the reverse, forfeits some or all of the adaptive loop. That is a legitimate procurement risk if your strategy values second sources. UEC is the hedge, but it has not yet matched the integrated stack’s maturity in public references.
Reordering cost moves to the NIC. Spraying is not free. The NIC must hold and place out-of-order packets, so its buffer sizing and packet-rate limits matter. Fabric behaviour under very small messages, such as many tiny all-to-all exchanges in mixture-of-experts layers, is a place to measure carefully, because per-packet overheads show up there and not in large all-reduce tests.
Telemetry and rate control can oscillate. Any closed-loop controller can overshoot. Feedback that is too aggressive lowers throughput with sawtooth behaviour, and feedback that is too slow allows queues to build. Vendors tune this for known topologies. If you deviate, for example with an unusual oversubscription ratio or very long cables, retest.
Multiplane complexity. Eight planes mean eight times as many links to cable, label, monitor and keep consistent. Misconfiguring one plane, for example with mismatched firmware or a wrong lane map, produces a partial slowdown that average-bandwidth dashboards may not reveal. Per-plane monitoring and automated cabling validation become mandatory.
Optics reliability is still being proven. CPO promises fewer lasers and longer time between interruptions, but those are design claims. Fleet data from several years at scale is the evidence that matters. Early adopters should negotiate replacement and spares terms in light of the fact that failed co-packaged engines may mean swapping a switch.
Power and cooling. Faster switches and CPO systems push power density up, and the Quantum-X Photonics line is liquid-cooled. Network planning therefore intersects with facility planning; see our analysis of 800 V DC power architecture for AI data centers for the other half of that conversation.
Benchmark skepticism. Vendor comparisons against “off-the-shelf Ethernet” rarely state the baseline’s tuning. Treat order-of-magnitude gains as signals that the baseline was untuned, and require reproducible tests on your own workload.
Practical Recommendations
Start from the workload, not the brand. If you operate fewer than a few thousand GPUs, a well-tuned RoCEv2 fabric with proper ECN configuration may be adequate, and the premium for an integrated stack is harder to justify. Between roughly ten thousand and a hundred thousand GPUs, collision-driven tail latency becomes the dominant cost, and an adaptive fabric, whether Spectrum-X or InfiniBand, usually pays for itself. Beyond that scale, topology becomes the story, and multiplane designs with high-radix switches are how vendors reach hundreds of thousands of endpoints.
Plan for three phases. Phase one is a proof of concept with your own models, measuring tail behaviour, multi-tenant interference and failure recovery rather than peak bandwidth. Phase two is a staged rollout of one pod with full telemetry, per-plane monitoring and automated cable validation. Phase three is a scale-out decision that revisits optics: pluggable for serviceability, co-packaged for power and density once fleet data supports it.
Checklist before signing:
- Ask for the exact baseline configuration behind every performance claim and rerun NCCL tests yourself.
- Measure p99 and run-to-run variance for all-reduce and all-to-all, not only the mean.
- Test concurrent multi-tenant jobs to quantify interference.
- Inject link and switch failures and time recovery to full throughput.
- Confirm firmware, SuperNIC and switch version compatibility matrices in writing.
- Decide your multi-vendor strategy explicitly, and check UEC support commitments.
- Check power, cooling and spares terms for CPO systems.
- For multi-site plans, confirm which parallelism layers will cross the long link.
- Verify what your storage and front-end networks will share.
Frequently Asked Questions
What is Spectrum-X and is it just a switch?
Spectrum-X is an Ethernet platform for AI, not a single chip. It combines Spectrum switch ASICs, such as the 51.2 Tb/s Spectrum-4 in the SN5600, with SuperNICs including BlueField-3 and ConnectX-8 and newer, plus management software. The benefits come from the pair: the switch sprays packets and reports telemetry, and the NIC reorders packets and adjusts rates. A Spectrum switch alone is a fast Ethernet switch without the adaptive loop.
Is Ethernet now better than InfiniBand for AI clusters?
Neither is universally better. InfiniBand has a long record in single-purpose training clusters and offers in-network aggregation. Spectrum-X narrows the performance gap with adaptive routing and congestion control while keeping Ethernet operations, and has public references at 100,000-GPU scale such as xAI Colossus. Meta and Oracle adopted it in 2025. The right choice depends on workload, team skills, storage sharing and sourcing strategy, so benchmark both on your own jobs.
How is Spectrum-X different from Ultra Ethernet?
Both use packet spraying, endpoint reordering and rate control that does not rely on a lossless network. Ultra Ethernet is an open multi-vendor standard, with its 1.0 specification published in June 2025, so devices from different suppliers can interoperate. Spectrum-X is one vendor’s integrated platform with co-designed hardware. NVIDIA participates in the consortium, but I could not verify the exact UEC compliance of shipping Spectrum-X products, so check with the vendor.
What does co-packaged optics change?
Co-packaged optics place the optical engine inside the switch package instead of in a pluggable faceplate module. This shortens the electrical path, which NVIDIA says cuts power per 1.6 Tb/s port by about five times and reduces the number of lasers by four times. The trade-off is serviceability, since a failed engine is part of the switch. The reliability claims are design targets awaiting multi-year fleet evidence.
What is Spectrum-XGS used for?
Spectrum-XGS is the scale-across product for linking separate data centers into one logical AI system. It adds distance-aware congestion control for long-haul links, where latency is milliseconds. NVIDIA claims up to 1.9 times lower multi-site latency. It cannot beat the speed of light, so latency-tolerant workloads, such as pipeline stages and inference replication, benefit, while tightly coupled tensor-parallel layers should stay within a single site.
Do I need Spectrum-X for a small GPU cluster?
Usually not. Under a few thousand GPUs, collision-induced slowdowns are smaller and a carefully tuned RoCEv2 network with ECN and sensible PFC settings often performs adequately at lower cost. The adaptive features pay off as scale, job concurrency and tail sensitivity rise. If you expect to grow quickly or share the fabric among many tenants, buying the adaptive fabric early avoids a disruptive migration later.
Further Reading
- RoCE v2 vs InfiniBand vs Ultra Ethernet: AI cluster congestion control
- NVLink vs InfiniBand vs RoCE for NCCL all-reduce and cluster design
- Memory wall and optical interconnect for physical AI
- NVIDIA: xAI Colossus on Spectrum-X (press release)
- NVIDIA developer blog: Powering next-generation AI networking with SuperNICs
- ServeTheHome: Spectrum-X multiplane architecture at Hot Chips 2026
- Network World: Ultra Ethernet Consortium publishes 1.0 specification
By Riju — about
