ROCm vs CUDA in 2026: AMD Instinct MI300X/MI350 for LLM Training and Inference

For most of the last decade the ROCm vs CUDA question had a boring answer: CUDA, because everything else was a science project. In 2026 that answer is no longer automatic. A single AMD Instinct MI355X carries 288 GB of HBM3E at 8 TB/s, which is enough to hold a 70-billion-parameter model in FP8 with room left for a large key-value cache, and AMD’s software stack has moved from “port it yourself” toward “pip install and run vLLM”. The hardware argument is easy to state. The software argument is where teams still get hurt.

This matters now because inference economics are dominated by memory capacity and bandwidth, the two places where AMD’s recent parts are strongest, while training at scale still leans on the maturity of collectives, kernels and tooling, where NVIDIA’s lead is real but narrowing. AMD also re-based its entire software distribution this year, so older advice about ROCm install paths and versions is stale.

You will leave with a mechanism-level picture of both stacks, worked memory and bandwidth arithmetic for MI300X and MI350-class parts, a practical HIP porting workflow, an honest RCCL-versus-NCCL comparison, and a decision framework for when each platform is the lower-risk choice.

What this covers: the state of play in late 2026, the layered architecture of each stack, HIP porting and kernel libraries, collectives and scale-out, worked inference and training numbers, failure modes, and a recommendation checklist.

Context and Background

NVIDIA’s CUDA is not just a compiler. It is a runtime, a driver model, a set of math libraries (cuBLAS, cuDNN, cuFFT), a collective library (NCCL), a profiler suite, and close to two decades of third-party code written against all of them. Every major LLM framework treats CUDA as the reference backend: new attention kernels, quantization schemes and fused operators usually land there first. That network effect, not raw FLOPS, is the moat.

AMD’s answer is ROCm (Radeon Open Compute), an open-source stack whose central abstraction is HIP, a C++ runtime API and kernel language that deliberately mirrors CUDA’s. HIP code compiles for AMD GPUs through LLVM, and with thin headers the same source can also compile for NVIDIA GPUs. The math libraries have parallel names: hipBLASLt for cuBLASLt, MIOpen for cuDNN, RCCL for NCCL, rocFFT for cuFFT. Because PyTorch’s ROCm builds expose AMD GPUs through the familiar torch.cuda namespace, most model code does not change at all.

The hardware has caught up on paper. According to AMD’s product pages, the Instinct MI300X (CDNA 3, gfx942) has 304 compute units, 192 GB of HBM3, 5.3 TB/s of peak bandwidth, a 750 W typical board power and a peak of 1.3 PFLOPS dense FP16 and 2.61 PFLOPS dense FP8. The MI355X (CDNA 4, gfx950) has 256 compute units, 288 GB of HBM3E, 8 TB/s, 1,400 W typical board power, 2.5 PFLOPS dense FP16 and 5 PFLOPS dense FP8, plus new MXFP6 and MXFP4 matrix formats at 10.1 PFLOPS. These are vendor-stated theoretical peaks, not measured throughput.

Two 2026 developments change the picture. First, AMD announced its MI400 generation and the Helios 72-GPU rack at Advancing AI 2026 in July, with press coverage citing roughly 420 to 430 GB of HBM4 per MI455X (reports disagree between 423 GB and 432 GB, so check AMD’s final datasheet); treat rack-level comparative claims against NVIDIA as AMD’s own. Second, AMD restructured ROCm: the legacy 7.x line gave way to the “ROCm Core SDK” built on a new build system called TheRock, and the 10.x series shipped this year. The current release at the time of writing is Core SDK 10.1.0, dated 2026-10-05 in the official documentation, listing support for MI350 series (MI355X, MI350X, MI350P), MI300 series (MI325X, MI300X, MI300A), and older MI200 and MI100 parts. For the serving-side context that determines why memory matters so much, see our explainer on chunked prefill and prefill-decode interference, and for the NVIDIA-side interconnect story, our comparison of NVLink, InfiniBand and RoCE for NCCL collectives.

The official ROCm documentation is the authority for supported-hardware matrices, and the MI355X product page lists the peak specifications used throughout this article.

The 2026 baseline in one table

Property	MI300X	MI355X	Source
Architecture target	CDNA 3, gfx942	CDNA 4, gfx950	AMD ROCm docs
Compute units	304	256	AMD product pages
Memory	192 GB HBM3	288 GB HBM3E	AMD product pages
Peak bandwidth	5.3 TB/s	8 TB/s	AMD product pages
Dense FP16 (peak)	1.3 PFLOPS	2.5 PFLOPS	AMD product pages
Dense FP8 (peak)	2.61 PFLOPS	5 PFLOPS	AMD product pages
FP4 / FP6 matrix	not supported	10.1 PFLOPS each	AMD product page
Board power	750 W	1,400 W	AMD product pages
Infinity Fabric links	8 x 128 GB/s	7 x 144 GB/s	AMD product pages
Last-level cache	256 MB	256 MB	AMD product pages

A note on the MI355X versus MI350X split: both are CDNA 4 parts with the same 288 GB of memory, but the MI355X is the higher-power, liquid-cooled variant and the MI350X targets air-cooled deployments. The figures above are MI355X.

The Core Architecture: Two Stacks, Same Shape

Short answer: CUDA and ROCm are structurally parallel stacks. Framework code calls a device API, which dispatches to vendor math libraries, collective libraries and a compiler that emits GPU machine code. The differences that matter are library maturity, kernel availability for new algorithms, wavefront width (64 on AMD CDNA versus 32 on NVIDIA), and the polish of the collective and profiling tooling.

Figure 1: The ROCm and CUDA stacks diverge below the framework layer; AMD adds AITER and Composable Kernel as a fast path for LLM operators.

Figure 1 shows why porting at the framework level is cheap and porting below it is not. Model code, optimizers and even most distributed training loops sit above the torch.cuda namespace. On a ROCm build of PyTorch, torch.version.hip is set, torch.cuda.is_available() returns true when an Instinct GPU is present, and a .to("cuda") places tensors on the AMD device. The backend selection happens when the wheel is built, not in your code.

Layer one: compilers and the execution model

CUDA compiles through nvcc to PTX (a virtual ISA) and then SASS (actual machine code), with PTX giving NVIDIA forward compatibility across architectures. HIP compiles through hipcc, a Clang wrapper, directly to AMDGPU machine code for a specific target such as gfx942 (MI300 series) or gfx950 (MI350 series). There is no PTX-equivalent stable virtual ISA; you build fat binaries for the architectures you ship. In practice this means a container built for gfx942 will not run its custom kernels on gfx950 unless you compiled for both.

The execution model differs in a way that bites hand-written kernels. NVIDIA schedules threads in warps of 32; AMD CDNA GPUs schedule in wavefronts of 64. Any kernel that hardcodes 32 in a shuffle mask, a ballot, a reduction tree or a shared-memory layout will compute wrong answers or leave half the lanes idle on CDNA. The HIP runtime exposes warpSize as a variable precisely so code can adapt, but a lot of inherited CUDA code was written with the literal.

Layer two: libraries and the LLM fast path

For dense GEMM, hipBLASLt is AMD’s counterpart to cuBLASLt, handling the FP16, BF16 and FP8 matrix multiplies that dominate transformer compute. Attention is where the gap historically opened. FlashAttention (see our explainer on IO-aware attention and tiling) was written for NVIDIA’s memory hierarchy, and AMD ships its own implementations built on Composable Kernel (CK) and Triton. AMD’s newer AITER library bundles tuned operators for attention, fused MoE, GEMM and normalization for inference engines, and vLLM and SGLang can call into it on Instinct hardware.

The practical consequence is a two-tier reality. The popular path, a recent PyTorch plus vLLM or SGLang running a mainstream open-weight model, is well trodden on Instinct and ships in AMD-published containers. The long tail, a research kernel released last month in CUDA and Triton, may be missing or slow on ROCm for weeks or months.

Layer three: distribution and packaging in the 10.x era

The ROCm Core SDK transition guide documents the packaging change. The install prefix moves from /opt/rocm-7.x/ to /opt/rocm/core-10.1, package names change to an amdrocm-<component> scheme, some packages merge (hipBLAS and rocBLAS become amdrocm-blas), ROCm SMI is removed in favour of AMD SMI, and Python wheels can be installed from a ROCm package index. If your Dockerfiles or Ansible roles hardcode rocm- package names or /opt/rocm-7.x paths, they will break on upgrade. A third-party summary reports that ROCm 10 launched on August 27, 2026 and that AMD claims average gains of 3.3x for inference and 2.4x for training versus ROCm 7-era configurations on the same hardware; per that summary these are AMD figures for selected MI355X configurations and should not be read as a general multiplier. I could not verify them against a primary AMD document in this run.

Where the “just works” claim holds, and where it does not

It holds when four conditions are met together: the GPU architecture is on the supported list, the OS and kernel are on the supported list, the framework build is the one AMD validated (Core SDK 10.1.0 lists PyTorch 2.14.0, JAX 0.11.1, vLLM 0.29.0 and SGLang 0.5.18), and the model uses mainstream operators. Break any one and you move from “install” to “debug”. That support matrix is the single most useful document for evaluating the platform, and it is worth reading before any hardware purchase.

Deeper Analysis: HIP Porting, Collectives and Worked Numbers

The architecture view tells you what exists. The walk-through below tells you what it costs: how much work a CUDA codebase takes to move, how collectives behave when you leave a single node, and what the arithmetic of LLM serving says about which chip fits which job.

HIP porting: the mechanical 90 percent and the human 10 percent

HIP porting is mostly a text transformation. The hipify-perl script and the Clang-based hipify-clang rewrite CUDA runtime calls (cudaMalloc becomes hipMalloc, cudaMemcpyAsync becomes hipMemcpyAsync), kernel launch syntax and library calls (cublasSgemm becomes hipblasSgemm). Because HIP was designed as a near-clone of the CUDA runtime API, the mechanical conversion usually compiles with few edits for straightforward kernels.

Figure 2: The HIP porting loop. Hipify handles the mechanical rewrite; wavefront-size and inline-assembly fixes are manual and feed back into the HIP source.

A minimal flow, which I have not executed on hardware in this run, so treat it as a template rather than verified output:

# 1. Convert CUDA source to HIP (perl-based, no Clang dependency)
hipify-perl saxpy.cu > saxpy.hip.cpp

# 2. Compile for the Instinct targets you actually deploy
hipcc --offload-arch=gfx942 --offload-arch=gfx950 -O3 saxpy.hip.cpp -o saxpy

# 3. Confirm what the device reports
rocminfo | grep -i gfx

// A wavefront-safe reduction: never hardcode 32
__device__ float wave_sum(float v) {
    for (int offset = warpSize / 2; offset > 0; offset >>= 1) {
        v += __shfl_down(v, offset);   // warpSize is 64 on CDNA
    }
    return v;
}

The human 10 percent is where schedules slip. The recurring items are these.

• Wavefront width. Reductions, ballots, __shfl_sync masks that assume 32 lanes, and shared-memory bank assumptions need review. Mask types also differ: CUDA’s 32-bit masks do not cover 64 lanes.
• Inline PTX. There is no automatic translation of PTX to AMD ISA. Each asm volatile("...") block must be rewritten with HIP intrinsics or AMDGPU assembly.
• Tensor-core intrinsics. WMMA and Hopper or Blackwell-specific instructions (TMA, wgmma, tcgen05-style paths) have different AMD equivalents, the MFMA matrix instructions, with different tile shapes. Libraries such as Composable Kernel exist so you do not write these by hand.
• Library gaps. Not every cuDNN, cuSPARSE or CUTLASS feature has a drop-in equivalent, and hipify will happily emit calls that compile but are slow.
• Performance, not correctness. A kernel tuned for NVIDIA’s 228 KB shared memory per SM and 32-wide warps will run correctly and perform poorly on CDNA until retuned. Correctness is a day of work; parity is a quarter.

ROCm 7.1’s release notes (October 2025) show AMD continuing to close API gaps, adding HIP APIs for CUDA compatibility such as nested tile partitioning within cooperative groups, and reducing HIP module-load latency. The direction is clear, but the end state is “compatible subset”, not “identical”.

For teams with no custom kernels, none of this applies. A PyTorch model plus stock vLLM requires no HIP porting. Porting is a cost only for organizations that own CUDA extensions: custom attention variants, fused optimizers, bespoke quantization kernels, or point-cloud and simulation code. If that is you, inventory your .cu files before you read another benchmark.

RCCL versus NCCL: same API, different fabric

RCCL (ROCm Communication Collectives Library) implements the NCCL API: ncclAllReduce, ncclAllGather, ncclReduceScatter, and so on, so PyTorch’s torch.distributed with the “nccl” backend name works unmodified on ROCm; the backend string stays nccl and dispatches to RCCL. Environment variables such as NCCL_DEBUG=INFO are honoured, and RCCL adds its own tuning knobs.

Figure 3: Intra-node RCCL traffic rides Infinity Fabric links; inter-node traffic leaves through NICs. Ring and tree algorithms are chosen from the detected topology.

The API match hides a topology difference that matters for how you design clusters. An NVIDIA HGX node connects its eight GPUs through NVSwitch, giving every GPU full-bandwidth access to every other. AMD’s MI300X-class platforms use a point-to-point Infinity Fabric mesh: each GPU has 8 links at 128 GB/s peak per AMD’s specification page, but in an eight-GPU node each GPU is directly connected to the other seven rather than through a switch, so the bandwidth available between any specific pair is a fraction of the aggregate. For ring-style all-reduce this is fine, because each step is neighbour-to-neighbour; for patterns that stress a single pair, such as certain all-to-all steps in mixture-of-experts routing, the effective per-pair bandwidth is lower than a switched fabric would provide. The MI355X lists 7 links at 144 GB/s each. Scale-out is separate: the MI355X page lists one scale-out link per GPU at 128 GB/s peak, typically a 400 Gb/s-class NIC per GPU over RoCE or InfiniBand, which is where the network-design guidance in our NVLink vs InfiniBand vs RoCE article applies directly.

Two maturity caveats are worth stating plainly. First, the ROCm Core SDK 10.1.0 release notes list RCCL 2.30.7 and carry a known issue in which LL-protocol all-reduce might produce incorrect results on MI350 series GPUs. A silent-wrong-answer bug in a collective is the most dangerous class of defect in training, because loss curves diverge slowly and blame lands on the data. Check the current known-issues list and pin protocol selection if your version is affected. Second, NCCL benefits from years of tuning against NVIDIA’s NVLink and InfiniBand SHARP in-network reduction; RCCL’s at-scale behaviour on tens of thousands of GPUs has fewer public data points.

Worked numbers: why memory decides inference

Decode-phase LLM inference is memory-bandwidth-bound at small batch sizes: each generated token requires streaming essentially all model weights from HBM once. That gives a clean upper bound, tokens per second per sequence at batch 1 equals bandwidth divided by weight bytes. These are theoretical ceilings from vendor peak bandwidth, not measured numbers; real kernels reach perhaps 60 to 80 percent of peak on good days (my estimate, not a sourced figure).

For a 70-billion-parameter dense model:

Precision	Weight size	MI300X (5.3 TB/s)	MI355X (8 TB/s)
FP16/BF16	140 GB	about 38 tok/s ceiling	about 57 tok/s ceiling
FP8	70 GB	about 76 tok/s ceiling	about 114 tok/s ceiling
FP4 (MXFP4)	35 GB	not native	about 229 tok/s ceiling

Capacity matters as much as bandwidth. A Llama-3-70B-class model has 80 layers, 8 key-value heads and a head dimension of 128. In FP16 the KV cache costs 2 (K and V) x 80 x 8 x 128 x 2 bytes = 327,680 bytes, about 320 KiB per token. A 32,000-token context therefore consumes about 10.5 GB per sequence. On a single MI300X (192 GB) holding 140 GB of FP16 weights, the remaining roughly 50 GB (before activations and runtime overhead) supports only about four such long sequences. Quantize the weights to FP8 and the free space becomes about 120 GB, around eleven sequences, nearly tripling concurrency. On a 288 GB MI355X with FP8 weights, about 218 GB remains, enough for roughly twenty 32k-token sequences before overhead, or more if the KV cache itself is also quantized.

That is the core inference argument for large-HBM parts: a 70B model fits on one device with no tensor parallelism, so you pay no all-reduce latency per layer and need no NVLink-class fabric for serving. A 405B-parameter model is 405 GB in FP8 and will not fit on one device of either generation, but fits comfortably on an eight-GPU node (1,536 GB on MI300X, 2,304 GB on MI355X), leaving large KV headroom. For the scheduler behaviour that fills this memory, see chunked prefill and prefill-decode interference.

Compute-bound phases (prefill and large-batch training) follow a different rule. Divide peak FLOPS by bandwidth to get the arithmetic intensity at which a kernel stops being memory-bound: MI300X FP16 is 1.3e15 / 5.3e12, about 245 FLOP per byte; MI355X FP16 is 2.5e15 / 8e12, about 312; MI355X FP8 is 5e15 / 8e12, about 625. A GEMM must reuse each byte that many times to approach the peak. Large training matmuls clear that bar; decode attention never does, which is why the quoted PFLOPS are nearly irrelevant to token latency.

Training: what the public evidence supports

Training is where claims need the most skepticism. The one clean, audited public data point comes from MLPerf Training v5.1, where AMD’s technical write-up reports an MI355X Llama 2 70B LoRA fine-tuning time-to-train of 10.18 minutes (submission ID 5.1-0018), a 2.8x improvement over MI300X, with nine partners (Asus, Cisco, Dell, Giga Computing, Krai, MangoBoost, MiTAC, QCT and Supermicro) submitting AMD results. Two caveats from the same write-up deserve emphasis. Both of AMD’s benchmarks used FP8, and AMD did not submit FP4 results, so MI355X’s headline FP4 peak was not exercised in the audited training results. And no multi-node AMD numbers were reported for the benchmarks discussed, so MLPerf does not tell you how a 4,000-GPU job behaves.

Fine-tuning a 70B model with LoRA on a handful of GPUs is a different regime from pre-training a frontier model across thousands. The former is dominated by single-node GEMM, attention and memory behaviour, which suits large-HBM parts. The latter is dominated by collective efficiency, straggler tolerance, checkpoint I/O and the operational maturity of the cluster. Public evidence for the first regime on AMD is solid and improving; evidence for the second is thinner, though large AMD deployments exist and several hyperscalers named as Helios partners suggest it is being exercised privately.

A useful way to compare is by training memory budget. With mixed-precision Adam, the per-parameter state is roughly 16 bytes (2 for BF16 weights, 2 for gradients, 4 for FP32 master weights, 8 for the two Adam moments), a standard accounting from the ZeRO literature. A 70B model therefore needs about 1.12 TB of optimizer-and-weight state before activations. Sharded across a single eight-GPU MI300X node (1,536 GB total) it fits with about 400 GB left for activations; on an eight-GPU MI355X node (2,304 GB) the headroom is about 1.18 TB. More headroom means less activation recomputation and less pipeline parallelism, both of which cost throughput. This is a derived estimate, not a benchmark: it ignores fragmentation, communication buffers and framework overhead.

Choosing a path: the decision picture

Figure 4: A workload-first decision flow. Memory-fit serving favours AMD; custom CUDA kernels and very large training runs raise AMD’s migration cost.

Scenario	Leans AMD Instinct	Leans NVIDIA CUDA
Serving a 70B-class model, long context	Yes: single-GPU fit at FP8 on MI300X and above	If you already run TensorRT-LLM and its tuned stack
Serving a very large MoE	Yes, if node memory holds experts	If you need NVL72-scale switched domains today
Fine-tuning with LoRA or QLoRA	Yes: MLPerf evidence, big HBM	Equal; ecosystem breadth favours NVIDIA
Pre-training at thousands of GPUs	Needs a pilot and a named escalation path	Lower operational risk
Heavy custom CUDA or CUTLASS kernels	Only with porting budget	Yes
Research on brand-new architectures	Expect lag on novel kernels	Yes

The same hardware logic that favours large HBM for dense LLMs also shows up in hybrid state-space models and long-context designs, which change the compute-to-memory ratio again; our write-ups on Mamba and hybrid architectures and native sparse attention cover how those shifts interact with memory bandwidth.

Total cost: the number the spec sheet leaves out

Power and cooling reshape the comparison. The MI355X is listed at 1,400 W typical board power against 750 W for MI300X. An eight-GPU node’s accelerators alone draw 11.2 kW at the MI355X rating (8 x 1,400 W), before CPUs, NICs and fans, which is why high-power Instinct deployments are generally liquid-cooled. If your data hall is built for 10 kW racks, the cost of retrofitting cooling can exceed the cost difference between GPU vendors. Conversely, performance per watt can favour the newer part: FP8 peak per watt is 2.61e15/750, about 3.5 TFLOPS per watt on MI300X, versus 5e15/1400, about 3.6 on MI355X at the dense FP8 rating, nearly flat, with the real efficiency gain arriving only when FP4 or FP6 formats are usable (10.1e15/1400, about 7.2 TFLOPS per watt). Peak ratings are not application efficiency, so benchmark your own model before committing capital.

Engineering time is the other hidden line. A port that takes two engineers three months costs on the order of half an engineer-year, which at typical loaded costs can equal the price of several accelerators. If your fleet is 16 GPUs, that port rarely pays; if it is 2,000, it almost always does. This scale-dependence is why the same technical facts lead different organizations to opposite, correct decisions.

Trade-offs, Gotchas, and What Goes Wrong

The failures I see repeated on AMD deployments are rarely about peak performance. They are about the edges of the support matrix and the speed of the surrounding ecosystem.

Version-matrix drift. ROCm, the kernel driver, the framework wheel, the inference engine and the container base image must align. Core SDK 10.1.0 validates PyTorch 2.14.0, vLLM 0.29.0 and SGLang 0.5.18; a newer vLLM may use an operator path that the ROCm kernels do not yet cover, and the fallback is silently slower. Pin versions, use the AMD-published containers as a baseline, and treat upgrades as controlled migrations with rollback. A third-party review of the ROCm 10 launch gives the same advice for production systems with custom kernels: migrate with representative load testing.

Fallback performance cliffs. When a fused kernel is missing, frameworks often fall back to an unfused PyTorch path. The model still runs and produces correct output; throughput can fall by a large factor with no error message. Always profile a representative request mix and compare against a published or internal baseline rather than trusting “it runs”.

Silent numerical issues. The RCCL LL-protocol known issue noted earlier is an example. Run a deterministic numerical regression, for example a fixed-seed loss comparison for a few hundred steps against a reference run, after every stack change.

Quantization format mismatch. FP8 comes in more than one encoding. AMD’s earlier MI300 generation and the newer MI350 generation differ in supported FP8 variants and in the OCP microscaling formats (MXFP8, MXFP6, MXFP4) that only CDNA 4 handles natively. A checkpoint quantized with an NVIDIA-specific recipe (for example, tied to a particular scale layout) may need re-quantization. Do not assume a checkpoint is portable.

Profiling gaps. NVIDIA’s Nsight tools are polished. ROCm’s profilers (ROCprofiler-SDK and related tools) have improved, with the 10.1 release notes listing kernel replay in beta and HIP event tracing, but debugging a stall in a multi-node job still takes more manual correlation.

Cooling and power surprises. Higher-TBP parts need facility planning, covered above.

Support-lifecycle risk. AMD removes things. The 10.x transition removed ROCm SMI in favour of AMD SMI, moved several tools into an extras package, and dropped TensorFlow 2.19.1 support in the 10.1 notes. If your tooling depends on a deprecated component, find out before an upgrade window.

Anti-patterns. Buying hardware based on a vendor peak-FLOPS table. Porting every CUDA extension on day one instead of identifying the three that matter. Running a pilot on a single node and extrapolating to a cluster. Comparing AMD FP8 against NVIDIA FP16, or vice versa, and calling it a fair test. Treating vendor-reported rack comparisons, such as AMD’s claims for Helios against NVIDIA’s next-generation rack, as independent results.

Practical Recommendations

If your workload is inference of open-weight models in the 30B to 400B range, an AMD pilot is justified in 2026, because memory capacity directly reduces parallelism overhead and raises concurrency. If your workload is large-scale pre-training with a deep library of custom CUDA, the safer plan is to stay on CUDA for the production run and evaluate AMD on fine-tuning or secondary workloads first. If you are building new capacity and your procurement leverage matters, a credible AMD qualification is itself valuable even if you buy mostly NVIDIA.

Run the pilot like an experiment rather than a demo. Define the metric (tokens per second per dollar at a fixed latency SLO, for example p95 time-to-first-token under 500 ms and inter-token latency under 50 ms), fix the model and quantization, and measure on both platforms with the same traffic replay. Include the engineering hours spent in the total.

Use this checklist before committing:

• Confirm the GPU architecture, OS, kernel, framework and engine versions appear in the ROCm support matrix for your target release.
• Inventory every .cu, .cuh, inline-PTX and CUTLASS dependency; classify each as drop, replace-with-library, or port.
• Verify that your model’s operators (attention variant, MoE routing, quantization format) have ROCm kernels in the exact framework version you will run.
• Run rccl-tests or a framework all-reduce benchmark on your actual topology, and record achieved bus bandwidth against theoretical.
• Check the current RCCL and ROCm known-issues pages for correctness bugs affecting your GPU generation.
• Compute power and cooling requirements per node from TBP, not from the previous generation.
• Pin container images, keep rollback images, and add a numerical regression test to CI.
• Measure on your own traffic; treat all vendor comparisons, including those cited here, as hypotheses.

For the control-oriented side of this site’s readers, the same discipline of measuring against a model of the plant before trusting a controller appears in our piece on model predictive control with digital twins; GPU selection benefits from the same habit of constraint-aware, measured planning.

Frequently Asked Questions

Is ROCm as good as CUDA in 2026?

It depends on the workload. For mainstream open-weight LLM inference and fine-tuning on Instinct hardware, ROCm is production-viable, with validated PyTorch, vLLM and SGLang releases and audited MLPerf results. CUDA still leads in breadth: new research kernels, tooling polish, third-party library support and large-scale training track record. The gap is narrowest in popular-model serving and widest for custom kernels and novel architectures.

Can I run CUDA code on AMD GPUs?

Not directly. You convert CUDA source to HIP with hipify-perl or hipify-clang, then compile with hipcc for your target architecture, such as gfx942 for MI300 series or gfx950 for MI350 series. Simple kernels convert mechanically. Code with inline PTX, hardcoded 32-wide warp assumptions, or NVIDIA-specific tensor-core intrinsics needs manual work. PyTorch model code usually needs no changes.

Does PyTorch work on AMD Instinct GPUs?

Yes. ROCm builds of PyTorch expose AMD GPUs through the same torch.cuda API, so torch.cuda.is_available() returns true and .to("cuda") targets the Instinct device. torch.version.hip distinguishes the build. ROCm Core SDK 10.1.0 lists PyTorch 2.14.0 as supported. You must install a ROCm-specific wheel or container, not the default CUDA wheel.

Is RCCL a drop-in replacement for NCCL?

At the API level, yes: RCCL implements the NCCL interface, and PyTorch’s “nccl” backend name dispatches to RCCL on ROCm. Behaviour differs underneath. AMD nodes use point-to-point Infinity Fabric rather than a switched NVLink fabric, tuning knobs differ, and at-scale public data is thinner. Always benchmark all-reduce on your topology, and check known issues for your RCCL version.

Which is better for LLM inference, MI300X or MI355X?

MI355X has more memory (288 GB versus 192 GB), more bandwidth (8 TB/s versus 5.3 TB/s) and native FP4 and FP6, so it raises the ceiling on both concurrency and single-stream speed. MI300X remains cost-effective for models up to about 70B in FP8. By the bandwidth-bound decode ceiling, MI355X offers about 1.5x higher tokens per second per stream at equal precision, before software effects. Power draw nearly doubles.

Should I wait for MI400 and Helios?

Only if your timeline allows and your workload needs rack-scale 72-GPU domains. AMD announced Helios at Advancing AI 2026 and describes it as in production, but I could not verify general availability dates, and AMD’s comparisons to NVIDIA’s next rack are its own claims. For inference capacity needed in the next two quarters, MI300X and MI355X are shippable now, and software investment transfers across generations.

Further Reading

• Chunked prefill and prefill-decode interference in vLLM and SGLang: how serving schedulers consume the memory discussed above.
• NVLink vs InfiniBand vs RoCE for NCCL all-reduce: cluster network design that applies equally to RCCL.
• FlashAttention and IO-aware attention: the attention kernels that sit at the heart of the porting gap.
• Model predictive control with digital twins: measured, constraint-aware planning for industrial systems.
• ROCm documentation and release notes: the support matrix and known issues.
• AMD’s MLPerf Training v5.1 technical write-up: the audited results cited here.

By Riju — about