NVFP4 vs MXFP4: 4-Bit Floating Point for LLM Training and Inference (2026)

Both formats store every number in four bits, use the same element type, and run on the same Blackwell tensor cores. Yet the NVFP4 vs MXFP4 question keeps producing different answers in practice: NVIDIA reports that a model trained in MXFP4 needed about 36% more training tokens to reach the loss NVFP4 reached, while OpenAI shipped its open-weight gpt-oss models in MXFP4 and they run well on a single 80 GB GPU. The difference is not the 4-bit element. It is the scale factor wrapped around every small block of elements, and a handful of design choices about how that scale is stored, shared, and rounded.

This matters now because 4-bit floating point has moved from research curiosity to the default low-precision path on current NVIDIA hardware, and from inference into pretraining. Picking the wrong format costs you either accuracy you cannot recover or memory you did not need to spend.

This article derives the difference from first principles, walks through the quantization math with a worked example and runnable emulation code, summarizes the published NVFP4 training recipe, and ends with a decision framework you can apply to your own hardware and quality bar.

What this covers: the E2M1 element, how microscaling works, the exact structural differences between the two formats, why scale precision dominates error, the NVFP4 training recipe, deployment realities, failure modes, and a decision matrix.

Context and Background

For most of the deep learning era, precision shrank in discrete steps. FP32 gave way to FP16 and BF16 for training, then FP8 arrived with Hopper. Each halving cut memory traffic and roughly doubled peak tensor throughput. Inference went further down a separate path, using integer weight-only quantization (INT8, then INT4) because decoding is bound by memory bandwidth and weights dominate the bytes moved. If you want the integer-side picture, our comparison of GGUF, AWQ, GPTQ, and FP8 quantization formats covers what those schemes trade away.

Four bits is where plain floating point breaks. A 4-bit float with a sign, two exponent bits, and one mantissa bit (E2M1) can represent only a handful of magnitudes. The dynamic range is far too narrow to hold a tensor whose values span several orders of magnitude, and a single outlier forces every other value toward zero. The fix, which predates both formats here, is block scaling: split the tensor into small blocks, store one scale per block, and let the 4-bit elements describe values only relative to that scale. Block floating point is an old idea in signal processing; the modern deep-learning versions are called microscaling.

In 2023 the Open Compute Project published the Microscaling Formats (MX) Specification v1.0, developed by a group that includes AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm. It defines a family of formats (MXFP8, MXFP6, MXFP4, and MXINT8) that all share the same block layout: 32 elements and one shared 8-bit power-of-two scale. The specification is open and license-free, which is why you now see MX support beyond NVIDIA. AMD’s Instinct MI350 series, built on CDNA 4, advertises native MX support, though you should check the current AMD documentation for exactly which MX types and rates apply to your part.

NVIDIA’s Blackwell generation, with fifth-generation Tensor Cores, implements the MX formats natively and adds its own variant, NVFP4. NVFP4 keeps the same 4-bit element but changes the scaling to be finer and more precise. NVIDIA positions it as the higher-accuracy option and has published both inference results and a pretraining recipe for it. The rest of this article is about why that finer scaling matters, what it costs, and when the portable MX variant is the better choice.

One framing note before the details. Most “FP4” marketing numbers describe peak tensor throughput, and peak throughput is not what your model achieves. Real models are limited by outlier behavior, attention, normalization, and communication. Anything this article cites from a vendor is labeled as reported, and anything I computed myself is labeled as an illustrative emulation. If you work on the kernel side, the memory-hierarchy discussion in our FlashAttention 2, 3, and 4 explainer is the right companion, because attention is where 4-bit formats struggle most.

The Reference Architecture: One Element, Two Scaling Schemes

Direct answer: MXFP4 and NVFP4 both store 4-bit E2M1 elements. MXFP4 shares one power-of-two (E8M0) scale across 32 elements. NVFP4 shares one FP8 E4M3 scale across just 16 elements, plus a second FP32 scale per tensor. The finer block and the non-power-of-two scale cut quantization error at a cost of about 0.25 extra bits per value.

Figure 1: Structure of an MXFP4 block (left) and NVFP4 two-level scaling (right). Both share the same 4-bit element; only the scale hierarchy differs.

Figure 1 shows the two layouts side by side. In MXFP4 each group of 32 values carries one 8-bit scale. In NVFP4 each group of 16 values carries one 8-bit scale, and that scale is itself expressed relative to a single FP32 number for the whole tensor. The dequantized value in both cases is the 4-bit element multiplied by the scale, with NVFP4 applying the per-tensor factor on top.

The E2M1 element: only eight magnitudes

The element is identical in both formats. E2M1 has one sign bit, two exponent bits, and one mantissa bit. The positive values it can represent are 0, 0.5, 1, 1.5, 2, 3, 4, and 6, and the negatives mirror them. There is no infinity and no NaN in the element; those concerns move to the scale. That is eight magnitudes plus sign, which is sixteen codes.

Notice the spacing. Between 0 and 1 the grid steps by 0.5. Between 1 and 2 it steps by 0.5. Between 2 and 4 it steps by 1, and between 4 and 6 it steps by 2. The relative error is roughly constant (it is floating point), but the absolute error is largest near the top of the range. A value that lands between 4 and 6 can be off by up to 1 in scaled units. This non-uniformity matters later, because the best scale is not always the one that maps the block maximum exactly onto 6.

Why the scale format matters more than the element

Since the element is shared, all of the accuracy difference between the formats comes from three structural choices.

First, block size. MXFP4 uses 32 elements per scale; NVFP4 uses 16. An outlier inflates the scale for every element in its block, pushing small values toward zero. With 16 elements, an outlier harms half as many neighbors. NVIDIA’s own description is that the smaller block gives twice as many opportunities to match the local dynamic range.

Second, scale precision. The MX scale is E8M0: eight exponent bits, no mantissa, no sign. It can only express powers of two, from 2 to the power of minus 127 up to 2 to the power of 127. When the ideal scale for a block is, say, 1.3 times a power of two, the format must round to a power of two, and either the block maximum overflows past 6 and clips, or the grid is stretched and resolution is wasted. NVFP4’s block scale is E4M3, an FP8 type with three mantissa bits, so it can express fractional scales like 1.375. The scale lands within a few percent of ideal instead of up to a factor of the square root of two off.

Third, the second level. E4M3 has a limited range (maximum 448, and a small minimum), so tensors whose natural block scales fall outside that window would saturate. NVFP4 solves this by dividing everything by a per-tensor FP32 scale first, which recenters the block scales into E4M3’s usable range. MXFP4 does not need this because E8M0 already spans an enormous range; it pays for that range by being coarse.

Storage overhead and the effective bit width

Counting scale bits is simple arithmetic. MXFP4 stores 32 four-bit elements plus an eight-bit scale: 136 bits per 32 values, or 4.25 bits per value. NVFP4 stores 16 four-bit elements plus an eight-bit scale: 72 bits per 16 values, or 4.5 bits per value, plus one FP32 scalar per tensor that is negligible for large tensors.

So NVFP4 costs about six percent more memory than MXFP4 for the same number of weights. Against FP16 it still shrinks the model by roughly 3.5 times, and NVIDIA reports about 1.8 times smaller than FP8. These figures are the vendor’s and they ignore any layers you keep in higher precision, which in practice you will.

Hardware execution

What makes these formats practical rather than merely compact is that the tensor core handles the scaling in hardware. On Blackwell the fifth-generation Tensor Cores take packed 4-bit operands with their block scales, apply the scales during the matrix multiply, and accumulate at higher precision. You do not dequantize to FP16 and then multiply; that would forfeit the throughput gain. NVIDIA’s inference article lists peak low-precision dense throughput of about 9 petaflops for B200 and 15 for GB300 against 1.9 for H100, which are peak numbers at the lowest supported precision, not application speedups.

Figure 2: The NVFP4 quantization pipeline. A global amax sets the FP32 tensor scale, per-block amax values are rounded into E4M3 scales, elements are rounded to the E2M1 grid, packed two per byte, and consumed by a scaled tensor core GEMM.

Figure 2 traces a tensor through that pipeline. The cost that does not appear in peak-throughput tables is the quantization step itself: computing block maxima, rounding scales, and packing bytes. Production stacks fuse this into the preceding kernel so activations are quantized on the way out of the previous operation instead of making an extra trip through memory. That fusion is why NVIDIA frames W4A4 (4-bit weights and 4-bit activations) as an end-to-end kernel problem rather than a single instruction.

Deeper Analysis: Error Math, a Worked Example, and Training

A worked example, by hand

Take a block whose real values are 0.1, -0.3, 0.7, 1.1, -0.2, 0.05, 0.9, and 2.6, with the other values in the block near zero. The block maximum is 2.6.

For MXFP4 the shared scale must be a power of two. The common rule picks an exponent from the block maximum and the element format’s largest exponent; for E2M1 that works out to a scale of 0.5 for this block. Dividing by 0.5 gives 0.2, -0.6, 1.4, 2.2, -0.4, 0.1, 1.8, and 5.2. Rounding each to the E2M1 grid and multiplying back gives approximately 0, -0.25, 0.75, 1.0, -0.25, 0, 1.0, and 3.0. The largest value, 2.6, comes back as 3.0, an error of 0.4, because 5.2 is nearer to 6 than to 4 on a coarse grid.

For NVFP4 the block scale can be fractional. Mapping the maximum exactly to 6 gives a scale of 2.6 divided by 6, about 0.433, and E4M3 can store that to within a few percent. The same values divided by 0.433 are 0.23, -0.69, 1.62, 2.54, -0.46, 0.12, 2.08, and 6.0. After rounding to the grid and rescaling the reconstruction is approximately 0, -0.217, 0.65, 1.3, -0.217, 0, 0.867, and 2.6. The block maximum is reproduced exactly and the others are closer overall.

This is a single hand-picked block and proves nothing statistically; its job is to show the mechanism. Rounding a power-of-two scale cannot track the block maximum, while a fractional scale can. NVIDIA’s inference blog reports a similar single-example comparison where E4M3 scaling gave a mean squared error of 0.08 against 0.72 for E8M0 scaling, and a separate NVIDIA research write-up states roughly 88% lower quantization error than power-of-two scaling. The blog does not publish its measurement setup, so treat those as illustrative of direction rather than a general constant.

Emulating both formats yourself

The following script emulates both formats in NumPy so you can measure error on your own tensors. It is a simulation of the number formats, not of the hardware: it ignores packing, the exact tie-breaking rules in the specification, and denormal handling in the scales. Use it for intuition and for sanity-checking the direction of an effect, never to claim hardware accuracy.

import numpy as np

GRID = np.array([0, .5, 1, 1.5, 2, 3, 4, 6.0])  # positive E2M1 magnitudes

def rn_fp4(x):
    s, a = np.sign(x), np.abs(x).clip(0, 6)
    return s * GRID[np.abs(a[..., None] - GRID).argmin(-1)]

def e4m3_round(x):
    x = np.clip(x, 0, 448); out = np.zeros_like(x); m = x > 0
    e = np.maximum(np.floor(np.log2(x[m])), -6)   # min normal exponent
    step = 2.0 ** (e - 3)                          # 3 mantissa bits
    out[m] = np.round(x[m] / step) * step
    return np.clip(out, 0, 448)

def mxfp4(x, bs=32):
    x = x.reshape(-1, bs)
    amax = np.abs(x).max(1, keepdims=True)
    s = 2.0 ** (np.floor(np.log2(amax)) - 2)       # E8M0: power of two only
    return (rn_fp4(x / s) * s).ravel()

def nvfp4(x, bs=16):
    x = x.reshape(-1, bs)
    ts = np.abs(x).max() / (6 * 448)               # FP32 per-tensor scale
    amax = np.abs(x).max(1, keepdims=True)
    s = e4m3_round(amax / 6 / ts) * ts             # E4M3 block scale
    s = np.where(s == 0, 1, s)
    return (rn_fp4(x / s) * s).ravel()

rng = np.random.default_rng(0)
n = 1 << 20
gauss = rng.standard_normal(n)
outl = rng.standard_normal(n) * (1 + 9 * (rng.random(n) < 0.002))
for name, x in [("gaussian", gauss), ("with outliers", outl)]:
    for f, tag in [(mxfp4, "MXFP4"), (nvfp4, "NVFP4")]:
        y = f(x); mse = ((x - y) ** 2).mean()
        print(name, tag, "MSE %.5f" % mse,
              "SQNR %.2f dB" % (10 * np.log10((x**2).mean() / mse)))

On my run of this emulation (seed 0, one million samples) the output was as follows. These are my own illustrative numbers from a toy simulation, not a benchmark of real model weights.

Synthetic data	Format	MSE	SQNR (dB)
Gaussian	MXFP4	0.01324	18.79
Gaussian	NVFP4	0.00904	20.44
Gaussian with rare 10x outliers	MXFP4	0.01983	17.78
Gaussian with rare 10x outliers	NVFP4	0.01062	20.49

Two patterns are worth taking away, with the caveat that synthetic data is only suggestive. NVFP4 wins on clean Gaussian data by a modest margin, because finer blocks and a precise scale both help. The gap widens when outliers are present: NVFP4’s error barely moves while MXFP4’s grows noticeably, since the 32-wide block with a coarse scale lets each outlier damage more neighbors. Real LLM activations are known for heavy-tailed outliers in specific channels, which is why this is the regime that matters.

What the published training evidence says

The strongest primary evidence on training is NVIDIA’s paper “Pretraining Large Language Models with NVFP4” (arXiv 2509.25149). The authors trained a 12-billion-parameter model on 10 trillion tokens, which they describe as the longest publicly documented 4-bit training run. The result is that the NVFP4 run reached training loss and downstream accuracy comparable to an FP8 baseline. All the figures in this section are reported by NVIDIA in that paper, and I have not reproduced them.

On downstream tasks for the 12B model, the paper’s table shows FP8 against NVFP4 results that are close but not identical. MMLU was 77.36 for FP8 and 76.57 for NVFP4. MMLU-Pro (5-shot) was essentially tied at 62.62 against 62.58. MATH was 83.32 against 81.48, GSM8k with chain of thought was 89.08 against 92.27 (NVFP4 higher), HumanEval+ was 59.93 against 57.43, MBPP+ was 59.11 against 55.91, and Winogrande was 80.58 against 78.77. The pattern is a small deficit on most tasks and a gain on one, with code benchmarks showing the largest gaps. “Comparable” is a fair word, but it is not “identical,” and code generation is the area to watch.

Validation loss tells a similar story. The paper reports relative loss error against FP8 staying below 1% during the stable phase of training and widening to slightly above 1.5% as the learning rate decayed. The authors also found that switching the forward pass to higher precision late in the run (at 8.2 trillion tokens) reduced the gap from about 1.5% to 0.5%. That is a useful practical lever: you can pay for precision only in the final stretch where the loss gap opens.

The head-to-head: NVFP4 vs MXFP4 in pretraining

The same paper contains the direct comparison most people cite. On an 8-billion-parameter model trained on 1 trillion tokens, the authors report a relative loss error of roughly 1.5% for NVFP4 and roughly 2.5% for MXFP4, and state that MXFP4 matches NVFP4’s loss only when trained on 36% more tokens. That is a compute and data cost, not a rounding detail: 36% more tokens is 36% more training time at fixed throughput.

Be careful about what this does and does not show. It is a single study from the vendor that sells the better-performing format, on one model size and one data budget, with a Hadamard transform size of 32 chosen for MXFP4 to match its block. Independent replication across model families would strengthen the claim. It is still mechanistically plausible, for the reasons derived above, and it is the best direct measurement currently available.

The training recipe, piece by piece

NVFP4 training is not “cast everything to 4 bits.” The paper combines four techniques, and each addresses a specific numerical problem.

Two-dimensional weight scaling. In a linear layer the forward pass uses weights along one axis and the backward pass uses the transposed weights along the other. If weights are quantized in 1-by-16 blocks, the forward and backward copies get different scales, so the backward pass sees a slightly different weight than the forward pass did. The recipe quantizes weights in 16-by-16 blocks so a block and its transpose share scales, giving consistent representations in both passes. Activations and gradients still use 1-by-16 blocks.

Random Hadamard transforms. Outliers in gradient and activation tensors inflate block scales. A Hadamard transform is an orthogonal mixing matrix; applied to the inputs of the weight-gradient GEMM, it spreads an outlier’s energy across many elements, making the distribution more Gaussian and easier to quantize. The paper applies it only to the weight-gradient inputs, with a 16-by-16 matrix. It reports that a 4-by-4 matrix increased loss and a 128-by-128 matrix gave only a minor benefit, so 16 is a pragmatic balance.

Stochastic rounding. Round-to-nearest is biased in a systematic way for gradients, because it consistently drops small contributions. Stochastic rounding rounds up or down with probability proportional to proximity, making the expected quantized value equal to the true value. The recipe uses it for gradients and round-to-nearest-even for weights and activations.

Selective high precision. A few layers are fragile. The 12B configuration keeps the first two blocks and the last eight blocks in BF16, which the paper counts as about 16% of the linear layers, and recommends keeping under about 15% in higher precision, mostly toward the end of the network. An ablation suggests the last four blocks alone may suffice. This means “4-bit training” in practice is mixed precision with 4-bit doing the bulk of the matrix math.

Figure 3: The NVFP4 training recipe. Weights use 16×16 block scales and activations 1×16 in the forward pass; gradients use stochastic rounding, and the weight-gradient path applies a random Hadamard transform. End layers remain in BF16.

Figure 3 summarizes where each technique plugs in. Notice that the master weights stay in higher precision and are re-quantized each step, which is the standard mixed-precision pattern.

Follow-on research: pushing error down further

Several later papers refine the same direction, and they show where the remaining error lives. The “Four Over Six” work observes that because floating-point grids are non-uniform, values near a block’s maximum suffer the largest absolute error; it adaptively scales some blocks so the maximum maps to 4 rather than 6, making the effective grid more evenly spaced near the top. The authors report that this brings pretraining loss closer to BF16 on a Nemotron 3 Nano 30B-A3B architecture than current NVFP4 recipes. A separate paper, “Quartet II”, argues that element-wise stochastic rounding costs representation quality, and proposes moving the randomness into the block scales instead. It reports a quadratic error about 2.5 times lower than stochastic rounding in a Gaussian test and training-loss improvements on small models. It also notes that the 4-over-6 scale selection is biased if used in the backward pass, so the two ideas are applied in different passes. These are research results on small and mid-size models; I treat them as promising rather than settled.

Deployment: Inference, Memory, and Real Models

Post-training quantization versus quantization-aware training

Inference is the easier use case, because you only need the forward pass. Two routes exist. Post-training quantization (PTQ) takes a finished BF16 or FP8 checkpoint, calibrates scales, and converts. Quantization-aware training (QAT) fine-tunes with the quantizer in the loop, so the weights adapt to the grid. PTQ is cheap and often enough for large models; QAT recovers accuracy when PTQ falls short, at the cost of training compute and data.

NVIDIA reports PTQ results for DeepSeek-R1-0528 going from FP8 to NVFP4: MMLU-Pro 85 to 84, GPQA Diamond 81 to 80, Humanity’s Last Exam 15 to 14, LiveCodeBench 77 to 76, SciCode 40 to 40, Math-500 98 to 98, and AIME 2024 89 to 91. The vendor summarizes this as one point or less of degradation on key language tasks. These are NVIDIA-reported numbers on a single, very large mixture-of-experts reasoning model, and they are rounded to whole percentages. Smaller dense models usually tolerate aggressive quantization less well than large ones, so do not assume the result transfers.

MXFP4 in the wild: gpt-oss

The best-known production MXFP4 deployment is OpenAI’s gpt-oss pair, released under Apache 2.0 as mixture-of-experts transformers. Secondary documentation describes the larger model as about 117 billion total parameters with about 5.1 billion active per token, and the smaller as about 21 billion total with about 3.6 billion active. Both use MXFP4 for the mixture-of-experts weights, which are the bulk of the parameters, at about 4.25 bits per parameter, and that is what lets the larger model fit on a single 80 GB GPU and the smaller one run in roughly 16 GB. I verified the Apache 2.0 license and mixture-of-experts design from the paper abstract; the parameter counts and memory figures come from a secondary summary, so confirm them against the official model card before you size hardware.

The gpt-oss case teaches two lessons. First, MXFP4 is good enough that a frontier lab chose it for a public release, which means the accuracy penalty versus a finer format is manageable when the weights are mixture-of-experts feed-forward layers that tolerate quantization. Second, it chose the open, portable format, which matters if you want the same checkpoint to run on more than one vendor’s accelerators.

Where each format shows up elsewhere

NVIDIA’s research write-up reports that the DeepSeek-V4 model uses NVFP4 with 16-element blocks in a quantization-aware setup, while gpt-oss uses MXFP4 with 32-element blocks. I could not verify the DeepSeek-V4 claim against a primary DeepSeek source, so treat it as reported by a secondary page. The same write-up reports an NVFP4 key-value (KV) cache up to about 50% smaller than FP8 with under 1% accuracy loss, and about 5% better than an MXFP4 KV cache. For attention, it describes SageAttention3, which stretches the softmax probability map to fill the 4-bit range before quantizing, raising its cosine similarity from 93.3% to 99.5%, and reports 99.52% attention accuracy for NVFP4 versus 98.37% for MXFP4. It also reports about 1,038 TOPS on an RTX 5090, roughly 5 times FlashAttention-2. These are reported numbers I have not independently reproduced. The same page concedes that FP4 attention still loses more accuracy than FP4 matrix multiplies, which matches the general experience.

Figure 4: A hardware-and-goal decision flow for choosing between NVFP4, MXFP4, and staying at FP8 or integer weight-only quantization.

Figure 4 encodes the decision logic developed in the rest of this section: hardware support first, quality bar second, and training versus inference third.

Throughput and energy claims

NVIDIA reports up to about 3 times higher peak inference throughput on Blackwell relative to FP8 and about 1.9 times faster pretraining on Llama 3.1 405B, though the source for the pretraining figure is a secondary summary and the exact configuration is not stated. NVIDIA’s inference blog also claims up to 25 times better energy per token for Blackwell and up to 50 times for Blackwell Ultra versus Hopper on a 1.8-trillion-parameter mixture-of-experts model, derived from 10, 0.4, and 0.2 joules per token. These combine hardware generation, software stack, and precision changes, so they should not be read as the gain from the number format alone. An independent measurement in the Quartet II paper found up to 2.5 times speedup on a B200 for 22B-shaped linear layers, and end-to-end gains of about 1.48 to 1.68 times on B200 for 3.3B to 11B models. That is a more modest and arguably more realistic picture of what an application sees.

A decision matrix

Situation	Better choice	Reasoning
Pretraining on Blackwell, quality-critical	NVFP4 with the full recipe	Measured token-efficiency advantage over MXFP4 in the NVIDIA study; recipe is documented
Serving a large MoE on Blackwell, accuracy matters	NVFP4 (PTQ, then QAT if needed)	Finer scale tracks outliers; reported near-FP8 accuracy on a large reasoning model
One checkpoint across NVIDIA and AMD accelerators	MXFP4	Open OCP specification; portable layout
Smallest footprint, tolerance for some loss	MXFP4	4.25 bits per value versus 4.5
Hopper or older GPUs	FP8 or integer weight-only	No native FP4 tensor core support; check your stack
Attention and softmax paths	Keep higher precision unless using specialized kernels	FP4 attention loses more accuracy than FP4 GEMMs

Trade-offs, Gotchas, and What Goes Wrong

Not everything should be 4-bit. The recipe itself keeps a fraction of layers in BF16, and attention, embeddings, normalization, and the output head are common candidates. A model that is “NVFP4” usually means the large linear layers are NVFP4. Any memory estimate that multiplies parameter count by 4.5 bits will be too low. Compute your footprint from the layer-by-layer precision map, not from the headline format.

Scale overhead is real but small. The extra 0.25 bits per value for NVFP4 over MXFP4 is about six percent. That is rarely decisive for capacity, but it is not zero when you are trying to fit one more model on a GPU. Also remember the activations and KV cache: the weights are only part of the memory budget at long context.

Outliers still win sometimes. Finer blocks reduce but do not eliminate the outlier problem. A single extreme value in a 16-element block still crushes its neighbors. That is why Hadamard rotations, smoothing, and rotation-based methods exist alongside block scaling. If your model has massive activation outliers in a few channels, expect to need them.

Calibration quality dominates PTQ. For post-training quantization, scales are computed from calibration data. If that data does not resemble production traffic, activation scales will be wrong in exactly the places that matter, and quality will drop in ways a clean benchmark does not show. Evaluate on your own workload, with long contexts and the languages and domains you serve.

Benchmarks hide regression. Reported accuracy changes of one percent are averages. The NVFP4 pretraining table shows code benchmarks (HumanEval+ and MBPP+) trailing FP8 by more than two points, while a math benchmark improved. If your product is code generation, the aggregate is not your number. Also be aware of contamination in public benchmarks, which can mask or exaggerate differences.

Training instability appears late. The loss gap widened during learning-rate decay in the 12B run. A run that looks healthy for the first several trillion tokens can drift at the end, so budget for a higher-precision finishing phase, as the paper’s late-run forward-pass switch suggests.

Stochastic rounding and reproducibility. Stochastic rounding adds randomness to the backward pass. Bit-exact reproducibility across hardware and kernel versions becomes harder, which complicates debugging a divergence. Fix seeds, log scale statistics, and keep an FP8 reference run for comparison.

Software maturity varies. FP4 kernels, framework integration, and export tooling are moving quickly. A format that is supported in one library version may behave differently in the next, and the open MX specification leaves some behaviors, such as rounding modes, to implementations. Pin versions and re-validate after upgrades.

Edge devices are a different world. If you are targeting microcontrollers or phone-class chips, none of this applies directly; those devices rely on integer arithmetic and tiny models. Our TinyML, TFLite Micro, and CMSIS-NN INT8 tutorial covers the integer path, where the same block-scaling ideas appear in simpler forms.

Practical Recommendations

My view, which is opinion informed by the evidence above and not a vendor claim: treat NVFP4 as the default 4-bit floating point format on Blackwell when quality matters, and treat MXFP4 as the right choice when portability or a smaller footprint outweighs a measurable accuracy gap. The two formats are close relatives. The cost of switching between them for inference is mostly a re-quantization and re-validation exercise, not a re-architecture.

For inference, start with PTQ from your best checkpoint, measure on your own tasks, and only move to QAT if a specific capability regresses. Compare against an FP8 baseline at the same model size, not against the vendor’s headline. For training, adopt the full recipe or none of it: the techniques interact, and removing the Hadamard transform or the BF16 end layers is where reported instabilities come from. Run a smaller FP8 versus NVFP4 pilot first and compare loss curves at equal tokens before committing a large run.

Also decide what you will do about attention. If your workload is long-context, the KV cache and attention kernel precision may dominate your memory and your error budget more than the weights do.

Checklist before you ship a 4-bit model:

• Confirm the target hardware has native FP4 tensor core support and that your serving stack uses it.
• Write down the per-layer precision map and compute memory from it.
• Calibrate on data that matches production traffic, including long contexts.
• Evaluate on your own task suite, including code and long-context retrieval, not only aggregate benchmarks.
• Keep an FP8 reference deployment for A/B comparison.
• For training, keep BF16 in the first and last blocks, use stochastic rounding for gradients, and plan a higher-precision finish.
• Pin library and kernel versions and re-run evaluations after every upgrade.
• Verify every vendor number you plan to quote against the primary source.

Frequently Asked Questions

What is the main difference between NVFP4 and MXFP4?

Both use the same 4-bit E2M1 element. MXFP4 shares one power-of-two E8M0 scale across 32 values. NVFP4 shares one FP8 E4M3 scale across 16 values and adds an FP32 per-tensor scale. The smaller block and the fractional scale let NVFP4 track local value ranges more closely, which lowers quantization error. The price is about 4.5 bits per value instead of about 4.25.

Is NVFP4 more accurate than MXFP4?

Published evidence says yes, with caveats. NVIDIA reports that on an 8B model trained on 1T tokens, MXFP4 needed about 36% more tokens to match NVFP4’s loss, and its relative loss error was about 2.5% versus 1.5%. That is vendor research on one setup. My own NumPy emulation on synthetic data also favored NVFP4, especially with outliers, but synthetic data is only suggestive.

Can I run NVFP4 or MXFP4 on Hopper or older GPUs?

Not natively with the speedup. Blackwell’s fifth-generation Tensor Cores execute both formats directly in hardware, including the scaling. On earlier GPUs you can store weights in 4 bits and dequantize before computing, which saves memory and bandwidth but not tensor core throughput. Check your inference framework’s documentation, because support and performance vary by version and kernel.

Is it possible to train an LLM entirely in 4-bit?

Not entirely. NVIDIA’s published recipe trained a 12B model on 10T tokens with NVFP4 matrix multiplies, but kept about 16% of linear layers (the first two and last eight blocks) in BF16, used 16-by-16 weight scaling, Hadamard transforms on weight-gradient inputs, and stochastic rounding for gradients. Master weights and many non-matmul operations stay in higher precision, so it is mixed-precision training with 4-bit doing most of the heavy math.

Which format should I choose for deploying a model across different vendors?

MXFP4, because it comes from the open OCP Microscaling Formats specification supported by multiple companies, and AMD’s MI350 series advertises native MX support. NVFP4 is NVIDIA’s own variant. If a single checkpoint must run on both vendors, MXFP4 is the safer common denominator. If you only target Blackwell and care about accuracy, NVFP4 is the stronger default.

Does 4-bit floating point hurt attention and long-context quality?

Often more than it hurts matrix multiplies. Attention involves a softmax whose probability map has a skewed distribution that fits a 4-bit grid poorly, and the KV cache accumulates error over long contexts. Research kernels such as SageAttention3 reshape the softmax map to improve this, and an NVFP4 KV cache is reported to keep losses under 1%, but most production stacks keep attention in higher precision.

Further Reading

• FlashAttention 2, 3, and 4: IO-aware attention, tiling, and FP8 explained
• GGUF vs AWQ vs GPTQ vs FP8: LLM quantization formats compared
• TinyML on microcontrollers: TFLite Micro, INT8 quantization, and CMSIS-NN
• Introducing NVFP4 for efficient and accurate low-precision inference (NVIDIA Developer Blog)
• Pretraining Large Language Models with NVFP4 (arXiv 2509.25149)

By Riju — about