Mixed-Precision Training Explained: FP16 vs BF16 vs FP8, Loss Scaling, Master Weights and What Breaks
Add 0.00003 to a BF16 weight of 0.02 a thousand times and the weight does not move at all. In FP32 the same loop ends at 0.05. That single experiment explains most of what mixed precision training is really about. The low-precision formats are not just “smaller floats”. Each one runs out of something different: exponent range, mantissa resolution or accumulation width. This matters now because FP8 is routine on Hopper-class GPUs, Blackwell adds hardware microscaling, and teams are quietly shipping 4-bit training recipes. Yet the failure modes are the same ones described in 2017, just at a smaller bit budget. You will leave with exact numbers for every format, the dynamic loss scaling algorithm, a per-parameter memory ledger, working PyTorch code and a debugging flow for NaNs and loss spikes.
What this covers: IEEE binary16, bfloat16, TF32 and OCP FP8 bit layouts; why FP16 gradients underflow; loss scaling; FP32 master weights; PyTorch AMP; FP8 scaling recipes; MX microscaling; DeepSeek-V3’s FP8 design; and what breaks.
Context and Background
Neural networks trained in 32-bit floating point for most of the deep learning era. The change began in October 2017, when Paulius Micikevicius and colleagues at NVIDIA and Baidu posted Mixed Precision Training, later published at ICLR 2018. The paper stored weights, activations and gradients in IEEE half precision and proposed three techniques to keep accuracy: an FP32 master copy of the weights, loss scaling, and FP16 arithmetic that accumulates into FP32. Volta’s Tensor Cores, which multiply FP16 matrices and accumulate into FP32, made the approach fast.
Google’s bfloat16 then offered a different trade. It keeps FP32’s 8-bit exponent and cuts the mantissa to 7 bits, so it has FP32’s range with far less resolution. A 2019 Intel and Facebook study, “A Study of BFLOAT16 for Deep Learning Training”, reported that BF16 matched FP32 results across image, speech, language and recommendation tasks without the hyperparameter tuning FP16 needs. NVIDIA added BF16 and TF32 Tensor Core support with Ampere, and BF16 became the default for large language model pre-training.
FP8 arrived next. The September 2022 paper “FP8 Formats for Deep Learning” by NVIDIA, Arm and Intel authors proposed two encodings, E4M3 and E5M2, which became the Open Compute Project’s OCP 8-bit Floating Point Specification (OFP8) in 2023. The OCP Microscaling (MX) Specification followed, adding a shared power-of-two scale for every block of 32 elements. Hardware followed the standards: Transformer Engine enables FP8 on compute capability 8.9 and above, and MXFP8 and NVFP4 on compute capability 10.0, which is Blackwell.
The incumbent tooling today is PyTorch’s torch.amp module for FP16 and BF16, NVIDIA Transformer Engine and PyTorch’s torchao for FP8, and framework-level recipes in Megatron-style trainers. If you also need to reason about how the same formats behave at inference, our FP8 vs INT8 vs INT4 LLM quantization benchmark covers that side.
The thesis of this article is that a mixed-precision recipe spends three separate budgets: range, resolution and accumulation. FP16 fails on range, so it needs loss scaling. BF16 fails on resolution, so it needs FP32 master weights. FP8 fails on both, so it needs fine-grained scaling and high-precision accumulation. Most training failures become easy to diagnose once you know which budget ran out.
How Mixed Precision Training Works: Range, Resolution and Accumulation
Mixed precision training runs the expensive matrix multiplications in a low-precision format such as FP16, BF16 or FP8, while keeping numerically fragile state in FP32. That state includes master weights, optimizer moments, reductions, softmax and normalization. Loss scaling or per-block scaling keeps small values representable, and FP32 accumulation keeps long dot products accurate.

Figure 1: One step of FP16 mixed precision training, from FP32 master weights to the overflow-checked optimizer update.
Figure 1 shows the loop that every FP16 recipe implements. The FP32 master weights are cast to a low-precision working copy. The forward pass runs under autocast, and the loss is computed in FP32. The loss is multiplied by a scale factor S before backpropagation, so every gradient is scaled by S through the chain rule. The gradients are then divided by S into FP32 and checked for Inf or NaN. A clean step updates the FP32 master weights; an overflowed step is skipped and S is reduced. BF16 recipes usually drop the scaling boxes but keep the rest.
The bit layouts decide which budget runs out first
Every format below is sign, exponent and mantissa. The exponent width sets dynamic range. The mantissa width sets relative resolution, measured by machine epsilon, the gap between 1.0 and the next representable number. We computed every value with NumPy and the ml_dtypes package, and cross-checked against torch.finfo in PyTorch 2.15.0.
| Format | Sign / exp / mantissa bits | Max finite | Min normal | Min subnormal | Machine epsilon |
|---|---|---|---|---|---|
| FP32 (binary32) | 1 / 8 / 23 | 3.40282e38 | 1.17549e-38 | 1.4e-45 | 1.19209e-7 |
| TF32 | 1 / 8 / 10 | about 3.40e38 | 1.17549e-38 | n/a in practice | 9.765625e-4 |
| FP16 (binary16) | 1 / 5 / 10 | 65,504 | 6.1035e-5 (2^-14) | 5.96e-8 (2^-24) | 9.765625e-4 (2^-10) |
| BF16 | 1 / 8 / 7 | 3.38953e38 | 1.17549e-38 | 9.18e-41 (2^-133) | 7.8125e-3 (2^-7) |
| FP8 E4M3 (OCP) | 1 / 4 / 3 | 448 | 0.015625 (2^-6) | 0.001953125 (2^-9) | 0.125 (2^-3) |
| FP8 E5M2 (OCP) | 1 / 5 / 2 | 57,344 | 6.1035e-5 (2^-14) | 1.526e-5 (2^-16) | 0.25 (2^-2) |
| FP4 E2M1 (MX element) | 1 / 2 / 1 | 6 | 1.0 | 0.5 | 0.5 |
Two rows deserve a closer look. TF32 is not a storage format. It is the input precision NVIDIA Tensor Cores use when they multiply FP32 matrices, so it is 19 meaningful bits inside a 32-bit container. In PyTorch 2.15.0, torch.get_float32_matmul_precision() still returns "highest" by default, which means TF32 matmuls stay off unless you opt in.
E4M3 bends the IEEE rules. As the FP8 paper puts it, E4M3 has no infinities and only one mantissa bit pattern for NaN, which buys it the range up to 448. E5M2 follows IEEE conventions and keeps infinities. One practical consequence: a value of 500 cast to E4M3 is not representable. Whether it saturates to 448 or becomes NaN depends on the conversion mode. The ml_dtypes cast we tested returns NaN, so saturation is a choice your kernel or library must make explicitly.
Why FP16 gradients underflow and BF16 gradients do not
Counted in powers of two, FP16’s normal range spans about 30 binades (2^-14 to 65,504), or 40 if you include subnormals. That sounds like plenty until you look at real gradients. The ICLR paper’s histogram of activation gradients for a Multibox SSD detector showed that 67% of the values were zero, and that values in the 2^-27 to 2^-24 range mattered for accuracy. Anything below 2^-24, about 5.96e-8, flushes to zero in FP16.
The fix is arithmetic, not magic. Multiplying the loss by S shifts every gradient exponent up by log2(S). We cast a set of FP32 gradients to FP16 to show the effect. Without scaling, 1e-9 and 1e-8 become exactly 0, and 5e-8 becomes 6e-8, which is the lone subnormal step nearest to it. Multiplied by 65,536 first, they become 6.557e-5, 6.552e-4 and 3.277e-3, all comfortably normal. The SSD network diverged without scaling and matched FP32 with a scale factor of just 8.
BF16 has the same 8-bit exponent as FP32, so its range runs from about 9.2e-41 to 3.4e38. Gradients that would underflow FP16 survive BF16 without any scaling. That is the core of the bf16 vs fp16 decision: BF16 removes the range problem entirely and trades it for a resolution problem.
Why FP32 master weights still matter, even with BF16
Resolution bites at the optimizer update. When you add a small update to a large weight, the hardware must align the binary points first. If the update is smaller than half a unit in the last place of the weight, it rounds away. The ICLR paper put the FP16 threshold at a weight-to-update ratio of about 2,048, because FP16 has 10 explicit mantissa bits.
BF16 has only 7 explicit mantissa bits, so the threshold drops to about 256. A weight of 0.02 sits in the binade from 2^-6 to 2^-5, where the BF16 spacing is 2^-13, about 1.22e-4. Any update below half that spacing, about 6.1e-5, vanishes. We verified this directly. Adding 3e-5 to 0.02 in BF16 returns the original value; repeated 1,000 times, the BF16 weight stays at 0.0200195 while the FP32 weight reaches 0.05. FP16 handles the same update, because its spacing near 0.02 is eight times finer.
This is why “pure BF16” training, with weights and optimizer state all in BF16, is risky late in a cosine schedule. As the learning rate decays, more updates fall under the rounding threshold and training stalls silently. The ICLR paper saw the extreme case in a Mandarin speech model, where updating FP16 weights directly cost 80% relative accuracy, while an FP32 master copy matched the FP32 baseline. For more on how optimizers shape these update magnitudes, see our deep-learning optimizers explainer covering AdamW, Lion and Muon.
Accumulation is the third budget
A matrix multiply of inner dimension K sums K products. If the running sum is kept in the input format, each addition rounds, and the error grows with K. The ICLR authors found some FP16 models only matched FP32 when dot products accumulated in FP32. They also recommended doing large reductions, such as normalization statistics and softmax sums, in FP32.
FP16 also overflows during accumulation. Summing 4,096 products of value 16 gives 65,536, which is already Inf in FP16. An unnormalized exponent overflows even sooner: in FP16, exp(11) is about 59,870, while exp(12) is Inf, because ln(65,504) is about 11.09. That is why softmax subtracts the row maximum before exponentiating, and why frameworks run softmax in FP32.
Accumulation also explains where the speedup comes from. Halving the bytes per value halves the traffic between high-bandwidth memory and the on-chip SRAM, and many training kernels are limited by that traffic rather than by arithmetic. Our GPU memory physics explainer on DRAM, SRAM and kernel fusion shows why bytes moved, not FLOPs, often set the ceiling. Mixed precision training therefore buys speed twice: faster Tensor Core math for the GEMMs and fewer bytes moved for everything else.
Walk-Through: From FP16 Loss Scaling to BF16 to FP8
This section walks the three generations of recipe in order. Each one adds a mechanism that pays off a specific budget. The code is current for PyTorch 2.15.0 and Transformer Engine 2.21.0, the latest releases we could install on 20 November 2026.
Dynamic loss scaling, step by step
A constant loss scale works if you know your gradient statistics. The ICLR paper trained networks with constant factors from 8 to 32K, and noted that many networks needed no factor at all. Its conclusion pointed to automating the choice by watching for overflows and skipping updates. Dynamic loss scaling does exactly that, and it is what torch.amp.GradScaler implements.

Figure 2: The dynamic loss scaling loop, using GradScaler’s default constants.
Figure 2 shows the logic. The scale starts high, at 65,536 in PyTorch’s default. Each step, the scaled gradients are checked for Inf or NaN. An overflow skips the optimizer step, multiplies the scale by 0.5 and resets a counter. A clean step updates the weights and increments the counter; after 2,000 consecutive clean steps, the scale doubles. The scale therefore hovers just below the overflow boundary, which maximizes how many small gradients survive.
We verified the constructor signature in PyTorch 2.15.0: GradScaler(device="cuda", init_scale=65536.0, growth_factor=2.0, backoff_factor=0.5, growth_interval=2000, enabled=True). The older torch.cuda.amp.GradScaler(...) form now emits a deprecation message pointing to torch.amp.GradScaler("cuda", ...).
A skipped step is not a bug. In a healthy run you will see a burst of skips at startup while the scale falls from 65,536 to a workable value, then a skip roughly every 2,000 steps as the scale probes upward and backs off. A run that skips every step, or whose scale collapses toward 1, has a real overflow that scaling cannot fix.
Here is the canonical FP16 loop with pytorch amp autocast and gradient clipping.
import torch
from torch import nn
device = "cuda"
model = MyTransformer().to(device) # parameters stay FP32 (master weights)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)
scaler = torch.amp.GradScaler("cuda") # init_scale=2**16, growth_interval=2000
for batch in loader:
x, y = batch["x"].to(device), batch["y"].to(device)
opt.zero_grad(set_to_none=True)
with torch.autocast(device_type="cuda", dtype=torch.float16):
logits = model(x) # matmuls run in FP16
loss = nn.functional.cross_entropy( # autocast runs this in FP32
logits.view(-1, logits.size(-1)), y.view(-1))
scaler.scale(loss).backward() # gradients are multiplied by S
scaler.unscale_(opt) # divide by S before clipping
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
scaler.step(opt) # skipped if Inf/NaN was found
scaler.update() # backoff or growth of S
Three details in that loop are easy to get wrong. First, unscale_ must run before clipping, otherwise the clipping threshold applies to scaled gradients. The ICLR paper made the same point: unscale before clipping or any gradient-related computation, and no hyperparameter needs retuning. Second, PyTorch documents that unscale_ should be called only once per optimizer per step. Third, the forward pass must sit inside autocast but the backward call should not; autocast records the dtype choices so the backward pass reuses them.
BF16 drops the scaler but keeps the master weights
With BF16, the range problem disappears, so most recipes remove GradScaler entirely. The loop gets shorter:
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
logits = model(x)
loss = nn.functional.cross_entropy(logits.view(-1, logits.size(-1)), y.view(-1))
loss.backward() # no loss scaling needed for BF16
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
opt.step()
Note what autocast does and does not do. It casts inputs to eligible ops, mostly matmuls and convolutions, on the fly. It does not change the parameter dtype. Your nn.Parameter tensors remain FP32, so the master weights, gradients and AdamW moments stay FP32 automatically. The price is a transient low-precision copy of each weight during the forward pass, which autocast caches within the region.
Autocast’s op lists are device-specific. On CUDA, PyTorch’s AMP reference lists softmax, log_softmax, layer_norm, the common loss functions and several reductions among the ops that autocast to FP32. On CPU the lists differ. In our CPU test under BF16 autocast, softmax and layer_norm returned BF16 tensors. Do not assume a sensitive op is protected; check its output dtype on your target device.
The memory ledger per parameter
The question “how much memory does mixed precision training save?” has a surprising answer for the optimizer: often nothing. The ZeRO paper’s accounting for mixed-precision Adam is 2 bytes of FP16 weights, 2 bytes of FP16 gradients and 12 bytes of FP32 optimizer state (master weights, momentum and variance), for 16 bytes per parameter. That equals pure FP32 AdamW. The savings come from activations and from faster matmuls, not from the weights.
| Recipe (AdamW) | Weights | Grads | Master | Moments m + v | Bytes/param | 7B model |
|---|---|---|---|---|---|---|
| FP32 baseline | 4 | 4 | 0 | 8 | 16 | 112 GB |
| FP16/BF16 + FP32 master (ZeRO accounting) | 2 | 2 | 4 | 8 | 16 | 112 GB |
| PyTorch autocast, FP32 params | 4 | 4 | 0 | 8 | 16, plus a transient 2-byte copy | 112 GB+ |
| DeepSeek-V3-style: FP32 master and grads, BF16 moments | 0 extra | 4 | 4 | 4 | 12, plus low-precision working copies | 84 GB+ |
| Pure BF16 weights, grads and moments | 2 | 2 | 0 | 4 | 8 | 56 GB |
The 7B column is plain multiplication and excludes activations, temporary buffers and fragmentation. The DeepSeek-V3 row is our illustrative reading of their report, which says master weights and gradients stay in FP32 and AdamW moments are tracked in BF16. Pure BF16 halves the ledger, but it carries the update stagnation risk shown earlier. Sharding this state across data-parallel ranks, as ZeRO and FSDP do, usually matters more than the format; our distributed training parallelism guide to FSDP, ZeRO and NCCL walks through that math.
FP8 training: per-tensor scaling, delayed vs current
FP8 training has two problems FP16 never had. E4M3’s range is 448 down to about 0.002, only about 18 binades including subnormals. And with 3 mantissa bits, its relative error is coarse; in our test, 432 rounds to 448 and 232 rounds to 224. A single global loss scale cannot fit every tensor into that window, so FP8 recipes scale each tensor, or each block, separately.

Figure 3: Four ways to choose FP8 scale factors before a Tensor Core GEMM, from coarse per-tensor to hardware microscaling.
Figure 3 places the options side by side. All four take a higher-precision tensor, divide it by a scale so its largest value lands near the FP8 maximum, run an FP8 GEMM with wider accumulation, and return BF16 or FP32 output. They differ in where the scale comes from and how many elements share it.
Transformer Engine’s DelayedScaling recipe uses a scale from previous iterations. Its source documents the defaults: margin=0, fp8_format=Format.HYBRID, amax_history_len=1024 and amax_compute_algo="max". HYBRID means E4M3 in the forward pass and E5M2 in the backward pass, since gradients need range more than resolution. The scale is computed as (FP8_MAX / amax) / 2**margin, where amax is the largest absolute value seen in the history window. Delayed scaling is cheap because no extra pass over the tensor is needed before quantizing. Its weakness is a sudden outlier that exceeds the history, which then saturates.
Float8CurrentScaling computes amax from the tensor being quantized, at the cost of that extra pass. It has the same HYBRID default. In Transformer Engine 2.21.0, the old fp8_autocast context manager is deprecated in favor of autocast(enabled=..., recipe=..., amax_reduction_group=...). A minimal FP8 setup looks like this:
import torch
import transformer_engine.pytorch as te
from transformer_engine.common.recipe import DelayedScaling, Float8CurrentScaling, Format
layer = te.Linear(4096, 4096, bias=True, params_dtype=torch.bfloat16).cuda()
recipe = DelayedScaling(fp8_format=Format.HYBRID, amax_history_len=1024, amax_compute_algo="max")
# recipe = Float8CurrentScaling(fp8_format=Format.HYBRID) # amax from the live tensor
x = torch.randn(8192, 4096, device="cuda", dtype=torch.bfloat16, requires_grad=True)
with te.autocast(enabled=True, recipe=recipe):
y = layer(x) # FP8 GEMM inside, BF16 output
y.float().sum().backward() # backward GEMMs use the recipe's backward format
The Transformer Engine docstring adds two constraints worth knowing. FP8 Linear layers need both tensor dimensions divisible by 16, which usually means padding sequence length to a multiple of 16. And with amax reduction enabled, a module must not be invoked more than once inside a single autocast region, because the reduction happens on exit.
Fine-grained scaling: DeepSeek-V3 and MX microscaling
Per-tensor scaling is fragile because one outlier sets the scale for millions of values. The DeepSeek-V3 technical report addressed this with fine-grained quantization. Activations are scaled per 1×128 tile, meaning per token per 128 channels. Weights are scaled per 128×128 block. With smaller groups, an outlier only coarsens its own tile. Because scales are local, DeepSeek used E4M3 for all tensors, including gradients, instead of the hybrid E4M3/E5M2 split.
The report also found an accumulation problem. On H800 GPUs, FP8 GEMM accumulation retained only around 14 bits. For K = 4096, their preliminary test showed a maximum relative error of nearly 2%. Their fix promotes partial sums to FP32 registers on CUDA cores every N_C = 128 elements, equivalent to 4 WGMMA instructions, where the per-group scales are also applied. The embedding, output head, MoE gating, normalization and attention kept BF16 or FP32. The team reported relative loss error versus a BF16 baseline consistently below 0.25% on validation runs of about 1 trillion tokens. Our MoE routing deep dive on DeepSeek-V3 and Mixtral covers the architecture those GEMMs serve.
Transformer Engine now ships this idea as Float8BlockScaling. Its docstring says scales are FP32 containers constrained to powers of two by default, and it fixes split accumulation on for numerical accuracy. Its support check requires compute capability 9.0 and CUDA 12.9 or newer.
The OCP MX specification standardizes the same principle in hardware. Every concrete MX format uses a block of 32 elements sharing one E8M0 scale, an 8-bit power-of-two exponent. MXFP8 uses E4M3 or E5M2 elements, MXFP6 uses E2M3 or E3M2, MXFP4 uses E2M1, and MXINT8 uses 8-bit integers. In Transformer Engine, MXFP8BlockScaling defaults to E4M3, and its support check requires compute capability 10.0, the datacenter Blackwell generation. The same check reports MXFP8 as not yet supported on 12.0-class parts. One subtle point from the MX paper: a block scale follows one axis, so quantizing and transposing do not commute. Transformer Engine therefore quantizes both the tensor and its transpose from the high-precision source, rather than re-quantizing a quantized copy.
Stochastic rounding and the 4-bit frontier
Round-to-nearest is unbiased per value but biased in aggregate when updates are consistently smaller than the spacing. That is the BF16 stagnation from earlier. Stochastic rounding rounds up or down with probability proportional to distance, so the expected value of the rounded number equals the true number. Over many steps, small updates accumulate correctly on average, at the cost of added noise.
NVIDIA’s NVFP4BlockScaling recipe in Transformer Engine uses stochastic rounding on gradients. It also uses 16×16 two-dimensional block scales for weights and 16×16 random Hadamard transforms to smooth outliers. Its two-level scaling pairs an E4M3 scale per 16 elements with a per-tensor FP32 scale. The accompanying paper, “Pretraining Large Language Models with NVFP4”, reports a 12-billion-parameter model trained on 10 trillion tokens with loss and downstream accuracy comparable to an FP8 baseline. We compare those 4-bit formats in detail in NVFP4 vs MXFP4 for Blackwell training and inference.
Trade-offs, Gotchas, and What Goes Wrong
Low precision rarely fails loudly at first, which is the most dangerous property of mixed precision training. It fails as a skipped step, a plateau or a spike that recovers twice and then does not. Knowing which budget broke shortens the search.

Figure 4: A triage flow for numerical failures, starting from the symptom and ending at the budget that ran out.
Figure 4 starts from the symptom. A loss scale that falls every step means a real overflow, so find the first Inf. A spike that recovers often points to attention logit growth. An early plateau often means updates are rounding away. In every branch, a short BF16 or FP32 control run is the fastest way to separate numerics from data or optimizer bugs.
Overflow in attention logits. Attention scores are dot products of queries and keys, and they grow as weights grow. “Scaling Vision Transformers to 22 Billion Parameters” reported attention logit growth as a source of instability, and “Small-scale proxies for large-scale Transformer training instabilities” reproduced it at small scale. Their mitigations included normalizing queries and keys (QK-layernorm) and a z-loss on output logits. In FP16, exp() overflows just above 11, so any softmax that is not computed in FP32 with max subtraction is a liability. Fused attention kernels handle this internally; hand-written attention often does not. We cover the IO-aware kernels in our FlashAttention 2, 3 and 4 explainer.
Silent stagnation. Pure BF16 weights stop learning when updates drop below half a unit in the last place. The loss looks merely “converged”. Compare against a run with FP32 master weights before trusting a plateau late in a schedule.
Gradient clipping on scaled gradients. Forgetting unscale_ before clipping clips gradients that are S times too large. The run trains, but effectively with a much smaller learning rate whenever clipping triggers.
E4M3 saturation versus NaN. Because E4M3 has no infinity, an out-of-range value either saturates to 448 or becomes NaN, depending on the conversion. A NaN here usually means an amax history too short or too stale for a sudden outlier.
Transpose double quantization. With block scales, the transpose of a quantized tensor is not the quantized transpose. Custom kernels that re-quantize an FP8 tensor for the backward pass add a second rounding error.
Accumulation width. A GEMM that accumulates FP8 products in a narrow internal register loses accuracy as K grows. DeepSeek’s measured error of nearly 2% at K = 4096 on H800 is the cautionary number.
Cost of the fixes. Current scaling adds a pass over each tensor. Block scaling adds scale tensors and promotion work. FP32 master weights cost 4 bytes per parameter. None of these is free, and speedups depend heavily on model shape; do not budget on a generic “2x” figure.
Practical Recommendations
Start with BF16 autocast and FP32 parameters on any GPU that supports BF16 Tensor Cores. It removes loss scaling, keeps master weights and optimizer state in FP32 automatically, and is the lowest-risk form of mixed precision training. Use FP16 with GradScaler only on hardware without fast BF16, or when you need FP16’s finer resolution, for example in some fine-tuning or inference-matched workflows.
Move to FP8 once the BF16 run is stable and profiled. Treat FP8 as a throughput and activation-memory optimization, not an optimizer-memory optimization. Keep embeddings, the output head, normalization, softmax and routing in higher precision, as DeepSeek-V3 did. Prefer current or block scaling over delayed scaling if your model has outliers, and accept the small extra cost. On Blackwell-class datacenter GPUs, MXFP8 gives you hardware block scales; on Hopper, Float8BlockScaling or a DeepSeek-style recipe is the closest equivalent.
The decision matrix below summarizes the default recipe by hardware generation, based on the Transformer Engine support checks and PyTorch APIs verified above.
| GPU generation | Default compute format | Scaling mechanism | Master weights and moments | When to deviate |
|---|---|---|---|---|
| Volta and Turing | FP16 autocast | GradScaler dynamic loss scaling | FP32 | No BF16 Tensor Cores, so FP16 is the only fast path |
| Ampere | BF16 autocast | None needed | FP32 | FP16 plus GradScaler if a model is resolution-sensitive |
| Ada and Hopper | BF16, then FP8 for large GEMMs | Delayed, current or block scaling | FP32, moments optionally BF16 | Keep BF16 if loss deviates from the control run |
| Datacenter Blackwell | BF16, FP8 or MXFP8 | MX block scales of 32 with E8M0 | FP32 | NVFP4 only with a tested recipe and a control run |
Always keep a control. Run a short BF16 baseline alongside any FP8 change and compare loss curves over the same tokens. In mixed precision training, a matched control is the only reliable way to tell a numerics regression from ordinary run-to-run noise. DeepSeek’s acceptance bar, relative loss error below 0.25% against BF16, is a reasonable benchmark to borrow.
A short checklist before a long run:
- Parameters are FP32 or have FP32 master copies; AdamW moments are FP32 unless tested in BF16.
- FP16 runs call
scaler.unscale_before clipping and log the scale every N steps. - Softmax, normalization, loss and large reductions run in FP32; verify output dtypes on the target device.
- FP8 layers have dimensions divisible by 16 and keep the first and last layers in BF16.
- Logged signals include loss scale, skipped steps, gradient norm, max attention logit and amax per layer.
- A BF16 control run exists for every precision change.
Frequently Asked Questions
What is mixed precision training in simple terms?
Mixed precision training uses two or more number formats in one training run. The heavy matrix multiplications run in a 16-bit or 8-bit format because Tensor Cores process those much faster and they halve activation memory. Values that need accuracy, such as master weights, optimizer moments, softmax and normalization statistics, stay in FP32. Extra mechanisms keep the low-precision parts safe: loss scaling for FP16, per-block scaling for FP8, and FP32 accumulation inside matrix multiplications. Done correctly, it matches FP32 accuracy while training faster.
Is BF16 better than FP16 for training?
For most large-model pre-training, yes. BF16 keeps FP32’s 8-bit exponent, so its range reaches about 3.4e38 and gradients do not underflow; you can drop loss scaling. FP16 has 10 mantissa bits to BF16’s 7, so FP16 is about eight times finer near any value. That makes FP16 more accurate when values fit its range, which is up to 65,504. The practical choice is BF16 when your hardware supports it, with FP32 master weights to compensate for its coarse resolution.
Do I still need loss scaling with BF16?
Usually not. Loss scaling exists because FP16’s smallest subnormal is 2^-24, about 5.96e-8, and many gradients fall below that. BF16’s smallest subnormal is 2^-133, about 9.2e-41, so the same gradients remain representable. PyTorch’s common BF16 recipe therefore calls loss.backward() without a GradScaler. Loss scaling would not hurt correctness, but it adds overhead and solves a problem BF16 does not have. FP8 is different: it needs per-tensor or per-block scaling, which is a more granular form of the same idea.
How much memory does mixed precision training save?
Less than people expect for weights and optimizer state. With AdamW, the standard mixed-precision layout costs 16 bytes per parameter: 2 for 16-bit weights, 2 for gradients and 12 for FP32 master weights and moments. That equals FP32 training. The real savings come from activations stored in 16 or 8 bits, which often dominate memory at large batch sizes. Storing AdamW moments in BF16, as DeepSeek-V3 did, cuts the ledger to roughly 12 bytes, while pure BF16 reaches 8 at a stability cost.
Why does my FP16 training loss become NaN?
Most FP16 NaNs come from overflow, not underflow. The largest FP16 value is 65,504, so a large activation, an unnormalized exponent above about 11, or a long FP16 accumulation can produce Inf, and Inf minus Inf gives NaN. Check whether GradScaler’s scale is falling every step; if so, the overflow happens regardless of scaling. Register forward hooks to find the first non-finite tensor. Then keep that op in FP32, use a fused attention kernel, add QK normalization, or switch to BF16 for its wider range.
Does FP8 training work on any GPU?
No. Transformer Engine’s support check enables FP8 on compute capability 8.9 and above, which covers Ada and Hopper. On Ada it also requires CUDA 12.1 and a recent cuBLASLt. MXFP8 and NVFP4 recipes require compute capability 10.0, the datacenter Blackwell generation. FP8 block scaling needs compute capability 9.0 with CUDA 12.9 or newer. Older GPUs such as A100 support BF16 and TF32 but not FP8 Tensor Core math. Check your framework’s own support function rather than relying on marketing names.
Further Reading
- FP8 vs INT8 vs INT4: an LLM quantization benchmark for inference — how the same low-precision formats behave once training is done.
- Distributed training parallelism: data, tensor, pipeline, FSDP, ZeRO and NCCL — how to shard the 16-bytes-per-parameter ledger across GPUs.
- GPU memory physics: DRAM vs SRAM, FlashAttention and kernel fusion — why fewer bytes per value translate into speed.
- NVFP4 vs MXFP4 for 4-bit training and inference on Blackwell — the next step below FP8.
- External: Mixed Precision Training, Micikevicius et al., ICLR 2018 and Microscaling Data Formats for Deep Learning, Rouhani et al., 2023.
- External: PyTorch automatic mixed precision reference and NVIDIA Transformer Engine on GitHub.
By Riju — about
