FP8 vs INT8 vs INT4 LLM Quantization Benchmark (2026)

FP8 vs INT8 vs INT4 LLM Quantization Benchmark (2026)

FP8 vs INT8 vs INT4 Quantization: The 2026 LLM Benchmark Guide, Plus NVFP4

Most quantization advice still ranks formats by bit count, as if fewer bits meant a simple slide down one quality dial. The published evidence says otherwise. In the largest open study so far, FP8 weight-and-activation quantization was effectively lossless on Llama 3.1 at 8B, 70B and 405B. Weight-only INT4 recovered about 99 percent of baseline accuracy. Tuned INT8 activations cost 1 to 3 percent per task. So the real fp8 vs int8 vs int4 quantization question is not “how many bits can I lose”. It is “which tensors am I compressing, and on which tensor cores”. In late 2026 the answer has a fourth branch, because Blackwell’s NVFP4 and the OCP MXFP4 format have made 4-bit floating point a production option. This guide gives you verified numbers, the memory and bandwidth arithmetic behind them, and a benchmark method you can run on your own stack.

What this covers: format internals (E4M3, INT8, INT4 groups, NVFP4 and MXFP4), the scaling granularity that decides accuracy, published accuracy and speed results, a reproducible llm-compressor and vLLM benchmark, a decision matrix by GPU generation, KV cache quantization, and the failure modes that bite in production.

Context and Background

A large language model spends most of its decode time moving bytes, not doing arithmetic. Every generated token reads every active weight from high-bandwidth memory (HBM) once. It also reads the growing key-value (KV) cache of all previous tokens. Quantization shrinks the bytes per number, so it changes memory footprint, decode speed and accuracy all at once. Picking a format is really picking where in that triangle you want to land.

The research lineage is short and well documented. LLM.int8() (Dettmers et al., 2022) showed that transformers past roughly 6.7B parameters develop large-magnitude “outlier features” in a few hidden dimensions. Naive 8-bit activation quantization breaks on those outliers. GPTQ (Frantar et al., ICLR 2023) used approximate second-order information to quantize a 175B model to 3 or 4 bits in about four GPU hours. SmoothQuant (Xiao et al., 2022) moved activation outliers into the weights with a mathematically equivalent rescaling, making INT8 W8A8 practical. AWQ (Lin et al., MLSys 2024 Best Paper) showed that protecting about 1 percent of salient weight channels, chosen by looking at activations, recovers most 4-bit accuracy.

On the floating-point side, the 2022 paper “FP8 Formats for Deep Learning” by NVIDIA, Arm and Intel engineers defined the E4M3 and E5M2 encodings that Hopper, Ada and later GPUs implement. The Open Compute Project (OCP) then standardized them as OFP8. In 2023 the OCP Microscaling (MX) v1.0 specification added block-scaled MXFP8, MXFP6, MXFP4 and MXINT8. NVIDIA’s Blackwell generation added its own NVFP4 variant with smaller blocks and finer scales. The format menu is now wider than three options.

The serving side has matured too. vLLM, SGLang and TensorRT-LLM all load FP8, INT8 and INT4 checkpoints. llm-compressor (from the vLLM project) and NVIDIA Model Optimizer (ModelOpt, formerly TensorRT Model Optimizer) produce them. If you are choosing an engine as well as a format, our vLLM vs SGLang vs TensorRT-LLM comparison covers the scheduling and kernel differences. They matter as much as the format itself. The canonical reference for which recipe each engine accelerates is the vLLM quantization documentation. It changes often, so check it against your installed version.

This post replaces our June 2026 version. That version used directional ranges without sources. Every number below either cites a primary source or carries an “illustrative” label.

How FP8, INT8 and INT4 Quantization Actually Differ

Direct answer: FP8 (E4M3) is a floating-point format whose exponent bits absorb activation outliers. That lets you quantize weights and activations to 8 bits with almost no accuracy loss on Ada, Hopper and Blackwell. INT8 needs outlier smoothing to match it. INT4 is almost always weight-only, trading a small accuracy loss for about 4x less weight memory.

FP8 vs INT8 vs INT4 quantization paths from a BF16 checkpoint to the tensor cores that execute each format

Figure 1: How FP8, INT8, INT4 and FP4 branch from a BF16 checkpoint, which tensors each one compresses, and which hardware runs it natively.

The diagram starts from a BF16 checkpoint and splits on the first real decision: which tensors get compressed. Compressing both weights and activations (W8A8) lets the matrix multiply itself run in 8-bit on the tensor cores. Compressing only weights (W4A16, W8A16) saves memory and bandwidth, but the kernel dequantizes back to 16-bit before multiplying. Block-scaled 4-bit floating point (W4A4) is the newest branch and needs Blackwell-class FP4 tensor cores. The bottom row shows where each path executes natively, and that row usually decides the format before accuracy does.

Floating point spends bits on range, integers spend them on uniform steps

FP8 E4M3 uses 1 sign bit, 4 exponent bits and 3 mantissa bits. Per the FP8 formats paper, its largest normal value is 448. It has no infinity encoding, so it can reclaim more bit patterns for finite values. E5M2 trades a mantissa bit for range, reaching 57,344. It is used mainly for gradients in training. Because the exponent is per value, E4M3 spaces representable numbers logarithmically. It keeps relative precision for both a 0.01 activation and a 300 activation in the same tensor.

INT8 spaces its 256 levels evenly across a range set by a scale factor. That is ideal when values cluster in a known band. It is unforgiving when one channel spikes to 50x the others. The scale must stretch to cover the spike, and every ordinary value then lands on only a few coarse levels. This is the outlier problem LLM.int8() documented. It explains most of the gap between FP8 and INT8 activation accuracy.

INT4 has only 16 levels. Applied to activations, it fails on most models without heavy rotation tricks (QuaRot and SpinQuant are research examples). So in practice “INT4” in LLM serving means W4A16. Weights are stored as 4-bit integers in groups of typically 128, each group with its own 16-bit scale. Activations stay in BF16 or FP16.

Peak throughput is equal; accuracy is not

A common misconception is that FP8 is “faster than INT8”. On an H100 SXM, NVIDIA’s spec sheet lists FP8 and INT8 tensor-core peaks as identical, 3,958 TFLOPS/TOPS with sparsity, so about 1,979 dense. FP16 and BF16 are half that. The H100 table has no INT4 row at all. Hopper dropped the INT4 tensor-core path that Ampere had, so 4-bit weights on Hopper are always dequantized in the kernel.

The FP8 vs INT8 choice on Hopper is therefore an accuracy and tooling choice, not a raw-compute choice. FP8 W8A8 needs no smoothing pass, tolerates dynamic per-token activation scales cheaply, and lands near lossless. INT8 W8A8 reaches the same math throughput but needs SmoothQuant or similar to protect accuracy. That is why FP8 became the default 8-bit format wherever hardware supports it. INT8 survives on Ampere and Turing, where FP8 tensor cores do not exist.

Weight-only INT4 wins on bandwidth, W8A8 wins on compute

Decode at low batch size is memory-bandwidth bound. Each step multiplies a handful of token vectors by enormous weight matrices. Arithmetic intensity is low, so tensor cores sit idle waiting for HBM. Cutting weight bytes by 4x cuts the time to stream them by close to 4x. That is why W4A16 dominates single-user latency. The Marlin paper (Frantar et al., PPoPP 2025) measured close to the ideal 4x kernel speedup over FP16 at batch sizes up to 16 to 32 on an A10. The advantage fell to about 1.5x at batch 128 as the workload turned compute bound.

At high concurrency the picture flips. Prefill and large-batch decode become compute bound, and W4A16 still pays for dequantization plus a 16-bit multiply. W8A8 runs the multiply itself at 2x the BF16 rate. This crossover is the single most important fact in the comparison. The “Give Me BF16 or Give Me Death” study (Kurtic et al., Neural Magic and IST Austria, arXiv 2411.02355) measured it directly with vLLM. W4A16 was the most cost-efficient choice for synchronous, one-request-at-a-time serving. W8A8 delivered the highest throughput under asynchronous, continuous-batching load. The crossover point depended on model size, hardware and latency target.

Scaling granularity decides accuracy more than bit width

Scaling granularity options for FP8 vs INT8 vs INT4 quantization and their metadata cost in bits per value

Figure 2: Scale granularity from per-tensor to 16-element micro-blocks, with the effective storage cost each adds.

Figure 2 lists the places a scale factor can live. A per-tensor scale is cheapest but fits outliers worst. Per-channel weight scales (one per output row) are standard for INT8 and FP8 weights. Per-token activation scales are computed at runtime for each row of the activation matrix, so a spiky token does not degrade its neighbours. INT4 weight-only uses group scales: with a 16-bit scale per 128 weights, effective storage is 4 + 16/128 = 4.125 bits per weight, slightly more with zero points.

The microscaling formats push this further. OCP MX uses blocks of 32 elements sharing an 8-bit power-of-two (E8M0) scale, which works out to 4.25 bits per value for MXFP4. NVFP4 uses blocks of 16 with an FP8 E4M3 scale plus a per-tensor FP32 scale. NVIDIA puts that at about 4.5 bits per value. The general lesson: two checkpoints with the same nominal bit width can differ a lot in quality because of granularity alone. A benchmark that reports “INT4” without the group size is under-specified.

Benchmark Results and a Method You Can Run

Direct answer: Published evidence ranks accuracy recovery as FP8 W8A8 (near lossless), then INT4 W4A16 and tuned INT8 W8A8 (roughly 97 to 99 percent), then NVFP4 (about 1 percent or less loss on the models NVIDIA reported). Speed depends on batch size: INT4 wins low-concurrency latency, W8A8 wins high-concurrency throughput.

What the primary sources actually measured

The table below collects verified results. Where a cell is a rule of thumb or arithmetic rather than a measurement, it says so. Do not compare rows from different studies as if they used the same hardware or tasks.

Format and scheme Weight memory vs BF16 Accuracy evidence Speed evidence Native hardware
FP8 W8A8 (E4M3) about 50% “Effectively lossless across all model scales” on Llama 3.1 8B/70B/405B (Kurtic et al.) vLLM docs: up to 1.6x throughput, 2x memory reduction Ada, Hopper, Blackwell; AMD CDNA3 (FNUZ variant) and CDNA4 (OCP)
INT8 W8A8 + SmoothQuant about 50% 1–3% average per-task drop when tuned (Kurtic et al.) SmoothQuant paper: up to 1.56x speedup, 2x memory reduction Turing and newer INT8 tensor cores
INT8 W8A16 about 50% Near lossless in practice; few systematic studies Bandwidth gain only, no compute gain Any GPU via dequant kernels
INT4 W4A16 (AWQ/GPTQ, g128) about 26% (arithmetic, 4.125 bits) About 99% average recovery for most sizes, bigger drops for the smallest models on reasoning (Kurtic et al.) GPTQ paper: 3.25x (A100), 4.5x (A6000) vs FP16; Marlin near 4x kernel speedup to batch 16–32 Any CUDA GPU; best kernels on Ampere+
NVFP4 W4A4 about 28% (4.5 bits); NVIDIA cites about 3.5x smaller than FP16 DeepSeek-R1-0528 FP8 to NVFP4: 1% or less degradation on key tasks; AIME 2024 91% vs 89% FP8 (NVIDIA) Vendor claims only; verify on your workload Blackwell (SM100+)
MXFP4 (OCP MX) about 27% (4.25 bits) gpt-oss models post-trained with MXFP4 MoE weights; all OpenAI evals used MXFP4 gpt-oss-120b fits a single 80 GB GPU Blackwell; AMD CDNA4; emulated elsewhere

Two patterns stand out. First, FP8 is the only format with strong evidence of being lossless as a full W8A8 scheme. That is why it is the default on modern GPUs. Second, INT4 weight-only and INT8 W8A8 end up close on accuracy but very different on speed, so the workload shape decides between them. NVFP4 is promising, but most evidence so far comes from NVIDIA and model vendors. Treat it as “validate before trusting”.

Also note what the INT4 recovery figure hides. Kurtic et al. report averages across task suites. Averages can mask a sharp drop on one capability, such as multi-step arithmetic, long-context retrieval or code. Your evaluation must include the capabilities your product depends on.

Memory arithmetic: a worked example

Format choice often changes deployment topology, not only speed. Take Llama 3.1 70B, roughly 70.6B parameters, with 80 layers, 8 KV heads and head dimension 128 (grouped-query attention). These figures are arithmetic, not measurements:

  • BF16 weights: 70.6B × 2 bytes ≈ 141 GB. Needs at least two 80 GB H100s before any KV cache.
  • FP8 or INT8 weights: ≈ 71 GB plus small scale tensors. Fits one 80 GB H100, but leaves under 10 GB for KV cache and activations. That is too tight for real concurrency. On a 141 GB H200 it leaves about 70 GB of headroom.
  • INT4 W4A16, group 128: 70.6B × 4.125 / 8 ≈ 36 GB, plus BF16 embeddings and the LM head that are usually left unquantized. Fits one 48 GB card with modest KV room.
  • NVFP4: about 4.5 bits ≈ 40 GB of quantized weights.

Now the KV cache. Per token it stores K and V for every layer: 2 × 80 layers × 8 heads × 128 dims × 2 bytes = 327,680 bytes, about 320 KiB in BF16. One 128K-token sequence therefore needs about 40 GiB of KV cache. That alone exceeds the INT4 weight footprint. An FP8 KV cache halves this to about 20 GiB. For long-context serving, KV precision can matter more than weight precision. Our KV cache optimization guide covers paging, eviction and offload in depth.

The bandwidth ceiling follows from the same arithmetic. An H100 SXM moves about 3.35 TB/s. A batch-1 decode step must at least stream all weights once. So the theoretical ceiling is about 3,350 / 71 ≈ 47 tokens/s for FP8 weights and about 3,350 / 36 ≈ 93 tokens/s for INT4 on one GPU. Real kernels reach a fraction of this. The ratio, roughly 2x, is the durable part. These are illustrative upper bounds, not benchmarks.

A reproducible benchmark pipeline

LLM quantization benchmark methodology pipeline for comparing FP8 vs INT8 vs INT4 against a BF16 baseline

Figure 3: The benchmark loop: pin the stack, baseline in BF16, quantize, gate on accuracy, then sweep latency and throughput and normalize to cost.

Figure 3 is the methodology we recommend. The key design choice is the accuracy gate before any speed measurement. A fast checkpoint that fails your quality threshold goes back to the recipe step (smaller group size, different algorithm, more layers skipped) and is never timed. That ordering keeps teams from falling in love with a throughput number they cannot ship.

Start by pinning everything except precision: model revision, GPU, driver, CUDA, engine version, tensor-parallel degree, prompt and output lengths, and sampling parameters. Comparing FP8 on one engine against INT4 on another measures the engines, not the formats.

Here is a minimal llm-compressor flow producing an FP8 and an INT4 checkpoint from the same base model. It follows the library’s documented oneshot API. Check the README for your installed version, since modifier names have shifted between releases.

# pip install llmcompressor  (versions change; pin what you test)
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier, GPTQModifier

MODEL = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="auto")
tok = AutoTokenizer.from_pretrained(MODEL)

# 1) FP8 W8A8: static per-channel weight scales, dynamic per-token activations.
#    No calibration data required.
fp8 = QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
oneshot(model=model, recipe=fp8)
model.save_pretrained("llama31-8b-fp8", save_compressed=True)
tok.save_pretrained("llama31-8b-fp8")

# 2) INT4 W4A16 with GPTQ (group size 128 by default in the scheme).
#    Reload a fresh BF16 model first; calibration data should match your traffic.
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="auto")
w4 = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
oneshot(
    model=model,
    dataset="open_platypus",          # replace with domain-matched samples
    recipe=w4,
    max_seq_length=2048,
    num_calibration_samples=512,
)
model.save_pretrained("llama31-8b-w4a16", save_compressed=True)
tok.save_pretrained("llama31-8b-w4a16")

Then gate on accuracy with the same harness for every variant, and only then measure speed at two operating points:

# Accuracy: identical tasks, shots and engine for every checkpoint
for ckpt in meta-llama/Llama-3.1-8B-Instruct llama31-8b-fp8 llama31-8b-w4a16; do
  lm_eval --model vllm \
    --model_args pretrained=$ckpt,max_model_len=4096,gpu_memory_utilization=0.85 \
    --tasks gsm8k,mmlu_pro,ifeval --batch_size auto --output_path results/$(basename $ckpt)
done

# Speed: serve each checkpoint, then sweep concurrency
vllm serve llama31-8b-fp8 --kv-cache-dtype fp8 --max-model-len 8192 &
vllm bench serve --model llama31-8b-fp8 --dataset-name random \
  --random-input-len 1024 --random-output-len 256 \
  --max-concurrency 1   --num-prompts 64
vllm bench serve --model llama31-8b-fp8 --dataset-name random \
  --random-input-len 1024 --random-output-len 256 \
  --max-concurrency 64  --num-prompts 1024

Report five numbers per checkpoint: task accuracy recovery as a percentage of BF16, time to first token (TTFT), time per output token (TPOT), output tokens per second at high concurrency, and peak GPU memory including KV cache. Then convert throughput into cost per million tokens using your actual GPU hourly price. A format that is 30 percent faster on a GPU that costs twice as much is not a win. Our inference SLO and capacity planning guide shows how to turn these into goodput under a latency SLO.

Methodology caveats

Calibration is where most published comparisons quietly diverge. Post-training quantization (PTQ) estimates scales from a calibration set, usually a few hundred samples. A generic chat dataset can under-represent the activations your domain triggers, such as code, tables, or non-English text. Results calibrated on generic data and tested on generic benchmarks will look better than your production traffic. Record the calibration dataset and sample count alongside every result.

Perplexity is a sensitive but incomplete signal. A checkpoint can hold perplexity within a percent and still lose a few points on multi-step math or instruction following. Use perplexity as a smoke test, not the gate. Also watch evaluation noise. Kurtic et al. found FP8 results often fell within the margin of error, so a single run showing FP8 “beating” BF16 is noise, not a finding. Run each evaluation with a fixed seed and report variance where tasks are small.

Finally, engine versions move kernels underneath you. A vLLM or TensorRT-LLM upgrade can change which kernel serves an INT4 checkpoint and shift throughput a lot without any change to the weights. Re-run speed sweeps after every engine upgrade.

The 4-Bit Floating Point Branch: NVFP4 and MXFP4

INT4 weight-only quantization dominated 4-bit serving through 2025 because no mainstream datacenter GPU could multiply 4-bit activations. Blackwell changed that. Its fifth-generation tensor cores natively execute FP4 (E2M1) matrix multiplies with hardware-managed block scales. That makes W4A4 a real option, with both bandwidth savings and compute savings.

The E2M1 element is tiny: 1 sign bit, 2 exponent bits, 1 mantissa bit. Its representable magnitudes are 0, 0.5, 1, 1.5, 2, 3, 4 and 6. On its own that is useless for neural network tensors. It works only because every small block carries its own scale, so the 16 representable levels are re-centred on each block’s local range.

The two block-scaled variants differ in exactly the way Figure 2 suggests. MXFP4, from the OCP MX v1.0 specification, shares one power-of-two E8M0 scale across 32 elements. Power-of-two scales are cheap in hardware, but they can only move the range in factors of 2. A block whose maximum sits just above a power of two wastes almost half its levels. NVFP4 shares an FP8 E4M3 scale across 16 elements and adds a per-tensor FP32 scale. The fractional scale and smaller block give a tighter fit. In an illustrative example, NVIDIA reports scale-encoding error of 0.08 for E4M3 versus 0.72 for E8M0. The cost is about 0.25 extra bits per value.

The evidence so far is encouraging but narrow. NVIDIA’s NVFP4 introduction reports 1 percent or less accuracy degradation when DeepSeek-R1-0528 is quantized from its native FP8 to NVFP4 with PTQ. It also reports memory about 3.5x smaller than FP16 and about 1.8x smaller than FP8. OpenAI’s gpt-oss models went further. Their MoE weights were post-trained in MXFP4, and all published evaluations used that format. As a result gpt-oss-120b (117B total, 5.1B active parameters) runs on a single 80 GB GPU, and gpt-oss-20b runs within 16 GB.

The distinction between post-training and quantization-aware training matters here. A model trained or post-trained in a 4-bit format has learned weights that tolerate it. A BF16 model cast to FP4 after the fact has not. Expect PTQ to FP4 to be riskier than PTQ to FP8 or INT4 weight-only, and expect quantization-aware recipes to matter more at 4 bits than they did at 8. For a full treatment of the two formats, see our NVFP4 vs MXFP4 deep dive.

The llm-compressor scheme guide reflects this new hardware tiering. It recommends NVFP4 or MXFP4 for maximum compression on Blackwell (compute capability 10.0), with FP8 or MXFP8 as the balanced alternative. On Hopper it recommends FP8 W8A8 for throughput and W4A8 with FP8 activations as a mixed option. On Ampere it recommends W4A16 for memory and INT8 W8A8 for compute, and on Turing INT8 W8A8. FP8 W8A8 needs compute capability 8.9 (Ada) or newer in vLLM. On Ampere, vLLM can still load FP8 checkpoints as weight-only W8A16 via Marlin kernels, which saves memory but not compute.

Choosing a Format: Decision Matrix by Hardware and Workload

Decision tree for FP8 vs INT8 vs INT4 quantization by GPU generation and serving workload

Figure 4: Choose by GPU generation first, then by workload shape and accuracy budget; add FP8 KV cache wherever context is long.

Figure 4 encodes the decision as most teams should make it. Hardware first, because a format without native tensor-core support loses most of its speed advantage. Workload second, because the W4A16 versus W8A8 crossover depends on concurrency. Accuracy budget third, because only 4-bit floating point adds meaningful new risk on modern hardware.

Situation First choice Fallback Why
Hopper or Ada, high-concurrency API FP8 W8A8 + FP8 KV cache INT4 W4A16 if memory bound Near lossless; 2x BF16 math rate; no smoothing pass
Hopper, single-user or agent loop latency INT4 W4A16 (AWQ or GPTQ) W4A8 with FP8 activations Decode is bandwidth bound; 4x fewer weight bytes
Blackwell, maximum density NVFP4 after validation FP8 or MXFP8 Native FP4 math; about 3.5x smaller than FP16
Ampere (A100, A10, A6000) INT4 W4A16 via Marlin INT8 W8A8 + SmoothQuant No FP8 tensor cores; INT8 is the compute path
Model too big for one GPU at 8-bit INT4 W4A16 or NVFP4 FP8 across tensor-parallel GPUs Avoids inter-GPU communication
CPU, Apple silicon, edge box GGUF Q4_K_M GGUF Q8_0 llama.cpp kernels; no tensor-core formats
Accuracy-critical, regulated output FP8 W8A8 BF16 Strongest evidence of losslessness

The GGUF row deserves a note. llama.cpp uses its own block formats, not the GPU schemes above. Q8_0 stores 32-weight blocks with a 16-bit scale, giving 8.5 bits per weight. Q4_0 works out to 4.5 bits. The mixed “K-quants” such as Q4_K_M land around 4.8 bits on Llama-class models, according to llama.cpp’s own tables. An importance matrix (imatrix) computed from calibration text improves them, just as calibration improves AWQ and GPTQ. The format comparison is covered in our GGUF vs AWQ vs GPTQ vs FP8 guide.

AMD hardware fits the same matrix with one wrinkle. CDNA3 (MI300 series) implements an FP8 variant called FNUZ. It has no infinities, NaN uses a single encoding, and the exponent bias differs from OCP E4M3. CDNA4 implements OCP FP8, per AMD’s ROCm documentation. Checkpoints and scales are not always bit-compatible across the two. A checkpoint calibrated for one may need conversion or re-quantization for the other. Check your engine’s ROCm path before assuming an NVIDIA FP8 checkpoint runs unchanged.

Mixture-of-experts (MoE) models shift the calculus toward 4-bit weights. Only a few experts fire per token, so the active compute per token is small while the total weight footprint stays huge. Memory, not math, is usually the binding constraint. That is why gpt-oss shipped MXFP4 expert weights. It is also why 4-bit expert weights with 8-bit attention layers is a common mixed recipe.

Trade-offs, Gotchas, and What Goes Wrong

Outlier layers break uniform recipes. Many models have a few sensitive layers: the first and last transformer blocks, the LM head, and sometimes specific down-projections. Quantizing everything uniformly can cost several points that skipping two or three layers would recover. Every major toolkit supports an ignore list. Use per-layer sensitivity analysis rather than guessing.

Small models suffer more. Kurtic et al. saw larger INT4 drops for the smallest models on reasoning tasks. Larger networks carry more redundancy. A recipe validated on 70B does not transfer to 8B, and an 8B result is pessimistic for 70B. Evaluate the exact checkpoint you ship.

Reasoning and long-context tasks degrade first. Errors compound over long chains of thought and long retrieval spans. A checkpoint that holds MMLU can still drift on 30-step derivations or needle-in-a-haystack retrieval at 100K tokens. Include at least one long-generation and one long-context task in your gate.

FP8 E4M3 saturates silently. With no infinity, values beyond 448 times the scale clamp to the maximum. A stale static activation scale, calibrated on short prompts and then hit with an unusual input, clips rather than overflows. You get quietly wrong outputs instead of NaNs. Dynamic per-token activation scaling avoids this at small runtime cost, and it is why FP8_DYNAMIC is the common default.

Weight-only formats do not speed up prefill. If your workload is retrieval-augmented generation with 20K-token prompts and 200-token answers, prefill dominates and is compute bound. W4A16 saves memory there but may be slower than FP8 W8A8 end to end.

KV cache quantization is a separate risk. FP8 KV cache halves cache memory and usually costs little. But the default scale of 1.0 with no calibration works only when values fit E4M3’s range. vLLM supports calibrated per-tensor and per-attention-head scales via llm-compressor. It can also skip sensitive layers, such as sliding-window layers, with --kv-cache-dtype-skip-layers. Calibrate, especially for models with large attention-logit ranges.

Kernel coverage lags formats. A checkpoint format can be supported in an engine’s loader long before it has a fast kernel on your GPU. The fallback kernel may be slower than BF16. Always measure, never infer speed from bit width.

Benchmark contamination and vendor framing. Vendor accuracy claims are usually measured on the vendor’s best recipe and the tasks that model was tuned for. That does not make them wrong. It makes them an upper bound.

Practical Recommendations

Default to FP8 W8A8 on any Ada, Hopper or Blackwell GPU unless memory forces you lower. It has the strongest published evidence of being lossless, needs no calibration data in its dynamic form, and doubles the math rate over BF16. Pair it with an FP8 KV cache for any context beyond a few thousand tokens.

Reach for INT4 W4A16 when the model does not fit at 8-bit, or when you serve latency-sensitive traffic at low concurrency. Use AWQ or GPTQ with group size 128. Drop to 64 if accuracy lands just short. Calibrate on samples from your real traffic. On Ampere, INT4 via Marlin kernels is the strongest option; INT8 W8A8 with SmoothQuant is the compute-bound alternative.

Treat NVFP4 and MXFP4 as Blackwell-specific wins that require validation. Prefer checkpoints whose authors post-trained or published evaluations in that format over your own PTQ. Run the full accuracy gate before production.

Whatever you choose, keep the BF16 checkpoint and the evaluation harness under version control next to the quantized artifact. Re-run both after every engine upgrade.

Checklist before shipping a quantized model:

  • [ ] Hardware support confirmed for the format’s compute path, not only its loader.
  • [ ] BF16 baseline measured on the same engine, GPU and settings.
  • [ ] Calibration set drawn from production-like traffic; sample count recorded.
  • [ ] Accuracy gate includes reasoning, long-context and your domain tasks.
  • [ ] Sensitive layers identified and excluded where needed (LM head at minimum).
  • [ ] Latency at batch 1–8 and throughput at high concurrency both measured.
  • [ ] KV cache precision chosen and calibrated for long contexts.
  • [ ] Cost per million tokens computed against real GPU pricing.
  • [ ] Canary evaluation and rollback path ready after deployment.

Frequently Asked Questions

Is FP8 better than INT8 for LLM inference?

On GPUs with FP8 tensor cores (Ada, Hopper, Blackwell, and AMD CDNA3 or later), usually yes. Peak math rates are identical on H100, so the advantage is accuracy and simplicity. FP8’s exponent bits absorb activation outliers. Large-scale evaluation found FP8 W8A8 effectively lossless on Llama 3.1, while tuned INT8 W8A8 lost 1 to 3 percent per task. On Ampere and Turing, which lack FP8 tensor cores, INT8 W8A8 with SmoothQuant remains the right 8-bit compute format.

Does INT4 quantization hurt LLM accuracy?

A little, unevenly. Weight-only INT4 with AWQ or GPTQ at group size 128 recovered about 99 percent of baseline accuracy on average across most Llama 3.1 sizes in published work. Small models lost more on reasoning tasks. Averages hide variance, so multi-step math, code and long-context retrieval tend to degrade first. Evaluate those capabilities specifically. A smaller group size or skipping sensitive layers usually recovers most of a shortfall without leaving 4-bit.

What is the difference between W8A8 and W4A16?

The labels give the bit width of weights (W) and activations (A). W8A8 quantizes both, so the matrix multiply runs in 8-bit on tensor cores. That speeds up compute-bound phases like prefill and high-concurrency decode. W4A16 stores 4-bit weights but dequantizes them to 16-bit before multiplying. It cuts memory and bandwidth by about 4x but does not accelerate the math. In practice W4A16 wins low-batch latency and W8A8 wins high-concurrency throughput.

Can I run FP8 models on an A100?

You can load them, but you will not get FP8 compute. The A100 (Ampere, compute capability 8.0) has no FP8 tensor cores. vLLM can serve FP8 checkpoints on Ampere as weight-only W8A16 using Marlin kernels. That halves weight memory versus BF16 but runs the multiply in 16-bit. If you are on A100s and need compute speedup, INT8 W8A8 is the native option. For memory savings, INT4 W4A16 goes further than FP8 weight-only.

What is NVFP4 and how is it different from MXFP4?

Both are 4-bit floating-point formats using E2M1 elements with shared block scales. MXFP4, from the OCP Microscaling specification, uses 32-element blocks with a power-of-two E8M0 scale, about 4.25 bits per value. NVFP4, NVIDIA’s Blackwell format, uses 16-element blocks with an FP8 E4M3 scale plus a per-tensor FP32 scale, about 4.5 bits per value. NVFP4’s finer, fractional scaling fits local ranges better, which generally helps post-training quantization accuracy.

Should I quantize the KV cache too?

For long contexts, almost always. The KV cache grows linearly with sequence length and batch size. For Llama 3.1 70B it costs about 320 KiB per token in BF16, roughly 40 GiB for a single 128K-token sequence. An FP8 KV cache halves that, letting you serve more concurrent requests or longer contexts on the same GPU. Calibrate the scales rather than relying on defaults, and consider skipping especially sensitive layers.

Further Reading

By Riju — about

4 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *