INT4 vs INT8 vs FP8 on Edge NPUs: The 2026 Quantization Trade-off

Every edge NPU roadmap slide in 2026 shows the same seductive bar chart: drop from INT8 to INT4
and throughput doubles, memory halves, and power drops accordingly. What the slide never shows is
the accuracy cliff that shows up three layers deep in a vision transformer, or the KV-cache
overflow that silently corrupts an on-device LLM’s long-context reasoning. The real question
practitioners face isn’t “how low can we go” — it’s int4 vs int8 edge npu accuracy: which
precision survives contact with your actual model, your actual hardware, and your actual
calibration data. FP8 complicates the picture further, sitting in an odd middle ground that some
2026 silicon treats as a first-class citizen and others don’t support at all. This matters now
because the NPUs shipping this year — Hailo-10H, Jetson Thor, Qualcomm’s sixth-gen Hexagon,
Apple’s M4-class Neural Engine — have diverged sharply in which formats they accelerate natively,
turning “just quantize it” into a hardware-specific engineering decision.

What this covers: the numerics of symmetric/asymmetric, per-tensor/per-channel/per-group
quantization; PTQ vs QAT mechanics including GPTQ and AWQ; how INT4, INT8, and FP8 (E4M3/E5M2,
plus microscaled MXFP4/NVFP4) actually differ at the bit level; a 2026 hardware support matrix
across Hailo, Jetson, Qualcomm, Apple, and Coral; real vs illustrative benchmark numbers for both
CNN/ViT vision models and edge LLMs; and the failure modes — outlier channels, activation
overflow, KV-cache degradation — that quantization tutorials routinely skip.

Context and Background

Post-training quantization moved from a research curiosity to a deployment default around
2020–2022, when TensorFlow Lite, ONNX Runtime, and TensorRT all shipped INT8 calibration
pipelines as first-class features. INT8 became the incumbent format for edge vision because it
offered a good compromise: roughly 4x memory reduction versus FP32, 2–4x throughput gains on
integer-native NPUs, and — for most CNNs — under 1% top-1 accuracy loss when calibrated properly.
By 2023–2024, INT4 weight-only quantization (GPTQ, AWQ) became the default for shrinking large
language models onto GPUs with limited VRAM, and vendors began pushing the same techniques down to
edge silicon.

2026 is the year that trend fully lands on-device. Edge NPUs now ship with native INT4 compute
paths (Hailo-10H, Qualcomm Hexagon), and NVIDIA’s Blackwell-based Jetson Thor introduces a
hardware Transformer Engine that dynamically switches between FP8 and NVFP4 at runtime — a
capability inherited from datacenter Blackwell GPUs. Apple’s Core ML stack has standardized on
4-bit block-wise palettization for on-device LLMs on the Neural Engine. The incumbent players in
this space — TensorRT Model Optimizer, ONNX Runtime’s QDQ (quantize-dequantize) format, and
Qualcomm’s QAIRT/QNN toolchain — have converged on similar calibration workflows even as the
underlying hardware diverges. For architectural context on how these NPUs compare on raw
throughput and power, see our Hailo-10H vs Jetson Orin Nano comparison.
On the standards side, the Open Compute Project’s OCP Microscaling Formats (MX) specification
is what NVFP4 and MXFP4 build on, giving FP4/FP8 a shared, vendor-neutral numeric definition
instead of each silicon vendor inventing its own.

Quantization Numerics: What INT4, INT8, and FP8 Actually Do to Your Weights

Direct answer: INT4 and INT8 are fixed-point formats that map a float range to a small integer
range via a scale (and optionally a zero-point), trading dynamic range for density; FP8 keeps a
floating-point exponent/mantissa split (commonly E4M3 or E5M2) so it preserves relative precision
across a wider range at roughly INT8’s storage cost, which is why LLM activations tolerate FP8
better than INT4 without extra tricks.

Symmetric vs asymmetric quantization

Symmetric quantization maps a float range [-a, a] to an integer range [-127, 127] (for INT8)
using a single scale factor: q = round(x / scale), with scale = a / 127. There’s no
zero-point, so dequantization is a single multiply — cheap on NPU dequant units. It works well
when the underlying tensor distribution is roughly centered on zero, which is common for weights
after batch normalization folding. Asymmetric quantization instead maps [min, max] to
[0, 255] (or [-128, 127]) using both a scale and a zero-point offset: q = round(x / scale) +
zero_point. This captures skewed distributions — post-ReLU activations, which are strictly
non-negative, are the textbook case — at the cost of an extra add during dequantization. Most 2026
toolchains (TensorRT ModelOpt, QAIRT) default to symmetric for weights and asymmetric for
activations, because weight distributions are usually zero-centered while activation
distributions after nonlinearities are not.

Per-tensor, per-channel, and per-group granularity

The coarsest option, per-tensor quantization, uses one scale for an entire weight tensor. It’s the
cheapest to compute and dequantize, but it’s fragile: if one output channel has weights an order
of magnitude larger than the rest (common in trained CNNs, especially in the last layers before a
classifier head), the shared scale either clips the outlier channel or wastes precision on every
other channel. Per-channel quantization fixes this by assigning an independent scale to each
output channel of a convolution or linear layer — the extra scale values are cheap to store
(one float per channel) and NPU compilers fold them into the dequantization step at negligible
runtime cost. This is now the de facto standard for INT8 weight quantization across every major
edge toolchain.

Per-group (also called blockwise or groupwise) quantization goes one level finer, splitting each
channel’s weight vector into groups of, typically, 32, 64, or 128 elements and giving each group
its own scale. This is essential for INT4, where the four-bit range (16 distinct values per
symmetric encoding, effectively ±7 with sign) is too coarse to represent an entire channel’s
dynamic range without severe error. Apple’s Core ML 4-bit palettization and Qualcomm’s INT4 weight
paths both use groupwise scales for this reason. The trade-off is storage overhead: at group size
32 with an FP16 scale per group, INT4 weights carry roughly 0.5 extra bits per parameter for scale
metadata — still a large win over INT8, but not the full 4x reduction the raw bit-width suggests.

FP8: two formats, one confusing name

“FP8” is not one format. The two variants that matter on 2026 edge silicon are E4M3 (4 exponent
bits, 3 mantissa bits) and E5M2 (5 exponent bits, 2 mantissa bits). E4M3 has a smaller dynamic
range (roughly ±448) but higher precision, making it the default for weights and forward-pass
activations. E5M2 has a wider dynamic range matching FP16’s exponent field, making it better
suited to gradients during training or activations with heavy-tailed distributions — a scenario
edge inference rarely hits, which is why inference-only NPUs (Jetson Thor’s Transformer Engine
included) lean on E4M3 almost exclusively. Unlike INT8, FP8 doesn’t need a per-tensor scale to
represent small values precisely near zero — the floating-point exponent already does that — so
FP8 quantization error tends to be more uniform across a tensor’s dynamic range than INT8’s, which
is one reason FP8 activations preserve LLM accuracy better than INT8 activations at comparable bit
width. NVIDIA’s NVFP4 and the OCP’s MXFP4 push this idea further with microscaling: a 4-bit
floating-point element (E2M1) shares an 8-bit exponent scale across a block of 16 or 32 values,
recovering much of FP8’s dynamic-range behavior at INT4’s storage cost — but only on hardware with
a native microscaling dequant unit, currently limited to Blackwell-generation silicon like Jetson
Thor.

PTQ vs QAT: two paths to the same bit width

Post-training quantization (PTQ) takes a fully trained FP32 or FP16 model, runs a small calibration
dataset (typically 100–1,000 samples) through it to observe activation ranges, computes scales
using a calibration algorithm (min-max, percentile clipping, or KL-divergence/entropy
minimization as used in TensorRT’s classic INT8 calibrator), and converts weights and activations
to the target format — no retraining required. PTQ is fast (minutes to hours) and is the default
path for INT8 and most FP8 deployments, where accuracy loss is typically small enough to tolerate.

Quantization-aware training (QAT) instead inserts “fake quantize” nodes into the training graph
that simulate quantization error (round-to-nearest plus clipping) during the forward pass, while
gradients flow through a straight-through estimator (STE) that treats the non-differentiable
rounding operation as identity for backpropagation purposes. The model’s weights adapt to the
quantization noise over several epochs of fine-tuning, which is why QAT recovers most of the
accuracy INT4 loses under PTQ — at the cost of needing labeled training data, compute for
fine-tuning, and engineering time. For LLM weight-only quantization specifically, the field has
largely moved to more efficient alternatives that sit between pure PTQ and full QAT: GPTQ, which
quantizes weights layer-by-layer using second-order (Hessian-based) error compensation so that
quantizing one weight adjusts the remaining unquantized weights in that layer to cancel error, and
AWQ (activation-aware weight quantization), which identifies the small fraction of “salient”
weight channels most responsible for output magnitude — typically under 1% of channels — and
preserves their precision (or scales them to survive quantization) while aggressively quantizing
the rest. Hailo’s compiler toolchain for its 10H accelerator explicitly builds on GPTQ-style
layer-wise correction combined with QuaRot-style rotation techniques that redistribute outlier
energy across channels before quantizing, which is part of how Hailo claims usable INT4 LLM
accuracy without full retraining.

Figure 1: The quantization numerics pipeline from FP32 weights through calibration, scale
computation, and NPU dequantization.

The pipeline above shows the common path every framework follows: a calibration pass over
representative data establishes per-tensor, per-channel, or per-group ranges; those ranges become
scale (and optionally zero-point) values; weights and activations are quantized into packed
low-bit storage; and at inference time the NPU’s integer MAC array accumulates in a wider
intermediate format (typically INT32 for INT8×INT8 or FP16/FP32 for FP8 products) before
dequantizing back to a usable activation range for the next layer.

Hardware Reality: What Edge NPUs Actually Support in 2026

The theory above is precision-agnostic, but real deployments are constrained by what silicon can
execute natively versus what it has to emulate — and emulation kills the latency win quantization
was supposed to deliver. The support matrix in 2026 is genuinely fragmented across vendors.

Hailo-10H: INT4/INT8 native, LLM-focused compiler

Hailo’s 10H accelerator, aimed at edge LLM and VLM workloads (and shipping in products like the
Raspberry Pi AI HAT+ 2), advertises 40 TOPS of INT8 performance and an equivalent 40 TOPS INT4
figure for its specialized paths, with the Hailo Dataflow Compiler dispatching mixed INT4/INT8
precision automatically — INT4 weights get expanded and processed through the same INT8 compute
units rather than requiring separate INT4 silicon. Independent accuracy figures reported for
1.5B–8B parameter LLMs quantized this way show roughly 1–2 percentage points of degradation
against FP16 baselines on standard benchmarks (MMLU, ARC, HellaSwag) — worth treating as
representative rather than universal, since exact numbers depend heavily on the specific model
architecture and calibration set.

Jetson Orin vs Jetson Thor: the FP8 generational split

This is the sharpest hardware discontinuity in the current landscape. Jetson Orin’s Ampere-class
GPU supports INT8 as its primary accelerated integer path through TensorRT, with FP8 support
present but not hardware-native in the same way — FP8 operations on Orin largely run through
software emulation paths rather than a dedicated tensor-core datatype, limiting the throughput
benefit. Jetson Thor, built on the Blackwell architecture, changes this: it ships a hardware
Transformer Engine that natively executes FP8 (E4M3) and NVFP4, dynamically switching between them
per-layer based on sensitivity, with NVIDIA recommending NVFP4 as the default precision for
Thor-class silicon (SM110+) because it delivers roughly 4x memory reduction with native hardware
decode rather than software unpacking. TensorRT Edge-LLM, introduced alongside JetPack 7.1,
formalizes INT4, NVFP4, and FP8 as first-class supported precisions specifically for on-device LLM
and VLM serving on Thor.

Qualcomm Hexagon: INT4/INT8 native on the HTP

Qualcomm’s Hexagon Tensor Processor (HTP), the NPU block inside Snapdragon SoCs, executes INT8
weights with INT8/INT16 activations as its baseline path, and its Hexagon Matrix Extension (HMX)
unit adds native INT4 support that doubles tensor throughput for INT4 layers relative to INT8 —
important because it means INT4 on Hexagon isn’t a storage trick layered on INT8 compute, it’s a
genuinely faster execution path. The Snapdragon X2 Elite generation (announced CES 2026) pushes
Hexagon NPU throughput to roughly 80 TOPS. Qualcomm’s QAIRT (Qualcomm AI Engine Direct, the
successor to SNPE) and Qualcomm AI Hub both expose INT4 and INT8 quantization workflows, and
QAIRT integrates with ONNX Runtime and Windows ML, which matters for the growing set of Snapdragon
Windows-on-Arm devices doing on-device inference.

Apple Neural Engine: INT4 works, but with sharp edges

Apple’s Core ML stack, since macOS Sequoia, supports 4-bit block-wise linear quantization
(palettization) as a first-class weight-compression path for the Neural Engine (ANE), and the
M4-generation ANE (roughly 38 TOPS) is explicitly positioned by Apple’s ML research team as
capable of serving multi-billion-parameter LLMs locally when combined with 4-bit quantization —
their published Llama 3.1 on-device work is the reference implementation. But field reports from
2026 are consistent on one point worth flagging honestly: INT4 is the practical floor on ANE.
Multiple independent implementations report that dropping to 3-bit weights produces unusable
output, and some INT4 configurations have failed to place cleanly on specific hardware generations
(reports of INT4 optimization failures on M3 Max specifically), which suggests ANE’s INT4 support,
while functional, is less uniformly robust across the product line than Qualcomm’s or Hailo’s. One
implementation cited in public write-ups achieved 99.78% operator placement on ANE with 4-bit
quantization, indicating that when it works, it works well — but “when it works” is doing real
work in that sentence, and per-model validation on target hardware isn’t optional.

Coral Edge TPU: the INT8-only holdout

Google’s Coral Edge TPU, still deployed in large volumes for vision workloads despite the chip’s
age, remains INT8-only at the hardware level — it has no INT4 or FP8 execution path at all. Models
must be fully INT8 quantized (both weights and activations) via TensorFlow Lite’s quantization
tooling to run on Edge TPU silicon; anything else falls back to CPU execution, which defeats the
purpose. Coral is a useful baseline for exactly this reason: it represents the “quantization is a
solved, boring problem” end of the spectrum that INT4/FP8 hardware is trying to move past.

Figure 2: 2026 edge NPU landscape and each platform’s native low-precision support.

Deeper Analysis: Vision Models vs Edge LLMs Behave Differently Under Quantization

Vision models (CNNs and ViTs) and LLMs respond to aggressive quantization in structurally
different ways, and conflating the two is the single most common mistake in quantization planning.

CNN weight distributions are relatively well-behaved after batch-norm folding — most channels
cluster in a narrow range, and per-channel INT8 PTQ typically holds accuracy within a fraction of
a percentage point of FP32 for architectures like MobileNet, EfficientNet, and YOLO-family
detectors. Vision Transformers (ViTs) are harder: their attention and layer-norm activations
produce large outlier values in specific channels — a well-documented phenomenon in transformer
literature generally — which per-tensor INT8 quantization handles poorly and even per-channel
INT8 sometimes struggles with on the activation side (activations don’t get the same
“per-channel” luxury as weights in most NPU dequant hardware, which usually applies a single scale
per activation tensor for compute efficiency). This is precisely why techniques like QuaRot and
SmoothQuant, which mathematically redistribute outlier magnitude between weights and activations
before quantization (using a rotation or a per-channel smoothing factor), have become standard
pre-processing steps ahead of aggressive edge quantization for transformer-based vision and
language models alike.

LLMs add a third quantization surface that vision models don’t have: the KV-cache. Beyond
weight and activation quantization, long-context LLM serving on edge NPUs increasingly quantizes
the key-value cache itself (commonly to INT8 or FP8) to fit longer context windows in limited
on-device memory. This is a distinct failure surface from weight quantization — a poorly
calibrated KV-cache quantization scheme degrades coherence specifically in long-context tasks
(document QA, multi-turn conversation) while leaving short-prompt benchmarks looking fine, which
means short-context evaluation can mask a KV-cache quantization bug entirely.

Illustrative benchmark comparison

The table below combines cited, real figures where available with clearly labeled illustrative
estimates for configurations where no controlled, published benchmark was available for this
specific hardware/model/precision combination. Treat the illustrative rows as directional, not as
numbers to cite elsewhere without your own validation.

Config	Model class	Precision	Accuracy delta vs FP16	Relative latency	Source
Hailo-10H	1.5B–8B LLM	INT4/INT8 mixed	~1–2 pts (MMLU/ARC/HellaSwag)	Baseline (1.0x)	Hailo vendor benchmarks
Jetson Orin	INT8 CNN detector	INT8	<0.5 pt (illustrative, typical for YOLO-class PTQ)	~1.0x vs FP16 baseline	Illustrative — representative of published TensorRT INT8 PTQ results
Jetson Thor	LLM/VLM serving	FP8 (E4M3)	Smaller than INT8 at matched bit budget (illustrative)	Faster than Orin INT8 (vendor claim, exact figure workload-dependent)	NVIDIA TensorRT Edge-LLM docs
Jetson Thor	LLM serving	NVFP4	Larger than FP8, recoverable with QAT/GPTQ (illustrative)	~4x memory reduction vs FP16 (vendor claim)	NVIDIA Blackwell Transformer Engine docs
Qualcomm Hexagon HTP	Vision + LLM	INT4	Workload-dependent; larger than INT8 without groupwise calibration (illustrative)	~2x INT8 throughput on INT4 layers (vendor claim)	Qualcomm AI Hub docs
Apple ANE (M4)	On-device LLM	INT4 block-wise	Small with correct group size; unusable if pushed to INT3 (reported)	Enables multi-billion-param local serving (vendor claim)	Apple ML research
Coral Edge TPU	CNN vision	INT8 only	Small, well-characterized (mature toolchain)	N/A — only supported path	TFLite quantization docs

Methodology caveat: rows marked “illustrative” reflect general, well-documented patterns in the
quantization literature (INT4 degrades more than INT8 without correction; FP8 activations are more
robust than INT8 at matched precision; groupwise calibration narrows the INT4 gap) rather than a
single controlled benchmark run on identical hardware, models, and calibration sets. Anyone citing
exact percentage-point numbers from this table for a specific model should re-run calibration on
their own target model and hardware — quantization accuracy loss is not a hardware constant, it is
a function of model architecture, calibration data quality, and granularity choice, and it can
vary by 2–3x between two CNN architectures quantized with the identical pipeline.

Figure 3: Post-training quantization and quantization-aware training as parallel paths from a
trained model to a deployed edge NPU checkpoint.

Trade-offs, Gotchas, and What Goes Wrong

Outlier channels are the number-one silent failure mode. A single weight channel with a magnitude
10–50x the tensor average — common in the final projection layers of transformers — forces
per-tensor quantization to either clip that channel (destroying its signal) or blow up the shared
scale (wasting precision everywhere else). Per-channel quantization fixes weights but not
activations in most NPU compilers, since activation scales are typically computed and applied
per-tensor for dequantization-hardware simplicity. If your accuracy drop is concentrated in a few
specific outputs rather than spread uniformly, suspect an unhandled activation outlier before
blaming the bit-width itself.

Activation overflow is a related but distinct bug class: INT8×INT8 multiply-accumulate naturally
overflows a 16-bit intermediate for long accumulation chains, which is why NPU MAC arrays
accumulate in INT32 (or wider) before requantizing — but if a compiler misconfigures accumulation
width when targeting INT4 (where the multiply result range is smaller per-element but summed over
more elements to match a layer’s channel count), it can silently saturate. This shows up as
accuracy that degrades specifically on inputs with high activation magnitude — bright,
high-contrast images for vision models, or long prompts for LLMs — while calibration-set accuracy
looks fine, because calibration sets rarely stress the true tail of the input distribution.

KV-cache quantization for edge LLMs interacts badly with long-context use cases specifically. A
model quantized to INT4 weights with an unquantized or lightly quantized KV-cache can look
completely healthy on short-prompt benchmarks (the standard MMLU/ARC/HellaSwag suite is
short-context) while degrading materially on document summarization or multi-turn chat — exactly
the workloads edge LLM deployments are often built for. Evaluate on your actual context-length
distribution, not just the standard short-context academic benchmarks vendors report.

Calibration-data mismatch is the most common practitioner error, independent of hardware. A
calibration set that doesn’t represent production input distribution (wrong lighting conditions
for a vision model, wrong language mix or prompt style for an LLM) produces scales that are
locally optimal for the calibration set and poorly matched to production traffic — this failure
mode is invisible in offline validation against a held-out split drawn from the same
(mis-matched) distribution, and only appears once the model meets real-world inputs.

Mixed-precision compiler bugs are a 2026-specific risk: because Hailo, Thor, and Hexagon all now
do automatic per-layer precision assignment (deciding which layers get INT4 vs INT8, or FP8 vs
NVFP4), a compiler version bump can silently change which layers get the aggressive precision,
shifting your accuracy/latency point without any change to your model or calibration data. Pin
compiler/SDK versions for production builds and re-validate accuracy after any toolchain upgrade —
treat it the same as any other build-tool version bump. Finally, cross-precision comparisons
between vendors are rarely apples-to-apples: Hailo’s “40 TOPS INT4” and Qualcomm’s INT4 throughput
figure are not measuring the same workload, batch size, or sparsity assumptions, so raw TOPS
numbers across the hardware matrix above should inform shortlisting, not final procurement
decisions — always benchmark your actual model on actual candidate hardware.

Practical Recommendations

Start every quantization project with per-channel INT8 PTQ as the accuracy baseline — it is fast,
well-supported everywhere including Coral’s INT8-only path, and tells you whether your model
architecture is quantization-friendly before you spend engineering time on INT4 or FP8. If INT8
already misses your accuracy or latency target, move to INT4 with groupwise calibration (group
size 64–128 is a reasonable starting point) rather than jumping straight to per-tensor INT4, and
budget time for a QAT or GPTQ/AWQ correction pass if PTQ INT4 alone doesn’t clear your accuracy
bar. For LLM deployment specifically, prefer FP8 over INT4 activations when your target hardware
has native FP8 support (Jetson Thor, and increasingly other 2026 silicon) — the floating-point
exponent handles activation dynamic range more gracefully than fixed-point INT8/INT4 without
extra outlier-smoothing steps. Always validate on your real context-length distribution if the KV
cache is quantized, not just short-prompt academic benchmarks. Match your target hardware’s native
support before choosing a format on paper — a format that “works” only through software emulation
gives up most of the latency and power win that justified quantizing in the first place. For
runtime-level trade-offs between ONNX Runtime, TFLite, ExecuTorch, and Core ML deployment targets,
see our ONNX vs TFLite vs ExecuTorch vs Core ML comparison,
and for LLM-runtime-specific quantization behavior, see our
comparison of llama.cpp, MLC, and ONNX Runtime for on-device LLMs.

Pre-deployment checklist:
– Baseline with per-channel INT8 PTQ before attempting INT4 or FP8.
– Use groupwise (not per-tensor) scales for any INT4 weight quantization.
– Apply outlier-redistribution (QuaRot/SmoothQuant-style) before quantizing transformer-based
vision or language models.
– Validate on your production input distribution, not just the calibration or academic benchmark
set.
– If quantizing the KV-cache, test with your actual long-context workload explicitly.
– Pin compiler/SDK versions and re-validate accuracy after any toolchain upgrade.
– Confirm native hardware support for your chosen format before committing — check vendor docs,
not marketing TOPS figures alone.

Frequently Asked Questions

Is INT4 always faster than INT8 on edge NPUs?

Not necessarily. INT4 is only faster when the NPU has a genuinely native INT4 compute path — true
on Qualcomm’s Hexagon HMX and Hailo’s mixed-precision dispatch — where it roughly doubles
throughput per the vendor’s own figures. On hardware without native INT4 execution, INT4 storage
still has to be unpacked to INT8 or FP16 before compute, which can reduce memory bandwidth
pressure without improving raw compute latency, or in worst cases add unpacking overhead that
partially offsets the storage win.

When should I use FP8 instead of INT8 for edge LLMs?

Use FP8 when your target hardware has native FP8 tensor-core support — Jetson Thor’s Transformer
Engine is the clearest current example — because FP8’s floating-point exponent handles activation
dynamic range more gracefully than INT8’s fixed-point scale, typically preserving more accuracy at
a similar storage cost. On hardware without native FP8 (most Jetson Orin deployments, most mobile
NPUs as of 2026), INT8 remains the more broadly supported and better-optimized path.

Does per-channel quantization eliminate the need for QAT?

No. Per-channel quantization fixes a specific problem — weight distributions that vary sharply
between output channels — but it doesn’t recover accuracy lost to aggressive bit-width reduction
itself. INT4 models frequently still need QAT, GPTQ, or AWQ-style correction even with per-channel
or per-group scales, particularly for architectures with heavy activation outliers like
transformers.

Can I mix precisions within a single model?

Yes, and 2026 compiler toolchains increasingly do this automatically — Hailo’s and Thor’s
compilers both assign precision per-layer based on sensitivity analysis, keeping accuracy-critical
layers (often the first and last layers of a network) at higher precision while quantizing the
bulk of the network more aggressively. This mixed-precision approach typically recovers most of
the accuracy gap between full INT8 and full INT4 while retaining much of INT4’s memory and
throughput benefit.

How much calibration data do I actually need for PTQ?

Most toolchains recommend 100–1,000 representative samples for INT8 PTQ calibration, with
diminishing returns beyond a few hundred for well-behaved models. INT4 and mixed-precision
calibration (especially GPTQ, which processes calibration data layer-by-layer with Hessian-based
correction) can benefit from a larger and more diverse set, particularly if your production input
distribution has meaningfully different statistics across sub-populations — different lighting
regimes for vision, different languages or prompt styles for LLMs.

Is Coral Edge TPU worth using in 2026 given it’s INT8-only?

For workloads that already fit comfortably in INT8’s accuracy and latency envelope — mature CNN
detectors and classifiers, mostly — Coral remains a low-cost, low-power, well-understood option.
It’s the wrong choice for anything requiring INT4/FP8-level memory savings or for LLM/VLM
workloads, where its lack of native low-bit-width or floating-point-inference paths puts it well
behind Hailo, Qualcomm, and Jetson Thor.

Figure 4: A practical decision path for choosing between INT4, INT8, and FP8 based on workload
type and target hardware capability.

Further Reading

• Hailo-10H vs Jetson Orin Nano for edge computer vision —
  how these two NPU families compare on throughput, power, and toolchain maturity beyond
  quantization alone.
• ONNX vs TFLite vs ExecuTorch vs Core ML —
  runtime-level differences that affect how quantized models actually get deployed and served.
• On-device LLM runtimes: llama.cpp vs MLC vs ONNX Runtime —
  a deeper look at how these runtimes implement INT4/FP8 weight quantization for edge LLM serving.
• NVIDIA TensorRT Model Optimizer documentation —
  authoritative reference for FP8, INT4, and NVFP4 quantization workflows on Jetson and datacenter
  GPUs.
• OCP Microscaling Formats (MX) Specification v1.0 —
  the vendor-neutral standard underlying NVFP4/MXFP4 microscaled low-precision formats.

By Riju — about