Vision Transformer vs ConvNeXt vs Swin: Choosing a Vision Backbone for Industrial Inspection

Every few months a new benchmark table declares one family of vision models the winner, and every few months a plant engineer discovers that the winner does not fit on the camera box, does not survive INT8 quantization, or overfits the 400 defect images the line actually produced. The vision transformer vs ConvNeXt question is rarely about ImageNet accuracy. It is about inductive bias, token arithmetic, pretraining data, and what your edge accelerator can compile.

This matters now because the three families have converged in quality while diverging sharply in deployment behavior. A plain Vision Transformer (ViT), a hierarchical Swin Transformer, and a modernized ConvNeXt can land within a point or two of each other on a benchmark, yet differ by an order of magnitude in memory at high resolution and by months of integration effort on an NPU.

You will leave with the parameter and FLOP arithmetic behind each backbone, a worked windowed-attention complexity example, the ConvNeXt design changes explained, the role of self-supervised pretraining for small defect datasets, a quantization and latency view for edge targets, a runnable timm benchmarking script, and a decision matrix.

What this covers: inductive bias, ViT patchify math, Swin window complexity, ConvNeXt modernizations, DINOv2 and MAE pretraining, resolution scaling, edge quantization, distillation, and how to choose.

Context and Background

Convolutional networks dominated vision from AlexNet through ResNet and EfficientNet. In October 2020 Dosovitskiy and colleagues showed that a pure transformer applied to image patches, the ViT, matches or beats strong convolutional baselines when pretrained on very large datasets such as JFT-300M and ImageNet-21k. The catch was in the caveat: with only ImageNet-1k, ViTs trailed comparably sized ResNets, because attention lacks the built-in assumptions that make convolutions sample efficient.

Two responses followed. The first was to give transformers the missing structure. Swin Transformer (Liu et al., 2021) introduced hierarchical feature maps and shifted local windows, which made transformers practical as general-purpose backbones for detection and segmentation. The second was to ask whether the transformer’s advantage was the attention mechanism at all. ConvNeXt (Liu et al., 2022, “A ConvNet for the 2020s”) took a ResNet and applied transformer-era training recipes and macro design choices step by step, ending with a pure convolutional model that competes with Swin at matched FLOPs.

In parallel, self-supervised learning changed the economics. Methods such as masked autoencoders (MAE) and DINOv2 produce general features without labels, which matters enormously when your labeled defect set is small. We will return to that.

For industrial work the question has a practical frame. Your inputs are often high resolution (several megapixels), your defects are small, your positives are rare, and your hardware is a fixed-function NPU or a Jetson-class GPU. Our earlier piece on industrial machine vision defect detection at the edge covers the system around the model; this article zooms into the backbone choice. For the detector heads that sit on top of these backbones, see the comparative analysis of state-of-the-art object detection models.

A note on scope: the numbers below for parameters and FLOPs come from the original papers and the official ConvNeXt repository, or from first-principles arithmetic shown in the text. Latency numbers are never quoted without a method, because they depend on your compiler, driver, batch size, and precision. Where a figure is illustrative, it is labeled as such.

For the primary sources, read the ViT paper, the Swin paper, and the ConvNeXt paper.

The Core Argument: Backbones Differ in Inductive Bias, Token Cost, and Compilability

The short answer: ConvNeXt, Swin, and ViT differ in how much structure they assume (locality and translation equivariance), how attention cost grows with resolution, and how cleanly their operators compile to edge hardware. Convolution-heavy ConvNeXt wins on small data and NPUs, Swin on dense prediction accuracy, and plain ViT on pretrained-feature leverage.

Figure 1: The three backbone families from input image to feature maps. ViT keeps a single scale with global attention, Swin builds a pyramid with local windows, ConvNeXt builds a pyramid with depthwise convolutions.

Figure 1 sums up the structural differences. A ViT patchifies once and keeps the same token grid through all layers. Swin and ConvNeXt both reduce resolution in four stages, producing the feature pyramid that detectors and segmenters expect. This is why ViT needed adapter tricks (such as ViTDet-style simple feature pyramids) to serve dense tasks, while the hierarchical models dropped in directly.

Inductive bias: what the architecture assumes for free

A convolution bakes in two assumptions. Locality: a pixel is most related to its neighbors, so each output depends on a small window. Translation equivariance: a scratch detector works the same anywhere in the image, because the same weights slide everywhere. These assumptions are almost exactly right for natural images and for most industrial surfaces. A scratch on a metal sheet is a scratch whether it is at the top left or bottom right.

Self-attention assumes neither. Every token can attend to every other token with weights computed from content, and positional information must be injected through learned or fixed position embeddings. This is more flexible, and it lets the model relate a defect at one corner of a part to a fixture mark at the other. But flexibility has a price: the network must learn from data what convolutions get for free. That is the root of the classic data-efficiency gap, and it explains why ViTs needed JFT-300M or ImageNet-21k scale to shine in the original paper.

Swin sits in between. Its window attention reintroduces locality (tokens attend within a window), and its relative position bias reintroduces a form of translation equivariance, since the bias depends on relative offsets within the window. Shifted windows then let information flow between windows across layers. ConvNeXt keeps the convolutional biases and borrows only the training and macro-design lessons from transformers.

The practical consequence for defect data

Industrial defect sets are small, imbalanced, and expensive to label. In that regime the architecture prior is a form of free labeled data. Given only a few thousand images, a convolutional or hybrid backbone initialized from a strong pretrained checkpoint typically fine-tunes more stably than a randomly initialized or weakly pretrained ViT. That statement is a general tendency reported across the literature, not a law, and the exception is important: a ViT pretrained with self-supervision on a huge corpus (DINOv2) carries so much general structure that it often matches or beats convolutional backbones even with little labeled data, and we cover that below.

Where the cost lives

For a ViT with D channels and n tokens, each layer costs about 12nD² multiply-accumulates (MACs) in the linear projections and MLP, plus 2n²D for the attention matrix products. The first term is linear in n; the second is quadratic. At 224×224 with 16×16 patches, n is 196 and the quadratic term is a rounding error. At a megapixel it dominates. That crossover is the single most important fact for industrial users, because inspection images are not 224 pixels wide. We work the numbers in the next section.

The final structural point is operator support. Convolutions, depthwise convolutions, LayerNorm, and GELU are well supported across TensorRT, OpenVINO, and most vendor NPU compilers. Softmax attention, window partitioning and reverse, cyclic shifts (torch.roll), and dynamic reshapes are the operators that cause fallbacks to CPU or accuracy collapse under INT8. We detail that in the edge section.

ViT Patchify Mechanics, Parameter and FLOP Arithmetic

Understanding the numbers lets you predict cost for any input size instead of trusting a benchmark table.

Patchify and the token grid

A ViT splits an H x W image into non-overlapping P x P patches, giving n = (H/P)(W/P) tokens. Each patch is flattened (P x P x 3 values) and linearly projected to D dimensions. This is equivalent to a single convolution with kernel P and stride P. A class token is prepended and learned position embeddings are added.

For ViT-B/16 (P = 16, D = 768, 12 layers, 12 heads, MLP width 3072) at 224×224: n = 14 x 14 = 196 patch tokens, plus one class token, so 197.

Parameters per layer

One encoder layer has four D x D projections in attention (Q, K, V, output), which is about 4D² parameters, and a two-layer MLP with expansion 4, which is 8D². The total is about 12D² per layer, ignoring biases and norms. With D = 768, D² = 589,824, so 12D² is about 7.08M. Twelve layers give about 85M. Add the patch embedding (16 x 16 x 3 x 768 = 589,824), position embeddings (197 x 768 = about 151k), and a 1000-class head (about 769k) and you reach roughly 86M. That matches the widely reported 86M for ViT-B/16.

FLOPs per layer, and the quadratic term

In MACs, the linear parts cost 12 n D² per layer. The attention score matrix QK^T costs n²D, and applying it to V costs another n²D, so 2n²D. For ViT-B at n = 196:

Linear: 12 x 196 x 589,824 = about 1.387 GMACs
Attention: 2 x 196² x 768 = about 59 MMACs
Per layer about 1.446 GMACs; times 12 about 17.35 GMACs, plus patch embedding about 0.12 GMACs, giving about 17.5 GMACs.

This agrees with the roughly 17.6 GFLOPs commonly quoted for ViT-B/16 at 224 (in the convention where one multiply-accumulate counts as one FLOP, which is also how the ConvNeXt and Swin papers report numbers). Watch this convention: some tools report true floating-point operations, which is twice as large.

Now scale the input to 1024×1024, a modest inspection crop. With P = 16, n = 64 x 64 = 4,096 tokens.

Linear: 12 x 4,096 x 589,824 = about 29.0 GMACs per layer
Attention: 2 x 4,096² x 768 = about 25.8 GMACs per layer
Total per layer about 54.8 GMACs; times 12 about 658 GMACs.

So a 21x increase in pixels (about 20.9) cost about 38x more compute, and attention became nearly half the budget. Memory is worse: the attention matrix is 4,096² = 16.8M entries per head per layer. With 12 heads in fp16 (2 bytes) that is about 403 MB per layer if materialized. FlashAttention-style fused kernels avoid materializing it, which rescues memory, but not the quadratic compute, and fused kernels are a GPU story, not an NPU story.

Position embeddings and resolution transfer

Learned absolute position embeddings are tied to the training grid. To run at a new resolution you interpolate them (bicubic over the 2D grid), which works acceptably but degrades if the shift is large. Modern checkpoints sometimes use relative or rotary position encodings to ease this. In ViT-based pipelines for inspection, plan on fine-tuning at the deployment resolution rather than hoping interpolation is enough.

Figure 2: Patchify pipeline and where cost scales. Tokens grow with image area; the linear terms scale with n while attention scales with n squared.

Figure 2 traces the flow from image to tokens to per-layer cost. Notice that the patch size is a design lever: halving P quadruples n and multiplies attention cost sixteen-fold. Small defects push you toward small patches, which is exactly where global attention becomes unaffordable. That tension is what Swin was designed to resolve.

Swin Transformer: Windowed Attention and Why Complexity Becomes Linear

The direct-answer version: Swin Transformer restricts self-attention to fixed M x M windows (M = 7 by default), so attention cost grows linearly with the number of tokens instead of quadratically. Shifted windows in alternating layers let neighboring windows exchange information, and a four-stage hierarchy produces the feature pyramids detectors need.

Architecture in brief

Swin-T uses a 4×4 patch embedding (so a 224 image becomes a 56×56 grid of tokens), then four stages with channel widths C, 2C, 4C, 8C where C = 96, depths of 2, 2, 6, 2 blocks, and 3, 6, 12, 24 attention heads. Between stages, a patch-merging layer concatenates each 2×2 neighborhood of tokens and projects it, halving resolution and doubling channels. Window size is 7 in the original configuration. At 224 input the four stages operate on 56×56, 28×28, 14×14, and 7×7 token grids, so the final stage has exactly one 7×7 window, which is effectively global attention.

Each block pair alternates regular window attention (W-MSA) with shifted window attention (SW-MSA). The shift displaces the window grid by half a window (M/2), so tokens at the border of one window sit in the middle of a window in the next layer. The implementation uses a cyclic shift (torch.roll) with an attention mask so that wrapped-around tokens do not attend to unrelated regions. Relative position bias, a learned table indexed by the offset between token pairs inside a window, is added to the attention logits.

Reported ImageNet-1k results at 224 (original paper, ImageNet-1k training): Swin-T about 29M parameters, 4.5 GFLOPs, 81.3% top-1; Swin-B about 88M parameters, 15.4 GFLOPs, 83.5% top-1. The ConvNeXt repository reports ConvNeXt-T at 28M parameters, 4.5 GFLOPs, and 82.1% top-1, and ConvNeXt-B at 89M, 15.4 GFLOPs, and 83.8%. Matched FLOPs, matched accuracy within a point or so: that is the headline finding of the ConvNeXt paper.

Worked example: global vs windowed attention

The Swin paper gives the complexity of a global multi-head self-attention layer on an h x w grid of C-channel tokens as 4hwC² + 2(hw)²C, and of a window-based layer as 4hwC² + 2M²hwC. The first term (the Q, K, V, and output projections) is identical. Only the attention matrix term changes: (hw)² becomes M² x hw.

Take Swin-T stage 1 on a standard 224 input: h = w = 56, hw = 3,136, C = 96, M = 7.

Global attention term: 2 x 3,136² x 96 = about 1.89 GMACs
Windowed term: 2 x 49 x 3,136 x 96 = about 29.5 MMACs
Ratio: hw / M² = 3,136 / 49 = 64

Now the same stage for a 1024×1024 input, where stage 1 has a 256×256 grid, hw = 65,536:

Global: 2 x 65,536² x 96 = about 825 GMACs, for a single layer
Windowed: 2 x 49 x 65,536 x 96 = about 0.62 GMACs
Linear projections (4hwC²): 4 x 65,536 x 9,216 = about 2.4 GMACs
Ratio global/windowed: 65,536 / 49 = about 1,337

At that resolution, the windowed attention is about a quarter of the projection cost. The whole stage scales linearly with image area, which is why a Swin backbone at 1024 input is feasible while a naive global-attention ViT at the same token count is not. These are arithmetic results from the paper’s formulas, not measurements; real latency also depends on memory traffic from the window partition, roll, and reverse operations, which the FLOP count ignores.

Figure 3: Windowed attention with a shifted window layer. Tokens attend inside 7×7 windows; the shift in the next block connects neighboring windows, so cost is linear in token count.

Figure 3 depicts the alternating pattern: layer l partitions into aligned windows, layer l+1 shifts the partition so information crosses former boundaries. After several block pairs the receptive field grows well beyond a single window, and patch merging enlarges it further.

ConvNeXt: What the Modernization Actually Changed

ConvNeXt answers a precise question: if you take a standard ResNet and adopt, one at a time, the design choices that made transformers successful, how far does a pure convolutional network get? The answer, at ImageNet-scale, is that it matches Swin at similar FLOPs, while keeping convolution’s simplicity and hardware support.

The roadmap from ResNet-50

The authors started with ResNet-50 and a modernized training recipe borrowed from DeiT and Swin: AdamW, longer training (300 epochs), Mixup, CutMix, RandAugment, random erasing, stochastic depth, label smoothing. The recipe alone improved ResNet-50 by a few points of top-1 accuracy (reported at roughly 76 to 79 percent in the paper), a useful warning that many “architecture wins” in older comparisons were really recipe wins. Then they changed the architecture:

Stage compute ratio. Shift blocks per stage from ResNet’s (3, 4, 6, 3) to (3, 3, 9, 3) to mirror Swin-T’s 1:1:3:1 ratio.
Patchify stem. Replace the 7×7 stride-2 convolution plus max pooling with a 4×4, stride-4 convolution, the same non-overlapping patch idea as ViT and Swin.
Depthwise convolution. Replace the 3×3 grouped convolution with a depthwise convolution (ResNeXt-ify), separating spatial mixing from channel mixing like attention versus MLP. Width was increased to compensate.
Inverted bottleneck. Expand the hidden dimension by 4x in the middle, as in the transformer MLP block, instead of squeezing it.
Large kernels. Move the depthwise convolution up and use a 7×7 kernel, matching Swin’s 7×7 window and restoring a large receptive field cheaply.
Micro design. Replace ReLU with GELU, keep a single activation per block, keep fewer normalization layers, swap BatchNorm for LayerNorm, and use separate downsampling layers between stages.

The final block is simple: a 7×7 depthwise convolution, LayerNorm, a 1×1 convolution expanding by 4x, GELU, and a 1×1 convolution projecting back, with a residual connection (and optional layer scale and stochastic depth).

Why this block is hardware-friendly

Every operator in a ConvNeXt block has mature support. Depthwise 7×7 convolutions run on GPUs through cuDNN, on NPUs as depthwise convolution primitives, and on CPUs through oneDNN. The 1×1 convolutions are matrix multiplies. LayerNorm is the one operator worth watching: some NPU toolchains implement it in higher precision or as a fused op, and a few older stacks lack it, in which case folding it or replacing it with BatchNorm-style normalization is a known deployment adaptation (at some accuracy cost that must be measured, not assumed).

A caution on large depthwise kernels: depthwise convolutions have low arithmetic intensity, meaning few FLOPs per byte moved. On a GPU they often underutilize compute, so ConvNeXt’s real latency per FLOP is worse than a dense-convolution ResNet’s. FLOPs overstate efficiency here; measure the model on your target.

Variants you will actually meet

ConvNeXt comes as T, S, B, L, and XL sizes. Reported ImageNet-1k results at 224 for the family: T 28M parameters and 4.5 GFLOPs at 82.1%, S 50M and 8.7 GFLOPs at 83.1%, B 89M and 15.4 GFLOPs at 83.8%, L 198M and 34.4 GFLOPs at 84.3%. A successor, ConvNeXt V2, adds a Global Response Normalization layer and pairs the architecture with a fully convolutional masked autoencoder pretraining scheme; check the official repository for its exact numbers before relying on them, as I have not re-verified them here. Smaller industrial deployments commonly use T or even smaller variants (Atto/Femto-class models exist in timm; verify the exact names in your timm version).

Self-Supervised Pretraining and the Small-Defect-Dataset Regime

The backbone architecture sets the ceiling; the pretraining sets how much of that ceiling you reach with 500 labeled images.

MAE and masked image modeling

A masked autoencoder (He et al., 2021) masks a high fraction (75 percent) of image patches, encodes only the visible ones with a ViT encoder, and reconstructs the missing pixels with a lightweight decoder. Because the encoder sees only 25 percent of tokens, pretraining is cheap. MAE-pretrained ViTs fine-tune very well on ImageNet, but the features are not strongly linearly separable out of the box; they need fine-tuning. For industrial teams, the appeal is that you can run MAE pretraining on your own unlabeled factory images, which you have in abundance, then fine-tune on the few labeled ones. This domain-adaptive pretraining is often the highest-leverage move for odd modalities such as X-ray, thermal, or microscopy imagery where web-pretrained features transfer poorly.

DINOv2 and frozen general-purpose features

DINOv2 (Oquab et al., 2023, Meta) trains ViTs with a self-distillation objective combining an image-level DINO loss and a patch-level iBOT masked prediction loss, plus a KoLeo regularizer. The authors curated LVD-142M, a dataset of 142 million images assembled by retrieving images similar to curated sources from a pool of about 1.2 billion uncurated web images, without using text or metadata. Their largest model, ViT-g, has roughly 1.1B parameters, and smaller models (ViT-S, B, L) are distilled from it. Pretraining mostly runs at 224 resolution with a short high-resolution phase at 518.

The practical value is that DINOv2 features work well with a frozen backbone and a light head (linear probe, small decoder, or k-NN). For defect detection that opens a data-efficient path: freeze a DINOv2 ViT-S or ViT-B, extract patch features, and train a small head or even do patch-level anomaly scoring against features from good parts. You can then label very few defects. Check the license of the specific checkpoint before commercial use; the original DINOv2 code and weights carried a non-commercial license at release, and later releases changed terms for some models, so read the current license on the official repository rather than assuming.

Data-efficiency regimes

A rough rule of thumb, which is a heuristic and not a measured law, divides the space into three regimes.

Under a few hundred labeled images per class. Use a frozen or lightly tuned strongly pretrained feature extractor (DINOv2 ViT-S/B, or ConvNeXt with a good checkpoint) with a small head, or patch-feature anomaly detection. Full fine-tuning of a large model will overfit.
Thousands of labeled images. Fine-tune ConvNeXt-T/S or Swin-T/S from a good checkpoint, with strong augmentation. Convolutional priors help most here.
Tens of thousands or more, plus unlabeled data. Domain-adaptive self-supervised pretraining then fine-tuning; architecture matters less than data quality and label consistency.

The label noise problem often dominates all of this. Industrial annotators disagree on borderline defects, and no backbone fixes inconsistent labels. Spend time on a labeling guide before comparing architectures.

Resolution Scaling for Small Defects

Defect size, not object size, sets your input resolution. If a hairline crack is 3 pixels wide in a 4,000-pixel-wide image, downsampling to 224 erases it. You have four options, and each interacts differently with the three backbones.

Downsample and hope. Works only if the defect remains several pixels wide at network input. A convolutional or hierarchical model downsamples internally by a factor of 32 at the last stage, so a feature at stride 32 summarizes a 32×32 region; the defect must still dominate that region’s signal.

Tile the image. Cut the frame into overlapping crops (for example 512×512 or 640×640) and run the backbone on each. Cost is linear in the number of tiles for every architecture, which neutralizes the quadratic penalty of global attention because each tile has a bounded token count. Tiling is often the right answer for ViT: at 512×512 with P = 16 you have 1,024 tokens, and the quadratic term is manageable. The costs are stitching logic, duplicate detections at tile borders, and loss of context across tiles.

Run at native resolution with a hierarchical backbone. This is where Swin and ConvNeXt shine: cost scales roughly linearly with pixels, and the multi-scale pyramid supports small-object detection. Plan for padding to a multiple of the window size for Swin (window 7 times 32 total stride means 224-pixel multiples at the coarsest level, unless the implementation pads internally).

Two-stage coarse-to-fine. A cheap model finds candidate regions at low resolution, and a stronger model examines high-resolution crops around them. This is common in production because most images are defect-free and the cheap first stage rejects the bulk.

Edge Deployment: Quantization, Compilers, and Latency

Parameters and FLOPs are weak predictors of edge latency. What matters is operator coverage, memory traffic, and precision behavior.

Figure 4: Compile path from trained backbone to edge runtime. Attention, window reshaping, and normalization layers are where quantization and operator support problems appear.

Why attention is painful to quantize

Post-training INT8 quantization works well for convolutions because their activation distributions are tame. Transformers have a known problem: certain channels in LayerNorm outputs and residual streams carry large-magnitude outliers (documented in the literature on transformer quantization, for example in LLM work and in ViT-specific PTQ papers). A per-tensor INT8 scale that accommodates the outliers crushes the resolution available to everything else; a scale that fits the bulk clips the outliers and damages accuracy. Softmax is also sensitive: its output is a probability distribution with a huge dynamic range, and its inputs are exponentiated, so small quantization errors in the logits become large errors in the weights. GELU and LayerNorm usually stay in higher precision or use integer approximations on some NPUs.

Mitigations exist: per-channel or per-token scales, keeping softmax, LayerNorm, and the residual path in FP16, quantization-aware training (QAT), and ViT-specific PTQ methods. They add engineering work and compiler dependence. A ConvNeXt block, in contrast, is mostly 1×1 and depthwise convolutions where standard PTQ with calibration works, apart from the LayerNorm. Where possible, quantize ConvNeXt with a representative calibration set of 100 to 500 real production images and verify the accuracy drop on your holdout set. Our companion article on INT4 vs INT8 vs FP8 quantization for edge NPUs covers the precision trade-offs in depth.

Measuring latency properly

A benchmark you can trust has these properties: fixed input resolution matching production, batch size matching production (often 1), warmup iterations discarded, GPU synchronization around timers, the exact precision you will ship, and percentile reporting (p50 and p99), not the mean. Include pre- and post-processing, because image decode, resize, and non-maximum suppression can match a small model’s inference time. Also measure power and thermal behavior over minutes, since edge devices throttle.

I am deliberately not quoting millisecond numbers for these models here. They would depend entirely on a device, compiler version, and precision, and a number lifted from a vendor blog is not transferable to your line. The script in the next section lets you collect your own in minutes.

Distillation as the edge bridge

When the best accuracy comes from a heavy backbone (a large Swin or a DINOv2 ViT-L) and the target is a small NPU, distill. Train the large model as a teacher, then train a small ConvNeXt or other compact convolutional student to match its logits or intermediate features on a large pool of unlabeled production images. DINOv2 itself uses distillation from ViT-g to smaller models, and the paper reports the distilled ViT-L beating a scratch-trained ViT-L across its benchmarks, evidence that the approach transfers knowledge not easily learned from labels alone. For industrial inspection, distilling into a convolutional student also removes the attention quantization problem from your deployment path while keeping most of the teacher’s behavior. Verify on defect classes, not just aggregate accuracy: rare classes are the first casualty of distillation if the unlabeled pool under-represents them.

Runnable Walk-Through: Counting Parameters, FLOPs, and Latency with timm

The following script counts parameters, estimates FLOPs at a chosen resolution, and benchmarks latency for representative backbones. It uses timm for model definitions and fvcore for FLOP counting. Note that fvcore counts one multiply-accumulate as one flop, matching the paper convention, and it may not count every operator (attention matmuls are supported in recent versions, but check the warnings printed for skipped ops).

import time
import torch
import timm
from fvcore.nn import FlopCountAnalysis

MODELS = {
    "vit_base_patch16_224": 224,
    "swin_tiny_patch4_window7_224": 224,
    "convnext_tiny": 224,
}

def count_params(model):
    return sum(p.numel() for p in model.parameters()) / 1e6

def gflops(model, res, device):
    x = torch.randn(1, 3, res, res, device=device)
    fca = FlopCountAnalysis(model, x)
    fca.unsupported_ops_warnings(True)
    return fca.total() / 1e9

@torch.inference_mode()
def latency_ms(model, res, device, iters=100, warmup=20):
    x = torch.randn(1, 3, res, res, device=device)
    for _ in range(warmup):
        model(x)
    if device == "cuda":
        torch.cuda.synchronize()
    times = []
    for _ in range(iters):
        t0 = time.perf_counter()
        model(x)
        if device == "cuda":
            torch.cuda.synchronize()
        times.append((time.perf_counter() - t0) * 1000)
    times.sort()
    return times[len(times) // 2], times[int(len(times) * 0.99)]

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"device={device}")
for name, res in MODELS.items():
    # pretrained=False: we only measure cost, not accuracy
    m = timm.create_model(name, pretrained=False).eval().to(device)
    p = count_params(m)
    g = gflops(m, res, device)
    p50, p99 = latency_ms(m, res, device)
    print(f"{name:34s} {p:6.1f}M params  {g:6.2f} GMACs  "
          f"p50={p50:6.2f} ms  p99={p99:6.2f} ms")

To explore resolution scaling, pass img_size=1024 to timm.create_model for the ViT (and window_size adjusted for Swin so that the feature map at each stage divides evenly), then rerun the FLOP counter and watch the attention term grow. For ConvNeXt no change is needed because convolutions accept any input size. To measure the edge path, export with torch.onnx.export, compile with TensorRT’s trtexec --fp16 (or --int8 with a calibration cache), and compare the engine’s reported latency to the PyTorch figure; the gap is often large and informative.

A short sanity check on your results: ViT-B/16 at 224 should land near 17.5 GMACs, matching the arithmetic earlier, and ConvNeXt-T and Swin-T should each land near 4.5 GMACs, matching the published tables. If they do not, the counter is skipping operators or your convention differs.

Decision Matrix: Choosing by Dataset Size, Latency, and Accelerator

No backbone wins everywhere. The table encodes the trade-offs discussed above as defaults; override them with your measurements.

Situation	Best default	Why	Watch out for
Few hundred labeled images, GPU available	DINOv2 ViT-S/B frozen features + small head	Strong general features, minimal labels	License check, patch-size resolution limits
Few thousand labels, Jetson or desktop GPU	ConvNeXt-T/S fine-tuned	Convolutional prior, stable fine-tuning, simple export	Depthwise conv underutilizes GPU; measure latency
Dense prediction (segmentation, detection) at high resolution	Swin-T/S or ConvNeXt-T/S as backbone	Native feature pyramid, linear cost in pixels	Window padding; roll ops in export
Fixed-function convolution NPU, INT8 required	ConvNeXt-T (or smaller), or distilled convnet student	Operators well supported, PTQ-friendly apart from LayerNorm	Verify LayerNorm support in your compiler
Cloud or server GPU, large unlabeled corpus	ViT with MAE/DINOv2-style pretraining	Scales with data, strong transfer	Quadratic cost beyond about 1k tokens per tile
Very large images, small defects	Tiling + ConvNeXt, or two-stage coarse-to-fine	Bounded cost per tile	Tile-border duplicates, lost global context
Need best accuracy then deploy tiny	Distill big ViT/Swin into small ConvNeXt	Keeps accuracy, removes attention from deployment	Rare classes underrepresented in transfer set

A note on where YOLO fits, since the slug mentions it: YOLO-family detectors are complete detection systems with their own CNN-based backbones and heads, not a competitor to a pure feature backbone. In practice you pick between a YOLO-style integrated detector (fast to deploy, well-supported on edge) and a heavier backbone plus a dedicated detection or segmentation head (more accuracy headroom, more integration work). The object detection comparison discusses that decision.

Trade-offs, Gotchas, and What Goes Wrong

Benchmarks mislead on your data. ImageNet accuracy ranks these models within a narrow band, and the ranking is sensitive to training recipe. ConvNeXt’s own paper shows that recipe changes alone moved a ResNet by several points. Your factory images share little with ImageNet, so rank candidates on a held-out set of your own data, including hard negatives.

FLOPs are not latency. Depthwise convolutions, window reshapes, rolls, and softmax are memory-bound. A Swin-T and a ConvNeXt-T both have about 4.5 GFLOPs, yet their device latencies can differ substantially in either direction depending on the runtime. Treat FLOPs as an upper-level screen only.

Quantization cliffs. A model that scores well in FP32 can lose many points after naive INT8. Transformer outliers and softmax sensitivity are the usual culprits. Always evaluate the quantized model on the defect classes you care about, especially rare ones; aggregate accuracy hides collapses in a minority class.

Resolution mismatch. Position embeddings or relative bias tables trained at one grid and applied to another can silently degrade. Windowed models also fail on odd input sizes without padding. Fix input size early and test at that size.

Spurious cues. All backbones can latch onto shortcuts: lighting rigs that correlate with defect classes, a specific camera’s noise, a fixture visible only on defective parts. Strong pretrained features make it easier to reach high validation scores through such cues. Randomize capture conditions and audit with saliency tools.

Practical Recommendations

Begin with the constraint set, not the leaderboard. Write down the target accelerator and its supported operators, the shipping precision, the input resolution implied by your smallest defect, the labeled data you have, and the p99 latency budget. Those five facts eliminate most candidates before any training.

Then run a bake-off with a fixed protocol: the same data splits, the same augmentation policy, the same training budget, and checkpoints with comparable pretraining. Start with ConvNeXt-T, Swin-T, and a DINOv2-pretrained ViT-S or ViT-B, because they cover the main design philosophies at similar cost. Measure accuracy, per-class recall on rare defects, and quantized accuracy, then compile for the real device and measure p50 and p99.

If the heavy model wins on accuracy but the device cannot run it, distill into a compact convolutional student. If your labels are scarce, invest in self-supervised pretraining on unlabeled factory images before spending on larger architectures.

Fix resolution from defect size, then choose tiling or hierarchical backbones accordingly.
Prefer ConvNeXt when the NPU is convolution-centric or when labels are scarce and checkpoints are not DINOv2-class.
Prefer Swin when you need a pyramid for dense prediction and the toolchain handles windowed attention.
Prefer a DINOv2-style ViT as a frozen feature extractor when labels are very scarce and a GPU is available.
Export to ONNX and compile on the target before committing; read the unsupported-op report.
Quantize with real calibration data and evaluate rare-class recall afterward.
Log checkpoint, license, resolution, and precision with every model release.

Frequently Asked Questions

Is ConvNeXt better than a vision transformer?

Not categorically. At matched FLOPs on ImageNet, ConvNeXt performs on par with Swin Transformer, according to its authors, and it is simpler to deploy because it is purely convolutional. A ViT pretrained with a large self-supervised method like DINOv2 can deliver stronger frozen features, though. The better choice depends on your labeled data volume, target accelerator, and whether you need a feature pyramid. Always compare candidates on your own held-out defects before committing.

Why is Swin Transformer more efficient than a plain ViT at high resolution?

Swin computes self-attention only inside fixed windows of 7×7 tokens, so the attention cost grows linearly with the number of tokens, while a plain ViT’s global attention grows quadratically. At a 256×256 token grid, the window approach is about 1,300 times cheaper in the attention term than global attention for that layer. Shifted windows in alternating layers then pass information between neighboring windows, restoring long-range communication gradually.

Which vision backbone is best for edge deployment?

For convolution-centric NPUs and INT8-only pipelines, ConvNeXt-T or smaller convolutional models are usually the safest, because their operators are widely supported and quantize well apart from LayerNorm. Transformers can run well on GPUs with TensorRT and FP16. Attention softmax, window reshaping, and outlier activations make INT8 harder. Confirm by exporting to ONNX, compiling for your device, and checking the unsupported-operator report before choosing.

How much labeled data do I need to fine-tune a vision transformer for defect detection?

There is no universal threshold, and results depend on pretraining and defect complexity. As a heuristic, with only hundreds of labeled images, avoid full fine-tuning of large ViTs; use frozen self-supervised features such as DINOv2 with a small head, or anomaly detection on good-part features. With thousands of labels, fine-tuning ConvNeXt or Swin from a strong checkpoint is typically stable. Measure with cross-validation, not intuition.

Can I use DINOv2 or MAE features for industrial inspection?

Yes, and they are often the most data-efficient option. DINOv2 gives strong frozen patch features usable with linear probes or anomaly scoring, and MAE lets you pretrain on your own unlabeled images, which helps for X-ray, thermal, or microscopy data where web-pretrained features transfer poorly. Check the license of the specific checkpoint for commercial use, and validate that features capture your defect scale at the patch size.

Does a bigger backbone always improve defect detection?

No. Image quality, lighting, label consistency, and resolution usually matter more than backbone size. A larger model also raises latency, memory, and quantization difficulty, and it can overfit small datasets. Gains from scaling are real on large, diverse datasets, but on a typical factory dataset the efficient route is a mid-sized pretrained backbone, careful data work, and, if needed, distillation into a smaller deployable student.