Neural Network Pruning for LLMs and Edge Models: Unstructured vs Structured vs 2:4 Sparsity

Here is the uncomfortable fact about neural network pruning: you can often delete 50 to 90 percent of a model’s weights and lose almost no accuracy, yet the pruned model frequently runs no faster than the dense one. Zeros are free to store in a paper’s sparsity table and expensive to exploit on real silicon. A GPU or microcontroller is built to stream dense tiles through fixed-function units, and a scattering of zeros inside those tiles saves nothing unless the hardware knows where they are.

That gap between “sparse on paper” and “fast in production” is the entire story of pruning in 2026. For large language models (LLMs) it matters because decoding is bound by memory bandwidth, so every byte of weight you can avoid reading is latency you get back. For edge models it matters because flash, RAM and milliwatts are all scarce.

This guide gives you the mental model, the arithmetic and the code to decide which kind of sparsity to use, and when to skip pruning altogether.

What this covers: the three families of pruning (unstructured, structured, semi-structured 2:4), why only some of them speed anything up, how one-shot LLM methods such as SparseGPT and Wanda work, how pruning interacts with quantization, worked memory and FLOP numbers, runnable PyTorch code, and a decision guide against quantization and distillation.

Context and Background

Pruning is older than deep learning’s current boom. Han and colleagues’ “Deep Compression” paper reported shrinking AlexNet by 35x (240 MB to 6.9 MB) and VGG-16 by 49x (552 MB to 11.3 MB) with no reported accuracy loss on ImageNet, by chaining three stages: pruning (9x to 13x fewer connections), quantization (32 bits down to 5 bits per connection) and Huffman coding. Two lessons from that paper still hold. Pruning and quantization compound, and the storage win depends on the encoding you choose for the surviving weights.

The theoretical case got stronger with the lottery ticket hypothesis (Frankle and Carbin, 2018). Their abstract reports that standard pruning can remove over 90 percent of parameters without hurting accuracy, and that dense, randomly initialised networks contain “winning tickets”: subnetworks, often less than 10 to 20 percent of the original size on the MNIST and CIFAR10 architectures tested, that train in a similar number of iterations to comparable accuracy. The practical caveat is that finding those tickets required repeated train-prune-rewind cycles on small models. It explains why pruning works; it is not a recipe for pruning a 70-billion-parameter model.

Large language models changed the constraints. Retraining a 175-billion-parameter model to recover from pruning is out of reach for most teams, which pushed research toward one-shot, post-training methods. SparseGPT (Frantar and Alistarh, 2023) reported pruning OPT-175B and BLOOM-176B to at least 50 percent sparsity in one shot with no retraining, in under 4.5 hours per model, with negligible perplexity increase at up to 60 percent unstructured sparsity. Wanda (Sun et al., 2023) then showed that a far cheaper score, weight magnitude times input activation norm, is competitive on LLaMA models.

At the hardware end, NVIDIA’s Ampere generation introduced sparse tensor cores that exploit a fixed 2:4 pattern: in every group of four contiguous weights, at least two are zero. NVIDIA’s paper on the topic states that this pattern gives up to twice the peak math throughput of dense matrix units. Note the wording: peak math throughput, not end-to-end speedup. Much of this article is about the distance between those two.

If you work at the edge, the companion piece on TinyML on microcontrollers with TFLite Micro, INT8 quantization and CMSIS-NN covers the quantization half of the story. The constraints there make some pruning styles useful and others pointless, as we will see. For the broader compression landscape, the model compression overview in Deep Compression remains a good primary source to read alongside this post.

The Three Families of Pruning and Why Hardware Decides

The short answer: neural network pruning comes in three families, and the only question that matters for speed is whether the hardware can skip the zeros without per-element bookkeeping. Unstructured pruning removes individual weights and needs sparse formats that rarely beat dense kernels. Structured pruning removes whole channels, heads or layers, producing a smaller dense model that runs fast anywhere. Semi-structured 2:4 sparsity constrains the pattern just enough for dedicated hardware to exploit it.

Figure 1: The three pruning families, what each removes, and the route each takes to a real speedup.

The figure sorts pruning by granularity. At the fine end, a single weight is the unit of removal. At the coarse end, an entire neuron, convolution filter, attention head or transformer block is. The 2:4 pattern sits between them: fine-grained within a group of four, but regular enough that a compressed format with tiny metadata can describe it.

Unstructured pruning: maximum freedom, minimum hardware support

Unstructured pruning scores every weight independently and removes the lowest scorers. The classic criterion is magnitude: sort by absolute value, zero the smallest fraction, optionally fine-tune. Because the choice is unconstrained, it preserves accuracy best at a given sparsity level. At 50 percent it is nearly free for most pretrained models; at 80 to 90 percent it still works for many vision networks, consistent with the lottery ticket and Deep Compression results.

The trouble starts when you try to run it. A weight matrix with random zeros still occupies a dense rectangle in memory unless you convert it to a sparse format such as compressed sparse row (CSR). CSR stores each nonzero value plus its column index, and a row pointer array. Reading it requires irregular, indirect memory access, which GPUs and CPUs handle poorly compared with dense tiled access.

Here is the storage arithmetic for a 7-billion-parameter model in FP16 (illustrative, using 32-bit indices):

Representation	Bytes per kept weight	Total at 50% sparsity	Total at 90% sparsity
Dense FP16	2	14.0 GB	14.0 GB
CSR, FP16 value + int32 index	6	21.0 GB	4.2 GB
CSR, FP16 value + int16 index	4	14.0 GB	2.8 GB

At 50 percent sparsity, a 32-bit-index CSR file is 50 percent larger than the dense one. The break-even density with 6 bytes per nonzero is one third, which means about 67 percent sparsity before storage even improves. Compute follows the same pattern: sparse-matrix kernels must beat dense GPU kernels that run near peak, and at moderate sparsity they do not.

That is why unstructured pruning is a compression technique for storage and transfer (ship a smaller file, decompress to dense at load) far more often than an inference-acceleration technique. It does shine on hardware designed for it, such as certain CPU inference engines and research accelerators with zero-skipping logic, but those are specific deployments, not a default.

Structured pruning: smaller dense networks

Structured pruning deletes whole structural units so the remaining tensors are simply smaller. Remove 30 percent of the output filters in a convolution layer and the next layer’s input channels shrink to match. Remove attention heads and the projection matrices shrink. Remove whole transformer blocks and depth falls.

The hardware story is the best of any family: nothing special is required. A model with fewer channels is a smaller dense model, and every runtime, compiler and accelerator benefits. The FLOP arithmetic is also better than intuition suggests. For a convolution with C_in input channels, C_out output channels, kernel size k and output map H x W, cost is proportional to C_in x C_out x k^2 x H x W. Pruning half the channels of two adjacent layers cuts both factors, so the cost of the second layer falls to 25 percent, not 50.

The price is accuracy. Structured pruning is a much harsher constraint than unstructured pruning, because you cannot keep an important weight inside an unimportant filter. At equal parameter count, structured models are usually less accurate before recovery training. That is why the successful LLM results lean on distillation or continued training afterwards.

Published examples show how much recovery is needed. Sheared-LLaMA pruned LLaMA2-7B down to 1.3B and 2.7B parameter models and reports needing only 3 percent of the compute of training such models from scratch. SliceGPT, which deletes rows and columns of weight matrices after an orthogonal transformation, reports removing up to 25 percent of parameters (including embeddings) while retaining about 99 percent of dense zero-shot performance for LLAMA2-70B and OPT 66B, and about 90 percent for Phi-2. It reports LLAMA2-70B inference compute dropping to 64 percent of dense on 24 GB consumer GPUs and 66 percent on 40 GB A100s.

NVIDIA’s Minitron work combines depth, width, attention and MLP pruning with knowledge distillation. Its abstract reports compressing the Nemotron-4 family by 2x to 4x, needing up to 40x fewer training tokens per derived model than training from scratch, and about 1.8x lower compute cost for the full 15B, 8B and 4B family. Those are paper claims about a specific family, not guarantees for your model.

Semi-structured 2:4 sparsity: the hardware-aligned middle

The 2:4 pattern is a compromise engineered into silicon. Within every group of four consecutive weights along the reduction dimension, exactly two (at most two nonzero) survive, so sparsity is fixed at 50 percent. The sparse tensor core receives only the two kept values plus a small index describing their positions, and uses the index to select the matching activations. It performs half the multiply-accumulates for the same logical matrix product.

Why 2:4 and not 1:4 or 4:8? Two of four makes the metadata cheap (each kept value needs a 2-bit position within its group, as NVIDIA documents it) and gives the pruning step enough freedom that accuracy loss is modest. The arithmetic is below, and the diagram shows the dataflow.

Figure 2: How a sparse tensor core consumes a compressed 2:4 weight tile: values and metadata in, activation selection, half the multiplies.

Storage for the same 7B FP16 model under 2:4: 3.5 billion kept values at 2 bytes is 7.0 GB, plus metadata at 2 bits per kept value, which is 3.5e9 x 2 / 8 = 0.875 GB, for 7.875 GB total. That is about 56 percent of the 14 GB dense size, deterministically, with no dependence on how the zeros fall. Compare that with the 21 GB CSR case at the same 50 percent sparsity.

What Actually Gets Faster: The Hardware Reality

The short answer: a 2:4 sparse model can approach 2x on the matrix-multiply math, but end-to-end gains reported in practice are far smaller, typically tens of percent, because other operations, memory traffic and kernel overheads do not shrink. Treat the 2x figure as a ceiling and measure your own workload.

Peak versus measured numbers

NVIDIA’s own paper says sparse tensor cores deliver up to twice the peak math throughput of dense units. The measured numbers published by the PyTorch team on A100 hardware are more modest, and instructive:

Measurement (A100)	Reported result
nn.Linear forward, dense vs 2:4 sparse (PyTorch tutorial example)	0.870 ms vs 0.630 ms, 1.382x
BERT on SQuAD, batch size 16, eager	1.23x
BERT on SQuAD, batch size 4, eager	0.71x (slower)
BERT, best case with torch.compile	up to about 2x total speedup per the tutorial text
ViT-L MLP linear layers, forward + backward	about 1.3x
DINOv2 ViT-L training, wall time	about 6% faster
Segment-Anything inference, dense matmuls replaced	about 10% end-to-end
Kernel timing, ViT-L shapes (13008x4096x1024)	dense GEMM 538 microseconds, sparse GEMM 387 microseconds

Three patterns stand out. First, the pure GEMM speedup in the last row is about 1.39x, well below 2x, because sparse kernels have less arithmetic intensity headroom and extra index handling. Second, small batches can lose: the BERT batch-4 eager result is 0.71x, slower than dense. Third, whole-model speedups shrink further because layer norms, softmax, attention score computation and elementwise operations remain dense.

Accuracy in the same BERT SQuAD experiment moved from 78.53 to 78.44 exact match and 86.93 to 86.49 F1 after 2:4 pruning with fine-tuning, so the recovery step worked. (The tutorial’s prose and table differ slightly in the last decimals; the table values are used here.)

The PyTorch authors also note that the compression step itself can be a bottleneck in training: cuSPARSELt compression took roughly 380 microseconds against a 151 microsecond budget, so they wrote a custom prune-and-compress kernel taking 36 microseconds. This matters if you plan sparse training rather than sparse inference.

Why decoding is where sparsity is most tempting

LLM token generation at small batch size is memory-bandwidth-bound, because each token must read all the weights once. A back-of-envelope (illustrative; real kernels never hit peak bandwidth): a 7B FP16 model reads 14 GB per token. At roughly 2 TB/s of HBM bandwidth that is 7 ms per token, or about 143 tokens per second. Reading 7.875 GB under 2:4 gives an ideal 3.9 ms, a 1.78x upper bound.

The catch is that this requires a kernel that reads compressed weights and feeds sparse tensor cores efficiently at batch size one, and that the rest of the system (KV-cache reads, attention) is unchanged. The same tutorial warns about small batches. If your serving stack already uses INT4 weights, the gap closes, as the next section shows.

Where the sparse path is supported

The PyTorch tutorial requires PyTorch 2.1 or later, an NVIDIA GPU with compute capability 8.0 or higher, and FP16 weights for its example path, and it was tested on an A100 80GB. It explicitly warns that newer architectures may not show similar speedups. I did not verify per-architecture support and performance for Hopper or Blackwell parts, or the status of 2:4 in vLLM and TensorRT-LLM this run, so check vendor release notes before committing.

One-Shot Pruning for LLMs: Magnitude, SparseGPT and Wanda

The short answer: LLMs are pruned after training, in one shot, using a small calibration set, because retraining is too expensive. The methods differ in how they score weights: magnitude uses the weight alone, Wanda multiplies by input activation norm, and SparseGPT solves a layer-wise reconstruction problem and updates the remaining weights.

Figure 3: A one-shot LLM pruning pipeline: calibration activations feed a per-layer score, a mask is chosen, and optional recovery follows.

Layer-wise formulation

All three methods share a framing. For each linear layer with weight matrix W and calibration inputs X, find a sparse W-hat that minimises the layer’s output error, the squared norm of (W X minus W-hat X). Doing this layer by layer avoids end-to-end backpropagation, so memory stays within one layer’s footprint and a single GPU can prune very large models.

Plain magnitude pruning ignores X. It works acceptably on small vision networks but degrades badly on LLMs, because transformer activations contain a few input features with very large magnitude (outlier features). A small weight attached to a huge activation matters more than a large weight attached to a near-zero activation.

Wanda: weights and activations

Wanda scores each weight by the product of its absolute value and the L2 norm of the corresponding input feature across the calibration tokens. Per the paper, it requires no retraining and no weight update, and compares weights within each output row. The method is motivated by the observed large-magnitude features in LLMs. It was evaluated on LLaMA and LLaMA-2, clearly outperforming magnitude pruning and competitive with methods that perform intensive weight updates.

A small worked example makes the difference concrete. Take one output row with four weights and the L2 norms of their input activations over a calibration set:

Index	0	1	2	3
Weight	0.8	-0.5	0.3	0.1
Input norm	0.2	4.0	1.0	0.5
Magnitude score	0.8	0.5	0.3	0.1
Wanda score	0.16	2.00	0.30	0.05

For a 2:4 group, magnitude keeps indices 0 and 1. Wanda keeps indices 1 and 2. The largest weight, 0.8, attaches to a feature that is nearly silent (norm 0.2), so its contribution to the output is small; Wanda drops it and keeps the 0.3 weight on a more active feature. The numbers are invented for illustration, but the mechanism is exactly the one the method exploits. Because Wanda only needs a forward pass for the activation norms and a sort, it is cheap.

SparseGPT: reconstruction with weight updates

SparseGPT frames pruning as sparse regression per layer. It processes columns in blocks, chooses which weights to zero using approximate second-order information derived from the calibration Hessian (the matrix X X-transposed), and then adjusts the remaining weights in the row to compensate for the removed ones. That compensation is why it holds up better at higher sparsity than pure scoring methods.

The paper’s headline results, from its abstract: pruning OPT-175B and BLOOM-176B to at least 50 percent in one shot, under 4.5 hours each, 60 percent unstructured sparsity with negligible perplexity increase, and support for the semi-structured 2:4 and 4:8 patterns. Over 100 billion weights can be skipped at inference time in those models.

Which one to pick

Method	Needs calibration data	Updates remaining weights	Cost	Typical use
Magnitude	No	No	Seconds	Baseline, vision CNNs
Wanda	Yes, small	No	Minutes for a 7B-class model (my estimate, unverified)	Fast 50% or 2:4 pruning
SparseGPT	Yes	Yes	Hours at 175B scale per its paper	Higher sparsity, best one-shot quality

All three can emit 2:4 masks by selecting the top two scores in each group of four. Quality at 2:4 is noticeably worse than at 50 percent unstructured for the same method, because the constraint removes freedom. Expect a larger perplexity penalty and plan to recover it.

Recovery fine-tuning

Fine-tuning after pruning restores accuracy with the mask frozen. NVIDIA’s recipe for 2:4 on Ampere is: train dense, prune to 2:4, retrain with the same optimizer and hyperparameters while keeping the mask fixed. For tasks sensitive to weight changes, a progressive schedule helps: compute a 25 percent mask, fine-tune, then the full 50 percent mask and fine-tune again. NVIDIA reports progressive sparsity reaching higher accuracy than one-shot with the same fine-tuning epochs in its experiments.

For LLMs, full fine-tuning is expensive, so teams use low-rank adapters on top of the frozen sparse weights, or distillation from the dense teacher. Both keep the zeros in place because adapters add a separate low-rank path.

Pruning and Quantization Together

The short answer: pruning and quantization are complementary, but the gains do not multiply cleanly, because lower bit widths make sparse-format metadata a larger share of the footprint and because fewer weights means each remaining weight carries more information and is more sensitive to rounding error.

Deep Compression applied both and reported the combined 35x to 49x results above, but those were with Huffman coding and unstructured sparsity on convolutional networks, where the storage format could be tailored. For deployment on GPUs with sparse tensor cores, the format is fixed, so do the arithmetic.

For the same 7B model (illustrative, assuming 2 bits of metadata per kept value regardless of precision, which I have not verified for every datatype):

Configuration	Weight bytes	Share of dense FP16
Dense FP16	14.0 GB	100%
2:4 FP16	7.875 GB	56%
Dense INT8	7.0 GB	50%
2:4 INT8	3.5 GB values + 0.875 GB metadata = 4.375 GB	31%
Dense INT4	3.5 GB	25%
2:4 INT4	1.75 GB values + 0.875 GB metadata = 2.625 GB	19%

Read the last two rows carefully. Going from dense INT4 to 2:4 INT4 saves 25 percent of bytes (3.5 to 2.625 GB), where 2:4 FP16 saves 44 percent over dense FP16. The metadata is a fixed cost, so the benefit of sparsity shrinks as values get smaller. Whether that 25 percent is worth the accuracy cost and the engineering is a real question, especially because INT4 weight-only quantization alone is often available with mature kernels.

Figure 4: Pruning and quantization compose, but metadata overhead and recovery cost erode the combined gain at low bit widths.

NVIDIA’s blog on structured sparsity gives a concrete case where both pay off: a “sparse-QAT” recipe that combines progressive sparsity, quantization (post-training then quantization-aware training to INT8) and knowledge distillation. In its search-relevance cases, a sparse INT8 model matched FP32 accuracy closely (0.8010 versus 0.8015 in one case; 0.7814 versus 0.7839 in another) with a 1.4x speedup over dense INT8, and Tencent reported 1.3x to 1.8x acceleration in offline services. These are one vendor’s reported deployments on encoder-style ranking models, not LLM decoding.

Order of operations

The sensible order is prune first, recover, then quantize with quantization-aware training or careful calibration, then verify. Quantizing first and pruning after is possible, and some one-shot tools do both in a single pass, but errors from each step compound, so evaluate the final artifact rather than each stage in isolation. The vector index quantization post on HNSW, DiskANN and IVF-PQ discusses a similar trade-off in a different domain: aggressive compression of stored vectors saves memory but needs a re-ranking step to recover quality.

Hands-On: PyTorch Pruning Code

The short answer: PyTorch gives you three levels: torch.nn.utils.prune for masks on a single layer, the torch.ao sparsifier for scheduled pruning with a mask squash step, and torch.sparse for converting a 2:4 pattern to a hardware-accelerated tensor. The code below runs on a CPU for the masking parts and needs an Ampere-or-newer GPU for the last part.

Magnitude pruning with a mask

import torch
import torch.nn as nn
import torch.nn.utils.prune as prune

layer = nn.Linear(1024, 4096)

# Unstructured: zero the 50% smallest-magnitude weights (adds weight_mask buffer)
prune.l1_unstructured(layer, name="weight", amount=0.5)
print("sparsity:", (layer.weight == 0).float().mean().item())

# Structured: remove 25% of output rows by L2 norm (still a mask, not a smaller tensor!)
prune.ln_structured(layer, name="weight", amount=0.25, n=2, dim=0)

# Make permanent: removes the reparametrisation, leaves zeros in the dense tensor
prune.remove(layer, "weight")

An important gotcha is in the comment: torch.nn.utils.prune produces masks over a dense tensor. A “structured” mask zeroes rows but does not shrink the matrix. To get the real speedup of structured pruning you must physically rebuild smaller layers, slicing weights, biases and the next layer’s input dimension consistently. Libraries such as torch-pruning automate the dependency graph; I did not verify their current status this run.

Building a 2:4 mask by hand

def mask_2_4(weight: torch.Tensor) -> torch.Tensor:
    """Keep the 2 largest-magnitude weights in each group of 4 along the last dim."""
    out, inp = weight.shape
    assert inp % 4 == 0
    groups = weight.abs().reshape(out, inp // 4, 4)
    topk = groups.topk(2, dim=-1).indices
    mask = torch.zeros_like(groups, dtype=torch.bool)
    mask.scatter_(-1, topk, True)
    return mask.reshape(out, inp)

w = torch.randn(4096, 1024)
m = mask_2_4(w)
assert m.reshape(4096, -1, 4).sum(-1).eq(2).all()
print("sparsity:", 1 - m.float().mean().item())   # 0.5

Replace weight.abs() with weight.abs() * act_norm (a per-column calibration statistic broadcast over rows) and you have a Wanda-style 2:4 scorer from the table above.

Scheduled pruning with torch.ao and conversion to a sparse tensor

The PyTorch tutorial uses WeightNormSparsifier with sparse_block_shape=(1, 4) and zeros_per_block=2 to target 2:4, with the workflow: prepare inserts mask parametrisations, step updates masks by weight norm, you fine-tune, and squash_mask fuses the mask into the weights.

import torch
from torch.ao.pruning import WeightNormSparsifier
from torch.sparse import to_sparse_semi_structured, SparseSemiStructuredTensor

model = build_model().half().cuda().eval()   # your fp16 model

sparse_config = [
    {"tensor_fqn": f"{fqn}.weight"}
    for fqn, m in model.named_modules()
    if isinstance(m, torch.nn.Linear) and "layer" in fqn   # skip task heads
]
sparsifier = WeightNormSparsifier(
    sparsity_level=1.0, sparse_block_shape=(1, 4), zeros_per_block=2
)
sparsifier.prepare(model, sparse_config)
sparsifier.step()              # compute 2:4 masks from weight magnitudes
# ... fine-tune here with the mask frozen ...
sparsifier.squash_mask()       # weights now contain real zeros in 2:4 layout

for fqn, m in model.named_modules():
    if isinstance(m, torch.nn.Linear) and "layer" in fqn:
        m.weight = torch.nn.Parameter(to_sparse_semi_structured(m.weight))

Requirements from the tutorial: PyTorch 2.1 or later, compute capability 8.0 or higher, FP16 weights, and layer shapes that satisfy the library’s constraints, which is why task-specific output heads are excluded. Always benchmark against the dense model at your real batch size, and try torch.compile, which in the tutorial’s BERT table changed the batch-4 result from a slowdown to roughly parity.

Edge Deployment: Where Pruning Helps and Where It Does Not

The short answer: on microcontrollers and most edge NPUs, structured pruning helps and unstructured pruning usually does not, because the runtimes execute dense kernels and sparse formats add code size and indexing overhead. Quantization to INT8 is the first lever, and structured pruning plus distillation is the second.

Consider a Cortex-M class device running a CMSIS-NN INT8 kernel. The kernels assume dense contiguous weights and use SIMD multiply-accumulate instructions over packed bytes. Zeroed weights still occupy flash and still get multiplied. A CSR layout would make each nonzero cost roughly the value byte plus an index byte, so break-even sits near 50 percent density before counting the extra instructions per nonzero, which defeats SIMD. Unstructured pruning therefore needs 80 to 90 percent sparsity to pay off in flash and almost never gives a latency gain.

Structured pruning avoids all of this. Removing 40 percent of channels in a keyword-spotting or vision model gives a smaller dense network with fewer MACs and a smaller activation arena, and the MCU’s runtime does not need to know it was ever pruned. The activation memory (the tensor arena) is often the binding constraint on a microcontroller, and channel pruning reduces it directly.

For federated and on-device training scenarios, sparsity can also cut communication: sending only the nonzero update entries lowers uplink cost. The trade-offs, including how masks interact with aggregation, are discussed in the post on federated learning for IoT with FedAvg and FedProx. Note that the privacy and convergence effects of sparsified updates are a separate question from inference speed.

A practical edge workflow is: start from a small architecture already designed for the target (do not prune a model that is the wrong shape), apply structured channel pruning with fine-tuning, quantize to INT8 with quantization-aware training, then measure latency and peak RAM on the actual device. Many teams find the architecture search or a smaller baseline beats pruning a larger one, and I treat that as practitioner opinion rather than a measured claim.

Trade-offs, Gotchas, and What Goes Wrong

Pruning fails in predictable ways. Knowing them in advance saves weeks.

Sparsity percentage is a vanity metric. A model reported as 90 percent sparse tells you nothing about speed without the pattern and the target hardware. Always ask what runtime executes it and whether the zeros are skipped.

Perplexity hides capability loss. One-shot LLM pruning papers lean on perplexity and zero-shot benchmarks, which can look healthy while reasoning, long-context retrieval, code generation or multilingual ability degrade. SparseGPT’s abstract speaks of negligible perplexity increase, not negligible loss on every downstream task. Evaluate on your tasks, including the long tail. A standardised view is the compression benchmark LLMCBench (arXiv 2410.21352), whose abstract notes that existing studies test on few models, datasets and metrics; I did not review its detailed sparse-versus-quantization results.

Calibration data matters. Wanda and SparseGPT use activation statistics. If calibration text is unrepresentative (English web text when production is code or another language), the mask is tuned to the wrong features.

Small batches and small matrices. The BERT batch-4 result of 0.71x shows sparse kernels can lose. Matrix shapes must meet alignment constraints, and layers too small to fill the sparse tensor core gain nothing. Embeddings, output heads, layer norms and attention softmax are typically left dense, capping end-to-end gains.

Compression overhead in training. If you prune during training, mask updates and compression can eat the savings, as the 380 microsecond versus 151 microsecond budget shows.

Dependency breakage in structured pruning. Residual connections force the channels of the layers they join to match, grouped and depthwise convolutions impose their own constraints, and attention head counts must stay compatible with kernels. Removing a channel in one place without updating all dependents breaks the graph silently or loudly.

Safety and alignment drift. Pruned and quantized models can behave differently on refusals and edge cases than the dense model. Re-run safety evaluations. This is a standard operational recommendation rather than a result I measured.

Reproducibility. Different pruning libraries seed, tie-break and order calibration differently. Pin the version, the calibration set and the random seed in your model card.

Practical Recommendations

Start from the constraint you are actually fighting, not from the technique.

If the problem is file size or download time, quantize first, then consider unstructured pruning with a compressed format plus entropy coding. Runtime speed is irrelevant here.

If the problem is GPU latency or cost for a transformer on Ampere or newer, test 2:4 sparsity only after you have INT8 or INT4 weight-only quantization in place, and compute the bandwidth arithmetic above for your actual batch size. If the ideal gain is under about 20 percent, skip it.

If the problem is fitting a model on a microcontroller or NPU, use structured pruning and a smaller architecture, then INT8. Skip unstructured sparsity unless your toolchain advertises a sparse kernel.

If you need a model that is meaningfully smaller and still strong, consider distillation or width and depth pruning followed by continued training, as in Sheared-LLaMA and Minitron, rather than a mask on the original. The cost is training compute, but you get a dense model every framework can serve.

A checklist to run before you commit:

Baseline the dense model at the real batch size, sequence length and precision on the real hardware.
Compute ideal bytes read and FLOPs for the pruned variant, including metadata.
Pick the pattern the runtime can actually exploit: structured, or 2:4 on supported GPUs.
Use a calibration set that matches production traffic.
Prune in steps (progressive) and fine-tune with the mask frozen.
Quantize after pruning and re-evaluate the final artifact on task metrics, not only perplexity.
Re-run latency tests with and without torch.compile or your compiler of choice.
Record versions, seeds and calibration data in the model card.

Frequently Asked Questions

What is neural network pruning?

Neural network pruning removes parameters that contribute little to a model’s output so it needs less memory and, if the hardware can exploit the zeros, less compute. Pruning can be unstructured (single weights), structured (whole channels, heads or layers) or semi-structured (patterns such as 2:4). Most pruned models need fine-tuning or distillation to recover accuracy, though one-shot LLM methods such as SparseGPT and Wanda avoid retraining.

Does pruning make a model faster?

Only if the runtime or hardware skips the removed work. Structured pruning yields a smaller dense network that speeds up everywhere. 2:4 sparsity uses sparse tensor cores on supported NVIDIA GPUs, with reported end-to-end gains in the tens of percent and kernel gains around 1.4x on A100, below the 2x peak. Unstructured sparsity rarely speeds up GPU inference at moderate sparsity levels.

What is 2:4 sparsity?

It is a fixed pattern in which at most two of every four consecutive weights are nonzero, giving 50 percent sparsity. The hardware stores the two kept values plus a small position index and performs half the multiplications. NVIDIA introduced it with Ampere sparse tensor cores, and describes up to twice the peak math throughput of dense units, with a train-prune-retrain recipe to keep accuracy.

How do SparseGPT and Wanda differ?

SparseGPT solves a layer-wise reconstruction problem with approximate second-order information and updates the surviving weights to compensate for the removed ones, which helps at higher sparsity but costs more compute. Wanda scores each weight by magnitude times input activation norm, needs no weight update, and is much cheaper. Both use a small calibration set and run without retraining.

Should I prune or quantize first?

Quantize first for the simplest, best-supported gain: weight-only INT8 or INT4 cuts memory 2x to 4x with mature kernels. Add pruning when you need more, and prune before the final quantization step so you can recover accuracy. Remember that sparsity metadata is a fixed cost, so the extra saving from 2:4 shrinks as bit width drops.

When should I use distillation instead of pruning?

Use distillation or prune-then-distill when you need a smaller, dense, portable model and can afford training compute. Sheared-LLaMA reports needing about 3 percent of from-scratch compute, and Minitron reports up to 40x fewer training tokens per derived model. Pure masking is better when you want to preserve the exact model and your hardware has a sparse path.

Neural Network Pruning for LLMs and Edge Models: Unstructured vs Structured vs 2:4 Sparsity (2026)