Mechanistic Interpretability: Sparse Autoencoders, Features and Circuits in LLMs (2026)

A large language model is a function you can run, copy, differentiate and probe at will, and yet nobody can say, in plain terms, why it produced one token instead of another. Every weight is visible and every activation can be logged. The difficulty is that the numbers do not come labelled. A single neuron in a trained transformer will fire for academic citations, for the word “the” in some contexts, and for a certain flavour of Korean text, all at once.

Mechanistic interpretability is the research program that tries to reverse engineer those networks into human-readable parts: features (directions in activation space that mean something) and circuits (the weights that connect features into algorithms). Between 2022 and 2025 it moved from toy models to production-scale language models, driven by sparse autoencoders and, more recently, attribution graphs built on transcoders.

What this covers: the residual stream view of a transformer, why superposition makes neurons polysemantic, how sparse autoencoders (SAEs) extract features, the loss functions and architectural variants, causal methods such as activation patching, induction heads and circuit tracing, steering, a small PyTorch SAE you can run, the open tooling, and an honest account of what the field cannot yet guarantee.

Context and Background

Interpretability has always had two camps. Post-hoc attribution methods such as saliency maps and SHAP-style scores ask which input tokens influenced an output. They are cheap and model-agnostic, but they say nothing about the computation in between. Probing classifiers train a small model on hidden states to test whether a concept is linearly decodable. A probe can tell you that “the sentence is in French” is recoverable from layer 12, but not whether the model uses that information, or how.

The mechanistic camp makes a stronger claim: that a trained network implements something like a program, and that the program can be recovered from the weights and activations with enough care. The foundational document is the transformer-circuits thread from Anthropic, which introduced the mathematical framing of a transformer as a sum of independent paths through a shared residual stream. Later work from OpenAI, Google DeepMind, EleutherAI-adjacent communities and many academic labs built on it.

Why does this matter now? Three pressures converge. First, models are deployed as agents that call tools, so a hidden objective or a latent jailbreak has consequences beyond text; see our analysis of agentic AI security and prompt injection for the attack surface. Second, evaluation by behaviour alone cannot rule out a model that behaves well only when it believes it is being tested. Third, architecture choices such as mixture-of-experts routing make the “which parameters did what” question harder, because only a fraction of the weights are active for any token.

The scale of the ambition should be stated carefully. As of this writing, interpretability gives partial, expensive, per-prompt explanations of parts of model behaviour. It does not give a complete account of any frontier model, and no published method certifies that a model is safe. Anthropic’s own scaling paper calls its results very preliminary and does not claim to have shown that the features it found are useful for safety. Keep that calibration in mind throughout.

For readers who want the primary literature, the Transformer Circuits thread at transformer-circuits.pub hosts the Anthropic papers discussed below, and the Gemma Scope paper is on arXiv.

The Residual Stream, Features and Superposition

Direct answer: Mechanistic interpretability treats a transformer as a residual stream, a shared vector that every attention head and MLP layer reads from and adds to. Meaning is stored as directions in that vector, called features. Because models represent more features than they have dimensions, features overlap in superposition, which is why individual neurons are polysemantic and why sparse autoencoders are needed to separate them.

Figure 1: The mechanistic interpretability pipeline. Activations are read from the residual stream, decomposed by a sparse autoencoder into features, then tested causally and assembled into circuits.

Figure 1 shows the pipeline used by most current work. A forward pass produces activations at every layer. A dictionary-learning model (the sparse autoencoder) is trained on those activations offline, producing a large set of sparse, nameable features. Researchers then validate features causally by ablating, patching or steering them, and finally trace how features at one layer produce features at the next, which yields circuits.

The residual stream view

In a decoder-only transformer, token embeddings enter a vector of width d_model, for example 4,096 in many mid-sized models. Each layer’s attention block and MLP block reads the vector, computes an update and adds it back. Nothing is overwritten; the final vector is the embedding plus a sum of every component’s output. Layer normalisation rescales but does not break this additive structure in any serious way.

This matters because it gives a clean decomposition. The logit for a token is a linear function of the final residual stream, so the contribution of any earlier component to that logit can be computed by projecting that component’s output through the unembedding matrix (with the final normalisation held fixed). Techniques such as the logit lens and direct logit attribution rely on exactly this property. Attention heads add a second structure: each head can be split into a QK circuit, which decides where to look, and an OV circuit, which decides what to copy once it has looked.

Features as directions

The working hypothesis, often called the linear representation hypothesis, is that a human-meaningful concept corresponds to a direction in activation space, and that the strength of the concept is the activation’s projection on that direction. Evidence comes from word-embedding arithmetic, from probing, and from the steering results discussed later. It is a hypothesis, not a theorem: some concepts appear to be encoded in circular or multi-dimensional subspaces, such as days of the week or position in a sequence, and the field treats “one feature, one direction” as a convenient default rather than a law.

Why neurons are polysemantic: superposition

If features were assigned to individual neurons, interpretability would be a matter of reading off labels. Empirically that is not what happens. In Anthropic’s one-layer study with a 512-neuron MLP, the median neuron scored 0 on human interpretability ratings, while the median learned feature scored 12 on the same rubric.

The explanation proposed in Anthropic’s Toy Models of Superposition paper is that a network can store more features than it has dimensions by giving them nearly orthogonal, rather than exactly orthogonal, directions. The cost is interference: activating one feature slightly activates its neighbours. A ReLU after the projection can filter much of that noise. The trade pays off when features are sparse, because with few features active at once, interference is rare.

The toy model is small enough to state exactly. Inputs x are n features, each zero with probability S (the sparsity) and otherwise uniform on [0, 1], each with an importance weight. The model compresses to m dimensions, h = Wx, and reconstructs x’ = ReLU(WᵀWx + b), trained with importance-weighted squared error. A purely linear version behaves like principal component analysis and keeps only the top-m features. The ReLU version, once inputs are sparse enough, starts packing extra features in.

The geometry is striking. With uniform importance, the paper reports features arranging into antipodal pairs (half a dimension per feature), triangles, tetrahedra and pentagons, with “dimensions per feature” values clustering at fractions such as 1/2, 2/3, 3/4 and 2/5. Each feature is in one of three states, not learned, learned in superposition, or given a dedicated dimension, and the transitions between states are sharp, which the authors describe as phase changes. The authors are explicit that how far this toy picture carries over to real networks is unclear. It is a motivating model, not a measurement of a frontier LLM.

A worked illustration, with illustrative numbers rather than measured ones: suppose a 4,096-wide residual stream must track 50,000 distinct concepts, and any given token activates about 100 of them. Exact orthogonality allows only 4,096 directions. Nearly orthogonal directions are nearly unlimited in high dimensions, so the model can assign all 50,000 and accept small cross-talk. With 100 active out of 50,000, the expected number of active neighbours that interfere with any given feature is small, and downstream nonlinearities threshold it away. That is the economics of superposition: capacity bought with sparsity.

The consequence for tooling is direct. If the true features are sparse and overlapping, then an overcomplete, sparsity-seeking decomposition of the activations should recover them. That is the sparse dictionary learning idea, and it is what sparse autoencoders implement.

Sparse Autoencoders: Dictionary Learning for Activations

A sparse autoencoder maps an activation vector x of width d_model to a much wider latent vector f(x) of width d_sae, forces most entries of f(x) to be zero, and reconstructs x as a sum of a few decoder columns. Each decoder column is a candidate feature direction; each latent’s activation is how strongly that feature is present.

The basic architecture and loss

The architecture in Anthropic’s Towards Monosemanticity paper is deliberately plain: subtract a pre-encoder bias, apply a linear encoder with a bias and a ReLU, then a linear decoder. The loss is mean squared reconstruction error plus an L1 penalty on the latent activations, optimised with Adam. In the follow-up Scaling Monosemanticity paper on Claude 3 Sonnet, the penalty is weighted by the norm of each decoder column, so the loss reads L = E[‖x − x̂‖² + λ Σᵢ fᵢ(x)·‖W_dec,i‖], with λ = 5 under that paper’s activation normalisation. The decoder-norm weighting matters: without it, the model can cheat the L1 penalty by shrinking activations and inflating decoder norms.

The first paper trained on 8 billion activation samples from a one-layer transformer with a 512-neuron MLP, trained on The Pile. Dictionaries ranged from 512 features (1x) to 131,072 (256x), across 90 learned dictionaries, and the authors analysed the 4,096-feature (8x) run most closely. At 4,096 features, replacing the MLP with the SAE output retained about 79% of the loss reduction the MLP provides, and the 131,072-feature dictionary retained 94.5%. The authors caution that such fractions can mislead because a long tail of features likely remains.

Figure 2: Sparse autoencoder training. Activations are encoded into a wide sparse code, decoded back, and the loss trades reconstruction against sparsity, with a separate path to revive dead latents.

Figure 2 sketches the training loop, including the maintenance step that every serious implementation needs. Latents that stop firing, known as dead latents, are a persistent failure. Anthropic’s one-layer work periodically resampled dead dictionary neurons, resetting their encoder weights toward inputs the SAE currently reconstructs poorly. At scale the problem is large: in the Claude 3 Sonnet study, roughly 2% of features were dead in the 1,048,576-feature SAE, about 35% in the 4,194,304-feature SAE, and about 65% in the 33,554,432-feature SAE. Those SAEs averaged fewer than 300 active features per token and explained at least 65% of activation variance.

Variants: TopK, JumpReLU and gating

The L1 penalty has a known flaw called shrinkage: it biases every active latent toward zero, so reconstructions are systematically too small. Newer variants attack sparsity directly.

OpenAI’s TopK autoencoder keeps only the k largest latents per token and zeros the rest. Fixing the number of active latents removes the sparsity coefficient from the tuning problem and improves the reconstruction-versus-sparsity frontier. In that paper, the authors trained a 16 million latent autoencoder on GPT-4 activations over 40 billion tokens and reported clean scaling laws in autoencoder size and sparsity, along with modifications that keep dead latents low even at large sizes.

Google DeepMind’s JumpReLU SAE replaces the ReLU with a thresholded activation: a latent is zero below a learned threshold and passes its pre-activation unchanged above it. This sidesteps shrinkage on the active latents and uses an L0-style penalty trained with straight-through gradient estimates. The Gemma Scope release uses JumpReLU SAEs across Gemma 2 2B and 9B on all layers and sub-layers, plus selected layers of Gemma 2 27B, with weights on Hugging Face (google/gemma-scope) and a demo on Neuronpedia.

Variant	Sparsity mechanism	Main strength	Main weakness
ReLU + L1	Penalty on activation magnitude	Simple, well understood	Shrinkage, sensitive coefficient
TopK	Hard cap of k active latents	Direct control of L0, fewer dead latents	Fixed k for every token regardless of difficulty
JumpReLU	Learned threshold per latent	Little shrinkage, strong frontier	Straight-through training is finicky
Gated SAE	Separate gate and magnitude paths	Reduces shrinkage	Extra parameters

Gated SAEs, a DeepMind proposal, appear in the table for completeness; the tuning guidance here is general rather than a verified benchmark claim.

What SAEs have actually found

The one-layer study documented features such as an Arabic-script feature (Pearson correlation 0.74 with a simple proxy), a DNA feature (0.8) and a base64 feature (0.38, which the authors attribute mainly to a broad proxy). Ablating the Arabic feature lowered the probability of Arabic tokens, and pinning it high made the model generate Arabic. Features also showed universality: counterparts in a second model trained with a different seed correlated at 0.85 to 0.92.

The scaling study found features that were multilingual and multimodal, and showed that larger SAEs learn rarer concepts. The authors report that loss follows approximately a power law in compute, that compute-optimal feature counts and training steps both follow power laws, and that a concept’s representation tracks its frequency in training data, possibly echoing Zipf’s law. They also note incompleteness: only about 60% of London boroughs had matching features in the 34M SAE.

From Features to Causes: Patching, Induction Heads and Circuits

Finding a feature that fires on a concept is a correlation. Mechanistic interpretability earns its name when the feature is shown to be causally involved in the output. The core instrument is intervention on activations.

Activation patching and causal tracing

Direct answer: Activation patching runs the model on two inputs, a clean one and a corrupted one, then copies an activation from one run into the other at a chosen layer and position. If copying that single activation restores the clean behaviour, that component carries the information the behaviour needs. It converts “this activation correlates with the answer” into “this activation is causally sufficient for the answer”.

Figure 3: Activation patching. The clean activation is cached, spliced into the corrupted run at one site, and the change in the output logit difference measures that site’s causal effect.

Figure 3 shows the procedure. The best-known application is causal tracing from the ROME work on factual recall, which corrupted the subject tokens of a prompt such as “The Eiffel Tower is in” with noise, then restored individual hidden states from the clean run to find where the fact was stored. It localised factual recall to mid-layer MLPs at the last subject token. The same recipe, applied with minimal prompt pairs rather than noise, is the standard tool for circuit discovery in small models.

A metric is needed to score each patch. A common choice is the logit difference between the correct and incorrect answer, normalised so that 0 is the corrupted baseline and 1 is the clean run. Patching one component and reading a normalised value of 0.8 means that site alone recovers 80% of the clean-versus-corrupt gap. Illustrative numbers like these are easy to misread: a high score for a single site shows sufficiency in that setting, while a low score may only mean the information is spread across redundant paths. The practitioner guide on interpreting patching results makes the point that denoising (clean into corrupt) and noising (corrupt into clean) test different properties, sufficiency versus necessity, and give different answers when circuits have backup components.

Path patching refines this by patching only the effect that flows along a specific edge, such as head A’s output into head B’s query, rather than the node’s whole output. Attribution patching approximates all single-site patches at once with a gradient, trading accuracy for a speed-up of roughly the number of sites. It is a first-pass filter, not a proof.

Induction heads: the first fully worked circuit

Induction heads are the textbook example of a circuit. Given a sequence containing … A B … A, an induction head attends from the second A to the token after the first A, namely B, and copies it forward so the model predicts B. The mechanism needs two heads in different layers. A previous-token head in an earlier layer writes “the token before me was A” into the residual stream at position B. An induction head in a later layer uses that signal in its key composition: its query at the second A matches keys at positions whose previous token was A, and its OV circuit then promotes B’s identity in the logits.

Anthropic’s In-context Learning and Induction Heads paper presented evidence that induction heads form at a sharp point during training, coinciding with a visible bump in the loss curve and a jump in in-context learning ability. The evidence for large models is correlational, and the paper itself frames it that way; for small attention-only models, the causal story is much stronger. Induction heads matter beyond pedagogy: they are a working demonstration that a discrete algorithm can be read from the weights, and that composition of heads across layers is how transformers build computations deeper than a single attention step.

Other circuits followed. The indirect object identification circuit in GPT-2 small involves a few dozen attention heads in groups with names such as duplicate-token heads, S-inhibition heads and name movers. It was recovered by patching and path patching, and it also demonstrated the weakness of the approach: manual circuit discovery is laborious, task-specific, and dependent on the researcher’s choice of prompt distribution.

Steering: using features as controls

If a feature direction is causal, writing to it should change behaviour. In the Claude 3 Sonnet study, a feature responding to the Golden Gate Bridge was clamped to ten times its maximum observed activation, after which the model began to identify itself as the bridge. This is the steering experiment in its most memorable form. Steering with SAE features is the strongest evidence the authors offered that features are causal handles and not just correlates, although they note that downstream effects can diverge from what activation patterns alone predict.

Simple vector steering predates SAEs. Activation addition methods compute a direction as the mean difference between activations on contrasting prompts and add a scaled copy during generation. The practical difficulty is the dose: too small does nothing, too large degrades fluency, and the useful window varies by layer and by concept.

Attribution Graphs and Circuit Tracing

Patching answers questions one site at a time. By 2025, the frontier shifted to building whole computation graphs for a single prompt, using replacement models whose components are interpretable by construction.

Transcoders and cross-layer transcoders

A transcoder is like an SAE, but instead of reconstructing a layer’s activation from itself, it learns to predict an MLP’s output from the MLP’s input through a sparse bottleneck. Because it approximates the computation and not just the representation, a transcoder’s features connect to each other through (approximately) linear paths, which makes edges between features computable.

Anthropic’s circuit tracing work extends this to cross-layer transcoders (CLTs). Each CLT feature reads from the residual stream at its own layer and writes to the MLP outputs of its own layer and every later layer. All layers’ features are trained jointly to reconstruct the MLP outputs under a sparsity penalty. Writing across layers lets one feature stand in for a quantity that the original model re-computes redundantly at several depths, which shortens the graphs.

Figure 4: Circuit tracing. A cross-layer transcoder replaces the MLPs, a local replacement model freezes attention and adds error nodes, and the attribution graph is pruned into an interpretable subgraph.

The local replacement model and the graph

Substituting CLT features for MLPs yields a replacement model, but an imperfect one. In the published methods paper, the largest 18-layer CLT reproduced the original model’s most likely next token on only about 50% of test prompts. To make it exact for one prompt, the authors build a local replacement model: it keeps the original model’s attention patterns and normalisation denominators frozen, and adds an error term at every token position and layer so the activations and logits match the original on that prompt.

The attribution graph then has four node types: output logits, active CLT features, prompt token embeddings, and error nodes. Error nodes represent the part of each MLP output the CLT does not explain and have no incoming edges. Edges are direct linear effects, and each feature’s activity is the sum of its incoming edges. Because the local model is linear in the paths between features once attention is frozen, edge weights can be computed exactly, and the graph is pruned to keep the most influential nodes. Researchers then group features into supernodes, label them, and validate the story with perturbation experiments on the real model.

The method was applied to Claude 3.5 Haiku for case studies such as addition, with most of the paper’s main case studies on a smaller 18-layer model. Claims about which circuit solves which problem should be read as findings about specific prompts on specific models, validated by intervention, not as general laws.

What the graphs cannot show

The authors list their limitations plainly. Attention patterns are held fixed, so the graph explains what attention moves but not why attention looks where it does (the QK side). Reconstruction error means some computation hides in the error nodes. Circuits that depend on a feature not firing, or on inhibition, are hard to capture. Graphs can be very complex, feature splitting and absorption mean features are often at the wrong level of abstraction, and global circuits across many contexts remain out of reach. Most fundamentally, it is uncertain whether transcoders use the same mechanisms as the original MLPs or merely correlate with their outputs, a question of mechanistic faithfulness.

Hands-On: Training and Using a Small SAE

The following code trains a TopK sparse autoencoder on residual-stream activations from GPT-2 small using TransformerLens, then runs an activation patch. It is a teaching-scale sketch: GPT-2 small has d_model = 768, and a 16x dictionary gives 12,288 latents. The hyperparameters are illustrative starting points, not tuned values, and you should expect to iterate. Install transformer_lens and torch first.

import torch, torch.nn as nn, torch.nn.functional as F
from transformer_lens import HookedTransformer

device = "cuda" if torch.cuda.is_available() else "cpu"
model = HookedTransformer.from_pretrained("gpt2").to(device)
LAYER, HOOK = 8, "blocks.8.hook_resid_post"

class TopKSAE(nn.Module):
    def __init__(self, d_model=768, expansion=16, k=32):
        super().__init__()
        d_sae = d_model * expansion
        self.k = k
        self.b_pre = nn.Parameter(torch.zeros(d_model))
        self.W_enc = nn.Parameter(torch.randn(d_model, d_sae) * d_model**-0.5)
        self.b_enc = nn.Parameter(torch.zeros(d_sae))
        self.W_dec = nn.Parameter(self.W_enc.data.T.clone())
        self.normalize_decoder()

    @torch.no_grad()
    def normalize_decoder(self):
        self.W_dec.data /= self.W_dec.data.norm(dim=1, keepdim=True)

    def encode(self, x):
        pre = (x - self.b_pre) @ self.W_enc + self.b_enc
        top = torch.topk(pre, self.k, dim=-1)
        f = torch.zeros_like(pre).scatter_(-1, top.indices, F.relu(top.values))
        return f

    def forward(self, x):
        f = self.encode(x)
        return f @ self.W_dec + self.b_pre, f

def get_batch(texts, seq=128):
    toks = model.to_tokens(texts, truncate=True)[:, :seq]
    _, cache = model.run_with_cache(toks, names_filter=HOOK)
    acts = cache[HOOK].reshape(-1, cache[HOOK].shape[-1])
    return acts

sae = TopKSAE().to(device)
opt = torch.optim.Adam(sae.parameters(), lr=4e-4)
fired = torch.zeros(sae.W_enc.shape[1], device=device)

def train_step(acts):
    global fired
    recon, f = sae(acts)
    loss = F.mse_loss(recon, acts)
    opt.zero_grad(); loss.backward(); opt.step()
    sae.normalize_decoder()
    fired = 0.999 * fired + (f > 0).float().mean(0)
    var_explained = 1 - ((recon - acts) ** 2).sum() / ((acts - acts.mean(0)) ** 2).sum()
    return loss.item(), var_explained.item()

# Replace `corpus` with batches of real text, for example from the Pile or OpenWebText.
# for step, texts in enumerate(corpus):
#     loss, ve = train_step(get_batch(texts))
#     if step % 200 == 0:
#         dead = (fired < 1e-5).float().mean().item()
#         print(step, round(loss, 4), round(ve, 3), "dead fraction", round(dead, 3))

Three details deserve attention. The decoder columns are renormalised to unit length after every step, which fixes the scale ambiguity between encoder activations and decoder norms. The fired exponential moving average gives a cheap dead-latent monitor; if the dead fraction climbs, the standard remedies are resampling dead latents toward high-loss inputs or adding an auxiliary loss that asks dead latents to reconstruct the residual error. And variance explained is the first number to log, but it is not enough: two SAEs with equal variance explained can differ greatly in how much downstream loss they cost when spliced back into the model.

That last point is the metric that matters. After training, splice the SAE reconstruction into the forward pass and measure the increase in next-token cross-entropy versus the unmodified model, the “loss recovered” figure used in the papers above.

def spliced_loss(tokens, sae):
    def hook(resid, hook):
        recon, _ = sae(resid)
        return recon
    return model.run_with_hooks(tokens, return_type="loss",
                                fwd_hooks=[(HOOK, hook)])

def activation_patch(clean_toks, corrupt_toks, answer_id, wrong_id, layer, pos):
    _, clean_cache = model.run_with_cache(clean_toks)
    name = f"blocks.{layer}.hook_resid_pre"
    def patch(resid, hook):
        resid[:, pos, :] = clean_cache[name][:, pos, :]
        return resid
    logits = model.run_with_hooks(corrupt_toks, fwd_hooks=[(name, patch)])
    last = logits[0, -1]
    return (last[answer_id] - last[wrong_id]).item()

To interpret a latent, take its top-activating tokens and contexts over a held-out corpus, look at its decoder column’s effect on logits by multiplying with the unembedding matrix, and then run an intervention: clamp it, ablate it, and check the output changes in the direction the label predicts. A latent that looks coherent but whose ablation changes nothing is a warning sign.

Tooling Landscape

You rarely need to build everything. TransformerLens, a library originally by Neel Nanda, loads dozens of open models with named hook points so that caching and patching are one-liners. SAELens, from the same community, standardises SAE training, loading and evaluation, and can pull pre-trained dictionaries. Neuronpedia hosts feature dashboards and, as noted in the Gemma Scope paper, an interactive demo for those SAEs. Gemma Scope itself is the most convenient large public SAE suite for experimentation. Anthropic released open-source tooling for generating attribution graphs on open-weights models alongside its circuit tracing paper; check the project’s repository for the current supported models, as that list changes.

Task	Typical tool	Notes
Cache and patch activations	TransformerLens	Named hooks, many open models
Train or load SAEs	SAELens	Supports multiple SAE architectures
Browse and label features	Neuronpedia	Hosts Gemma Scope dashboards
Attribution graphs	Circuit tracing tooling from Anthropic	Open-weights models only
Pre-trained SAEs	Gemma Scope on Hugging Face	Gemma 2 2B, 9B, selected 27B layers

Versions and APIs in this ecosystem move fast. Pin library versions in your environment and record the exact SAE checkpoint used, because a feature index is meaningless without the dictionary that defines it.

Trade-offs, Gotchas, and What Goes Wrong

Reconstruction error is not a rounding error. Splicing a typical SAE into a model raises its loss measurably, and the residual the SAE cannot explain is not random noise. Researchers have found that the error vectors themselves contain structure the model uses. If your explanation of a behaviour depends on the SAE reconstruction, the unexplained remainder can contain the real mechanism. The attribution-graph work addresses this with explicit error nodes, which makes the gap visible instead of hiding it.

Feature splitting and absorption. As dictionaries grow, a broad feature fragments into narrower ones. The one-layer paper showed a base64 feature splitting into three features at the next dictionary size. There is no canonical granularity, so a statement such as “the model has a feature for X” is relative to a dictionary size. Absorption is the sibling problem: a general feature fails to fire on cases where a more specific feature already covers the token, which breaks naive “this feature is on or off” readings.

Dead and dense latents. The Claude 3 Sonnet SAEs had 2% to 65% dead features depending on size, and dense latents that fire on a large fraction of tokens carry little interpretable content. Both inflate the apparent dictionary size.

Interpretability at low activation. Human-rated interpretability drops sharply at low activation strengths, where most activations fall. A feature that is clean at its top activations may be murky across its full range.

Faithfulness and the illusion of understanding. A neat label is a hypothesis about a feature, and automated labelling by another language model adds a second layer of uncertainty. Always test labels with interventions and with negative examples.

Compute and cost. Training SAEs on frontier-scale activations is expensive: the OpenAI TopK work used 40 billion tokens for a single 16 million latent autoencoder. Attribution graphs are produced per prompt and need manual analysis. This does not scale to auditing every output of a deployed system.

Distribution shift. An SAE trained on pre-training text may reconstruct poorly on code, rare languages or adversarial inputs, and those are often the cases a safety audit cares about most.

What Interpretability Can and Cannot Guarantee for Safety

The hopeful case is detection and intervention. If a model has a feature for deception, for a dangerous capability, or for the sense that it is being evaluated, a monitor could watch that feature at inference time, and steering could suppress it. Features related to code vulnerabilities, bias and sycophancy were reported in the Claude 3 Sonnet study as examples of safety-relevant directions, with the explicit caveat that finding them does not show they are useful for safety.

The limits are structural. Interpretability gives existence proofs, not absence proofs. Showing that a circuit performs a behaviour is feasible; showing that no circuit anywhere in the model performs a harmful behaviour would require a complete decomposition, which does not exist. Incomplete dictionaries (only about 60% of London boroughs had features in the 34M SAE) mean a missing feature is weak evidence of a missing concept. Adversarial settings are harder still: a model optimised against a specific interpretability monitor could, in principle, route around it, and no published method rules this out.

A sensible position is to treat interpretability as one layer in a defence-in-depth stack. It is a good source of hypotheses about failures, a useful debugging tool, and a promising complement to behavioural evaluation and red-teaming. It is not a certificate. For a view of how a deployed agent can be attacked despite an aligned model, see the agentic AI security analysis, and for the infrastructure side of training at scale, the post on ring attention and context parallelism shows how activation tensors for very long contexts are sharded, which is the same activation volume an SAE pipeline has to ingest.

There is also an engineering analogy worth drawing. Teams that ship software behind feature flags with progressive delivery accept that you cannot prove a change safe in advance, so you evaluate it on a small cohort, watch signals, and keep a kill switch. Feature-level monitoring and steering in a model is an analogous control plane: observe, gate, roll back. The analogy is conceptual, and no vendor currently documents such a runtime gate as a standard production practice.

Practical Recommendations

If you are starting out, begin with small open models where ground truth is checkable. Reproduce the induction head result in a two-layer attention-only model, then the indirect object identification circuit in GPT-2 small, before touching SAEs. Intuition for what a clean circuit looks like protects you from over-reading messy ones.

Use pre-trained SAEs such as Gemma Scope before training your own; most of the cost and tuning pain is in dictionary training. When you do train, choose TopK or JumpReLU over plain L1 unless you need simplicity, log dead-latent fraction and loss recovered from the first run, and hold out a test corpus that matches the distribution you care about.

Treat every explanation as a claim requiring an intervention. A feature label needs ablation or steering evidence; a circuit needs patching evidence on held-out prompts; an attribution graph needs perturbation experiments on the real model. Report the error share alongside any graph.

For teams in industry, the realistic near-term value is debugging and evaluation, not certification. Use interpretability to investigate surprising behaviour, to find spurious features a fine-tune has latched onto, and to build monitors you validate empirically against behavioural tests.

Checklist

• [ ] Pin the model, layer, hook point and exact SAE checkpoint for every claim
• [ ] Log variance explained, loss recovered, L0 and dead-latent fraction
• [ ] Evaluate on a held-out distribution that includes code and adversarial text
• [ ] Validate each feature label with an ablation or steering test
• [ ] Use both noising and denoising patches before calling a component necessary or sufficient
• [ ] Report error nodes or reconstruction gap next to any circuit diagram
• [ ] State dictionary size, since feature granularity depends on it
• [ ] Keep behavioural evaluation and red-teaming as independent checks

Frequently Asked Questions

What is mechanistic interpretability in simple terms?

It is the effort to reverse engineer a neural network the way you would disassemble a program. Instead of asking which inputs mattered, researchers try to identify the internal variables, called features, and the connections between them, called circuits, that compute an output. The aim is a causal, checkable account of how the model works, validated by interventions such as ablating or steering a feature, not just a plausible-sounding story about its behaviour.

What is superposition in neural networks?

Superposition is a network representing more features than it has dimensions by assigning them nearly orthogonal directions. It works because real-world features are sparse, so few are active at once and interference stays tolerable, especially with a nonlinearity such as ReLU filtering noise. It explains why single neurons respond to several unrelated concepts. Anthropic’s toy models show it emerging as a sharp phase change when features are sparse enough.

How do sparse autoencoders find features?

A sparse autoencoder is trained to reconstruct a model’s activations through a much wider layer in which only a few units are active. The sparsity pressure, from an L1 penalty, a TopK cap or a JumpReLU threshold, pushes each unit to represent one recurring pattern. Each decoder column then becomes a candidate feature direction, which researchers label by inspecting top-activating examples and testing with interventions.

What is the difference between activation patching and a probe?

A probe is a classifier trained on hidden states, so it shows only that information is decodable. Activation patching intervenes: it copies an activation from one run into another and checks whether the output changes. That tests causal use, not mere presence. A probe can succeed on information the model never uses, while patching can show a component is sufficient to restore a behaviour, so the two answer different questions and are often combined.

Can interpretability prove an AI model is safe?

No. Current methods give partial, per-prompt or per-feature explanations with measurable gaps: reconstruction error, incomplete dictionaries, frozen attention in attribution graphs and unclear faithfulness. They can reveal concerning mechanisms but cannot show that none exist. Anthropic’s own scaling paper describes its results as very preliminary and does not claim to show features are useful for safety. Treat interpretability as one layer of defence, alongside evaluation and red-teaming.

Are sparse autoencoder features the same as circuits?

No. Features are the nodes: directions in activation space that mean something. Circuits are the edges and structure: how features at one layer drive features at later layers through attention and MLP weights. SAEs on a single layer give you features only. Transcoders and cross-layer transcoders add the connections that let you build attribution graphs, which is the step from a vocabulary to an algorithm.

Further Reading

• Agentic AI security and prompt injection in 2026 for the deployment threats interpretability hopes to help detect
• Mixture-of-experts LLM architecture in 2026 for how sparse routing complicates interpretation
• Ring attention and context parallelism for long-context training for activation sharding at scale
• Feature flag architecture with OpenFeature and progressive delivery for the control-plane analogy
• Anthropic, Towards Monosemanticity, Toy Models of Superposition, Scaling Monosemanticity and Circuit Tracing methods
• OpenAI, Scaling and evaluating sparse autoencoders, and Google DeepMind, Gemma Scope

By Riju — about