Module: interp (the interpretability suite)
One window, a rail of tools down its left edge. The context window, the model explorer and designer, the llama.cpp traces / stepper / lens, agentpedia and the window's lens lab used to be six panes that did not know about each other. They are one subject — the same model read at the prompt, the weights and the activations — so they are one pane now.
Status: Phase 11 of the suite plan — frontend in
packages/core/src/modules/interp/.
The rail
| Group | Section | Body |
|---|---|---|
| — | Home | state tiles, a card per tool, research stages outside this window |
| Prompt | Context | interpretability context window |
| Prompt | Runs | agentpedia, with Runs / Harness / Forks / Duels as sub-tabs |
| Weights | Model | the model explorer's Inspect view |
| Weights | Attention | what every block's token mixer is — see below |
| Weights | Design | the model designer |
| Activations | Traces | llama.cpp traces |
| Activations | Stepper | the layer stepper |
| Activations | Lens | the trace lens, Compare lenses (logit lens vs fitted Jacobian lenses — see below), and the WebML lens lab (Reply / Guess / Compare) as sub-tabs |
| Activations | Heads | every attention head of a trace, scored — see below |
| Activations | Neurons | every MLP neuron of the window's GGUF in 3D; dose a few, compare replies, light the stack by a token — see below |
| Features | Features | SAE features of a trace, labelled and checked — see below |
| Features | Steer | push a feature into the model; default and steered side by side — see below |
| Features | Axis | the Assistant Axis built on a local GGUF: monitor drift, cap it — see below |
| Circuits | Graph | attribution graphs from Neuronpedia or circuit-tracer — see below |
| Circuits | NLA | a natural language autoencoder on local GGUFs: an activation in words, scored by reconstructing it — see below |
| — | Library | SAEs, transcoders and lenses; the GPU worker; Neuronpedia lookups — see below |
Sections and Home's cards both come from tools.ts, so the rail and the map cannot
disagree. Each body is registered as an embedded view (interp.<section>), so
show("Stepper") and openPane still reach it by name. Every section sits under a
shared strip naming the model, the model locus and the trace.
Attention
One scalar head count does not describe the models worth looking at. Gemma 4 windows
five of every six layers (four of five on E2B), gives the sixth twice the head width
and a quarter of the KV heads and drops its V projection; Gemma 4 E2B's last twenty
layers keep no KV cache of their own; Qwen3.5 replaces three of every four layers
with Gated DeltaNet; granite-h and LFM2 attend only every few blocks. The Attention
section draws all of that per layer, from GET /api/interpretability/attention
(backend/modules/interpretability/attention_layout.py).
How each layer is known
Every layer carries a basis, and the card shows it:
| Basis | Meaning | Example |
|---|---|---|
metadata | the GGUF states it per layer | gemma4.attention.sliding_window_pattern (an array, true = windowed), a per-layer head_count_kv with zeros, full_attention_interval, shared_kv_layers |
rule | the family's llama.cpp loader implies it | Gemma 3 stores no pattern; set_swa_pattern(6) windows all but the last of every six. Llama 4's 8192 chunk is a loader constant |
tensors | only the tensor directory shows it | ssm_* (Mamba), shortconv (LFM2), attn_qkv + ssm_* (DeltaNet), a missing attn_v (K=V), attn_sinks |
Where the metadata or rule and the tensors disagree, the layout reports a conflict and draws what the tensors show — the file is the physical fact. The K projection's width is cross-checked against KV heads × head width on every attention layer. A family with a stated window but no known rule is drawn global with a note, never guessed into a pattern.
set_swa_pattern's convention — of every n layers all but the last are windowed,
n = 0 windowing all — is shared with the WebGPU engine's layerWindow, and both
test suites read one table (backend/tests/fixtures/swa_patterns.json) so the two
cannot drift apart.
What it draws
- The ribbon: one column per block, coloured by mixer, height by head width (so Gemma 4's 512-wide global layers stand out from its 256-wide local ones), faded where the block reads another block's KV with an arc from that source, dashed where only the family rule says so, and marks for K=V, gated output, sinks and NoPE. A click publishes the model locus, so Model opens that block and Stepper can step it.
- The KV cache at 1k–max context, fp16, as built against what the same layers would cost all-global with their own caches. Each pair is on its own scale — the ratio is the point; the size is printed beneath. On Gemma 4 E2B it is 774 MB against 5.4 GB at 128k. An estimate: K and V at fp16, K=V counted once.
- How far back by each depth: an upper bound on the distance information can have travelled by the end of each layer — a windowed layer extends it by its window, a global or recurrent one reaches the whole context.
- The block card: the layer's facts, its basis, and a small causal mask of its kind.
In the Stepper, a sliding layer's attention map gets its window edge drawn over it — but only when the trace was recorded from the very file the layout was read from. The layout follows the agent's loaded model and a trace can be of any GGUF; one model's window over another's pass would be a confident wrong answer.
Heads
The stepper shows one head's matrix at a time; a 3B model has 672 heads. Heads
answers the question before it — which ones are worth opening — by scoring every head
of a trace's prompt pass (backend/modules/llamacpp/heads.py,
GET /api/llamacpp/traces/{id}/heads):
| Score | What it finds |
|---|---|
| induction | weight on the token that followed an earlier copy of the current token — the head that copies a continuation forward |
| previous token | weight on the position just before — the first half of an induction circuit |
| duplicate | weight on earlier copies of the current token itself |
| sink · self · entropy | the stepper's per-head summary: weight on position 0, on the diagonal, and how spread the head is |
The finder is a layer × head grid shaded by one score, the twelve highest beside it, and the selected head's card (step its layer, or ask the agent what it does).
Induction is measurable only on a prompt that repeats something. "The capital of
France is" repeats nothing, so on it induction and duplicate are null — greyed out,
never drawn as zero, which would read as "this model has no induction heads". Run
induction probe records the prompt that does: a random word sequence twice
(heads.probe_prompt), attention and l_out only, so it costs megabytes. On Llama
3.2 1B it found L10 H23 at 0.89 and L5 H10 at 0.82 against a median of 0.006, with the
strongest previous-token head early (L0 H2, 0.64) — the two halves of the circuit,
in the order they have to run.
Layer kinds (sliding / global …) are tagged on heads only when the attention layout
was read from the trace's own file (sameFile); a trace is of whatever GGUF it was
recorded on, not necessarily the agent's model.
The section's state is a module store (interp/heads/state.ts), so the digest carries
its strongest head and interp.heads ranks a trace's heads for the agent whether or
not the section is open.
Graph (attribution graphs)
An attribution graph explains one prediction as a network: Anthropic's circuit
tracing, as produced by circuit-tracer and shown on Neuronpedia. The bottom row
is the prompt's embeddings, the middle holds transcoder features and error nodes,
and the top is the output logits. Each signed edge says how much one node's
activation pushed another's.
Getting one. Import takes a Neuronpedia graph link
(neuronpedia.org/<model>/graph?slug=…) or <model>/<slug>. It reads the graph's
page from Neuronpedia's public API, then fetches the JSON file it names, capped at
64 MB. Open file takes a local circuit-tracer run, which is the same format.
Each graph is validated before it is kept: it needs metadata with prompt tokens,
nodes with ids, layers, positions and types, and links whose ends exist (more than
1% dangling refuses it). It is stored as $DATA/interp/graphs/<model>--<slug>.json
with a Library row of kind graph (backend/modules/interp/graphs.py).
Reading it (interp/graph/). The graph is drawn on a canvas, because the
featured Dallas → Austin graph alone carries 34,801 edges:
- Layout: token positions run left to right, and only the layers that hold a node get a row.
- Edges: accent for excitatory, danger for inhibitory, with opacity by weight.
Pruning is client-side and says what it does:
- Features are kept by circuit-tracer's cumulative
influence, which is smaller for more important nodes. - Edges are kept when they carry a set share of their target's strongest input, so a node's few real inputs survive in a weak neighbourhood.
Interacting:
- Click a node for its strongest inputs and outputs (each clickable) and its
Neuronpedia label. Labels are fetched by
{layer}-{sourceSet}when the graph came from Neuronpedia. - Ctrl-click pins a node, and Pinned only shows the reader's subgraph.
- Shift-click gathers nodes into a labelled supernode.
Pins and supernodes are written into the graph's own qParams, the way
Neuronpedia stores them, so they persist and travel with the file.
For the agent, interp.graph walks the strongest excitatory paths back from
the target logit to the embeddings, greedily and without reusing an edge. It
labels every step, names each path's weakest link, and can walk only the pinned
subgraph. A path is a summary, not the whole circuit, and the guide tells the
agent to say so.
Local generation is not built. It needs transcoders for a model this machine runs. The candidates are:
- Qwen3-0.6B: the transcoder set is 18.8 GB.
- Llama 3.2 1B: 17.3 GB.
- Gemma 3 1B IT: Gemma Scope 2 transcoders, about 150 MB per layer and roughly
4 GB for all 26, plus the gated Hugging Face model. It is also unconfirmed that
circuit-tracerreads Scope 2's folder format.
Each of those is a download to agree to first. The viewer reads circuit-tracer's output unchanged, so a local run, once possible, opens here like an import.
Fitted lenses (Jacobian / J++)
The logit lens reads a layer's residual through the final norm and the output
head: what the model would say if it stopped there. A Jacobian lens
(Anthropic's J-lens) first applies a fitted map J_ℓ ≈ E[∂h_final/∂h_ℓ], which
transports the residual to the final layer. It reads what the layer holds,
including concepts the model has not yet decided to say. Neuronpedia publishes
fitted lenses for many models (neuronpedia/jacobian-lens). The Library
downloads them (Gemma 3 1B IT's J-lens is 66 MB), and Use as a lens installs
one into the llama.cpp lens slot for a matching GGUF.
Installing. The file is a torch.save dict: J: {layer: fp16[d, d]},
d_model, n_prompts, and for J++ files a provenance. The worker converts it,
since it has torch and the backend does not. It loads with weights_only=True,
so the downloaded pickle cannot run code. The result is one J_<layer>.npy
(fp16) per layer plus a lens.json under
$DATA/llamacpp/lens/<gguf sha>/np-jlens-<dataset>/, the slot lens.py always
read. Install targets are only the GGUFs the lens matches. A lens file states no
width, so an unrelated GGUF (Whisper, a TTS model) is merely not contradicted,
and offering it would be offering a wrong reading.
Which residual "layer k" means was measured, not assumed. The J-lens file has no provenance key, so it does not say whether layer k is block k's output or its input. Both readings were tested against a full-depth trace:
- As block k's output (llama.cpp's
l_out-k): slightly better next-token agreement (0.149 vs 0.132), and the right reading. - Transposed: reads nothing (0.003), so
J @ his the right orientation.
The file holds layers 0–24 of Gemma 3 1B's 26 blocks. The final block has no map because it already is the final layer, and it reads through identity, exactly. Any other unmapped layer, the embedding for example, is an empty cell rather than a borrowed reading.
What it is for. The J-lens is not a better next-token predictor. In middle layers it matches the next token less often than the logit lens. What it does is surface concepts early. At the last position of two-hop prompts:
| Prompt | Token | Layer | Logit lens rank | J-lens rank |
|---|---|---|---|---|
| Mount Fuji → national language | " Japan" | 11 | 4,204 | 88 |
| Mount Fuji → national language | " Japanese" | 15 | 519 | 3 |
| Taj Mahal → currency | " India" | 13 | 4,667 | 113 |
| Colosseum → capital | " Rome" | 18 | 136 | 4 |
On the Fuji prompt the logit lens reads " quieran" at layers 8–10. The J-lens reads " country" at layers 9–12, " language" at 13–14, and " Japanese" from layer 20, against the logit lens's 23.
Compare lenses (a Lens sub-tab, interp/lenses/) shows that directly. It
takes one position of a trace and lays out every lens installed for the model
side by side, layer by layer. Track a token and it adds the token's rank per
layer under each lens, plus the first layer each lens has it in the top 10.
Only the logit lens is self-verified. Its last layer must reproduce the trace's
own result_output. A fitted lens deliberately does not reproduce the output, so
its grid reports verified: unavailable while the identity path confirms the
machinery.
J++ files share the same J structure and install the same way. Neuronpedia
publishes them only for larger models than this machine runs: Qwen3-4B,
Qwen3.5-0.8B and 2B, Gemma 4 31B, OLMo 32B. That path has not been tried on a
real J++ file.
Neurons
Neurons draws every MLP neuron of the window's WebML GGUF model in 3D. Each layer is a plate, the plates are stacked in depth, and you orbit the stack. It is inspired by webml-community's Neuron Studio and runs on our own engine, so it works with any architecture WebML runs (Llama, Qwen 2/3, Gemma 3, SmolLM3).
- The stack. A layer of
ffnneurons is a near-square grid filled row-major (neurons/grid.ts; Qwen3-4B's 9728 neurons make a 99 × 99 grid).neurons/scene.tsdraws one three.jsPointsper layer over a shared grid of positions. Every neuron on every plate is drawn; nothing is subsampled.- Every camera parameter (layer, zoom, pan, spread, yaw, pitch) eases to its target
on its own critically damped spring (
neurons/camera.ts). Traversing to a layer only moves the target. - Clicks are resolved by intersecting the pointer's ray with a plate's plane
analytically (
pickCell), never with a Raycaster over the points. - Frames are drawn on demand. A hidden pane gets no
requestAnimationFrame, so each invalidation also arms a timer that draws the settled frame. - All colours are theme tokens:
--ramp-pos/--ramp-negfor polarity, re-read when the theme changes.
- Steering. A recipe holds up to 32 neurons (
neurons/recipe.ts).- Each neuron edits the gated activation right before
down_proj, on every token:act = act × scale + offset. - Multiply sets the scale directly, and 0 ablates.
- Additive doses run from −3 to +3, in units of the layer's median residual norm divided by that neuron's down-column norm. So dose 1 moves the residual stream by about one typical residual norm at any depth.
- The medians are measured once per load over a neutral passage. The column norms
are read on the GPU as
W_down · e_n. See the engine for the kernels.
- Each neuron edits the gated activation right before
- Compare. Both replies decode greedily from the same prompt (thinking off), so the recipe is the only difference. With nothing active, the steered reply is the baseline itself.
- Heat. Click any token of either reply to replay the conversation, under that
reply's own edits, up to the position that predicted it. Each layer's activations
are read back there.
- The stack lights by signed activation, normalized per layer.
- Fired hardest lists the top neurons, across all layers or only the selected one; click a row to fly to that neuron.
- The replay re-tokenizes the reply's text. A reply that does not tokenize back to the tokens it was generated as can shift a position by one.
- Recipes persist per model file in the viewer's browser. Export writes
<model>.neurons.json(schema 1, naming the site and normalization), and Import refuses a recipe made on another file or shape. - Keys (section-scoped, never in a text box):
- Arrows: adjacent neuron. Alt + arrows: orbit.
- Page Up / Page Down: layer.
=/-: zoom. I: inspect.R: reset the view.A: add the selection. Mod+Z: undo.- Each key is an
interp.neurons.*command, reached throughneurons/actions.ts.
- Sound. Pentatonic hover, select, dose and layer tones on the mixer's Neurons strip. Muting is per viewer.
- Not here: Qwen3.5 (WebML has no Gated DeltaNet layers yet), so Neuron Studio's Qwen3.5-4B sample recipes do not transfer. The llama.cpp tracer cannot edit an MLP activation either, so this section is WebML only.
Features
What a sparse autoencoder reads in a trace. An SAE is a dictionary for one site of one
model: Gemma Scope 2's resid_post/layer_13_width_16k_l0_medium rewrites Gemma 3 1B
IT's residual stream after block 13 as a few of 16,384 directions. It runs in numpy on
the residuals a llama.cpp trace already holds (backend/modules/interp/sae.py). That
needs no torch and no GPU, about half a second for a 38-token trace. l_out is
llama.cpp's name for exactly that site: the block's output after its post-norms and
residual add.
The section shows:
- the token strip, shaded by the selected feature;
- the features at the selected token beside the trace's strongest;
- the selected feature's card: its activation across the prompt, then Neuronpedia's label and examples.
Selecting a token moves the shared locus, so the other tools follow.
Picking a trace no installed SAE fits (marked (no SAE) in the picker) says so by
model name and offers a trace that does fit, or the Library when none does.
Can these features be trusted here?
Gemma Scope was trained on bf16 activations of the Hugging Face model. A trace holds a quantized GGUF's activations as computed by llama.cpp. An SAE fed the wrong activations still returns features, confident and wrong. So there are three checks, cheapest first:
- Reconstruction, every encoding. FVU (
Σ‖x − x̂‖² / Σ‖x − x̄‖²) and cosine, with position 0 left out. Gemma's BOS residual is the attention sink, about 9× the norm of any other token, and SAEs are trained without it. The verdict isverifiedat FVU ≤ 0.4,driftedabove, andnot_checkedwith fewer than two tokens. - L0 against the trained L0. Features active per token, beside the
l0the SAE's config states. A wrong site or a scale error moves this first. - The label check, per feature (Check the label). It traces Neuronpedia's own top example through this GGUF, then compares the activation at Neuronpedia's peak token with Neuronpedia's bf16 value. The text is Neuronpedia's, so nothing of yours is sent. The temporary trace is deleted afterwards.
Measured on Gemma 3 1B IT Q8_0 with Scope 2 L13 16k medium:
- Reconstruction: FVU 0.31, cosine 0.995.
- L0: 60.0, against 60 trained.
- fp16 vs f32 trace storage: the same FVU (0.3105 vs 0.3100).
- Label checks: 8 of 8 matched at Neuronpedia's peak token, 7 of them within about 1%. For example, F90 "cities" on " Newark" read 588.7 here against 586.0.
One thing the checks surfaced: some features peak on the first token after BOS.
F371 reads 1,127 there against Neuronpedia's maximum of 240 elsewhere. That residual
is not an outlier (its norm is ordinary), so the activation is real. But Neuronpedia's
examples never show that position, so such features are marked first token rather
than hidden.
Labels come from Neuronpedia, whose sources are named
{layer}-gemmascope-2-res-{width} and host the medium-L0 dictionary of each
width. The small, big and seed variants share a width but not the features, so they
get no labels rather than someone else's (catalog.neuronpedia_source).
The fit check runs before encoding, and an incompatible SAE is refused rather than
encoded.
Steer
Add a feature's direction to the residual stream and read what the model does
with it (backend/modules/interp/steer.py).
How. A feature's decoder row is a direction. The tracer adds it, scaled, at the
SAE's layer through llama.cpp's control-vector API (llama_set_adapter_cvec).
llama.cpp adds layer il's row right after block il, before l_out is named.
So a steered trace's l_out includes the push, and the Features section, reading
that trace, shows the feature lit up. The control-vector buffer starts at layer 1,
which puts layer il at offset (il − 1) · n_embd; a test pins that offset.
Two traces per run.
- The default is a plain generation of the prompt.
- The steered run is a fork of the default on the same prompt tokens:
derivedFromnames the default, andedits: [{kind: "steer", feature, layer, strength, scale}]records the push.
Generation is greedy, so any difference is the vector's and not the sampler's. Trying another strength on the same prompt reuses the default run.
Strength is in the feature's own units. The scale added is strength × peak,
where peak is the feature's strongest activation on the prompt. When the
feature does not fire there, peak falls back to Neuronpedia's maximum. A raw
scale would mean nothing across features: Gemma 3 1B's residual norm at layer 13
is about 6,000, and a feature's peak is in the hundreds.
Measured on Gemma 3 1B IT Q8_0, with F90 ("cities states regions") on "My favourite thing to do on a weekend is":
| Strength | Steered continuation |
|---|---|
| default | "to curl up with a good book and a cup of tea." |
| 0.5× | "to spend time with my dog, Buster…" |
| 1× | "to visit the botanical gardens. It's a beautiful place…" |
| 1.5× | "to visit.! It's a place that has been on my mind…" |
| 2× and up | punctuation |
The slider therefore marks anything past ±1.5 as past coherent.
Did it take? Three readings, in the order they can go wrong:
- Arrived. The feature's mean activation on the prompt, default vs steered, judged against the sign of the push. A push the SAE does not read back as this feature is not steering with it, whatever the text does.
- First prediction. Each run's top token, the tokens that gained and lost the most log-probability (among tokens above 10⁻⁴ in either run), and the KL. This is the most sensitive reading. The France feature (F13769) at −1× on "The Eiffel Tower is in the capital of" moved " France" from 99.6% to 47%, and " Japan", " Germany" and " Tokyo" gained, while the greedy text still said France.
- Text. Both generations side by side, shaded from the first token where they differ.
Two findings shaped the design:
- BOS needs no special handling. llama.cpp adds the vector at every position, BOS included. A direct test decoding BOS alone before setting the vector produced the same text at 1×, and the same collapse at 2× and 4×.
- Steering does not touch the chat. It runs in the tracer only. The agent's
interp.steerdeclares a side effect, because it runs the model and writes two traces, so the permission engine sees it.
Axis (the Assistant Axis)
Lu et al. (2026) found that a chat model's persona has a leading direction in the residual stream. It is the mean activation of the model answering as its assistant, minus the mean over it answering in many other roles. Projections along it fall when a conversation pulls the model out of character, and flooring the projection keeps the model in character.
The published vectors are for 27–70B models, so this section builds the axis on the GGUF you run and runs everything through llama.cpp. Nothing is downloaded and nothing leaves the machine.
Building (backend/modules/interp/axis.py, axis_runner.py):
- Who answers. The model answers ten questions three times as its plain assistant (no system prompt, and two plain ones) and once in each of 30 roles, from a librarian to a sentient storm. You can also paste your own role list, one per line. Each answer is generated greedily through the model's own chat template, which llama.cpp reads from the GGUF.
- What is kept. The residual (
l_out) of every generated token at every layer. The process loads the model once and keeps onlyl_out, so a 1B model takes about 5 minutes on the CPU build here. - The axis. At each layer, the mean of the assistant's answers minus the mean of the roles' answers, normalised.
- The questions interleave persona prompts ("Tell me about yourself") with ordinary tasks ("How do I boil an egg?"). With persona questions alone, every assistant answer was "I am a large language model…", the bands sat too high, and a plain budgeting reply read 92% "below the floor". A role answers a task in character too, so the axis keeps measuring voice rather than topic.
The check comes before any reading. The last three questions are held out of the fit. For each layer the section reports:
- AUC, two ways: how often a held-out assistant token projects above a held-out role token, and the same for whole answers.
- d′ between those two token distributions.
- The cosine between the axis and PC1 of the role means.
The default layer is the one that separates single tokens best, never one picked in
advance. Block 0 and the last block are excluded: llama.cpp cannot add a vector
after block 0, and the last block's l_out is pruned to the output position. The
verdict is separates (token AUC ≥ 0.8), weak or not_checked.
The bands are percentiles of the held-out token projections. The floor is the assistant's 25th percentile. Each role's first answer is stored, so you can see whether a role was actually played: unlike the paper, answers are not filtered by a judge.
Monitoring. Three sources, each projected token by token and drawn over the assistant and role bands:
- A pasted conversation, as
user:/assistant:/system:lines. - A captured agent turn, rebuilt from its last round by agentpedia's
rebuild.pyand flattened for a small model's template. Tool results become marked user messages. - A trace. A trace is raw text with no chat template, so its level means less than its shape.
A conversation is read by its drift, not its level. The bands come from the model's own generated answers, and text it did not write, read back, sits lower even when it is plain assistant prose. Within one conversation that offset is shared, so the section labels each message with its median and reports how far the replies moved from the first one, in units of the gap between the assistant's and the roles' held-out medians. A value of −100% means "as far as a role sits from the assistant". The mean, the share below the floor and the trend are reported too, for the assistant's own messages first.
An agent turn from a remote model is read through the local GGUF. It shows how this model reads the conversation, not the remote model's own state.
Capping. A llama.cpp control vector can only add, so a floor costs two passes per answer token:
- Decode the token and read its projection at each band layer.
- If it is short of the floor, roll the token out of the KV cache and decode it again, adding the shortfall along each layer's axis.
The shortfall is counted bottom-up. What is added at one layer is still in the residual at the next, so the next layer's need is reduced by that push, read through the angle between the two layers' axes. That is linear and approximate, so a second correction pass is allowed, and every capped token's projection is read back from the pass that produced it rather than assumed.
Only answer tokens are floored. The floor was measured on answers; template markers, the system prompt and BOS (whose residual is many times the norm) sit below it by nature. Flooring them pushed on text the model did not write and wrecked the output (measured).
Agent: interp.axis returns the check (reported first), the default layer and
band, where the roles sit, and the section's last reading and cap. With traceId,
it reads that trace now. It is read-only: building and capping run in the section.
Measured on Gemma 3 1B IT, Q8_0, CPU (axis gemma-3-1b-it-Q8_0--20261028-091134):
| What | Result |
|---|---|
| Build | 330 answers × 32 tokens in 358 s |
| Check | held-out token AUC 0.997 at L12 (the default); 0.996 at L11, 0.995 at L14; 0.67 at L8 |
| PC1 of the role means vs the axis | cosine 0.16–0.24 at L10–12, 0.72–0.81 at L17–23. On this model the roles' main spread is not the axis in the layers where the axis separates best, unlike the paper's large models |
| Roles nearest / farthest | child, tour guide, drill sergeant, medieval knight, therapist / demon, noir detective, dragon, fisherman |
| A conversation sliding into role-play (budgeting → "you are a sea witch") | reply medians 87 → −115 → −88: the replies moved −48% of the gap, into the roles' interquartile band. The same opening kept as an assistant stayed flat (78 → 73) |
| A captured 9-round agent turn (8.4k tokens, last 4096 read) | replies steady, +3% of the gap |
| Cap, L10–14, "You are a ghost…" / "Tell me about yourself." | default: "(A slow, deliberate sigh, like the rustling of ancient parchment…"; capped: "(I'll do my best to channel a truly spectral presence – it's a bit… complicated! Here goes…) Alright, let's talk." All 48 answer tokens were floored, and 239 of 240 layer readings, read back, were at the floor (the last was 1.04 short). In the default run 0–6% were |
NLA (natural language autoencoders)
A natural language autoencoder is two fine-tunes of one model trained as an
autoencoder whose bottleneck is English (Fraser-Taliente et al., 2026). The
AV (activation verbalizer) is given one residual-stream vector in place of a
token's embedding and writes an explanation; the AR (activation reconstructor)
reads the explanation and predicts the vector back. The reconstruction's cosine to
the original scores the explanation. The section runs a pair entirely on
llama.cpp, with nothing fetched at runtime (backend/modules/interp/nla.py,
nla_runner.py):
- Base — the text (raw, or a conversation or agent turn through the base
model's chat template) is decoded once and every position's
l_outat the pair's layer kept. A trace of the base model brings its own vectors, prompt and generated tokens alike. - AV — the pair's canonical prompt (with Qwen's default system turn, which
llama.cpp's built-in ChatML does not write) is decoded up to the marker token
㈎; the vector, rescaled to the L2 norm the AV was trained at (150), goes in as that position's embedding through allama_batchwithembdset; the rest of the prompt follows. The marker must sit between the sidecar's two neighbour tokens, or the run is refused — an injection one slot off is verbalized as the marker character. The prefix stays in the KV cache; each sample rolls back to the marker. - AR — each explanation in the AR's template is decoded; the last token's
l_outat the pair's layer (no final norm — the AR stops at that block) times the value head is the reconstruction.
The three models load one at a time. The AR GGUF carries its value head as an extra
tensor (nla.value_head.weight, BF16) that stock llama.cpp refuses ("wrong number of
tensors"), so the first run writes a copy without it beside the file
(*.llama.gguf, its nla.role set to ar-backbone so discovery skips it) and reads
the head with the gguf reader.
The score's floor is not zero. Layer-K residuals share a large common
direction, so the AR's reconstruction of any text lands near any vector. Every
run also reconstructs a content-free sentence ("A token in the middle of an
ordinary English sentence.") and reports each position's cosine against that
floor (nullCos). Verdicts: carried at cos ≥ 0.8, partial at ≥ 0.5,
drifted below, or within 0.15 of the floor whatever the number. Positions in
the first ten (little context at that layer yet) and rare very-high-norm ones are
flagged early / high_norm, after the authors' own note.
The section: the pair and which of its three GGUFs were found; the source
(Text, Conversation, Agent turn, Trace) and which positions — comma separated,
negative from the end, or the last N; samples per position (the best-scoring is
kept) and the sampling temperature (1, as the authors run it). The run streams:
each explanation appears as it is written. The readout is a token strip with each
explained position underlined by its verdict (click another token to queue it for
the next run), and a card per position: the explanation, its cosine on a 0–1 track
with the floor and the 0.5 / 0.8 marks, other samples, and Rewrite and
re-score — any explanation, scored by the AR against the same stored vector.
Readouts are stored in $DATA/interp/nla/ (JSON, and the explained vectors as
.npz so a rewrite can be scored later).
Agent: interp.nla returns the readout in view (or a stored one by id): per
position, the explanation, cos, the floor, the verdict and flags; and whether the
pair's files are on disk. Read-only: explaining runs in the section.
Getting the pair. The only NLA pair small enough to run here is kitft's
Qwen2.5-7B-Instruct, layer 20. The pair's contract — the marker, its neighbours,
the injection norm, the prompts, the MSE scale — is copied into nla.PAIRS from the
checkpoints' nla_meta.yaml. The GGUFs are found among the models llama.cpp lists
(managed, LM Studio, extra folders), by header: the AR names itself (nla.role), the
AV carries the nla tag at full depth, the base is the same architecture and size
with neither. On this node they are Q4_K_M conversions in D:\.lmstudio\models:
bartowski's Qwen2.5-7B-Instruct (4.68 GB) and thomasgauthier's AV (4.68 GB) and AR
(3.41 GB). Before use they were checked against kitft's safetensors by ranged reads:
the norm weights and the value head byte-exact, quantized matrices at cos 0.997.
Measured on the CPU build (llama_cpp 0.3.34, Q4_K_M):
| What | Result |
|---|---|
| Speed | ~40–76 s for 1–3 positions end to end: base 4 s, AV ~8 tok/s (an explanation is ~140 tokens), AR ~6 s per explanation |
| "…the Eiffel Tower is in the city of Paris, and the Colosseum is in the city of" | 3 positions, mean cos 0.82, +0.41 above the floor; at " of" the AV wrote about a city pair and named Rome (cos 0.86, carried) |
| Control: six positions of three unrelated texts (landmarks, Python, grief) | matched explanations 0.84 on average; the same text, another position 0.64; another text 0.35; neutral sentences 0.38–0.44 |
| Rewrite at " of" | "Python … Fibonacci" 0.40 (below the 0.48 floor: drifted); a hand-written "about to be completed with Rome, Italy…" 0.82 (carried); the AV's own 0.86 |
| A trace ("Mount Fuji is in a country whose national language is" → " not English") | both positions carried (0.86, 0.84); the AV described a country-and-language question but did not name Japanese |
What is not built: "explain with NLA" from an axis vector or a feature direction
(no axis or SAE exists for Qwen2.5-7B on this node), larger pairs (Gemma 3 12B / 27B,
Llama 3.3 70B), and GPU offload — the llama_cpp build here is CPU-only.
Library
What the suite's later tools — features, steering, Jacobian lenses, circuits — stand
on. Backend in backend/modules/interp/, at /api/interp; the section is
interp/library/.
The worker
The backend has no torch, and keeps it that way. Anything that needs gradients or a
Hugging Face model runs in the worker: a long-lived process in its own managed
venv, $DATA/interp-venv, holding one model at a time.
-
Environment (
env.py). Package groups, each installed on demand from the section:base(torch with CUDA, transformers, accelerate, safetensors),sae(sae-lens),circuits(circuit-tracer, nnsight). torch is pinned to the version the notebook venv already has, so uv takes it from its cache rather than downloading ~2.5 GB again. Kept apart from the notebook venv so research dependencies cannot break notebooks. -
Process (
worker.py,worker_main.py). Started on demand and kept: loading a model is the slow part. JSON lines over stdin/stdout, with requests matched to replies by id. Spawned with blockingPopenand daemon reader threads, never asyncio subprocesses, which break under--reloadon Windows. Spawned-I, so nothing of the backend is importable inside it. Stopped in the app's lifespan shutdown, so a reload does not orphan a process holding VRAM. -
Every reply is one of three states:
ran, with aresult;failed, with the traceback verbatim;not_checked, with the reason it declined.
loadreads the Hugging Face cache only. A repo that is not cached is declined, never silently downloaded, unless the caller passesallowDownloadafter showing its size.
Measured on an RTX 4080 Laptop GPU:
- The worker answers
readyimmediately, and torch 2.11+cu128 imports on the firstping(about 6 s). Qwen/Qwen3-0.6Bloads from the cache in about 6 s, at 1.19 GB of VRAM in bf16.
Artifacts
A curated catalog (catalog.py), not free search: an SAE is only useful beside
the exact model it was trained on, and every publishing repo is laid out its own way.
Each entry says how a downloadable choice is formed:
- a folder per choice, such as Gemma Scope 2's
resid_post/layer_13_width_16k_l0_medium; - or the whole repo, for a transcoder set, where an attribution graph needs every layer.
Each entry also names which files make a usable artifact. Gemma Scope's ~800 MB
examples.safetensors stays on the Hub. Entries in the catalog:
| Entry | Repo |
|---|---|
| Gemma Scope 2, Gemma 3 1B/4B IT, residual SAEs | google/gemma-scope-2-{1b,4b}-it |
| Gemma Scope 2, Gemma 3 1B IT, transcoders | google/gemma-scope-2-1b-it |
| Qwen3 0.6B transcoders | mwhanna/qwen3-0.6b-transcoders-lowl0 |
| Llama 3.2 1B transcoders | mntss/transcoder-Llama-3.2-1B |
| Jacobian lenses (J-lens / J++) | neuronpedia/jacobian-lens |
| Assistant Axis vectors | lu-christina/assistant-axis-vectors (dataset) |
Nothing is written before its size is shown. Browsing reads sizes from the Hub's
tree API: one recursive listing, following its Link: rel="next" pages. A download
then takes two steps:
- Get shows the size, the file count and what the library would weigh afterwards.
- Download sends that size back as
confirmBytes. The backend refuses with 409 if the Hub now says otherwise, because consent was to that many bytes.
Downloads also check the budget (interp.libraryBudgetGb, 40 GB) before writing. Each
file is written to .part and renamed on success: a truncated params.safetensors
would otherwise load and produce garbage features instead of failing. The files keep
the Hub's own paths under $DATA/interp/artifacts/, with one row per artifact in
app.db (interp_artifacts).
Does it fit? Three answers
Each artifact's facts, its input width, dictionary width and layer, come from the
strongest source available. In order: the encoder's shape in the params.safetensors
header (read without loading the file), then the config (including a hook name such
as model.layers.13.output), then the folder name. Each fact records which, since a
width parsed from a folder name is the weaker claim. The facts are checked against two
models:
- the agent's GGUF, identified by its own name parts.
basename,size_labelandfinetunetogether givegemma-3-1b-it. Itsgeneral.base_modelis the model it was fine-tuned from: Gemma 3 1B IT's GGUF namesgemma-3-1b-ptthere. So an artifact trained on that parent isunknown, notcompatible. Matching on the parent once called a Gemma Scope SAE incompatible with the very model it reproduces within 1%. - the worker's loaded model.
compat.py returns one of three states:
- incompatible when a stated fact contradicts the model: another repo, an input
width that is not
d_model, or a layer the model does not have; - compatible when the trained-on repo matches and nothing contradicts it;
- unknown otherwise. A matching width alone is not a yes, because other models
share
d_model.
An SAE applied to the wrong model still produces plausible numbers, all of them wrong. This check is where that is caught.
Neuronpedia
neuronpedia.py is a read-only client for the public API. A feature lookup is a GET by
public id (/api/feature/ + model / source / index) and sends nothing of the user's.
- Cached in
app.dbfor a week. - Trimmed to a card: explanations, the logits the feature promotes and suppresses, its density, and the top examples, each windowed around its peak token.
The endpoints that would send the user's text are refused unless
interp.neuronpedia.allowText is on. The gate (require_text_allowed) exists before
any of them does, so the first one cannot be written without it. These endpoints are:
- testing a prompt;
- remote steering;
- NLA explanations;
- remote graph generation.
The agent reads all of this through interp.library. That tool reads and never
downloads: a download is consent to a size, and the user gives it.
The agent
The suite can be asked about. Every section publishes one digest of what the suite
is looking at as its agent context (interp/suite-state.ts): the model and where its
description came from, the attention pattern and its KV cost, the locus and what that
layer is, the latest captured agent turn, the in-window lens run, and caveats — a kind
known only by family rule, a conflict between sources, estimated token counts. It is
built from the module-level stores the tools already share, so it reads the same
whether the asked-about section is mounted, kept alive behind another, or not open.
The compact form (~600 characters) rides along with every chat turn while the suite is
visible; it is pinned under the per-pane workspace-context budget by a test.
| Piece | Where |
|---|---|
interp roster agent — "Interpretability guide" | backend/modules/agent/roster.py; scope interp, lens, llamacpp, agentpedia, kb, arxiv, preloads interp and lens |
interp tool group — interp.state (full digest), interp.attention (per-layer rows), interp.heads (a trace's heads, ranked), interp.features (SAE features + checks), interp.graph (circuit paths), interp.lenses (lens comparison), interp.steer (steer + compare; a side effect), interp.axis (assistant axis: check, drift, cap), interp.nla (NLA readouts: explanation, cos above the floor), interp.library (artifacts, fit, worker), interp.goto (section + locus) | declared on the suite pane, interp/agent-tools.ts; read-only, so never gated |
| Its primer — tool usage, how-a-fact-is-known, a glossary (SWA, chunked, NoPE, KV sharing, K=V, MLA, DeltaNet, SSM, sinks, lens, J-lens, MoE), pitfalls | backend/modules/agent/guides/interp.md, delivered once the group loads — costs nothing per turn until then |
Ask strip — the chat pinned to interp, its own sessions, no picker | a right region of the suite (interp.ask, key a), not a section: a section would replace the thing being asked about |
| Ask about this — in every section's strip, and Ask why on the attention block card | interp/ask.ts: opens the strip and sends a question naming the section and locus, addressed to interp |
test_interp_agent.py keeps the two halves honest: the guide must name exactly the
tools the frontend declares, so the model is never taught a tool that does not exist
or handed one nobody explained.
The primer is a group guide rather than a user skill on purpose: it ships with the code, versions with the tools it documents, and is delivered exactly when the group is loaded.
Two layout-shell features it introduced
Both are general; any pane can opt in.
sectionsLayout: 'rail'draws a pane's sections as a vertical rail (packages/ui/src/layout/SectionRail.tsx), grouped bySectionDecl.group, with vector icons named bySectionDecl.glyph(NAMED_GLYPHSinglyphs.tsx). Collapsing to icons only is remembered per viewer inlocalStorage.sectionsKeepAlive: truemounts a section on first visit and then hides it instead of unmounting it. A design with unsaved edits survives a look at the lens, and each kept body keeps its own section-scoped agent context.
Keep-alive means a section's verbs stay bound while it is hidden. The keymap
therefore gained a paneSection context key, and every single-key binding
into the suite names its section (whenSuiteSection in interp/ids.ts) — x
pressed in the lens must not delete a designer node. suite.test.ts enforces it.
Panes that switched their own sections internally (agentpedia, the lens lab) are
hosted unchanged: SectionScopeContext lets a host supply the value
usePaneSection() returns, one level down (interp/SubSections.tsx).
Migration
RENAMED_VIEWS (layout/serialize.ts) may now name a section:
interpretability.context, interpretability.architecture, agentpedia.hub and
webml.lens reopen as the suite on the matching section, collapsing to one pane.
VIEW_ALIASES keeps the retired ids and titles resolving for the agent, and an
alias open carrying params (an eval's {turnId}) now applies them. Command ids are
unchanged; section.show:llamacpp.server:{traces,lens,stepper} are kept as shims.
The llama.cpp pane keeps only Server and Models.
Contributions
| Kind | ID | Notes |
|---|---|---|
| Panel | interpretability.suite | Singleton document; rail + keep-alive |
| Panels (embedded) | interp.home … interp.library | Section bodies |
| Command | interp.open | Open the suite at Home |
| Setting | interp.libraryBudgetGb | Library disk budget, GB (40; 0 = no limit) |
| Setting | interp.neuronpedia.allowText | Allow endpoints that send your text to Neuronpedia (off) |
| Endpoints | /api/interp/library/* | Catalog, choices, download (ndjson, confirmBytes), artifacts with fit, delete |
| Endpoints | /api/interp/worker* | Env install, start / stop, status, ops (ping status load unload generate) |
| Endpoint | /api/interp/neuronpedia/feature | One feature card, cached |
| Endpoints | /api/interp/graphs* | List, import (Neuronpedia ref or JSON), read, qParams, summary |
| Endpoints | /api/interp/lens/targets, /api/interp/lens/install | Matching GGUFs for a lens; install one (worker conversion) |
| Endpoints | /api/interp/axis* | List + defaults, build (ndjson), a trace's projection, monitor (ndjson), cap (ndjson) |
| Endpoints | /api/interp/nla* | Pairs + files + readouts, a stored readout, explain (ndjson), score an explanation (ndjson) |
| Endpoint | /api/interp/steer | Default + steered generations (ndjson), then arrived / next-token / texts |
| Endpoints | /api/interp/features/* | Traces with fitting SAEs, SAEs per trace, encode, one feature's values, label check |