Module: interpretability
See exactly what the model is being handed. The agent loop rebuilds its context
every round — system prompt, tool guides, replayed history, the focused editor
buffer, the user turn, and a tool list that progressive disclosure recomputes as
the model calls load_tools — and until this module none of it was observable.
Status: implemented — frontend in
packages/core/src/modules/interpretability/, backend in
backend/modules/interpretability/, capture hook in
backend/modules/agent/orchestrator.py.
What it shows, and why that's the useful part
The interesting question about a local model is rarely "what did it say" — it's "what did it actually see". Three things drive agent behaviour that were previously invisible:
- The tool schemas. Serialized tool JSON is usually the single largest block of an agent's context, and progressive disclosure changes it every round. The pane breaks it down per group and per tool, in real tokens.
- Silent tool-budget truncation.
_select_toolscaps the list atTOOL_BUDGET(44) and drops the overflow with nothing but alogger.warning. If the model "ignored" a tool you were sure it had, this is why — the pane says so in a full-width warning, with the dropped count. - The gap between requested and real context length.
num_ctxis what we ask for; the model has what it has. Asking for more gets you the model's real limit silently. The header shows both.
Prompt, weights, activations — three surfaces, three modules
This module inspects the prompt. That is a real boundary and not a temporary one: activations, attention, KV cache contents and gradients are unavailable through the HTTP API of every provider it can talk to, which returns tokens and counts and nothing else. Everything on this page is measured from the request the backend is about to send.
The model explorer covers the weights: reading the GGUF directly gives the real tensor inventory — every name, shape and quantization — so the structure shown is measured rather than described. It stops at the file; nothing here runs a forward pass.
The computation between them is now covered too, and it lives next door in
llamacpp rather than here, because it needs a different
engine and a different install. A traced run captures ggml's per-node cb_eval
tensors into a snapshot on disk, scrubbed offline rather than streamed (one
pass with attention on is about a gigabyte), and is read in the traces section
of the llama.cpp pane.
That page also now carries the lens — the same snapshot read as words
rather than as numbers, a layer × position grid of what the model was disposed to
say at each point, which is Neuronpedia's Jacobian
Lens with the identity transport.
It needs no autograd and no fitted artifact, so the rule below still holds: this
backend has no torch and should not grow one. Its correctness argument is the
same shape as everything else here — at the last layer the lens must reproduce
the trace's own captured logits, and a grid that cannot show that says
unavailable rather than presenting itself as checked. That path still avoids autograd entirely — the backend has
no torch and should not grow one; heavy dependencies live in per-project uv envs
(backend/modules/training/envs.py) or optional extras, and tracing is
uv sync --extra llamacpp, lazy-imported and CPU-only.
The seam between this page and that one matters: a trace and a chat turn are
different runs until proven otherwise. Chat goes through a downloaded
llama-server binary, tracing through the wheel's own libllama, so the two may
only be shown together when both the build and the model hash agree
(traces.matches_run). Overlaying them regardless would put a forward pass under
an answer it did not produce — which is exactly the kind of confident wrongness
this module exists to refuse.
Research Compass & Connected Workflow
To make the dashboard approachable for beginners and fast for researchers, both interpretability panes host a unified Workflow Header and the Research Compass (interpretability.guide command or the 🧭 Compass button):
- Context Window (Prompt):
interpretability.context— measures exact token counts, prompt block kinds (system, history, buffer, tools), and tool budget truncation. Includes an onboarding launchpad with prompt suggestions and quick navigation when no turns have been captured. - Model Explorer (Weights):
interpretability.architecture— inspects real GGUF tensor directories, quantizations (Q4_K_M, Q8_0), and architecture specs. Features inline glossary tooltips for technical metrics (GQA vs MHA, KV cache footprint, SwiGLU gating, RoPE θ, MoE active fractions) and empty-state links to Hugging Face and llama.cpp. - Logit Lens & Stepper (Activations):
llamacpp.server— dynamic layer representations and execution stepping.
The 🧭 Compass modal maps the 7 stages of AI research (Literature, Datasets, Context Window, Model Explorer, Logit Lens, Evals, Training), answering which question each pane answers, when to use it, and common beginner pitfalls.
Merge interpretability.architecture and llamacpp.server into one window to keep prompt, weights, and activations within immediate reach.
Architecture
Capture is a read-only observer inside the agent loop. It runs once per round, just before the provider call, and is wrapped so that a failed capture can never cost the user their turn.
Two rings and a table
The ring is what the pane opens on: 25 turns, in process, gone on restart. The
agent_turns table behind it is what makes a turn answerable tomorrow — 5000 rows,
each the full snapshot. Both are written by the same capture_round; they differ
only in how long they last, so GET /turns/{turn_id} tries the ring and then falls
back to the table. Without that fallback every turn older than the ring lists in the
history and 404s when opened.
Both halves are read together by agentpedia, which steps one turn through what it was shown, what it sent, and what it did.
The third ring in the diagram is the telemetry one, and it is a different kind of
record: IoEvent is what went over the wire, where a RoundSnapshot is what the
model was shown. run_agent_loop stamps every request it makes with turn_id and
round (telemetry/turn.py, a contextvar rather than a parameter — the recording
happens in httpx's response hook, several layers below the loop that knows the
answer), which is what lets the two be lined up without matching them by timestamp.
A sub-turn restores its parent's stamp when it leaves rather than clearing it.
Two invariants hold the design together, both enforced in tests:
- Capture never raises. It runs inside the turn; a bad message list costs a snapshot, never an answer.
- Capture never mutates. It only reads
messages/tools. The pane's entire premise is that it shows the real prompt, so observing it cannot change it.
Multi-agent: delegation and peers
main is not the only agent that runs. It orchestrates two different ways, and the
pane distinguishes them because they have very different visibility.
agent.delegate | agent.ask_peer | |
|---|---|---|
| Runs on | this node, this browser connection | another user's node |
| Tools | full — relayed to your frontend, gated under the sub-agent's mode | none (tool-less, read-only) |
| Depth | one level (specialists set can_delegate: false) | n/a |
| Visible in this pane | fully — it's a real captured turn | opaque node only |
A delegated turn needs no special capture: run_delegate reuses
orchestrator.run_agent_loop, which is exactly where the capture hook lives, so the
sub-agent is recorded like any other turn with its own agentId. What it did need
was the link — parent_turn_id is threaded from run_delegate into the loop so the
pane can rebuild the handoff tree instead of showing unrelated siblings.
Each turn carries the identity its context was actually assembled from: agent name,
declared toolGroups scope, and permission mode. Scope is the operative field —
a specialist can only ever load tools from its declared groups, so "the agent
ignored my request" is frequently really "that agent was never given that
capability", and the pane can now show which.
toolGroups: null ≠ []null means unrestricted — every group loadable, which only main is. An empty
list means the opposite: no groups at all. They must never be collapsed.
Why peers stay opaque
A peer turn is recorded with its peerId and the prompt that was sent, and nothing
else — no rounds, no token counts. The peer's agent assembles its context on its
own machine, and this node genuinely cannot see it (nor should it; see
network). Recording the handoff anyway is what keeps the tree
honest: a delegation that leaves the node shows up as a leaf that says so, rather
than a silent gap the reader has to guess about.
Model explorer
Beside the context view sits the loaded model: what it is, next to what it was given. It is a thing you click through rather than a picture — selecting a stage of the stack shows the tensors that actually implement it.
The block is drawn once with a ×N multiplier rather than repeated — forty-eight identical rectangles say nothing the number doesn't — but the multiplier expands into a layer picker, so "block 17 specifically" is one click away.
Inspect is a canvas, and the stack is one node
Inspect used to be a vertical column of <button> rows inside a CSS grid track
declared minmax(190px, 260px). That single line was the whole problem: the diagram
could not exceed 260px in a pane of any width, there was no zoom, and the residual —
the one connection worth drawing — was a text row reading ⊕ residual that connected
nothing to nothing.
It is now a read-only @xyflow/react canvas (inspect/InspectCanvas.tsx) with pan,
zoom, a minimap and two fit controls, sitting inside a resizable split.
The node model is O(1) in the layer count. A 48-layer model has ~290 sublayers, and drawing them would be unreadable and a lie: the interesting fact about a stack of 48 decoder blocks is that they are identical, and 48 drawn boxes assert the opposite. So the stack is one node carrying a rail of N ticks, and exactly one block is ever expanded — the selected one. Six to eight nodes, ever, plus the rail.
The rail earns its place twice over. Tick height is that block's real bytes from
the GGUF inventory, so a model whose later blocks are heavier says so. Tick band is
its tensor-shape signature, computed by layerSignatures, so a model that alternates
its attention shape by layer — Gemma 4's blk.0.attn_q is 3840×4096 while
blk.17.attn_q is 3840×8192 — draws as stripes instead of as a grey varies
chip with a tooltip. The GGUF inventory is the only source that can see this at all:
every scalar source, GGUF metadata and a repo's config.json alike, reports a single
head count for the whole model.
Selection {stage, layer} is unchanged and remains the single source of truth;
inspect/graph.ts only provides a bijection to node ids, so the canvas is a
projection of the selection rather than a second copy of it. That is what let the
model-locus wiring survive the rewrite with no edit at all — the
lens still publishes a layer, the explorer still selects and opens that block, and the
canvas now additionally brings it into view. The focus is driven by a nonce rather
than the layer value, so clicking layer 15 in the lens twice re-centres instead of
being a no-op.
Layout is dagre with rankdir: 'TB', in a file deliberately separate from the
designer's LR one. designer/layout.ts already argues both sides: LR because node
editors read left to right, and TB for training/panels/ModelGraphPane because "a
traced module tree reads as a stack". Inspect is the second case by that file's own
criterion — and every canonical transformer figure is vertical, the residual only
reads as a spine when the block runs down the page, and the left column is now
narrow by default so the axis with room to spare is the vertical one.
flow/canvas/FlowCanvas.tsx said it was the only file importing @xyflow/react, and
designer/ModelCanvas.tsx argued its way in as the second. InspectCanvas makes the
same argument a third time in its own header rather than assuming it: ModelCanvas is
built around the designer's editable GraphNode — typed sockets,
socketsCompatible refusing a link at the wire, onConnect, frame membership, a
persisted layout sidecar. Inspect's nodes are read-only, derived, socketless and
unpersisted, so reusing it would mean every one of those invariants growing a "not in
inspect mode" branch. What the rule protects — the node model is the API, the engine
stays swappable — still holds: nothing outside that file imports the engine.
Zoom means two things, because the wheel already changes the zoom factor and a control that did the same would be redundant. Fit is the whole model and the way back. Focus is the selected node and what it connects to, on double-click — the useful verb on a graph is "show me this and its neighbours", not "make it 1.4×". On top of that the nodes read the viewport zoom and drop their fact rows below ~0.62, so zooming out shows the model's shape and zooming in reads its numbers.
Both sides resize now
SplitPane (packages/core/src/SplitPane.tsx) is a shared in-pane splitter, applied
to Inspect's two columns and to the designer's palette and inspector — which were
width: 176px and width: 320px, equally unresizable.
It is not packages/ui/src/layout/Sash.tsx: that one takes a SplitNode and
dispatches SET_SPLIT_SIZES to layoutStore because it divides the workspace. A
pane's own columns are not part of the layout tree, and core cannot import ui anyway.
Sizes live in localStorage (split-sizes.ts, mirroring llamacpp/pins.ts) rather
than in the layout store, which also keeps them out of an exported workspace — opening
somebody else's workspace should not resize the insides of your panes.
Two things in it are load-bearing:
- The drag does not
setState. A pointermove at 60 Hz re-rendering both children makes the React Flow canvas rebuild its node list sixty times a second. The live size is written straight to a--split-sizecustom property and React is told once, on pointerup. narrowBelowmeasures the container, not the viewport. It replaces a@media (width <= 720px)rule that was wrong by construction: a pane docked in a three-column workspace is narrow at any viewport width, and a maximised pane is wide on a small screen.
Two sources, and why they stay separate
| Source | Available for | What it gives |
|---|---|---|
| The model's own GGUF header | the node's own llama.cpp server, Ollama, LM Studio, or any server with interpretability.ggufPath set | The inventory: every tensor's name, shape, ggml quant type and byte size — and, read the same way, the description and the per-layer attention layout |
GGUF model_info via /api/show | Ollama | The description, written from the weights the server loaded |
config.json from a HF repo | any server whose weights file cannot be found | The description, but strictly of the repo |
The pane labels which it used (weights / repo, plus a gguf chip when the
inventory is present) rather than presenting them interchangeably.
/architecture reads the header itself (architecture.from_gguf_header) whenever the
weights file can be located — which is what an LM Studio model gets now. It used to
fall straight through to the repo's config.json for any server without a metadata
endpoint, describing the repo while the file it was converted into sat readable on
disk. Gaps are still filled from the repo the GGUF names as its base model, never
overriding a stated value. All three header readers share gguf.read_header_cached,
keyed on path, size and mtime so a re-downloaded file under the same name is re-read.
The split is load-bearing. A description can be approximated from a repo when the
weights are unreachable; an inventory cannot — a config.json lists no tensors —
so /tensors returns an error the pane renders as "no GGUF found" rather than a
plausible-looking list. A model can have the description without the inventory, and
the pane stays useful minus the tensor tables.
What the inventory catches that a schematic can't
Every scalar source reports one head count for the whole model. That is often
false. Gemma 4 alternates its attention shape by layer — blk.0.attn_q is 3840×4096
where blk.17.attn_q is 3840×8192 — so a "× 48" with a single head count beside it
actively misleads. The explorer compares tensor shapes across blocks and, when they
differ, marks the block frame varies and names the roles that change. Only the
inventory can see this.
A GGUF can also store its head counts per block — a hybrid (Jamba, Falcon-H1)
writes attention.head_count as an array with zeros on its state-space blocks.
architecture._int used to turn that array into None, so such a model drew no
attention at all. _scalar_or_max keeps the array as headsPerLayer /
kvHeadsPerLayer, reports the maximum as the headline with a note saying so, and
the attention node's subtitle uses the selected block's own count ("no attention
in this block" where it is zero). A uniform array collapses to its scalar.
With the per-layer attention layout loaded, the rail also carries each block's mixer: an accent cap on the global layers, a dashed cap on blocks with no attention at all (state-space, linear, convolution), none on the windowed majority. The open block's attention node is labelled by its own mixer — "Sliding-window attention · window 512 · KV from L13" — rather than by the model's one head count. Shape bands cannot do this alone: Gemma 3's windowed and global layers have identical tensor shapes.
Following the layer stepper
The explorer is the map; the llama.cpp stepper is a position on
it. While a trace is being stepped, the model locus carries the stepper's stage
and step as well as the layer, and the explorer selects that block, lights the
sub-block executing (attention, FFN, experts, norm — mapped in
inspect/stepper-stage.ts, because the two panes share the bus and nothing else),
and shows a following the stepper strip with a way back to it. The canvas
re-centres only when the block changes, so stepping from attention to FFN moves a
highlight rather than refitting the diagram under the reader.
It works in reverse too: while the stepper is driving, clicking a block or a sub-block on the diagram publishes a locus and the stepper's cursor jumps there.
It also reports the mixed quantization a K-quant build actually uses (a Gemma 4 12B QAT build is 338 F32 tensors, 328 Q4_0, and exactly one Q6_K — the embedding), and where the bytes sit: select Attention and the pane says what share of the file it is.
The window's logit lens
The explorer's lens so far has been the llama.cpp trace grid: a trace the node
captured, read as words. The second source is the model running in this window.
When a GGUF generates with the logit lens on (the playground's webml.lens, or a
{webllm} with :show: lens; see WebML), the engine reads out every
layer at every token, and the explorer draws the latest such reply under Whole
model and under a selected block (inspect/WindowLens.tsx).
- A layer × token grid, layer 0 at the top, the reply's last 32 tokens across. Each cell is the token the LM head would pick from the residual stream after that layer, shaded by its probability. Where it is the token finally chosen, it is in the accent, so a prediction settling reads as the accent running up a column to the last row. Hovering a cell gives the probability, the entropy and the stream's norm.
- A
settlesfooter: the first layer from which the chosen token leads at that layer and every later one. A dash means the last layer preferred another token (a sampled runner-up). - It fills while the reply is written. The window's engine keeps the run
(
webml/lens-run.ts) whichever surface asked for it, so neither the playground nor{webllm}reports in. One run is kept, the newest, cut to its last 512 tokens. - Rows link to blocks only when the layer counts agree. The explorer follows the node's model, and the window's is often another one, so the grid always names its own model. When the counts match, clicking a layer selects and opens that block and the selected block's row is outlined. When they differ, a note says so and the rows are not links. With no node model loaded, the empty state still shows the grid, unlinked.
Locating the weights
gguf.resolve_model_path tries, in order:
interpretability.ggufPath, which wins outright. Auto-discovery reads two undocumented on-disk layouts their owners are free to change, so there has to be a way to say "it is this file" that no upstream release can break.- The node's own llama.cpp server, when one is running and
its
--aliasmatches the configured model. This is the only source that is known rather than inferred: we chose the file. It is matched on the alias and not merely on "a server is running", becausellama-servercan be up while the agent talks to Ollama, and answering with the wrong model's tensors is exactly the plausible-looking wrongness this module exists to prevent. - Ollama's blob store — its manifest names the layer whose
mediaTypeis…image.model; that digest is the blob.OLLAMA_MODELSrelocates the root. - LM Studio's index (
~/.lmstudio/.internal/model-index-cache.json), whereindexedModelIdentifieris exactly the id its API serves. Its REST API reports a model's arch and quantization but not its path, so the index is the only route. TakingentryPointmatters: a multimodal repo ships anmmproj-*.ggufbeside the weights, and the first file in the directory is a real GGUF that parses cleanly and describes the wrong thing.
Step 4 is not gated on the provider being LM Studio — a third-party llama.cpp build and other OpenAI-dialect servers are often pointed at a model LM Studio also has indexed, and a hit there is a hit on the same file.
A quantized tensor's footprint depends on its block format, so gguf.py carries a
(elements per block, bytes per block) table. An unrecognized ggml type reports
no size rather than a guessed one, and the response's bytesComplete goes false
so the total reads as a floor. The table's correctness argument is that on a real
mixed-quant file, sum(tensor bytes) equals file_size - data_offset to the byte.
GGUF keys are namespaced per architecture (gemma2.block_count,
llama.attention.head_count_kv), so they're matched by suffix. A new family
works without a table entry.
gated is true for known-gated families (SwiGLU/GeGLU), false for known-dense
ones, and null for a family in neither list. "We don't know" and "we know it
isn't" are different claims, and only the second justifies drawing a
single-projection FFN. The same reasoning governs every optional field.
Model designer
Beside Inspect sits Design: the same pane, switched from reading a model to
drawing one. It is a Blender-shaped node editor —
drag RMSNorm, Attention, SwiGLU, wire a residual Add back around a block,
stack it ×N — and a real PyTorch nn.Module subclass materialises beside it on
every edit.
The two modes share a pane because of the bridge between them. Inspect answers "what is this model"; Design answers "what if it were different", and the obvious way to start a design is from the thing you are currently reading. Putting a workspace switch between a question and its follow-up would be the wrong seam.
What is taken from Blender, and what is refused
| Blender | Here |
|---|---|
| Colour-coded typed sockets | tensor / int / float / bool; a link between different types is refused at the wire |
| Node groups (Group input/output) | a group is a generated nn.Module subclass — the reason the metaphor fits at all |
| A socket that accepts many links | op.add, so a residual can fold several branches; every other input takes exactly one |
Mute a node (M) | ablation: it emits nothing, its input passes through, and it stops being counted |
| Reroute nodes, frames | the same, and purely cosmetic — codegen walks straight through them |
| Timings overlay | a cost overlay: each node's share of the parameter count |
| Left→right data flow | enforced by the dagre auto-layout (rankdir: 'LR') |
Not taken: Blender's implicit socket conversions. A float becoming a colour is a
convenience; a tensor silently becoming a differently-shaped tensor is the class of
bug this module exists to refuse, so a mismatch is a red wire you fix with an explicit
Reshape node.
The two things it shows that a drawn diagram cannot
The shape on every wire. [B, T, 2048], propagated from the Input node in pure
Python, in under a millisecond, on every keystroke. Batch and sequence stay symbolic,
because a symbolic dimension compared against a concrete one is an unknown rather than
a conflict.
Where the parameters actually are. Change d_model from 512 to 256 and the count
drops from 54.9M to 26.2M, every downstream shape re-labels, and the generated
__init__ signature changes with it — because config values stay variables in the
emitted class rather than being frozen into it.
Those numbers are our arithmetic, and the pane chips them estimated. The backend
has no torch and should not grow one, so measuring the real thing means instantiating
the module in a training project's venv — a separate, slower question, and a separate
answer. Where they disagree, the estimate is the one that is wrong.
The code pane is highlighted, per line
Read mode is syntax-highlighted by designer/highlight.ts, which parses the generated
module with the same lezer grammar CodeMirror uses and hands back one token array per
line. Spans go inside the existing per-line DOM rather than replacing it with an
editor view, because the pane does two things per line that an editor would have to
re-derive as decorations: light the lines a hovered node generated, and render the
gutter number.
Two things are bugs if missed, and both are pinned by tests:
- A styled range can span newlines (a docstring, a bracketed expression), so every
emitted run is split on
\n. Otherwise the line count moves and every# horrible:node=marker points at the wrong line. - Highlighting may not alter the text.
highlightTreeonly reports styled ranges; the gaps between them are real source and must be emitted too. The test rejoins every token and compares against the original — a dropped gap would still render as plausible-looking code.
Edit mode stays a plain textarea. A textarea overlaid on a highlighted <pre> is the
prettier trick and it depends on two elements agreeing about font metrics to the pixel;
when they disagree the caret drifts from the text, which is a far worse failure than a
button press.
Edits go both ways
The code pane is editable. Change a number in the generated module, hit Apply, and
it lands on the canvas — because the .py is the structural source of truth and the
graph is a projection of it.
Three rules make that safe rather than merely clever:
- The sync is explicit. Ctrl-S or Apply, never per keystroke. Parsing mid-word would replace your graph with whatever half-typed source currently means.
- Your positions survive. Every emitted line carries
# horrible:node=<id>, so re-reading the file recovers node identity and the layout sidecar still describes the graph. A round trip that re-arranged the canvas would make the pane useless for anything but a glance. - A file we cannot read changes nothing. It is read whole or not at all: the graph stays, your edit stays, and the reason is shown. Half a graph silently replacing a whole one is the worst outcome available here.
Source that does not match the shape the generator emits is not refused — it is
preserved verbatim as a custom.module node, and the pane names the classes it kept
that way. An opaque import nobody is told about is indistinguishable from a wrong one.
The guarantee under test is emit(parse(emit(g))) == emit(g), byte for byte, for every
template. It is a stronger promise than IR equality in the way that matters: parse and
regenerate and nothing churns, so a save never looks like a change you did not make.
Blocks are editable, not just instantiable
A group is one generated nn.Module subclass, so editing one is editing a class. The
breadcrumb under the toolbar is Blender's context path — the model, then whichever
groups you have stepped into. Double-click a block (or Edit contents in the
inspector) to go in; click a crumb to come back out. The canvas is handed one level and
knows nothing about groups; the rules live in designer/scope.ts as pure functions.
Two ways to make one:
- New group drops an empty one here — already wired input to output, because a group whose output has nothing connected to it cannot be generated at all — and steps into it.
- Group selection is Blender's Ctrl-G: the selected nodes move inside, the wires that crossed the boundary reconnect to the group's own terminals, and an instance lands where the selection was. Select several the way any node editor does — Ctrl/Cmd-click to add one, or Shift-drag a box across the canvas.
That is what forward(self, x) is, and what the ×N stack depends on. So a selection
that would need two inputs, or that feeds the rest of the graph from two nodes,
is refused with the count rather than wired to whichever one came first. Widening the
selection is the fix. For the same reason, the graph's own Input and Output nodes can
never go inside a group — they are the boundary it sits within, and it brings its own.
Renaming a group renames its class, and two groups compiling to the same class name are refused — the second definition would silently replace the first, and every instance of the first would quietly start running the second's code. A group nothing instantiates still shows shapes on its wires while you build it, but its parameters stay out of the headline count: a class the model never instantiates holds no weights. The model panel lists every group with its instance count and offers to delete the unused ones, since they are still written into the file.
Forking the model you are looking at
The inspected model sits at the top of the palette's starting points, named after
whatever the Inspect tab is showing. It turns that model's GGUF (or config.json)
metadata into an editable graph — the bridge the two modes share a pane for.
The difficulty is that ModelArchitecture
is deliberately full of holes: a dimension the metadata did not state stays null
rather than being guessed. A graph has no holes. So the import splits the difference
along one line:
| Width, depth, vocabulary, head count | Required. Missing any of them, there is no import — and the reason names which. A hidden size we invented would produce a plausible model of nothing |
| Gated or dense, RMSNorm or LayerNorm, RoPE or not | Assumed when absent, and every assumption is listed in a panel that stays until you close it. These only shape the graph, and the result is a starting point you edit, not a measurement |
The panel ends with the parameter count our arithmetic derives beside the one the metadata states. On Llama 3.2 3B those differ by 12% — and the whole difference is the output head, which that model ties to its embedding while the graph gives it its own matrix. So the import says that, in those words. A gap it cannot account for is reported as exactly that: a wrong import that stays quiet is worse than one that says it does not add up.
Two RoPE-shaped caveats worth knowing. Absent rotary parameters are ambiguous — they mean either "this model has none" or "the metadata omitted them" — and the import assumes RoPE anyway, because a decoder with no positional encoding at all is certainly wrong. And head width is left derived (hidden ÷ heads) unless the metadata explicitly decoupled them, which keeps the generated class parametric; a model like Gemma that decouples them is called out in the panel.
Running it for real
Everything above is the pane's own arithmetic. Run it builds the generated module in a training project's venv and runs one batch through it — the only thing that can turn an estimate into a measurement. It reports three states, never two:
| What it means | |
|---|---|
| Ran | Output shape, parameter count and torch version, measured. If the measurement disagrees with the estimate, the pane says so and by how much — the measurement is right and the cost overlay was wrong |
| Failed | The traceback, verbatim. Not summarised: the traceback is the answer, and paraphrasing throws away the line number |
| Not checked | No project, no venv, or no torch — said out loud, with the reason, and with a reminder that the numbers are still estimates |
That third state is the point. A model reported as validated when nothing validated it is worse than not offering the button at all — the same rule the hardware probe exists to enforce.
The probe runs as a subprocess in the project's venv, never in the backend, which has
no torch and should not grow one.
The script it runs is self-contained and does not import horrible_train: an
evals-owned project carries only its benchmark's requirements, and a probe that failed
on a technicality about our own helper package would read as the model being broken.
test_model_graph.py builds each template in real PyTorch and asserts
sum(p.numel()) equals what shapes.py predicted. All three currently agree to the
parameter — including the mixture-of-experts one. The test skips when torch is absent,
because the backend genuinely does not have it.
Tracing a module that already exists
The other importer reads a description of a model; Trace reads the model.
torch.fx.symbolic_trace records what a forward actually does, in order, with real
propagated shapes, and that becomes an editable graph. It runs in a training project's
venv, like the probe and for the same reason.
A trace is in torch's vocabulary, not ours, so some of it will not map onto the
twenty-one node types. Those arrive as custom.module placeholders that raise
NotImplementedError, named after the class they stand for.
The tempting placeholder returns its input unchanged. It would compile, run, train — and be a different model than the one you traced, wrong in a way nothing would ever report. Raising means the design refuses to run until you fill in the blanks, which is worth far more than one that quietly computes something else. The pane names every placeholder rather than only counting them.
Keyboard
The canvas takes Blender's verbs, declared through
the keymap — never a keydown listener in a
component:
Shift-A | focus the node search |
M | mute (ablate) the selected node |
X | delete it |
H | collapse it |
Home | frame everything |
Tab / Shift-Tab | enter / leave a group |
Ctrl-G | fold the selection into a group |
Every one is scoped to the pane and guarded with !textInput: unguarded, typing a
d_model value in the inspector would delete the selected node. A binding naming
paneFocus already beats an unscoped global, so the pane does not take capture —
which matters, because the same pane hosts Inspect mode, where swallowing the keyboard
would be wrong.
Reroutes and frames
Two more borrowings, both deliberately weightless. A reroute is a bend in a wire, drawn as a dot rather than a titled box — codegen walks straight through it, and a box would look like a node that does something. A frame is a labelled rectangle behind a set of nodes: pure cosmetics, living entirely in the layout sidecar, so the generated module is byte-identical before and after you draw one. Its rectangle is derived from where its members currently are, never stored, so dragging a member re-draws the frame around it and there is never a second competing fact about where the frame is.
Handing it to training
Send to project writes the design into a training project as model.py and adds a
notebook block that imports and instantiates it. No new execution path — the project's
own kernel, venv and Kaggle/Colab push all apply unchanged.
Two rules keep it from colliding with the recipe machinery that writes
into the same notebook. The block carries its own marker (horrible_model, not
horrible_recipe), because sharing the flag would make a recipe regenerate delete the
model definition and a model regenerate delete the trainer config — each silently. And
it lands above the recipe block, because a notebook where the trainer is configured
above the model it trains does not run.
Where a design lives
Two files under $HORRIBLE_DATA_DIR/model-graphs/: <name>.py — the generated module,
importing nothing from this app, yours to take anywhere — and <name>.graph.json,
holding the graph plus the canvas cosmetics Python has nowhere to put. Losing the
sidecar costs you a tidy layout; losing the .py costs you the model.
Block classification
The provider payload is an undifferentiated list of role/content dicts, and five
distinct parts of the prompt all arrive as role: system. Callers that assemble a
chat prompt (run_agent_turn and run_delegate) build it with
orchestrator.assemble_prompt. That function returns the messages together with a
parallel list of sources, and hands the sources to recorder.set_prompt_sources
before the loop's first capture. The sources travel beside the messages and never
inside them: an extra key on a provider message is a 400 on some dialects. With
sources present, the prompt prefix is labelled exactly.
Callers that pass no sources (evals, flow, the SDK) fall back to _classify_prompt.
It recovers the kinds from assembly order plus two content markers: <<<BUFFER, and
the skill catalog's header, which is imported from the module that writes it so that
rewording the header can't silently relabel the block. The heuristic cannot tell the
workspace index from a tool guide, and labels it guides.
| Kind | Source | Role |
|---|---|---|
system | spec.system_prompt | system |
guides | _guides_message — usage notes for preloaded groups | system |
skills | _skills_message — the skill catalog | system |
history | prior conversation replayed into the turn | user / assistant |
editor | _active_editor_message — the focused buffer | system |
user | this turn's prompt | user |
assistant | the model's output this turn (incl. tool calls) | assistant |
workspace | _workspace_context_message — the other visible panes | system |
tool_result | a tool's return value | tool |
skill_loaded | a use_skill result — the skill body the model asked for | tool |
tools_loaded | a load_tools result — the guides of the groups it loaded | tool |
mcp_result | an mcp-* tool's return value | tool |
nudge | the force-tool retry system message | system |
dropped_tools | the [tool budget] note naming groups cut to fit | system |
A tool result's kind is decided by the tool that produced it. The name comes from the
message's own tool_name, or failing that from its tool_call_id, matched back to
the assistant message that made the call.
Every block also carries hash, a short digest of its full, role-salted text. The
same block has the same hash in every round, so a round-by-round diff (the
trajectories Agent section) keys on the
hash rather than on position. The loop removes and re-appends its own notes, so
"index N is new" is not the same claim as "this text is new".
skills is deliberately not folded into guides: their cost profiles are
opposites. A guide is paid only on turns where its group is active, while every line
of the skill catalog rides every round — so merging them would hide the one block
whose growth the user most needs to see.
The prompt/loop boundary is pinned at round 0 so classification stays stable as the loop appends to the list.
Classification depends on the assembly order in run_agent_turn. Changing that
order means changing _classify_prompt in the same edit — a mislabelled block is
worse than no pane at all.
Token counting
Counts come from the model's real tokenizer via tokenizers (Rust, ~5 MB) — not
transformers, because the backend env has no torch and must not grow one. A bare
tokenizer.json fetched through the already-present huggingface-hub is all that
Tokenizer needs.
Which repo (tokenizer.resolve_repo, shared with the architecture diagram's
config.json fetch). The tokenizerSource reported on each turn names the step
that answered:
-
setting: theinterpretability.modelReposetting, if set. It wins outright. -
weights: the running GGUF's own identity (repo_from_weights), from the strongest claim available:general.source.repo_url, the file naming its own repo;- a
general.base_modelwhose repo name equals the file's ownbasename-size_label-finetune(a plain re-quantization); - for a file LM Studio serves, the Hub repo it was downloaded from. LM Studio's
models directory is laid out
<owner>/<repo>/<file>.gguf, and a GGUF repo has noconfig.json, so its card'sbase_modelis followed (at most three hops, never through a merge's multi-base list or a link the card calls a fine-tune) to the checkpoint that has one; - the file's own name under its
base_model's owner, if the Hub has that repo.
The result is memoized per model, misses included, because capture runs every round.
-
model: the model id, if it is shaped likeowner/name. -
family: a same-family default. The pane shows this as "approx".
A GGUF's general.base_model is its parent, not itself. An instruction-tuned
Gemma's GGUF names google/gemma-3-1b-pt there, the checkpoint it was fine-tuned
from. So base_model alone is never taken as the weights' repo; the file's name
parts decide, the same match the interp suite's
compatibility check makes for SAEs.
Step 2 runs before step 3 because an LM Studio model id is only repo-shaped.
google/gemma-4-12b-qat is LM Studio's catalog key and 401s on the Hub. Taking
it as the repo is what used to produce "Could not read config.json" and a chars/4
estimate for a model whose real checkpoint
(google/gemma-4-12B-it-qat-q4_0-unquantized) is public. When config.json still
can't be read, the error names which step chose the repo.
Auth order for the fetch:
- the Hugging Face connector's token, for gated repos (Gemma's are gated, and anonymous fetches 401);
- anonymous, which works for ungated families.
If no tokenizer can be had, counts fall back to a chars/4 estimate and the snapshot is
marked exact: false. The pane renders that as an "estimated" chip — an
estimate displayed as a precise number is exactly the failure this module exists
to prevent.
Contributions to the layout shell
| Kind | ID | Notes |
|---|---|---|
| Section | interpretability.suite → context | The context window — a section of the interpretability suite |
| Section | interpretability.suite → model / design | Model explorer and designer, two sections where they were two modes of one pane |
| Command | interpretability.open | Inspect context window (opens the suite's Context section) |
| Command | interpretability.openDiagram | Explore model structure (opens the suite's Model section) |
| Command | interpretability.guide | Opens the suite's Home, which replaced the Compass modal |
| Keybinding | designer verbs | Scoped paneFocus == 'interpretability.suite' && paneSection == 'design' && !textInput |
| Keybinding | mod+shift+x | → interpretability.open |
| Setting | interpretability.modelRepo | HF repo for exact tokens and the description |
| Setting | interpretability.ggufPath | Explicit weights file for the tensor inventory |
interpretability.budget was a compact role: 'tool' widget duplicating the
context panel's header. It is gone; the budget bar (and its "tools dropped" chip)
lives in the context view, which was always one region-strip click away.
RENAMED_VIEWS in layout/serialize.ts maps the retired id onto the suite's Context section, so a
workspace saved before the merge opens without a hole. interpretability.openDiagram
keeps its id through the rename — a command id is an API even when its title isn't.
The context window and the model explorer (and designer) are sections of the
interpretability suite. The panel ids above are retired and
resolve there through RENAMED_VIEWS and VIEW_ALIASES; the commands keep their
ids. The suite keeps visited sections mounted, so the context view's live /ws
subscription survives switching tabs.
Both panes are role: 'document': by the test in
windowing.mdx they are surfaces you read and work in,
not tiles you glance at.
Backend surface
| Method | Path | Purpose |
|---|---|---|
| GET | /api/interpretability/turns | Recent captured turns from the ring, newest first |
| GET | /api/interpretability/turns/history | Turn summaries from the durable table (limit, agent_id, since, roots_only) |
| GET | /api/interpretability/turns/{turn_id} | One turn in full — ring first, then the table |
| GET | /api/interpretability/turns/{turn_id}/requests | The turn's provider calls as summaries — literal, else rebuilt from the snapshot. See Literal requests |
| GET | /api/interpretability/requests/{request_id} | One provider call in full: messages, provenance segments, tools, params, response, usage |
| DELETE | /api/interpretability/turns | Clear the ring |
| GET | /api/interpretability/model | Model metadata + true context length |
| GET | /api/interpretability/architecture | Normalized model structure (the description) |
| GET | /api/interpretability/tensors | The GGUF tensor inventory; no fallback source |
| GET | /api/interpretability/attention | The per-layer attention layout from the GGUF header; no fallback source |
| GET | /api/interpretability/graph/specs | The designer's node catalog and templates |
| POST | /api/interpretability/graph/code | A design graph → a PyTorch module (stateless) |
| POST | /api/interpretability/graph/validate | Shape inference and parameter counts (stateless) |
| POST | /api/interpretability/graph/parse | A PyTorch module → a design graph (the round trip in) |
| POST | /api/interpretability/graph/probe | Build and run the module in a training project's venv |
| GET/PUT/DELETE | /api/interpretability/graph/{name} | A saved design |
/turns/history is declared before /turns/{turn_id}: FastAPI matches routes in
declaration order, and the other way round history resolves as a turn id — a 404 on
a route that exists. A summary carries no context blocks, only metadata and a round
count; content is fetched one turn at a time, which is also the boundary the MCP
export redacts at.
/api/interpretability/model reads Ollama's /api/show. The OpenAI dialect (LM
Studio, vLLM) has no equivalent, so it returns an error the pane renders as
"context length unknown" rather than guessing a number the budget bar would then
be wrong about.
The window a turn actually ran under
requestedNumCtx is what we ask for; modelContextLength is what the model has, and
the gap between them is the difference between "my prompt fit" and "my prompt was
silently truncated". Only the server knows, so run_agent_loop asks once as the turn
closes (window.context_length → recorder.finish_turn) and stamps it on the turn.
Every server reports it somewhere different, so the probe branches on the provider
kind, not just the dialect — asking /api/show alone would have made this dead on
an LM Studio box, which is the common local setup here:
| kind | asked | read |
|---|---|---|
ollama | POST /api/show | the *.context_length metadata key |
llamacpp | GET /props | default_generation_settings.n_ctx |
lmstudio | GET /api/v0/models/{id} | loaded_context_length, else max_context_length |
vllm | GET /v1/models | that model's max_model_len |
A hosted litellm provider is not probed at all: no endpoint of ours reports a remote
model's window, and a guess here would be rendered as a measurement. The probe runs in
the loop's finally on every turn, so it is capped at a 2s timeout and cached for
five minutes — negative answers included, since a server without that endpoint will not
grow one. It is cached rather than permanent because reloading a model at a different
n_ctx is a normal thing to do, and a stale denominator is the exact lie this is here
to stop telling.
The live path is the interpretability /ws channel, which pushes a round
event per round as it happens; the HTTP routes back-fill the pane on open.
Browser vs desktop
| Browser | Desktop | |
|---|---|---|
| Pane + live capture | ✅ | ✅ |
| Exact token counts | ✅ (needs one tokenizer fetch) | ✅ |
| Offline / no HF access | estimates, flagged | estimates, flagged |
No platform branching — the pane is the same component in both layouts.
Durable turns (agent_turns)
The recorder writes each turn through to an agent_turns table in app.db
(store.py) as well as into its ring. The two serve different jobs and neither
replaces the other:
Ring (recorder.py) | Table (store.py) | |
|---|---|---|
| Holds | 25 turns | 5000 turns (pruned) |
| Survives restart | no | yes |
| Read by | the pane | historical queries, the MCP export |
The pane never reads the database, so a slow or failed write cannot stall a turn. Every
store call swallows and logs its own errors for the same reason: observation must not
break the thing it observes, which is the same rule capture_round already follows.
Rounds are stored as a JSON blob rather than a child table — they are only ever read
whole, and a relational rounds table would need a migration every time
RoundSnapshot gains a field.
One subtlety worth knowing when querying it: total_tokens is the last round's
total, not the sum across rounds. Rounds are cumulative — each resends the whole
conversation plus whatever the loop appended — so summing them would multiply-count the
same context and report a turn as costing several times what it did.
agent_turns is pruned to 5000 rows by store.prune(), which runs from
requests.forget_turn every 200 finished turns. Before literal capture landed, nothing
called it.
Literal requests (ctx_requests)
A RoundSnapshot is a human-readable account of a round: blocks labelled by where they
came from, clipped, with tool_calls folded into text. It is not what the provider
received. Just before a request goes out, normalize_system_messages joins every
leading system message into one and turns a later system message (the forced-tool
nudge, the workspace index) into user. The forced-tool retry is a second provider
call inside the same round, and a snapshot has no place for it.
requests.py records the request itself. providers.chat_stream and providers.chat
call begin() right after normalizing, and finish() once the call returns or raises.
Each dialect path calls note_payload() with the body it actually sent, so the OpenAI
path's retry without stream_options after a 400 records the request the server
accepted. Nothing is recorded outside a turn: the hook reads (turn_id, round) from
telemetry/turn.py.
- One row per provider call.
attemptcounts calls within a round, so a forced retry isR2 · retry. A failed call is recorded withstatus: error, and the error still propagates. - Provenance survives the flattening.
normalize_system_messages_mappedreturns, for each message it sends, the input messages it was built from and their character offsets. The recorder's labels (kinds_for) ride on those offsets assegments, so "Skill catalog" is a span inside the one system message rather than a role. A message whose role changed carriesfromRole. - Bodies are content-addressed. Each message and each
toolsarray is a row inctx_bodies, keyed by a sha1 of its canonical JSON. A request stores only the list of hashes, so ten rounds that re-send the same history store it once. Tokens are counted once per body; the totals are estimates, beside the provider's reportedprompt_tokens. - Stored verbatim. Credentials travel in headers, not bodies. The one exception,
litellm's
api_keykeyword, is kept out ofparams. - Same retention as
agent_turns.prune()deletes requests whose turn has gone, then removes any body that no request references.
Each request pushes an interpretability /ws frame with event request, once when it
is sent and again when it finishes. A turn recorded before this existed has no rows. For
those turns both routes rebuild one request per round from the snapshot, using
agentpedia.rebuild, mark it reconstructed: true and list what the rebuild could not
know. The rebuild's id is r:{turn_id}:{round}.
Limits
- The pane's ring is in-process. 25 turns, cleared on backend restart. The ring
counts delegated sub-turns and peer nodes as entries, so a heavily-delegating session
fills it faster; an orphaned child (parent already evicted) is promoted to a root
rather than dropped. Turns are now also persisted to
app.db(see below), so what the pane shows is bounded while the underlying history is not. - A peer's context is never visible here. By design — see above.
- Block previews are clipped for transport. Conversation and tool results are cut at 4000 characters. Harness text is cut at 65536: the system prompt, the skill catalog, tool guides, the workspace index, the editor buffer, and loaded skill and tool-group bodies. That text is what a reader comes to learn from. The ring does not multiply it, because an unclipped block keeps the message's own string, so every round of a turn shares one copy. The persisted snapshot does repeat it once per round. Token counts always describe the full text, and clipped blocks say so.
- Counts are of what we send. Ollama applies its own chat template on top, so the true prompt is marginally larger than the sum of the blocks.