Skip to main content

Module: interpretability

See exactly what the model is being handed. The agent loop rebuilds its context every round — system prompt, tool guides, replayed history, the focused editor buffer, the user turn, and a tool list that progressive disclosure recomputes as the model calls load_tools — and until this module none of it was observable.

Status: implemented — frontend in packages/core/src/modules/interpretability/, backend in backend/modules/interpretability/, capture hook in backend/modules/agent/orchestrator.py.

What it shows, and why that's the useful part​

The interesting question about a local model is rarely "what did it say" — it's "what did it actually see". Three things drive agent behaviour that were previously invisible:

  • The tool schemas. Serialized tool JSON is usually the single largest block of an agent's context, and progressive disclosure changes it every round. The pane breaks it down per group and per tool, in real tokens.
  • Silent tool-budget truncation. _select_tools caps the list at TOOL_BUDGET (44) and drops the overflow with nothing but a logger.warning. If the model "ignored" a tool you were sure it had, this is why — the pane says so in a full-width warning, with the dropped count.
  • The gap between requested and real context length. num_ctx is what we ask for; the model has what it has. Asking for more gets you the model's real limit silently. The header shows both.

Prompt, weights, activations — three surfaces, three modules​

This module inspects the prompt. That is a real boundary and not a temporary one: activations, attention, KV cache contents and gradients are unavailable through the HTTP API of every provider it can talk to, which returns tokens and counts and nothing else. Everything on this page is measured from the request the backend is about to send.

The model explorer covers the weights: reading the GGUF directly gives the real tensor inventory — every name, shape and quantization — so the structure shown is measured rather than described. It stops at the file; nothing here runs a forward pass.

The computation between them is now covered too, and it lives next door in llamacpp rather than here, because it needs a different engine and a different install. A traced run captures ggml's per-node cb_eval tensors into a snapshot on disk, scrubbed offline rather than streamed (one pass with attention on is about a gigabyte), and is read in the traces section of the llama.cpp pane.

That page also now carries the lens — the same snapshot read as words rather than as numbers, a layer × position grid of what the model was disposed to say at each point, which is Neuronpedia's Jacobian Lens with the identity transport. It needs no autograd and no fitted artifact, so the rule below still holds: this backend has no torch and should not grow one. Its correctness argument is the same shape as everything else here — at the last layer the lens must reproduce the trace's own captured logits, and a grid that cannot show that says unavailable rather than presenting itself as checked. That path still avoids autograd entirely — the backend has no torch and should not grow one; heavy dependencies live in per-project uv envs (backend/modules/training/envs.py) or optional extras, and tracing is uv sync --extra llamacpp, lazy-imported and CPU-only.

The seam between this page and that one matters: a trace and a chat turn are different runs until proven otherwise. Chat goes through a downloaded llama-server binary, tracing through the wheel's own libllama, so the two may only be shown together when both the build and the model hash agree (traces.matches_run). Overlaying them regardless would put a forward pass under an answer it did not produce — which is exactly the kind of confident wrongness this module exists to refuse.

Research Compass & Connected Workflow​

To make the dashboard approachable for beginners and fast for researchers, both interpretability panes host a unified Workflow Header and the Research Compass (interpretability.guide command or the 🧭 Compass button):

  1. Context Window (Prompt): interpretability.context — measures exact token counts, prompt block kinds (system, history, buffer, tools), and tool budget truncation. Includes an onboarding launchpad with prompt suggestions and quick navigation when no turns have been captured.
  2. Model Explorer (Weights): interpretability.architecture — inspects real GGUF tensor directories, quantizations (Q4_K_M, Q8_0), and architecture specs. Features inline glossary tooltips for technical metrics (GQA vs MHA, KV cache footprint, SwiGLU gating, RoPE θ, MoE active fractions) and empty-state links to Hugging Face and llama.cpp.
  3. Logit Lens & Stepper (Activations): llamacpp.server — dynamic layer representations and execution stepping.

The 🧭 Compass modal maps the 7 stages of AI research (Literature, Datasets, Context Window, Model Explorer, Logit Lens, Evals, Training), answering which question each pane answers, when to use it, and common beginner pitfalls.

Merge interpretability.architecture and llamacpp.server into one window to keep prompt, weights, and activations within immediate reach.

Architecture​

Capture is a read-only observer inside the agent loop. It runs once per round, just before the provider call, and is wrapped so that a failed capture can never cost the user their turn.

Two rings and a table​

The ring is what the pane opens on: 25 turns, in process, gone on restart. The agent_turns table behind it is what makes a turn answerable tomorrow — 5000 rows, each the full snapshot. Both are written by the same capture_round; they differ only in how long they last, so GET /turns/{turn_id} tries the ring and then falls back to the table. Without that fallback every turn older than the ring lists in the history and 404s when opened.

Both halves are read together by agentpedia, which steps one turn through what it was shown, what it sent, and what it did.

The third ring in the diagram is the telemetry one, and it is a different kind of record: IoEvent is what went over the wire, where a RoundSnapshot is what the model was shown. run_agent_loop stamps every request it makes with turn_id and round (telemetry/turn.py, a contextvar rather than a parameter — the recording happens in httpx's response hook, several layers below the loop that knows the answer), which is what lets the two be lined up without matching them by timestamp. A sub-turn restores its parent's stamp when it leaves rather than clearing it.

Two invariants hold the design together, both enforced in tests:

  1. Capture never raises. It runs inside the turn; a bad message list costs a snapshot, never an answer.
  2. Capture never mutates. It only reads messages/tools. The pane's entire premise is that it shows the real prompt, so observing it cannot change it.

Multi-agent: delegation and peers​

main is not the only agent that runs. It orchestrates two different ways, and the pane distinguishes them because they have very different visibility.

agent.delegateagent.ask_peer
Runs onthis node, this browser connectionanother user's node
Toolsfull — relayed to your frontend, gated under the sub-agent's modenone (tool-less, read-only)
Depthone level (specialists set can_delegate: false)n/a
Visible in this panefully — it's a real captured turnopaque node only

A delegated turn needs no special capture: run_delegate reuses orchestrator.run_agent_loop, which is exactly where the capture hook lives, so the sub-agent is recorded like any other turn with its own agentId. What it did need was the link — parent_turn_id is threaded from run_delegate into the loop so the pane can rebuild the handoff tree instead of showing unrelated siblings.

Each turn carries the identity its context was actually assembled from: agent name, declared toolGroups scope, and permission mode. Scope is the operative field — a specialist can only ever load tools from its declared groups, so "the agent ignored my request" is frequently really "that agent was never given that capability", and the pane can now show which.

toolGroups: null ≠ []

null means unrestricted — every group loadable, which only main is. An empty list means the opposite: no groups at all. They must never be collapsed.

Why peers stay opaque​

A peer turn is recorded with its peerId and the prompt that was sent, and nothing else — no rounds, no token counts. The peer's agent assembles its context on its own machine, and this node genuinely cannot see it (nor should it; see network). Recording the handoff anyway is what keeps the tree honest: a delegation that leaves the node shows up as a leaf that says so, rather than a silent gap the reader has to guess about.

Model explorer​

Beside the context view sits the loaded model: what it is, next to what it was given. It is a thing you click through rather than a picture — selecting a stage of the stack shows the tensors that actually implement it.

The block is drawn once with a ×N multiplier rather than repeated — forty-eight identical rectangles say nothing the number doesn't — but the multiplier expands into a layer picker, so "block 17 specifically" is one click away.

Inspect is a canvas, and the stack is one node​

Inspect used to be a vertical column of <button> rows inside a CSS grid track declared minmax(190px, 260px). That single line was the whole problem: the diagram could not exceed 260px in a pane of any width, there was no zoom, and the residual — the one connection worth drawing — was a text row reading ⊕ residual that connected nothing to nothing.

It is now a read-only @xyflow/react canvas (inspect/InspectCanvas.tsx) with pan, zoom, a minimap and two fit controls, sitting inside a resizable split.

The node model is O(1) in the layer count. A 48-layer model has ~290 sublayers, and drawing them would be unreadable and a lie: the interesting fact about a stack of 48 decoder blocks is that they are identical, and 48 drawn boxes assert the opposite. So the stack is one node carrying a rail of N ticks, and exactly one block is ever expanded — the selected one. Six to eight nodes, ever, plus the rail.

The rail earns its place twice over. Tick height is that block's real bytes from the GGUF inventory, so a model whose later blocks are heavier says so. Tick band is its tensor-shape signature, computed by layerSignatures, so a model that alternates its attention shape by layer — Gemma 4's blk.0.attn_q is 3840×4096 while blk.17.attn_q is 3840×8192 — draws as stripes instead of as a grey varies chip with a tooltip. The GGUF inventory is the only source that can see this at all: every scalar source, GGUF metadata and a repo's config.json alike, reports a single head count for the whole model.

Selection {stage, layer} is unchanged and remains the single source of truth; inspect/graph.ts only provides a bijection to node ids, so the canvas is a projection of the selection rather than a second copy of it. That is what let the model-locus wiring survive the rewrite with no edit at all — the lens still publishes a layer, the explorer still selects and opens that block, and the canvas now additionally brings it into view. The focus is driven by a nonce rather than the layer value, so clicking layer 15 in the lens twice re-centres instead of being a no-op.

Layout is dagre with rankdir: 'TB', in a file deliberately separate from the designer's LR one. designer/layout.ts already argues both sides: LR because node editors read left to right, and TB for training/panels/ModelGraphPane because "a traced module tree reads as a stack". Inspect is the second case by that file's own criterion — and every canonical transformer figure is vertical, the residual only reads as a spine when the block runs down the page, and the left column is now narrow by default so the axis with room to spare is the vertical one.

The third importer of the graph engine

flow/canvas/FlowCanvas.tsx said it was the only file importing @xyflow/react, and designer/ModelCanvas.tsx argued its way in as the second. InspectCanvas makes the same argument a third time in its own header rather than assuming it: ModelCanvas is built around the designer's editable GraphNode — typed sockets, socketsCompatible refusing a link at the wire, onConnect, frame membership, a persisted layout sidecar. Inspect's nodes are read-only, derived, socketless and unpersisted, so reusing it would mean every one of those invariants growing a "not in inspect mode" branch. What the rule protects — the node model is the API, the engine stays swappable — still holds: nothing outside that file imports the engine.

Zoom means two things, because the wheel already changes the zoom factor and a control that did the same would be redundant. Fit is the whole model and the way back. Focus is the selected node and what it connects to, on double-click — the useful verb on a graph is "show me this and its neighbours", not "make it 1.4×". On top of that the nodes read the viewport zoom and drop their fact rows below ~0.62, so zooming out shows the model's shape and zooming in reads its numbers.

Both sides resize now​

SplitPane (packages/core/src/SplitPane.tsx) is a shared in-pane splitter, applied to Inspect's two columns and to the designer's palette and inspector — which were width: 176px and width: 320px, equally unresizable.

It is not packages/ui/src/layout/Sash.tsx: that one takes a SplitNode and dispatches SET_SPLIT_SIZES to layoutStore because it divides the workspace. A pane's own columns are not part of the layout tree, and core cannot import ui anyway. Sizes live in localStorage (split-sizes.ts, mirroring llamacpp/pins.ts) rather than in the layout store, which also keeps them out of an exported workspace — opening somebody else's workspace should not resize the insides of your panes.

Two things in it are load-bearing:

  • The drag does not setState. A pointermove at 60 Hz re-rendering both children makes the React Flow canvas rebuild its node list sixty times a second. The live size is written straight to a --split-size custom property and React is told once, on pointerup.
  • narrowBelow measures the container, not the viewport. It replaces a @media (width <= 720px) rule that was wrong by construction: a pane docked in a three-column workspace is narrow at any viewport width, and a maximised pane is wide on a small screen.

Two sources, and why they stay separate​

SourceAvailable forWhat it gives
The model's own GGUF headerthe node's own llama.cpp server, Ollama, LM Studio, or any server with interpretability.ggufPath setThe inventory: every tensor's name, shape, ggml quant type and byte size — and, read the same way, the description and the per-layer attention layout
GGUF model_info via /api/showOllamaThe description, written from the weights the server loaded
config.json from a HF repoany server whose weights file cannot be foundThe description, but strictly of the repo

The pane labels which it used (weights / repo, plus a gguf chip when the inventory is present) rather than presenting them interchangeably.

/architecture reads the header itself (architecture.from_gguf_header) whenever the weights file can be located — which is what an LM Studio model gets now. It used to fall straight through to the repo's config.json for any server without a metadata endpoint, describing the repo while the file it was converted into sat readable on disk. Gaps are still filled from the repo the GGUF names as its base model, never overriding a stated value. All three header readers share gguf.read_header_cached, keyed on path, size and mtime so a re-downloaded file under the same name is re-read.

The split is load-bearing. A description can be approximated from a repo when the weights are unreachable; an inventory cannot — a config.json lists no tensors — so /tensors returns an error the pane renders as "no GGUF found" rather than a plausible-looking list. A model can have the description without the inventory, and the pane stays useful minus the tensor tables.

What the inventory catches that a schematic can't​

Every scalar source reports one head count for the whole model. That is often false. Gemma 4 alternates its attention shape by layer — blk.0.attn_q is 3840×4096 where blk.17.attn_q is 3840×8192 — so a "× 48" with a single head count beside it actively misleads. The explorer compares tensor shapes across blocks and, when they differ, marks the block frame varies and names the roles that change. Only the inventory can see this.

A GGUF can also store its head counts per block — a hybrid (Jamba, Falcon-H1) writes attention.head_count as an array with zeros on its state-space blocks. architecture._int used to turn that array into None, so such a model drew no attention at all. _scalar_or_max keeps the array as headsPerLayer / kvHeadsPerLayer, reports the maximum as the headline with a note saying so, and the attention node's subtitle uses the selected block's own count ("no attention in this block" where it is zero). A uniform array collapses to its scalar.

With the per-layer attention layout loaded, the rail also carries each block's mixer: an accent cap on the global layers, a dashed cap on blocks with no attention at all (state-space, linear, convolution), none on the windowed majority. The open block's attention node is labelled by its own mixer — "Sliding-window attention · window 512 · KV from L13" — rather than by the model's one head count. Shape bands cannot do this alone: Gemma 3's windowed and global layers have identical tensor shapes.

Following the layer stepper​

The explorer is the map; the llama.cpp stepper is a position on it. While a trace is being stepped, the model locus carries the stepper's stage and step as well as the layer, and the explorer selects that block, lights the sub-block executing (attention, FFN, experts, norm — mapped in inspect/stepper-stage.ts, because the two panes share the bus and nothing else), and shows a following the stepper strip with a way back to it. The canvas re-centres only when the block changes, so stepping from attention to FFN moves a highlight rather than refitting the diagram under the reader.

It works in reverse too: while the stepper is driving, clicking a block or a sub-block on the diagram publishes a locus and the stepper's cursor jumps there.

It also reports the mixed quantization a K-quant build actually uses (a Gemma 4 12B QAT build is 338 F32 tensors, 328 Q4_0, and exactly one Q6_K — the embedding), and where the bytes sit: select Attention and the pane says what share of the file it is.

The window's logit lens​

The explorer's lens so far has been the llama.cpp trace grid: a trace the node captured, read as words. The second source is the model running in this window. When a GGUF generates with the logit lens on (the playground's webml.lens, or a {webllm} with :show: lens; see WebML), the engine reads out every layer at every token, and the explorer draws the latest such reply under Whole model and under a selected block (inspect/WindowLens.tsx).

  • A layer × token grid, layer 0 at the top, the reply's last 32 tokens across. Each cell is the token the LM head would pick from the residual stream after that layer, shaded by its probability. Where it is the token finally chosen, it is in the accent, so a prediction settling reads as the accent running up a column to the last row. Hovering a cell gives the probability, the entropy and the stream's norm.
  • A settles footer: the first layer from which the chosen token leads at that layer and every later one. A dash means the last layer preferred another token (a sampled runner-up).
  • It fills while the reply is written. The window's engine keeps the run (webml/lens-run.ts) whichever surface asked for it, so neither the playground nor {webllm} reports in. One run is kept, the newest, cut to its last 512 tokens.
  • Rows link to blocks only when the layer counts agree. The explorer follows the node's model, and the window's is often another one, so the grid always names its own model. When the counts match, clicking a layer selects and opens that block and the selected block's row is outlined. When they differ, a note says so and the rows are not links. With no node model loaded, the empty state still shows the grid, unlinked.

Locating the weights​

gguf.resolve_model_path tries, in order:

  1. interpretability.ggufPath, which wins outright. Auto-discovery reads two undocumented on-disk layouts their owners are free to change, so there has to be a way to say "it is this file" that no upstream release can break.
  2. The node's own llama.cpp server, when one is running and its --alias matches the configured model. This is the only source that is known rather than inferred: we chose the file. It is matched on the alias and not merely on "a server is running", because llama-server can be up while the agent talks to Ollama, and answering with the wrong model's tensors is exactly the plausible-looking wrongness this module exists to prevent.
  3. Ollama's blob store — its manifest names the layer whose mediaType is …image.model; that digest is the blob. OLLAMA_MODELS relocates the root.
  4. LM Studio's index (~/.lmstudio/.internal/model-index-cache.json), where indexedModelIdentifier is exactly the id its API serves. Its REST API reports a model's arch and quantization but not its path, so the index is the only route. Taking entryPoint matters: a multimodal repo ships an mmproj-*.gguf beside the weights, and the first file in the directory is a real GGUF that parses cleanly and describes the wrong thing.

Step 4 is not gated on the provider being LM Studio — a third-party llama.cpp build and other OpenAI-dialect servers are often pointed at a model LM Studio also has indexed, and a hit there is a hit on the same file.

A byte size is not derivable from a shape

A quantized tensor's footprint depends on its block format, so gguf.py carries a (elements per block, bytes per block) table. An unrecognized ggml type reports no size rather than a guessed one, and the response's bytesComplete goes false so the total reads as a floor. The table's correctness argument is that on a real mixed-quant file, sum(tensor bytes) equals file_size - data_offset to the byte.

GGUF keys are namespaced per architecture (gemma2.block_count, llama.attention.head_count_kv), so they're matched by suffix. A new family works without a table entry.

Three-state, not two

gated is true for known-gated families (SwiGLU/GeGLU), false for known-dense ones, and null for a family in neither list. "We don't know" and "we know it isn't" are different claims, and only the second justifies drawing a single-projection FFN. The same reasoning governs every optional field.

Model designer​

Beside Inspect sits Design: the same pane, switched from reading a model to drawing one. It is a Blender-shaped node editor — drag RMSNorm, Attention, SwiGLU, wire a residual Add back around a block, stack it ×N — and a real PyTorch nn.Module subclass materialises beside it on every edit.

The two modes share a pane because of the bridge between them. Inspect answers "what is this model"; Design answers "what if it were different", and the obvious way to start a design is from the thing you are currently reading. Putting a workspace switch between a question and its follow-up would be the wrong seam.

What is taken from Blender, and what is refused​

BlenderHere
Colour-coded typed socketstensor / int / float / bool; a link between different types is refused at the wire
Node groups (Group input/output)a group is a generated nn.Module subclass — the reason the metaphor fits at all
A socket that accepts many linksop.add, so a residual can fold several branches; every other input takes exactly one
Mute a node (M)ablation: it emits nothing, its input passes through, and it stops being counted
Reroute nodes, framesthe same, and purely cosmetic — codegen walks straight through them
Timings overlaya cost overlay: each node's share of the parameter count
Left→right data flowenforced by the dagre auto-layout (rankdir: 'LR')

Not taken: Blender's implicit socket conversions. A float becoming a colour is a convenience; a tensor silently becoming a differently-shaped tensor is the class of bug this module exists to refuse, so a mismatch is a red wire you fix with an explicit Reshape node.

The two things it shows that a drawn diagram cannot​

The shape on every wire. [B, T, 2048], propagated from the Input node in pure Python, in under a millisecond, on every keystroke. Batch and sequence stay symbolic, because a symbolic dimension compared against a concrete one is an unknown rather than a conflict.

Where the parameters actually are. Change d_model from 512 to 256 and the count drops from 54.9M to 26.2M, every downstream shape re-labels, and the generated __init__ signature changes with it — because config values stay variables in the emitted class rather than being frozen into it.

Estimated until something runs it

Those numbers are our arithmetic, and the pane chips them estimated. The backend has no torch and should not grow one, so measuring the real thing means instantiating the module in a training project's venv — a separate, slower question, and a separate answer. Where they disagree, the estimate is the one that is wrong.

The code pane is highlighted, per line​

Read mode is syntax-highlighted by designer/highlight.ts, which parses the generated module with the same lezer grammar CodeMirror uses and hands back one token array per line. Spans go inside the existing per-line DOM rather than replacing it with an editor view, because the pane does two things per line that an editor would have to re-derive as decorations: light the lines a hovered node generated, and render the gutter number.

Two things are bugs if missed, and both are pinned by tests:

  • A styled range can span newlines (a docstring, a bracketed expression), so every emitted run is split on \n. Otherwise the line count moves and every # horrible:node= marker points at the wrong line.
  • Highlighting may not alter the text. highlightTree only reports styled ranges; the gaps between them are real source and must be emitted too. The test rejoins every token and compares against the original — a dropped gap would still render as plausible-looking code.

Edit mode stays a plain textarea. A textarea overlaid on a highlighted <pre> is the prettier trick and it depends on two elements agreeing about font metrics to the pixel; when they disagree the caret drifts from the text, which is a far worse failure than a button press.

Edits go both ways​

The code pane is editable. Change a number in the generated module, hit Apply, and it lands on the canvas — because the .py is the structural source of truth and the graph is a projection of it.

Three rules make that safe rather than merely clever:

  • The sync is explicit. Ctrl-S or Apply, never per keystroke. Parsing mid-word would replace your graph with whatever half-typed source currently means.
  • Your positions survive. Every emitted line carries # horrible:node=<id>, so re-reading the file recovers node identity and the layout sidecar still describes the graph. A round trip that re-arranged the canvas would make the pane useless for anything but a glance.
  • A file we cannot read changes nothing. It is read whole or not at all: the graph stays, your edit stays, and the reason is shown. Half a graph silently replacing a whole one is the worst outcome available here.

Source that does not match the shape the generator emits is not refused — it is preserved verbatim as a custom.module node, and the pane names the classes it kept that way. An opaque import nobody is told about is indistinguishable from a wrong one.

The file is the fixed point

The guarantee under test is emit(parse(emit(g))) == emit(g), byte for byte, for every template. It is a stronger promise than IR equality in the way that matters: parse and regenerate and nothing churns, so a save never looks like a change you did not make.

Blocks are editable, not just instantiable​

A group is one generated nn.Module subclass, so editing one is editing a class. The breadcrumb under the toolbar is Blender's context path — the model, then whichever groups you have stepped into. Double-click a block (or Edit contents in the inspector) to go in; click a crumb to come back out. The canvas is handed one level and knows nothing about groups; the rules live in designer/scope.ts as pure functions.

Two ways to make one:

  • New group drops an empty one here — already wired input to output, because a group whose output has nothing connected to it cannot be generated at all — and steps into it.
  • Group selection is Blender's Ctrl-G: the selected nodes move inside, the wires that crossed the boundary reconnect to the group's own terminals, and an instance lands where the selection was. Select several the way any node editor does — Ctrl/Cmd-click to add one, or Shift-drag a box across the canvas.
A group takes one tensor and returns one

That is what forward(self, x) is, and what the ×N stack depends on. So a selection that would need two inputs, or that feeds the rest of the graph from two nodes, is refused with the count rather than wired to whichever one came first. Widening the selection is the fix. For the same reason, the graph's own Input and Output nodes can never go inside a group — they are the boundary it sits within, and it brings its own.

Renaming a group renames its class, and two groups compiling to the same class name are refused — the second definition would silently replace the first, and every instance of the first would quietly start running the second's code. A group nothing instantiates still shows shapes on its wires while you build it, but its parameters stay out of the headline count: a class the model never instantiates holds no weights. The model panel lists every group with its instance count and offers to delete the unused ones, since they are still written into the file.

Forking the model you are looking at​

The inspected model sits at the top of the palette's starting points, named after whatever the Inspect tab is showing. It turns that model's GGUF (or config.json) metadata into an editable graph — the bridge the two modes share a pane for.

The difficulty is that ModelArchitecture is deliberately full of holes: a dimension the metadata did not state stays null rather than being guessed. A graph has no holes. So the import splits the difference along one line:

Width, depth, vocabulary, head countRequired. Missing any of them, there is no import — and the reason names which. A hidden size we invented would produce a plausible model of nothing
Gated or dense, RMSNorm or LayerNorm, RoPE or notAssumed when absent, and every assumption is listed in a panel that stays until you close it. These only shape the graph, and the result is a starting point you edit, not a measurement
The import audits itself

The panel ends with the parameter count our arithmetic derives beside the one the metadata states. On Llama 3.2 3B those differ by 12% — and the whole difference is the output head, which that model ties to its embedding while the graph gives it its own matrix. So the import says that, in those words. A gap it cannot account for is reported as exactly that: a wrong import that stays quiet is worse than one that says it does not add up.

Two RoPE-shaped caveats worth knowing. Absent rotary parameters are ambiguous — they mean either "this model has none" or "the metadata omitted them" — and the import assumes RoPE anyway, because a decoder with no positional encoding at all is certainly wrong. And head width is left derived (hidden ÷ heads) unless the metadata explicitly decoupled them, which keeps the generated class parametric; a model like Gemma that decouples them is called out in the panel.

Running it for real​

Everything above is the pane's own arithmetic. Run it builds the generated module in a training project's venv and runs one batch through it — the only thing that can turn an estimate into a measurement. It reports three states, never two:

What it means
RanOutput shape, parameter count and torch version, measured. If the measurement disagrees with the estimate, the pane says so and by how much — the measurement is right and the cost overlay was wrong
FailedThe traceback, verbatim. Not summarised: the traceback is the answer, and paraphrasing throws away the line number
Not checkedNo project, no venv, or no torch — said out loud, with the reason, and with a reminder that the numbers are still estimates

That third state is the point. A model reported as validated when nothing validated it is worse than not offering the button at all — the same rule the hardware probe exists to enforce.

The probe runs as a subprocess in the project's venv, never in the backend, which has no torch and should not grow one. The script it runs is self-contained and does not import horrible_train: an evals-owned project carries only its benchmark's requirements, and a probe that failed on a technicality about our own helper package would read as the model being broken.

The estimate is checked against the real thing in CI

test_model_graph.py builds each template in real PyTorch and asserts sum(p.numel()) equals what shapes.py predicted. All three currently agree to the parameter — including the mixture-of-experts one. The test skips when torch is absent, because the backend genuinely does not have it.

Tracing a module that already exists​

The other importer reads a description of a model; Trace reads the model. torch.fx.symbolic_trace records what a forward actually does, in order, with real propagated shapes, and that becomes an editable graph. It runs in a training project's venv, like the probe and for the same reason.

A trace is in torch's vocabulary, not ours, so some of it will not map onto the twenty-one node types. Those arrive as custom.module placeholders that raise NotImplementedError, named after the class they stand for.

A pass-through stub would be the worst possible default

The tempting placeholder returns its input unchanged. It would compile, run, train — and be a different model than the one you traced, wrong in a way nothing would ever report. Raising means the design refuses to run until you fill in the blanks, which is worth far more than one that quietly computes something else. The pane names every placeholder rather than only counting them.

Keyboard​

The canvas takes Blender's verbs, declared through the keymap — never a keydown listener in a component:

Shift-Afocus the node search
Mmute (ablate) the selected node
Xdelete it
Hcollapse it
Homeframe everything
Tab / Shift-Tabenter / leave a group
Ctrl-Gfold the selection into a group

Every one is scoped to the pane and guarded with !textInput: unguarded, typing a d_model value in the inspector would delete the selected node. A binding naming paneFocus already beats an unscoped global, so the pane does not take capture — which matters, because the same pane hosts Inspect mode, where swallowing the keyboard would be wrong.

Reroutes and frames​

Two more borrowings, both deliberately weightless. A reroute is a bend in a wire, drawn as a dot rather than a titled box — codegen walks straight through it, and a box would look like a node that does something. A frame is a labelled rectangle behind a set of nodes: pure cosmetics, living entirely in the layout sidecar, so the generated module is byte-identical before and after you draw one. Its rectangle is derived from where its members currently are, never stored, so dragging a member re-draws the frame around it and there is never a second competing fact about where the frame is.

Handing it to training​

Send to project writes the design into a training project as model.py and adds a notebook block that imports and instantiates it. No new execution path — the project's own kernel, venv and Kaggle/Colab push all apply unchanged.

Two rules keep it from colliding with the recipe machinery that writes into the same notebook. The block carries its own marker (horrible_model, not horrible_recipe), because sharing the flag would make a recipe regenerate delete the model definition and a model regenerate delete the trainer config — each silently. And it lands above the recipe block, because a notebook where the trainer is configured above the model it trains does not run.

Where a design lives​

Two files under $HORRIBLE_DATA_DIR/model-graphs/: <name>.py — the generated module, importing nothing from this app, yours to take anywhere — and <name>.graph.json, holding the graph plus the canvas cosmetics Python has nowhere to put. Losing the sidecar costs you a tidy layout; losing the .py costs you the model.

Block classification​

The provider payload is an undifferentiated list of role/content dicts, and five distinct parts of the prompt all arrive as role: system. Callers that assemble a chat prompt (run_agent_turn and run_delegate) build it with orchestrator.assemble_prompt. That function returns the messages together with a parallel list of sources, and hands the sources to recorder.set_prompt_sources before the loop's first capture. The sources travel beside the messages and never inside them: an extra key on a provider message is a 400 on some dialects. With sources present, the prompt prefix is labelled exactly.

Callers that pass no sources (evals, flow, the SDK) fall back to _classify_prompt. It recovers the kinds from assembly order plus two content markers: <<<BUFFER, and the skill catalog's header, which is imported from the module that writes it so that rewording the header can't silently relabel the block. The heuristic cannot tell the workspace index from a tool guide, and labels it guides.

KindSourceRole
systemspec.system_promptsystem
guides_guides_message — usage notes for preloaded groupssystem
skills_skills_message — the skill catalogsystem
historyprior conversation replayed into the turnuser / assistant
editor_active_editor_message — the focused buffersystem
userthis turn's promptuser
assistantthe model's output this turn (incl. tool calls)assistant
workspace_workspace_context_message — the other visible panessystem
tool_resulta tool's return valuetool
skill_loadeda use_skill result — the skill body the model asked fortool
tools_loadeda load_tools result — the guides of the groups it loadedtool
mcp_resultan mcp-* tool's return valuetool
nudgethe force-tool retry system messagesystem
dropped_toolsthe [tool budget] note naming groups cut to fitsystem

A tool result's kind is decided by the tool that produced it. The name comes from the message's own tool_name, or failing that from its tool_call_id, matched back to the assistant message that made the call.

Every block also carries hash, a short digest of its full, role-salted text. The same block has the same hash in every round, so a round-by-round diff (the trajectories Agent section) keys on the hash rather than on position. The loop removes and re-appends its own notes, so "index N is new" is not the same claim as "this text is new".

skills is deliberately not folded into guides: their cost profiles are opposites. A guide is paid only on turns where its group is active, while every line of the skill catalog rides every round — so merging them would hide the one block whose growth the user most needs to see.

The prompt/loop boundary is pinned at round 0 so classification stays stable as the loop appends to the list.

Order is a contract

Classification depends on the assembly order in run_agent_turn. Changing that order means changing _classify_prompt in the same edit — a mislabelled block is worse than no pane at all.

Token counting​

Counts come from the model's real tokenizer via tokenizers (Rust, ~5 MB) — not transformers, because the backend env has no torch and must not grow one. A bare tokenizer.json fetched through the already-present huggingface-hub is all that Tokenizer needs.

Which repo (tokenizer.resolve_repo, shared with the architecture diagram's config.json fetch). The tokenizerSource reported on each turn names the step that answered:

  1. setting: the interpretability.modelRepo setting, if set. It wins outright.

  2. weights: the running GGUF's own identity (repo_from_weights), from the strongest claim available:

    • general.source.repo_url, the file naming its own repo;
    • a general.base_model whose repo name equals the file's own basename-size_label-finetune (a plain re-quantization);
    • for a file LM Studio serves, the Hub repo it was downloaded from. LM Studio's models directory is laid out <owner>/<repo>/<file>.gguf, and a GGUF repo has no config.json, so its card's base_model is followed (at most three hops, never through a merge's multi-base list or a link the card calls a fine-tune) to the checkpoint that has one;
    • the file's own name under its base_model's owner, if the Hub has that repo.

    The result is memoized per model, misses included, because capture runs every round.

  3. model: the model id, if it is shaped like owner/name.

  4. family: a same-family default. The pane shows this as "approx".

A GGUF's general.base_model is its parent, not itself. An instruction-tuned Gemma's GGUF names google/gemma-3-1b-pt there, the checkpoint it was fine-tuned from. So base_model alone is never taken as the weights' repo; the file's name parts decide, the same match the interp suite's compatibility check makes for SAEs.

Step 2 runs before step 3 because an LM Studio model id is only repo-shaped. google/gemma-4-12b-qat is LM Studio's catalog key and 401s on the Hub. Taking it as the repo is what used to produce "Could not read config.json" and a chars/4 estimate for a model whose real checkpoint (google/gemma-4-12B-it-qat-q4_0-unquantized) is public. When config.json still can't be read, the error names which step chose the repo.

Auth order for the fetch:

  1. the Hugging Face connector's token, for gated repos (Gemma's are gated, and anonymous fetches 401);
  2. anonymous, which works for ungated families.

If no tokenizer can be had, counts fall back to a chars/4 estimate and the snapshot is marked exact: false. The pane renders that as an "estimated" chip — an estimate displayed as a precise number is exactly the failure this module exists to prevent.

Contributions to the layout shell​

KindIDNotes
Sectioninterpretability.suite → contextThe context window — a section of the interpretability suite
Sectioninterpretability.suite → model / designModel explorer and designer, two sections where they were two modes of one pane
Commandinterpretability.openInspect context window (opens the suite's Context section)
Commandinterpretability.openDiagramExplore model structure (opens the suite's Model section)
Commandinterpretability.guideOpens the suite's Home, which replaced the Compass modal
Keybindingdesigner verbsScoped paneFocus == 'interpretability.suite' && paneSection == 'design' && !textInput
Keybindingmod+shift+x→ interpretability.open
Settinginterpretability.modelRepoHF repo for exact tokens and the description
Settinginterpretability.ggufPathExplicit weights file for the tensor inventory
Layouts saved before the budget widget was merged

interpretability.budget was a compact role: 'tool' widget duplicating the context panel's header. It is gone; the budget bar (and its "tools dropped" chip) lives in the context view, which was always one region-strip click away. RENAMED_VIEWS in layout/serialize.ts maps the retired id onto the suite's Context section, so a workspace saved before the merge opens without a hole. interpretability.openDiagram keeps its id through the rename — a command id is an API even when its title isn't.

Now sections of one window

The context window and the model explorer (and designer) are sections of the interpretability suite. The panel ids above are retired and resolve there through RENAMED_VIEWS and VIEW_ALIASES; the commands keep their ids. The suite keeps visited sections mounted, so the context view's live /ws subscription survives switching tabs.

Both panes are role: 'document': by the test in windowing.mdx they are surfaces you read and work in, not tiles you glance at.

Backend surface​

MethodPathPurpose
GET/api/interpretability/turnsRecent captured turns from the ring, newest first
GET/api/interpretability/turns/historyTurn summaries from the durable table (limit, agent_id, since, roots_only)
GET/api/interpretability/turns/{turn_id}One turn in full — ring first, then the table
GET/api/interpretability/turns/{turn_id}/requestsThe turn's provider calls as summaries — literal, else rebuilt from the snapshot. See Literal requests
GET/api/interpretability/requests/{request_id}One provider call in full: messages, provenance segments, tools, params, response, usage
DELETE/api/interpretability/turnsClear the ring
GET/api/interpretability/modelModel metadata + true context length
GET/api/interpretability/architectureNormalized model structure (the description)
GET/api/interpretability/tensorsThe GGUF tensor inventory; no fallback source
GET/api/interpretability/attentionThe per-layer attention layout from the GGUF header; no fallback source
GET/api/interpretability/graph/specsThe designer's node catalog and templates
POST/api/interpretability/graph/codeA design graph → a PyTorch module (stateless)
POST/api/interpretability/graph/validateShape inference and parameter counts (stateless)
POST/api/interpretability/graph/parseA PyTorch module → a design graph (the round trip in)
POST/api/interpretability/graph/probeBuild and run the module in a training project's venv
GET/PUT/DELETE/api/interpretability/graph/{name}A saved design

/turns/history is declared before /turns/{turn_id}: FastAPI matches routes in declaration order, and the other way round history resolves as a turn id — a 404 on a route that exists. A summary carries no context blocks, only metadata and a round count; content is fetched one turn at a time, which is also the boundary the MCP export redacts at.

/api/interpretability/model reads Ollama's /api/show. The OpenAI dialect (LM Studio, vLLM) has no equivalent, so it returns an error the pane renders as "context length unknown" rather than guessing a number the budget bar would then be wrong about.

The window a turn actually ran under​

requestedNumCtx is what we ask for; modelContextLength is what the model has, and the gap between them is the difference between "my prompt fit" and "my prompt was silently truncated". Only the server knows, so run_agent_loop asks once as the turn closes (window.context_length → recorder.finish_turn) and stamps it on the turn.

Every server reports it somewhere different, so the probe branches on the provider kind, not just the dialect — asking /api/show alone would have made this dead on an LM Studio box, which is the common local setup here:

kindaskedread
ollamaPOST /api/showthe *.context_length metadata key
llamacppGET /propsdefault_generation_settings.n_ctx
lmstudioGET /api/v0/models/{id}loaded_context_length, else max_context_length
vllmGET /v1/modelsthat model's max_model_len

A hosted litellm provider is not probed at all: no endpoint of ours reports a remote model's window, and a guess here would be rendered as a measurement. The probe runs in the loop's finally on every turn, so it is capped at a 2s timeout and cached for five minutes — negative answers included, since a server without that endpoint will not grow one. It is cached rather than permanent because reloading a model at a different n_ctx is a normal thing to do, and a stale denominator is the exact lie this is here to stop telling.

The live path is the interpretability /ws channel, which pushes a round event per round as it happens; the HTTP routes back-fill the pane on open.

Browser vs desktop​

BrowserDesktop
Pane + live capture✅✅
Exact token counts✅ (needs one tokenizer fetch)✅
Offline / no HF accessestimates, flaggedestimates, flagged

No platform branching — the pane is the same component in both layouts.

Durable turns (agent_turns)​

The recorder writes each turn through to an agent_turns table in app.db (store.py) as well as into its ring. The two serve different jobs and neither replaces the other:

Ring (recorder.py)Table (store.py)
Holds25 turns5000 turns (pruned)
Survives restartnoyes
Read bythe panehistorical queries, the MCP export

The pane never reads the database, so a slow or failed write cannot stall a turn. Every store call swallows and logs its own errors for the same reason: observation must not break the thing it observes, which is the same rule capture_round already follows.

Rounds are stored as a JSON blob rather than a child table — they are only ever read whole, and a relational rounds table would need a migration every time RoundSnapshot gains a field.

One subtlety worth knowing when querying it: total_tokens is the last round's total, not the sum across rounds. Rounds are cumulative — each resends the whole conversation plus whatever the loop appended — so summing them would multiply-count the same context and report a turn as costing several times what it did.

agent_turns is pruned to 5000 rows by store.prune(), which runs from requests.forget_turn every 200 finished turns. Before literal capture landed, nothing called it.

Literal requests (ctx_requests)​

A RoundSnapshot is a human-readable account of a round: blocks labelled by where they came from, clipped, with tool_calls folded into text. It is not what the provider received. Just before a request goes out, normalize_system_messages joins every leading system message into one and turns a later system message (the forced-tool nudge, the workspace index) into user. The forced-tool retry is a second provider call inside the same round, and a snapshot has no place for it.

requests.py records the request itself. providers.chat_stream and providers.chat call begin() right after normalizing, and finish() once the call returns or raises. Each dialect path calls note_payload() with the body it actually sent, so the OpenAI path's retry without stream_options after a 400 records the request the server accepted. Nothing is recorded outside a turn: the hook reads (turn_id, round) from telemetry/turn.py.

  • One row per provider call. attempt counts calls within a round, so a forced retry is R2 · retry. A failed call is recorded with status: error, and the error still propagates.
  • Provenance survives the flattening. normalize_system_messages_mapped returns, for each message it sends, the input messages it was built from and their character offsets. The recorder's labels (kinds_for) ride on those offsets as segments, so "Skill catalog" is a span inside the one system message rather than a role. A message whose role changed carries fromRole.
  • Bodies are content-addressed. Each message and each tools array is a row in ctx_bodies, keyed by a sha1 of its canonical JSON. A request stores only the list of hashes, so ten rounds that re-send the same history store it once. Tokens are counted once per body; the totals are estimates, beside the provider's reported prompt_tokens.
  • Stored verbatim. Credentials travel in headers, not bodies. The one exception, litellm's api_key keyword, is kept out of params.
  • Same retention as agent_turns. prune() deletes requests whose turn has gone, then removes any body that no request references.

Each request pushes an interpretability /ws frame with event request, once when it is sent and again when it finishes. A turn recorded before this existed has no rows. For those turns both routes rebuild one request per round from the snapshot, using agentpedia.rebuild, mark it reconstructed: true and list what the rebuild could not know. The rebuild's id is r:{turn_id}:{round}.

Limits​

  • The pane's ring is in-process. 25 turns, cleared on backend restart. The ring counts delegated sub-turns and peer nodes as entries, so a heavily-delegating session fills it faster; an orphaned child (parent already evicted) is promoted to a root rather than dropped. Turns are now also persisted to app.db (see below), so what the pane shows is bounded while the underlying history is not.
  • A peer's context is never visible here. By design — see above.
  • Block previews are clipped for transport. Conversation and tool results are cut at 4000 characters. Harness text is cut at 65536: the system prompt, the skill catalog, tool guides, the workspace index, the editor buffer, and loaded skill and tool-group bodies. That text is what a reader comes to learn from. The ring does not multiply it, because an unclipped block keeps the message's own string, so every round of a turn shares one copy. The persisted snapshot does repeat it once per round. Token counts always describe the full text, and clipped blocks say so.
  • Counts are of what we send. Ollama applies its own chat template on top, so the true prompt is marginally larger than the sum of the blocks.