Skip to main content

Module: llamacpp

The node serving its own weights. Ollama and LM Studio are applications the user installs and runs elsewhere; this module makes the backend itself fetch a llama-server build, keep a catalog of GGUF files, and supervise the process the agent talks to.

Status: implemented — backend in backend/modules/llamacpp/, frontend in packages/core/src/modules/llamacpp/, tests in backend/tests/test_llamacpp.py.

Agent tools​

This module had none until now, which made the fine-tuning agent's own prompt a description of a loop it could not run: it could convert a checkpoint and then had no way to serve the result — the step immediately before the eval it is told to run.

ToolDoes
llamacpp.statusWhether a local GGUF is being served, and which
llamacpp.list_modelsEvery servable GGUF — the managed directory plus whatever Ollama and LM Studio already have — with its base model where this node trained it
llamacpp.serveLoad a GGUF into the local server
llamacpp.stopStop it, freeing the memory

llamacpp.serve refuses rather than replacing a server that is already running a different model. llama-server holds one model at a time, so serving a second means stopping the first — and the first may be the model the user is chatting with, which is not a decision a tool call should take on their behalf. Asking for the model already loaded is neither an error nor a reload; it reports alreadyServing.

wait_ready returning False (rather than raising) when a model is too large to load is the trap on this path: reporting success there is how a caller comes to score a model that never came up and read the zero as the model's fault. serve turns it into an error.

Deliberately not exposed: installing or removing a build, and deleting a model. Fetching a multi-gigabyte binary or erasing weights belongs to a person looking at a disk-budget readout, not to a turn that inferred it would be helpful.

Note this inverts a rule recorded in two places. roster.py and training both said llamacpp must never appear in an agent's tool_groups, because a group is a tool name's prefix and that namespace held only settings keys — true until these tools existed, wrong after. Both comments are updated.

Why this is not just another provider entry​

Every other local provider answers "which model are you running?" with a name. It cannot tell you which file that name resolves to, what quantization it is, or how big it is — which is why interpretability has to reverse-engineer a path out of Ollama's blob store and LM Studio's undocumented index just to read a tensor table. Here the node picked the file, so the path is known rather than inferred, and the model explorer gets its highest-confidence source for free.

The second reason is that it removes the last hard dependency on somebody else's app. A packaged node can serve a model with nothing else installed.

Three artifacts, three lifetimes​

Where it comes fromWhere it livesWho owns it
The binaryUpstream GitHub release, downloaded on demand$HORRIBLE_DATA_DIR/llamacpp/bin/<tag>-<variant>/us
The weightsHugging Face, or already on disk$HORRIBLE_DATA_DIR/llamacpp/models/, or Ollama's / LM Studio's storemixed
The processSpawned by server.pypid + a loopback portus

Nothing is bundled. That is the same call as SearXNG, AssaultCube content and the DB-IP database: we support it, we fetch it on request, we ship none of it.

Asset selection is matched, not hardcoded​

Release asset names have changed shape several times, so pinning exact strings guarantees a break on some future release. select_asset matches on the tokens that carry meaning — OS, arch, accelerator — with two rules that are the whole test suite:

  • An accelerator build is never substituted for cpu. A CUDA archive's name contains every token a CPU one does, plus one. Downloading it onto a machine with no NVIDIA driver produces a binary that fails to load its runtime, which reads like a corrupt download.
  • The extension is per-OS. Upstream ships .zip for Windows and .tar.gz everywhere else. Matching only .zip — as this did until it was caught — reduced the whole module to a Windows feature, and reported it in upstream's voice: "release b10373 publishes no cpu build for ubuntu/x64". Both formats now match and _extract picks the reader by extension rather than by OS, because the extension is what the bytes actually are.
  • The name must start with llama-. Releases also carry cudart-llama-bin-win-cu12.4-x64.zip, the CUDA runtime redistributable, which matches every other token, is shorter than the real CUDA build, and contains no server at all.

When nothing matches, the error names the assets it did see, rather than failing blind on a release whose naming moved again.

The picker knows before you click install​

select_asset used to run only after the user picked a variant and hit Install — a release publishing no CUDA build for Linux surfaced as an error only once the download was already "starting." GET /llamacpp/install/variants runs the same match ahead of time, for every variant against the current release and this OS/arch, and the pane's <select> greys out (disabled) any option that comes back unavailable, labelled with the release tag. It reuses select_asset rather than a second notion of "available" that could disagree with what actually gates the download; a failed lookup (offline, GitHub down) leaves every option enabled, since "we could not ask" must never be shown as "unavailable."

Which variant, though — the hardware probe decides​

variant used to default to a flat cpu, which is correct on a machine with no GPU and silently wrong on every machine with one. It now defaults to "auto" and is resolved by the hardware probe, which still falls back to cpu whenever it could not determine what the machine has — the asymmetry above still holds, and "we could not ask" must never install a CUDA build.

The card alone does not decide the variant; the OS does too. Upstream publishes CUDA for Windows only — there is no ubuntu-cuda-* asset in any release, because Linux NVIDIA users are expected to build it or run the Vulkan build. So an NVIDIA card on Linux resolves to vulkan, with a reason string that says why. Getting this wrong is not a slow build, it is no install at all, and it only became reachable when the default moved from cpu to auto. test_upstream_publishes_no_linux_cuda_build pins the upstream fact, so if a Linux CUDA asset ever appears the special case can be deleted on a failing test rather than on a hunch.

The same probe supplies gpuLayers and threads at spawn and tokenCap for a trace. Those request fields are nullable, and the resolution is an is None check rather than a falsiness one: an explicit 0 is a request for pure CPU and has to survive.

A form must send the nulls, or none of the above happens. The pane originally initialized its own state to variant: 'cpu' and gpuLayers: 0 — plausible-looking defaults that are in fact concrete user instructions, so variant in ("", "auto") and gpuLayers is None never fired and the whole probe was bypassed by its primary caller. A machine with a 4090 installed a CPU build and offloaded nothing, with nothing in the UI to explain it. The pane now holds 'auto' and null, renders the resolved value as the input's placeholder, and prints the matching reasons string under the control. Three properties are load-bearing:

  • An empty box is a question, a typed number is an instruction. 0 in the GPU layers box still means pure CPU, and says so, rather than being silently reinterpreted as "auto".
  • The reading sits next to the decision. The probe's findings used to live only in the Settings page. The pane now shows the accelerator line above the Build card, with a Re-probe button, because that is where the choice is made.
  • Unknown is not none. With certain: false the line says the probe could not ask — the same rule the hardware settings section follows.

The offload preview — how many layers actually fit​

GET /models/layers (offload.py) sums the selected GGUF's tensor directory into per-block byte totals, and the pane draws the stack against the card's capacity with the GPU-layers slider on top of it. Everything it needs was already on disk — the directory carries a size and a block index per tensor — so the answer to "how many of these fit?" was one sum away and was simply never taken.

Two of llama.cpp's own rules are load-bearing, and reversing either draws a confident picture of the wrong thing:

  • The offloaded blocks are the LAST N. llm_load_tensors sets i_gpu_start = n_layer - n_gpu_layers. Taking the first N gives the same total (blocks are near-uniform) while marking the wrong end of the stack.
  • The output tensors only move once N exceeds the block count — what -ngl 99 is for. On a large-vocabulary model that is most of the file: Gemma 4 E4B keeps 2.9 GB of its 5.3 GB in the embedding and output head, so folding them into "layers" would be wrong by more than half. layerBytes and overheadBytes are therefore separate fields, and the bar shows the jump when the slider passes the end.

The KV cache is the one modelled quantity, and it earns its place: it is computed from attention.head_count_kv, so a GQA model's four-times-smaller cache is reported as such rather than inflated from the query head count. It is returned per token so the pane multiplies by the context size the user is still free to change — at 65k context on a 20B model the cache alone is 3 GB, which is the difference between fitting and an OOM at load, and baking a context into the server's answer would make it silently wrong the moment the field moved. It gets its own band on the bar rather than being added to the weights, because it is VRAM the file size does not account for.

What it does not claim: compute buffers, the CUDA context and driver overhead depend on the build and the batch size, not on the weights. Rather than a fudge factor hidden inside the total, a flat VRAM_RESERVE_BYTES is held back and named in the tooltip. And with no measured VRAM — a card reporting no memory size, a probe that could not ask, or unified memory where the "VRAM" is the RAM — the ceiling is simply not drawn and no verdict is given, because "unknown" rendering as "fits" is the exact failure the hardware module exists to prevent.

The arithmetic lives in offload.ts (pure, unit-tested) rather than in the component, including the Fit it suggestion, which is the largest count that still fits and is reported as layerCount + 1 when everything does — that being the value to actually send.

The probe's default does not know your context

hardware.defaults() decides gpuLayers from VRAM alone, so on a 12 GB card it says "all". The preview will show that overflowing once you raise the context, because it knows the model and the context and the probe knows neither. The preview is right; the default is a starting point.

The planner draws every block, not three totals​

The preview above began life as three stacked widths — weights on the GPU, KV cache, weights left in RAM — with a ceiling line across them. That drawing could state both of the rules in the previous section and show neither, because a single bar has no place to put a block index.

LayerColumns.tsx draws one column per decoder block, height from the real layerBytes[i], filled from the right. Dragging the slider then teaches the last-N rule without a sentence: the fill travels backwards down the stack, which is exactly what i_gpu_start = n_layer - n_gpu_layers does. The embedding and output head are a separate, wider column set apart from the rest, so the 2.9-of-5.3 GB fact is a thing you can see rather than a number in a paragraph, and it visibly stays empty until the count passes the block count.

The ceiling changed shape too. offloadColumns (pure, in offload.ts, tested beside estimateOffload) accumulates a running total in offload order — last block first, output tensors last — and the VRAM line crosses that curve at exactly the block where the model stops fitting. "Does it fit" becomes a position rather than a verdict.

Two honesty rules the columns inherit and one they add:

  • No measured VRAM means no ceiling and no verdict, unchanged from the bar.
  • A block whose bytes could not be sized still gets a tick, at a floor height. The rail is an index as well as a chart, and a missing column would shift every label after it.
  • A budget larger than the whole model draws no ceiling either. Clamping the line to 100% put it across the top of the chart, which reads as "this exactly fills the card" — an assertion of a crossing that does not exist. A model that fits entirely has no crossing point, and the honest drawing of no crossing is no line, plus a sentence saying so.

estimateOffload and offloadColumns are checked against each other at every slider position, so the legend and the picture cannot drift apart.

The drawing primitives are shared now​

Four of the visuals on these tabs used to be private to one component: a sparkline and a heat strip inside TracesSection.tsx, a progress bar in ServerPane.tsx, and a segmented budget bar in OffloadPreview.tsx — the last of which had two more call sites that had settled for a sentence instead. They live in packages/core/src/viz/ now, beside Primitives.tsx and DataList.tsx which set that precedent.

Three of them carry a rule that used to be a comment:

  • spark.ts — a gap is a gap. A pass whose record was summarized without a statistic has nothing to report, and joining the line across it draws a measurement that was never taken. A lone measured point between two gaps becomes a dot, because a one-point polyline draws nothing at all — silently.
  • Meter — threshold: null means no verdict. The "unknown VRAM draws no ceiling" rule is now in the type: there is deliberately no fits prop, so a caller cannot assert a verdict the component would have refused to draw.
  • ramp.ts returns a sign and an alpha, never a colour. The old cellColor returned rgb(110 190 255 / 34%), which renders identically on the light theme and slipped past no-hex-literals.test.ts only because that guard matches #rrggbb. Passing the ratchet is not the same as being themed. The sign becomes a data attribute and the alpha a custom property, resolved in CSS against two new per-theme tokens, --ramp-pos and --ramp-neg — the only two tokens whose job is to be read as a direction rather than as a status, which is why they are not --accent and --danger: a negative activation is not an error.

HeatCanvas is deliberately not added to the hex guard's CANVAS_EXEMPT bucket. That exemption exists because a CanvasRenderingContext2D cannot resolve a custom property — true, but it only covers a canvas with no way to ask. This one reads its ramp through getComputedStyle and repaints on a theme change. The cautionary case is training/panels/MetricsPane.tsx, which passed var(--accent, #539bf5) straight into uPlot: ctx.strokeStyle ignores an unparseable value without throwing, so that series was drawn in uPlot's default colour on all six themes while looking, in the source, fully themed. viz/uplot-theme.ts resolves tokens properly and that bug is fixed.

The machine band — what the probe found, and what it could not ask​

The Server section used to open with one line naming profile.primary. Three things the hardware probe had already measured were reaching the pane and being dropped:

  • accelerators[], not just primary. A machine with a discrete card and an integrated GPU showed one of them, so "why did it pick that one" had no answer on screen. Each device now gets a card with its own memory meter and a measured / reported mark taken from Accelerator.exact.
  • notes[] — the gaps. This is the hardware module's entire reason for existing and it had no renderer anywhere in the app. An absent nvidia-smi and an absent GPU produce the same empty list and are not the same fact; the backend is careful to distinguish them and the UI flattened it. They are now listed under "what the probe could not ask".
  • defaults.reasons for every auto-resolved field, not the two that had tooltips.

installs[] had the same problem: always fetched, never listed. A machine that has changed build variant a few times accumulates them — five, on the machine this was developed against, four of them stale CUDA builds — and POST /install/remove had no frontend caller at all. The list is now rendered whenever there is more than one, and the Remove button says why it is disabled while the server is up rather than letting the backend answer 409.

Finally running and ready are drawn as the two separate facts the API models them as. Loading a large GGUF takes tens of seconds during which the process is up and every request fails; collapsing that into one sentence made a normal load look like a fault and gave no way to tell it from a server that is genuinely stuck.

Verification says what it actually knows​

GitHub publishes a digest for release assets. When it is there, a mismatch aborts the install and nothing is written — an unverified half-install that a later newest_install() hands to the spawner is worse than no install. When it is absent, the digest we computed is recorded and the install is marked verified: false, which the pane renders as a chip. Recording a self-computed hash and calling it verification would be theatre.

The catalog​

Three origins, and the distinction is load-bearing:

  • managed — under our models directory. Downloadable, deletable, counted against the disk budget.
  • ollama / lmstudio — files those apps own. Serveable, never touched. llama-server memory-maps a GGUF read-only, so pointing it at LM Studio's copy costs nothing, and duplicating a 20 GB file to "own" it would be absurd. But a delete route that can reach into another app's store is a footgun, so the origin gates deletion — and is_managed resolves both paths, because a .. segment in a caller-supplied path is exactly how a delete route becomes an arbitrary-file-delete route.
  • extra — directories added via llamacpp.modelDirs.

mmproj-*.gguf is skipped everywhere. A multimodal repo ships its vision projector beside the weights: a real GGUF that parses cleanly and cannot be served as a chat model, so offering it in the picker is a guaranteed bad spawn. It is the same trap the model explorer hits from the other direction.

The quantization label is the most common tensor type, not general.file_type: that KV is an enum id whose meaning has shifted between llama.cpp releases, and a mixed-quant build (most K-quant files keep some tensors at F32) is the norm. Header facts are memoized on (path, size, mtime) — scanning a directory of 20 GB files on every status poll would otherwise re-read all of them.

Models you trained land here too​

The training module converts a finished checkpoint into a GGUF in the managed directory, so it shows up in this catalog like any download and is served the same way. That is the loop closing: train it here, serve it here, look inside it here. A LoRA result is labelled an adapter, because llama-server takes it with --lora beside a base model rather than serving it alone.

GET /models carries that provenance, not just the agent tool. Each row gets baseModel, projectId and isAdapter, joined from training.lineage on the GGUF path. Without it the pane — which is where a person actually picks a model — listed a fine-tune by whatever general.name the converter wrote: the adapter of a Qwen fine-tune appeared as a servable model called "Qwen/Qwen3-0.6B", and the merged model beside it as "Merged". The row now says "fine-tune of <base>", the adapter is tagged in the list and in the start picker, and a missing row reads as unknown provenance rather than as a claim about a base.

The disk budget is pre-flight​

A download is refused before a byte is written, against the size the server declares in a HEAD. Checking as it goes means discovering the disk is full 30 GB into a 40 GB transfer. The ceiling is llamacpp.diskBudgetGb (default 80).

Downloads write to a .part file and rename on success, so an interrupted transfer can never be picked up by the catalog scan as a servable model — a truncated GGUF fails at load time with an error that reads like a corrupt model.

Reading the catalogue​

The model list is grouped by origin with each group's root named, because "why is this file here" and "why can I not delete it" are the same question and origin is the whole answer — it was previously a bare tag on a row, sorted next to nothing. extraDirs is finally rendered, so a file from a user-configured directory says which one.

Two things that were sentences became pictures:

  • The disk budget is a segmented meter, one segment per model, hovering either way between a segment and its row. usedBytes of budgetBytes told you the total and never which file was responsible for it.
  • Quantization cost is a log-log scatter of parameters against sizeBytes. Plotted against each other the two numbers stop being two numbers and become a rate — bytes per parameter — and each quantization family falls on its own line through the origin, with guides drawn at 0.5, 1, 2 and 4. Q8_0 sits at about twice Q4_K_M's slope and F16 at four times, which is readable without knowing what any of the names mean. Log-log because a catalogue spans 0.5B to 70B and a linear axis puts every small model in one pixel; a constant ratio is still a straight line after the transform, so the property being read survives it. It refuses to draw under three points — a scatter of one is a dot with axes implying a trend nobody measured.

Each row also carries a context rail against the longest context in the catalogue, so the over-context warning on the Server tab is predictable before you type a number rather than after.

The server​

--jinja is passed explicitly, never inherited​

It selects the model's own chat template, which is what carries its tool-call syntax. Upstream currently defaults it on — verified on b10362, where a server started without it reports the same template — but the default has flipped before, --no-jinja exists, and it also reads LLAMA_ARG_JINJA from the environment. So whether the agent can call tools would otherwise depend on which build happened to be downloaded and what is in the user's env.

Without the model's template, llama-server accepts only "commonly used" templates and falls back to a generic one for the rest — and that failure is silent: 200, a perfectly fluent answer, and every tool in the schema simply never called, which is indistinguishable from "this small model is bad at tool use". Hence the explicit flag, and the test that pins it.

Ready is not the same as running​

Loading a large GGUF takes tens of seconds, during which the process is up and every request is refused. wait_ready polls /health; the status the pane reads distinguishes running from ready, and a process that dies during load reports its last log lines rather than a timeout.

Two more things that would otherwise bite​

  • The port is negotiated, not assumed. Hyper-V reserves ranges that swallow ordinary dev ports on this machine, and 8080 is a popular one, so the manager binds the default if it can and takes an ephemeral port if it cannot. This is why _endpoint_for checks the live manager before the saved config for this provider alone — a saved :8080 would point at nothing while the real server sat elsewhere.
  • Spawning goes through subprocess.Popen on a thread. Under uvicorn --reload on Windows the loop is a SelectorEventLoop, where asyncio subprocess creation raises — the same reason the LSP and PTY managers do it this way. The app's lifespan stops the server on shutdown, or an orphan survives a reload holding gigabytes of mapped weights and a bound port.

How it was started is recorded, so it can be put back​

spawn stores a SpawnSpec — model path, alias, port, context size, GPU layers, threads, extra args — and spawn_spec hands it to anyone who has to restore the server. There is exactly one process and one model, so anything that wants to borrow it has to be able to give it back: evals loads a GGUF to score it and then restores whatever was running.

Restoring from the spec rather than from defaults is the whole point. Defaults would bring the user's server back with gpu_layers reset, extra_args gone and context_size shrunk to 4096 — a machine that is quietly slower afterwards, with nothing said. The same failure hardware.defaults() exists to prevent.

Chat integration: four lines, not a new dialect​

llama-server speaks the OpenAI API, so this is one PROVIDERS entry with dialect="openai" and can_spawn=True. Everything already built on that dialect — streamed reasoning, tool-call assembly from indexed deltas, the tool_choice="required" retry — works unchanged.

A bespoke dialect would have meant six new branches in providers.py and would have silently lost that retry, which is gated on info.dialect == "openai".

It is deliberately not can_pull: weights are fetched by this module's catalog, not by asking the inference server to pull them the way Ollama does.

Speculative decoding​

A small draft model proposes tokens and the large target verifies a batch of them in one pass. It is local — -md/--model-draft takes a file path, and there is no remote-draft mode in llama.cpp, so a peer cannot supply draft tokens however convenient that would be.

Flags are probed, never hardcoded​

features.probe_flags() shells llama-server --help and reports which spelling this build wants, cached by binary path and mtime.

Two reasons, both silent failures otherwise. Upstream renamed the tuning flags under a --spec- prefix in 2026 (--draft-max → --spec-draft-n-max), and both spellings are in the wild; passing the wrong one does not warn — the server exits during load and wait_ready reports the generic "did not start", which reads as a broken model. And --rpc is a compile-time feature whose presence in upstream's release binaries has changed over time: b10453-cuda, which this module downloads, does have it, so a documented constant would have been true when written and quietly wrong later.

The probe answers in three states (hardware/probe.py's rule): present, absent, or could-not-ask. A knob the build does not advertise is dropped from the command line rather than swapped for the other spelling — its absence means the build uses its own default, which works, where a wrong flag does not.

A draft must share the target's tokenizer​

speculative.check_compatible compares tokenizer.ggml.model, tokenizer.ggml.pre, vocabulary size, a hash over a stride of the token list, and the BOS/EOS ids.

It refuses rather than warning, because the failure is invisible: the draft proposes token ids, so a mismatched tokenizer collapses the acceptance rate and makes generation slower than not drafting at all, while looking like it worked. A warning on a feature whose only symptom is "mysteriously slower" gets ignored. For the same reason an unreadable header is a refusal, not a pass, and compatibility is read from headers rather than inferred from names — files sharing a name prefix routinely have different tokenizers.

llamacpp.find_drafts lists compatible candidates smallest-first, and also reports whether the build can draft at all, which saves a doomed spawn.

Dividing the card​

hardware.defaults().gpu_layers stays 0-or-999 — every caller depends on that number, and this is the only place that needs to split a card between two models. speculative.spec_plan does the division, reusing offload.layer_plan and inheriting its honesty: that function reports weights and KV cache and explicitly declines to invent a compute-buffer allowance, so headroom is a caller-supplied parameter here too.

The draft goes on the card whole or not at all. Splitting an already-small model across host and device costs a PCIe round trip per token on the very path whose cheapness is the entire premise of drafting. If it will not fit alongside anything, spec_plan offers no plan and says why, rather than guessing — a guess here is an out-of-memory error at load.

Measuring it, because it may not help​

specbench.py is the deliverable, not garnish. On a single consumer card the draft's VRAM comes out of the target's KV cache, and drafting costs a forward pass per batch whether the guesses land or not; the trade can genuinely be negative.

It reports p50 and p10, never a mean (variance is the defining property — a mean hides that a code prompt got 2.4× and a prose prompt 0.8×), per prompt, never only aggregated (an aggregate over mostly-boilerplate prompts is how you convince yourself of a speedup you will never see), and A/B back-to-back on the same machine, since tokens/sec is not comparable across runs on a laptop that throttles. compare() may return a verdict of "slower" — that is a result.

The acceptance rate is included because it explains the outcome: 20% acceptance and slower is a model pairing problem, 70% and slower is a memory problem.

Per-agent provider selection​

model, temperature, contextSize, maxTokens and topP have been per-agent since the roster landed. provider and endpoint were not — which made "run the coder on the node's llama.cpp server and leave the orchestrator on Ollama" unexpressible, because the only per-agent knob was a model name, and a name means nothing on a server that does not have that model.

roster.resolve_provider(config, agent_id) resolves both, per key: agent.<id>.provider → agent.orchestrator.provider → the saved global config. An override naming an unknown provider is ignored, not fatal — a stale settings value must not take the agent down.

Every call site, or the override is a lie

run_agent_loop is called from three places: the orchestrator (orchestrator.py), local delegation (delegate.py), and the flow executor's Agent node. A site left on provider_for(config.provider) does not fail — it quietly runs that path on the global provider, which is how a delegate ends up on a different model than the one its settings show.

Contributions to the layout shell​

KindIDNotes
Panelllamacpp.server"llama.cpp" 🦙; singleton, role document — running a model, not reading one
Sectionllamacpp.server → serverDefault, key s — build + process + log
Sectionllamacpp.server → modelsKey m — catalog + downloads
Sectioninterpretability.suite → tracesThe activation scrubber — a section of the interpretability suite
Sectioninterpretability.suite → lensA trace read as words (the Trace lens tab)
Sectioninterpretability.suite → stepperThe layer debugger: a trace in execution order (see "The stepper" below)
Commandsection.show:llamacpp.server:{traces,lens,stepper}Kept as shims for those sections' old ids; they open the suite section
Commandllamacpp.openllama.cpp: Server and models
Commandllamacpp.openStepperllama.cpp: Step through a trace (/stepper)
Commandllamacpp.stepper.*Step into/back/over/out, continue (both ways), toggle breakpoint, play, restart — no-ops unless the Stepper section is mounted
Keybindingstepper verbs→/← step, ↓/F10 over, ↑/Shift+F10 back over, Shift+→ out, C/Shift+C continue, B breakpoint, Space play, Home restart — scoped paneFocus == 'interpretability.suite' && paneSection == 'stepper' && !textInput, because the suite keeps the stepper mounted behind its other sections
Commandllamacpp.traceSelectionllama.cpp: Trace editor selection (/trace-selection)
Settingllamacpp.modelDirsExtra GGUF directories to scan
Settingllamacpp.diskBudgetGbCeiling for the managed directory
Settingllamacpp.traceBudgetGbCeiling for stored traces (default 2 GB)
Settingllamacpp.lensCacheGbCeiling for the dequantized output-head cache (default 4 GB)

One pane, five sections — the build, the weights, the traces and the lens are one workflow in a strict order (no build, no server; no weights, nothing to serve; nothing served, nothing to look inside; nothing traced, nothing to read), so splitting them would give panes of which most are usually an instruction to open another. Traces and Lens are two readings of the same object rather than two objects, which is exactly what the sections primitive is for. The pane is also tabbed into the Lab frame beside the Hub and the model explorer: find weights → serve them → look at what you served is one working position.

Provider and endpoint overrides are edited in Settings → Agent orchestrator, beside the model override they have to agree with.

Tracing what you are actually writing​

llamacpp.traceSelection (/trace-selection) sends the editor's selection — or the whole buffer when nothing is selected — into the traces section as the prompt, then reveals that section. The prompt box was otherwise a fixed 'The capital of France is', which demonstrates the machinery and answers nothing anyone was wondering; fed the buffer instead, the token strip becomes your own identifiers and (with attention on) the attention map is a map over your code. It pairs naturally with the fact that traces apply no chat template: what is tokenized is exactly the text, which is what you want for a code fragment.

Four things it does deliberately:

  • The editor is reached through its registered service, EditorService (registry.getService('editor')), which gained getBufferSelection for this — the same seam the visualizer uses, never a deep import.
  • An unmounted buffer is declined, not guessed. A selection exists only in a live CodeMirror view; falling back to the persisted bytes would trace something other than what is on screen.
  • An untitled buffer is reported specifically. The editor tracks the active buffer by source URI, and an unsaved one has none — so the message says "needs a saved buffer" rather than "needs an open buffer", which would read as a bug when one is plainly open. See editor.
  • The origin is shown and clears on edit. The section prints "From foo.py (selection)"; typing in the box drops the label, because a textarea that silently repopulated itself is indistinguishable from a bug.

The handoff itself is trace-prompt.ts: a pending value plus a subscription, because revealing the section mounts it fresh (a prompt pushed first has no listener yet) while a push into an already-open section has no mount to read it. Either mechanism alone drops half the cases, and the read clears the value so a later visit does not resurrect an old selection over what you were typing.

Backend surface​

MethodPathPurpose
GET/api/llamacpp/statusBuild, process, readiness, log tail
POST/api/llamacpp/installDownload + unpack a build (NDJSON progress)
POST/api/llamacpp/install/removeDelete a build (refused while it runs)
GET/api/llamacpp/modelsCatalog + disk usage + starter repos
GET/api/llamacpp/repo?repo=GGUF files in a Hugging Face repo, with sizes
POST/api/llamacpp/models/downloadFetch one GGUF (NDJSON progress)
POST/api/llamacpp/models/deleteDelete a managed model (403 otherwise)
GET/api/llamacpp/models/layersPer-block byte sizes + KV cost per token, for the offload preview. Catalog-only — it opens a client-supplied path, so an uncatalogued one 404s rather than becoming an arbitrary-file read
POST/api/llamacpp/startSpawn on a chosen GGUF, optionally waiting for ready
POST/api/llamacpp/stopTerminate
GET/api/llamacpp/tracesStored traces, disk usage, and whether tracing is available at all
POST/api/llamacpp/traces/estimatePre-flight cost, from the GGUF header
POST/api/llamacpp/tracesRun one traced forward pass (NDJSON progress)
GET/api/llamacpp/traces/{id}Manifest, record list, tokens
GET/api/llamacpp/traces/{id}/record/{i}One record's values, decoded server-side (capped at _VALUE_CAP)
GET/api/llamacpp/traces/{id}/seriesOne node's statistic per forward pass, by name. Summarizes the whole record, unlike /record/{i}
GET/api/llamacpp/traces/{id}/tensorsThe raw blob, honouring Range
GET/api/llamacpp/traces/{id}/record/{i}/matrix?head=&cols=One 2-D plane read through the record's strides: tokens × features (max-abs pooled, full-resolution row norms) or one attention head's query × key matrix cropped to the KV in use
GET/api/llamacpp/traces/{id}/record/{i}/headsPer-head entropy / self-weight / first-token weight for an attention record
GET/api/llamacpp/traces/{id}/headsEvery head of a pass scored (heads.py): previous-token, induction, duplicate, sink, self, entropy — induction/duplicate null on a prompt with no repeats
POST/api/llamacpp/heads/probeRecord the induction probe on a GGUF: a random word sequence twice, attention + l_out only; streams like /traces
GET/api/llamacpp/traces/{id}/layer/{L}/delta?passIndex=What block L wrote into the residual stream, per token (relative change, cosine)
GET/api/llamacpp/traces/{id}/experts?passIndex=Mixture-of-experts routing: per-token expert choices and weights, per-layer counts. moe: false on a dense model
POST/api/llamacpp/traces/{id}/forkRe-run with some tokens replaced (NDJSON progress)
GET/api/llamacpp/traces/capture-setsThe named capture sets and their real ggml node patterns
GET/api/llamacpp/traces/{id}/lensThe layer × position readout
GET/api/llamacpp/traces/{id}/lens/trackOne vocabulary token's logit and rank at every cell
GET/api/llamacpp/traces/{id}/lensesWhich lenses apply to this trace's model
GET/api/llamacpp/models/vocabSearch a GGUF's own vocabulary — the token picker behind a swap
GET/api/llamacpp/traces/catalogThe llamacpp_traces rows in app.db — filterable by model hash or parent, which the directory listing cannot do
POST/api/llamacpp/traces/{id}/findingFile a lens reading into a library as a note. A 200 with error set is a refusal, not a failure
DELETE/api/llamacpp/traces/{id}Delete a trace

Progress rides NDJSON on the request, not a /ws channel — the shape /agent/pull already uses, and it keeps a multi-gigabyte transfer scoped to the request that asked for it: navigating away cancels it rather than leaving a broadcast running for nobody. The shared client helper is streamNdjson in packages/core/src/api.ts.

A Hugging Face token is used when the connector is connected, and not required: the great majority of GGUF repos are public, so gating weights on an OAuth flow nobody needs would be backwards.

Browser vs desktop​

No difference. The server is a backend child process and the pane is ordinary HTTP — both layouts drive the same node. The only asymmetry is the usual one: a browser layout pointed at someone else's backend manages that machine's models, which is what "the node serves its own weights" means.

Testing​

backend/tests/test_llamacpp.py. The tests worth keeping are the ones pinning failures that are otherwise silent:

  • --jinja present on every spawn, so the model's own chat template is never left to a build default or an env var.
  • An accelerator build is never selected for a cpu request, and cudart-* is never mistaken for a server.
  • A digest mismatch leaves no install behind.
  • An asset with no published digest installs as verified: false.
  • delete refuses a path outside the managed directory, including one that walks out of it via ...
  • The disk budget refuses before writing.
  • A per-agent provider override reaches resolution, an unknown one degrades to the configured provider, and an unoverridden agent is untouched.

The tracer​

Everything above serves tokens. This part serves activations: a traced forward pass, captured node by node out of ggml and written to disk.

It is a separate artifact from the chat path and stays that way. Chat runs a downloaded llama-server binary; tracing runs whatever libllama the llama-cpp-python wheel bundled, installed by uv sync --extra llamacpp and pinned with == rather than >=. The chat provider must not depend on the risky code, and the two llama.cpps genuinely disagree — which is why a trace may only be overlaid on a chat turn when both llamaBuild and modelSha match (traces.matches_run). A pane that overlaid them anyway would be showing a forward pass that did not produce the answer above it.

The watch window — pinned nodes, gdb-style​

The scrubber could show exactly one node at a time, so the two questions an interpretability session is made of — what does this node do as the model generates? and how do these two compare? — could only be answered by clicking back and forth and remembering numbers. Pinning answers both: pinned nodes stack in a Watch card above the list, all visible at once, re-resolved for whichever pass is on screen.

A pin holds a node NAME, not a record index, and that is the whole design. A watch in a debugger is an expression re-evaluated at each stop, not a pointer to one evaluation of it. So a pin survives switching pass (each generated token is its own forward pass with its own records under the same names) and survives opening a different trace of the same model — the closest thing here to re-running a program under gdb. Pins are keyed by model, not by trace, for the same reason: "watch layer 41's residual" is a fact about the architecture, and keying by trace id would throw the list away exactly when it became useful, on the second trace.

Three states are deliberate:

  • A pin naming a node this trace never captured is shown as unresolved, struck through with "not captured in this trace" — never silently dropped. It is a real finding: pin kq_soft_max-17 from an attention-on trace, open an attention-off one, and the gap tells you the capture set differed rather than pretending the node was never interesting.
  • A summary pin shows statistics and no strip. The rule that a summary carries statistics instead of a tensor is preserved rather than special-cased away.
  • The cap is 8, and it says why. Each pin is a request per pass change; an uncapped watch list turns switching pass into a request storm. pin all on a block fills up to the cap rather than dropping the ones that fit.

The sparkline is a second route, GET /traces/{id}/series?name=…, and it exists because the manifest cannot answer it: tracer._capture writes summary only for fidelity="summary" records, so exactly the records that have data carry no statistics. It summarizes the whole record where /record/{i} caps at _VALUE_CAP — a series is a comparison across passes, and a statistic over a prefix compared against one over a different prefix is not a comparison. This is measurable, not theoretical: on a 9-token prompt pass l_out-41 has 23040 values, and the capped route reports rms 0.894 against the whole record's 1.248. Where the record fits under the cap the two agree to the last digit. A pass with nothing measured contributes null and is drawn as a gap, never interpolated.

The node list gained filters to match — name search, layer range, and kind toggles built from a pure node-kind.ts classifier. Two details worth keeping:

  • Kind is derived on the frontend from the node name, not stamped on the record at write time, so it works on every trace already on disk rather than only on ones recorded after the change.
  • Narrowing the layer range hides the layerless nodes (inp_embd, result_norm, result_output) and leaving it untouched keeps them — mirroring TraceFilter.wanted in the tracer, which refuses to let a layer selection silently drop the two ends of the story.
  • Only the kinds present are offered as toggles. A Mamba trace showing an "attention" chip that matches nothing is worse than no chip.

The depth profile — the pass, without pinning anything​

A TraceRecord carries seven statistics (count, min, max, mean, rms, absMax, zeroFraction) and the pane rendered exactly one of them, for pinned nodes only — so the most basic question you can ask of a traced pass, does the residual grow with depth?, took eight manual pins and still could not be seen at a glance. The Profile card answers it in two views, above the record list.

Why it is a route and not a useMemo. The first cut computed it from records[].summary, already in the browser, and cost no request. Against every trace on disk it drew an empty chart reading 0 of 28 layers measured, and blamed the fidelity for it. tracer._capture writes summary only for summary-fidelity records — the ones that hold no data — so on an fp16 trace the manifest carries 170 records and zero statistics. Exactly the records with data have no statistics. That is the same fact that forced GET /series to exist, and GET /traces/{id}/profile?passIndex=&stat= is its sibling one axis over: every record of one pass, reduced server-side over its whole tensor rather than the _VALUE_CAP prefix get_record ships to a browser. Summarizing 322 records that way is 22 MB of activations, which is why traces.summarize_array is numpy and summarize now delegates to it — one implementation, not two that can disagree about the number the pane prints.

The pass and the statistic are the whole request; which role is on screen is not, because that is arrangement of a response already in hand. So:

  • One role against depth. A role is a record name with its block index removed, so kqv_out-3 and kqv_out-27 are the same role at two depths — which is the only way to compare them. Two naming schemes reach roleOf and only one is blk.N.: graph nodes named by llama.cpp's cb() carry the index as a -N suffix, and handling only the segment form left every activation name unchanged, so each depth read as its own role and nothing was profilable at all. Only roles appearing at more than one depth are offered; a node captured once is not a profile.
  • The pass as a fingerprint — node kind against depth as a heat grid, normalized per row, not globally. An attention score and an FFN activation are different quantities in different units, and one shared scale would render every row but the largest as uniformly blank, which reads as nothing happens there. Per-row normalization asks the only question a heat grid can honestly answer here: where along the depth is this quantity large? Several records of one kind landing on one layer (q, k and v are all attention) average rather than max, so one outlier cannot stand for the group.

The statistic selector is as much the point as the charts: rms answers "how big", zeroFraction "how sparse", absMax "is there an outlier". Those are different questions about the same pass, and six of the seven statistics previously had no renderer at all. The same selector now also drives the watch sparklines, which the series route already accepted as ?stat=.

A missing statistic is null, never 0. A summary-fidelity record can lack a statistic outright, and zero is a measurement — reporting it would draw a value that was never taken and drag every mean and every colour scale toward it. The route emits null and the grid skips it; a gap is drawn as a gap. The other half of the same rule is why the grid tests value !== null rather than truthiness: a genuine 0.0 is a reading and belongs on the chart.

A failed request is surfaced, not swallowed — an empty profile and a broken one look identical on screen otherwise, and only one of them is a bug worth seeing.

Architectures beyond the transformer​

DEFAULT_CAPTURE is a transformer capture set — attn_norm, kqv_out, ffn_*. llama.cpp also runs Mamba/Mamba2, RWKV and hybrids (Jamba, Falcon-H1, Nemotron-H), and on those none of those names exist: tracing one recorded the residual stream and nothing whatsoever about the mechanism, which reads as a broken tracer rather than as an unasked-for capture. tracer.capture_for(architecture) now adds an SSM or RWKV group, chosen by general.architecture read from the GGUF header at request time (routes._architecture, failing soft to the transformer defaults).

Two rules:

  • Union, never replacement. A hybrid interleaves attention blocks with SSM blocks, so swapping the list on an architecture match would blind the trace to half of every such model — quietly, since what remained would still look plausible.
  • The names were verified, not guessed. They come from the string literals in the installed llama.dll (b10448): ssm_conv1d is real and ssm_scan is not. A capture pattern matching nothing fails silently, which is the exact failure being fixed, so the patterns are prefixes (ssm_, time_mix_, channel_mix_) matched the same substring way Tracer.wanted already works. gguf._BLOCK_RULES gained ("ssm_", "ssm") too, or the model explorer files every Mamba tensor under "other".
Diffusion is not covered

Image diffusion is stable-diffusion.cpp — a different binary this module does not run, and nothing here traces it. Diffusion language models (LLaDA, Dream) do exist in llama.cpp, but they denoise the whole sequence iteratively, so their steps are not passIndex-as-generated-token and the tracer's autoregressive llama_decode loop does not express them. Both are open, not half-supported.

The SSM capture set is verified against the build's own strings and against Tracer.wanted, but not yet against real Mamba weights — no Mamba GGUF is present on the development machine.

Why snapshots, not a stream​

Attention scores alone are fp32 [n_kv, n_tokens, n_head] — roughly 33 MB per layer per pass at 512 tokens and 32 heads, about a gigabyte across 32 layers. A live tensor feed is therefore a websocket carrying a gigabyte a second to a pane that can render none of it. A run writes a snapshot and the pane scrubs it afterwards, which also makes a trace something you can keep, compare and delete.

$HORRIBLE_DATA_DIR/llamacpp/traces/<trace_id>/
manifest.json # provenance + every record (name, layer, pass, op, dtype,
# ne[4], nb[4], offset, length, fidelity)
tokens.json # the tokens the pass actually ran on
tensors.bin # one append-only blob, records in capture order

One blob rather than a file per layer — hassault's cube-grid precedent literally. Per-layer files give thousands of handles and still cannot address a single node inside a layer; blob plus Range gives both, and the range handling (including the suffix form, where bytes=-3 is the last three bytes) is copied from karaoke along with its tests.

The ggml ABI is the top risk, and it is contained three ways​

struct ggml_tensor is internal C with no stability promise, and llama-cpp-python declares the cb_eval callback type and zero ggml accessors — so the callback hands over a void * that nothing in the wheel can read. ggml_abi.py closes that gap as narrowly as possible:

  1. Mirror as little as possible. Only the first four fields (type, buffer, ne[4], nb[4]). Name, op and data are read through functions ggml exports — the same instinct as serving plane_order instead of hardcoding it.
  2. A self-check gates every trace. On the first tensor, nbytes is recomputed from (type, ne, nb) using ggml's own formula and compared against ggml_nbytes(), and a dimension count derived from ne against ggml_n_dims(). A field inserted upstream shifts the layout and the two disagree immediately. On mismatch the run aborts and writes nothing — a trace that parses but is garbage is worse than no trace.
  3. The tracer is a subprocess. python -m backend.modules.llamacpp.tracer <spec.json>, spawned with subprocess.Popen on a worker thread (never asyncio.create_subprocess_exec — under uvicorn --reload on Windows the loop cannot spawn subprocesses at all). A segfault inside a ggml callback is then an exit code the runner reports, not a dead FastAPI backend mid-turn. A non-zero exit with no error line is reported as exactly that.

Those exported functions live in ggml-base, and only there. A modern wheel splits the build into several shared libraries, and llama.dll re-exports none of the ggml API — nor does ggml.dll (the backend registry) or ggml-cpu.dll. Nor is the CDLL reliably reachable from llama_cpp._ggml under a fixed attribute name (it has been both lib and libggml). So _ggml_candidates() yields every plausible holder — each CDLL bound on _ggml, then *ggml-base* from the wheel's lib/ directory, then the old single-library llama_cpp.llama_cpp._lib — and each is probed for ggml_nbytes before being accepted. Picking a library by name instead surfaces later and further away, as AbiMismatch: ggml exports none of ggml_type_name out of bind(), which reads like an upstream ABI break rather than the wrong file being open.

Four things that fail silently if forgotten​

  • Flash attention must be disabled explicitly. With it on, ggml_flash_attn_ext is one fused node and the score matrix never materialises — attention capture would produce nothing, reading as a bug in the name filter rather than a missing flag. Recent llama.cpp auto-enables it, so the default is not safe; _disable_flash_attn sets flash_attn_type (or the older flash_attn bool, whichever this build has).
  • The CFUNCTYPE instance is held in a module global. Built inline, it is garbage-collected the moment the call that created it returns and the next graph node jumps into freed memory.
  • Quantized tensors are recorded as metadata only. Dequantizing in Python would be a second implementation of ggml's block formats — wrong in one corner and plausible everywhere else. There is no dequantizer here, and traces.decode refuses a quantized dtype rather than guessing.
  • No chat template is applied. The prompt is traced as raw text; rendering a template ourselves would put a different token sequence through the model than the one shown beside it. The manifest carries chatTemplate: false and says so.

Everything declares what it is​

The module's three-state honesty applies unchanged:

  • Every record carries a fidelity — full (the tensor), fp16 (a downcast of it), or summary (statistics instead of the tensor). A summary record has no bytes by construction: TraceWriter.append refuses a payload for one, the record route returns empty values, and the pane draws the statistics as statistics.
  • modelSha says what it hashed. Digesting 20 GB to open a pane would be absurd, so it is the GGUF's first megabyte plus its size, and every manifest carries modelShaScope: "gguf-header-1mib+size". It is enough to catch "a different model" and does not pretend to be a whole-file checksum.
  • The estimate is labelled an estimate. POST /traces/estimate costs the run from the GGUF header (block_count, embedding_length, attention.head_count) before it starts, because the difference between a 200 MB trace and a 12 GB one is one checkbox and a progress bar that has already started is too late. The manifest's blobBytes is what actually happened.

Cost control is three layers: a hard MAX_TRACE_TOKENS cap (512) regardless of what the caller asks for, attention off by default and per-layer opt-in, and a global byte budget (llamacpp.traceBudgetGb, default 2 GB) pruned oldest-first after every run.

Testing​

backend/tests/test_llamacpp_trace.py, and it needs no native library — bind() is duck-typed against a stub ggml, so the ABI check is exercised as arithmetic over ctypes structures built in Python memory. The tests pin: the self-check passing on a consistent tensor and failing on a mutated one (both the nbytes and the n_dims disagreement), the quantized block branch (misreading nb[0] as a stride overstates a q5_0 row by 32×), manifest/blob round-trip including a zero-length summary record, Range requests including the suffix and open-ended forms, server-side decoding of both dtypes, a summary record returning no values, oldest-first pruning, the overlay guard refusing when either the build or the model differs, and a trace id that tries to walk out of the trace directory with ...

The lens — a trace read as words​

The Traces section is a debugger: pin a node, watch its numbers. The Lens section (packages/core/src/modules/llamacpp/lens/, backend backend/modules/llamacpp/lens.py) asks the other question, the one Neuronpedia's Jacobian Lens asks — at every layer and every position, what is the model disposed to say?

The readout is unembed(norm(J_L · h)), where h is the residual stream at (layer, position) — which the tracer already captures as l_out-L — and J_L is a per-layer transport matrix. Anthropic's J-lens fits J_L = E[∂h_final/∂h_L] over a corpus, which needs autograd this backend deliberately does not have. Set J_L = I and the same machinery is the classic logit lens: no artifact, no download, works on every GGUF on the machine. That is what ships. LensSpec names the transport, so a fitted J is a directory of J_<layer>.npy plus a lens.json rather than a rewrite — and the identity lens is labelled "identity (logit lens)" everywhere so it is never mistaken for a J-space readout.

On a real 40-layer Mistral trace of The capital of France is, Paris enters the top 4000 around layer 16, reaches rank 513 by layer 24 and rank 1 by layer 28. That progression is the artifact; a single grid is a screenshot.

The self-check is the correctness argument​

Per-architecture details are exactly the kind that yield confident wrong numbers rather than errors: Gemma folds 1 + into its norm weights at convert time, Gemma 2 softcaps final logits, most models tie output.weight to token_embd.weight. So at the last layer the identity lens must reproduce the trace's own captured result_output, and result_norm independently checks the norm on its own — the two are separately wrong-able, and a disagreement that names neither is a worse answer than none.

verified is therefore three-valued, the module's standing rule: true, false (rendered as a banner, never as numbers), and unavailable — the trace captured no logits to check against. A fitted lens always reports unavailable, because a J-lens does not reproduce the model's output by design; reporting true for one would mean the check was measuring nothing. It still says whether the identity path agreed, which is what tells you the machinery is sound even when the lens on screen cannot be validated.

The check costs nothing: it rides the grid's own streaming pass as one extra column rather than a second pass over the output head.

Reading a pruned graph​

The rows are ragged, and that is not a bug in the trace. llama.cpp prunes its graph to what the pass needs, so on a prompt pass the last block's l_out is one column wide — only the final position's residual produces logits — while every earlier block is the full width, and result_output is one column covering that same final position. Two things follow, both silent when wrong:

  • A narrow tensor covers the tail of the sequence, so column_of maps a position to a column and returns None when there is none. A cell that was never computed renders blank; falling back to the nearest column would put the last token's readout under every earlier one and make the top of the grid look uncannily prescient.
  • The self-check addresses both tensors from the end. Taking column 0 of a one-wide result_output against position 0 of the residual compares the last token's logits with the first token's activation, disagrees wildly, and reports every real trace as unverified.

The output head is never materialized​

For gemma-3-12b the head is 262144 × 3840 — 2 GB in fp16. Weight chunks stream past a matrix of every cell in the grid stacked together, so a 40 × 6 grid costs one pass over the head rather than 240. Dequantization is llama.cpp's own numpy implementation (the gguf package, ~110 kB over numpy); a quantization it cannot read is named and refused, the same instinct as traces.decode refusing a dtype it never wrote.

The dequantized fp16 head is cached under $HORRIBLE_DATA_DIR/llamacpp/lens/<modelSha>/unembed.f16.npy when it fits llamacpp.lensCacheGb (4 GB default) — measured on a Q6_K 131k × 5120 head, that takes a grid from 7.9s to 2.0s. Two traps are pinned by tests: np.save appends .npy to a path that lacks it, so a temp file named unembed.f16.part lands as unembed.f16.part.npy and the rename after it finds nothing (a cache never installed, plus a gigabyte of litter per grid); and a cache whose shape does not match is deleted rather than used, which needs the memmap released first because on Windows an open mapping keeps the file undeletable.

modelSha keys the cache, which is the same key the trace carries — two quantizations of one model are two caches, as they must be.

Narrowing, and why it is a stride​

The grid is a <table> with one row per layer and one column per token, so a 262-layer trace over a 200-token prompt is 52,400 cells — a request measured in seconds and a picture nothing can be found in. getLensGrid has accepted layers and positions since it was written and the pane passed neither.

The control is a stride, not a window, and that is the interesting choice. What a logit lens is for is seeing where along the depth a prediction settles. A window over layers 0–32 of a 262-layer model answers a different question: it shows a sixth of the model in full detail and hides where the answer actually appeared. A stride keeps the full span and samples it, so the knee is still visible and you can then narrow to a window once you know roughly where to look.

Three rules in narrow.ts (pure, tested):

  • The last layer is always kept. It is the model's real output, and a stride that happened to skip it would draw a lens whose final row is a prediction the model never made.
  • The window is applied first, then the stride samples within it. Striding first and clipping after gives a different row count depending on where the window sits, so widening it could make the picture coarser — the opposite of what the control says it does.
  • Only an unnarrowed response tells us the true depth. A narrowed one reports the rows it was asked for; treating that as the depth would let the suggested stride ratchet itself down on every refresh.

A deep trace opens at a suggested stride rather than opening broken; touching the control stops it being suggested again.

The track ribbon — where a word gets decided​

LensTrack carries logits[layer][position] and ranks[layer][position], and the pane used only the ranks: a superscript in a cell and a tint behind it. The logits were fetched on every track and thrown away.

That is the whole point of a logit lens, and it had no picture. A word does not fade in evenly across the depth — it sits near the floor for two thirds of the stack and then climbs steeply over a handful of layers. A per-cell tint cannot show a knee. A curve can, and the knee is the finding.

TrackRibbon draws both encodings on purpose. The line is the logit, the quantity the model computes. The band beneath is the rank, log-scaled, which is what that logit is worth against every other token — a rising logit stuck at rank 4000 has decided nothing. The first layer at which the token is the top choice is called out as a badge, and a token that never wins says so rather than leaving the badge absent, which would read as a missing feature.

Swapping a token​

A grid alone cannot separate the model knows this from the prompt said it. POST /traces/{id}/fork replaces tokens and re-runs, and the two grids are diffed cell by cell.

Three decisions carry it:

  • Token ids, not text. TraceRequest.tokenIds bypasses tokenization entirely. Re-tokenizing an edited string can merge or split its neighbours, so the run would differ in more places than the one you changed and the diff would be unreadable.
  • Everything else is inherited from the parent — model, generation length, layer selection, fidelity, capture set. A fork that also changed those is not a counterfactual.
  • fromId is stamped from the parent, never trusted from the caller. It is the record of what was actually replaced, and a client sending a stale one would make the fork's own provenance wrong.

Only the prompt is forkable: a generated token is an output, and replacing one would be editing the answer rather than the question. The diff aligns by (layer, position) rather than by array index, because a fork can differ from its parent in either and comparing cells[2][3] to cells[2][3] of a grid with a different layer subset silently compares different layers.

The picker searches the GGUF's own vocabulary (GET /models/vocab), not an HF tokenizer's: interpretability/tokenizer.py's family fallback can hand back a different generation's vocabulary while looking exact, and a swap has to name a token the traced weights actually have. It distinguishes Italy (#16507) from Italy (#93355), which is the distinction a swap turns on.

Capture sets​

TraceRequest.capture is now reachable over HTTP, and GET /traces/capture-sets serves the named sets with their real ggml node patterns — the plane_order precedent: two lists of node names in two languages is one upstream rename away from a capture set that matches nothing and fails silently.

lens is the one that matters. The lens reads the residual stream and the output head and nothing else, so it drops the default six-nodes-per-block to one: a 40-layer, 6-token trace is megabytes rather than a gigabyte, and seconds rather than minutes. That is what makes swapping a token and looking again a loop rather than an errand. traces.nodes_per_layer teaches the pre-flight estimate about it, counting only the per-block nodes — inp_embd and the output head are captured once per pass, and charging them per layer would inflate the estimate by half.

The trace catalog — why a trace is in two places​

trace_catalog.py keeps one row per trace in app.db (llamacpp_traces) while the activations stay on disk. That is the split library_sources and karaoke_songs already use, and it is not redundancy: a directory of manifests is answerable only by this module's own code, whereas a table in app.db sits beside agent_turns and traj_steps, so "which traces were taken during that agent run" and "every fork of this trace" become one query in the database console.

Disk stays the authority. A row is a description of a directory and nothing depends on it — delete app.db and you lose the query surface, not an activation. Two hooks and a reconciler keep the two honest:

  • trace_runner.run_trace records the row at the one point every trace, forks included, is known to have finished.
  • traces.delete_trace drops it. Hooked there rather than at the routes because prune() also deletes — a budget eviction is the path nothing else would ever tell the catalog about.
  • sync() runs at startup and reconciles both ways. A node that traced before this table existed has directories and no rows, and an empty catalog reads as "no traces" rather than "not indexed yet"; a manual rm -rf leaves rows claiming traces that are gone.

The model locus​

packages/core/src/model-locus.ts is "which part of the model am I looking at", the way locus.ts is "which code am I looking at": a global {modelSha, traceId, layer, position, tokenId} that panes publish to and follow. Click layer 15 in the lens grid and the model explorer reveals blk.15's tensors — neither pane imports the other or knows it exists. Core rather than a module, for the same reason locus.ts is: it is the spine several surfaces hang off.

One deliberate difference from the code locus: the /ws mirror is inbound only. The code locus is bidirectional because two panes in two windows publish to it. Here every publisher that matters is either in this tab — the grid and the explorer, which a local store notifies synchronously — or on the backend (dash.lens.focus, the lens.focus tool). Sending our own clicks up would buy nothing and cost a frame per mouse-move over a grid. So setModelLocus does not send, there is deliberately no lens entry in app.py's inbound channel ladder (only the outbound push_lens_events fan-out), and consequently there is no self-echo to suppress — which is why this has no origin field where the code locus needs one.

setModelLocus merges; an inbound one replaces. The grid knows the layer and position but not which vocabulary token is pinned, so a local replace would make every publisher restate facts it does not own and the first to forget would silently clear a pin. A locus off the wire came from a publisher that stated the whole thing it meant, and merging would leave a stale layer under an agent's "look at the embedding".

Agent tools​

Two groups, both loaded on demand.

llamacpp gains the trace verbs it never had: llamacpp.trace records a pass and llamacpp.list_traces reads the catalog. trace defaults to the lens capture set with attention off — megabytes and seconds — and full: true opts into the whole mechanism; attention needs full, since the score matrix is the largest thing a trace can hold.

lens is the readout: lens.grid, lens.cell, lens.track_token, lens.fork, lens.focus, lens.layer and lens.save_finding. lens.layer is what one block did, in numbers an explanation can quote — the share of each token's residual it rewrote and the angle it turned, its sharpest content heads and its attention sinks (ranked separately: a head parked on the first token is low-entropy too, so ranking by entropy alone just re-lists the sinks), and on a mixture-of-experts model which experts it routed tokens to. It reuses the stepper's reads, so the agent and the pane cannot disagree about a block. lens.focus takes a stage and a passIndex, which move the stepper's cursor — "look at block 17's attention" puts the user's screen there. Two tools rather than a new group: the group already sits under the per-turn tool budget, and a separate stepper.* group would be one more load_tools round for the same question. The group name equals the tool prefix because _group_of splits on the dot and AgentTool.group does not name the group — the connectors rule, and the trap llamacpp itself inverted.

Three things carry the containment:

  • An unverified grid is reported as unverified, never as numbers. Every payload carries verified in the same three states the pane renders, and false / unavailable lead with a warning rather than burying it under the cells. A model whose norm convention we get wrong produces confident wrong words, not an error, and an agent summarising those into prose is how a wrong reading becomes a claim.
  • A grid too large to carry is refused, with its dimensions. A tool answer is prose in a context window, not a screen an eye skips; MAX_TOOL_CELLS bounds it and the error names the axes so a caller narrows rather than guesses.
  • lens.fork and lens.save_finding are side effects. A fork mmaps the weights for minutes and writes a trace that may prune an older one; a save writes to the user's library.

lens.track_token accepts text as well as an id, and does not strip it: a leading space is what makes " Paris" a token and "Paris" not one, so stripping deletes the very thing the error message tells you to add.

dash.lens​

The REPL facade, for sweeps — "which layer does the answer first appear at, across these forty prompts" is a loop and a max(), which is neither a click nor a chat turn. Backend-local like dash.io, so it needs no browser attached; the one exception is focus(), which pushes to browsers and still works with none, since the locus is stored and the next window to attach follows it.

Saving a finding​

findings.py renders a grid plus the reader's note as a note source in a library — the browser.save precedent, and the reason it matters is that a trace is pruned the moment the budget wants its bytes.

A note is deliberately not an artifact: the activations are not copied, because megabytes of fp16 have no business being embedded and the numbers mean nothing apart from the weights that produced them. What is written down is the reading — prompt, top token per layer, and the provenance to reproduce it (model sha, llama build, trace id, and any edits, so a fork's note says what was changed).

An unverified grid is refused rather than filed with a caveat. A caveat in a note body survives exactly as far as the first retrieval that quotes the numbers without it.

"Trace this" on someone else's surface​

The module contributes a contextMenu provider for two target kinds it does not own — editor.selection and library.chunk — so the verb reaches the text instead of requiring the palette and a focused editor. Neither the editor nor the library learns this module exists, which is the point of the context-menu registry (see context menus). The editor reads the selection from the live view rather than the persisted bytes, for the same reason llamacpp.traceSelection does: tracing something other than what is on screen is worse than declining.

The stepper — a forward pass stepped like source​

The Traces section is a record list you pin and scrub; the Lens is a grid of words. The Stepper is the third reading: execution order. It needs nothing new on disk, because a trace already is an instruction stream — the tracer appends records in the order ggml computed them, each tagged with its pass and its block. stepper/program.ts turns the record list into a program and gives every debugger verb a meaning:

verblands on
step into / backthe next / previous graph node
step overthe first node of the next frame (a block of one pass; the embedding and output head are frames of their own, so stepping over the last block lands on the head rather than skipping it)
reverse step overthe start of this frame, then of the previous one
step outthe first node of the next forward pass
continue / reverse continuethe next node a breakpoint stops on
playstep over on a timer, stopping at any breakpoint crossed inside a frame

Reverse execution is free — a snapshot has no side effects to undo — which is the one way this debugger is better than the real kind.

One classifier. A step's stage (attention, ffn, moe, residual, …) comes from node-kind.ts, the same function the Traces filter uses. The program is built in the browser from the record list GET /traces/{id} already returns, precisely so no second list of ggml node names exists to drift from the first.

Breakpoints, including ones no source debugger has​

  • block — stops on entry to the block only, so Continue does not stop six times inside it;
  • stage ("every attention node") and node (one name);
  • threshold — a stage's rms above a value, evaluated against the per-pass profile. A node whose statistic was not measured never fires: stopping on a guess is worse than not stopping;
  • lens — "stop where the lens first says ' Paris'". The stepper reads the identity lens at the selected position for every layer of the pass (one /lens?positions=&passIndex=&k=1 call) and offers each token that is ever top-1, with the block it first appears at. Verified on Llama-3.2-3B with the prompt "The capital of France is": ' Paris' first wins at block 19, Continue stops on l_out-19, and the timeline marks blocks 19–27.

Breakpoint marks on the timeline are drawn with the same HitContext Continue evaluates, so a mark is exactly where Continue would stop.

What each stage shows​

stepper/StageView.tsx picks the view that answers the question you stepped there to ask. Every read goes through the stride-aware routes; a summary record has no values and says so. Reads are cached for the session (stepper/load-cache.ts, last 200), because a recorded trace never changes: stepping back to a node already seen paints it at once instead of "Reading…". A failed read shows its last error at once and is still retried, since an error may be transient.

  • Attention (kq_soft_max, kq) — a strip of heads (the fill is each head's weight on position 0, the attention sink), then the selected head's query × key matrix on HeatCanvas, with the selected position's row outlined and its top keys listed. kq is labelled as raw pre-softmax scores, where a query may score keys after it — causality is applied by the next node.
  • Residual (l_out) — what the block wrote: the share of each token's vector it rewrote and the angle the vector turned, and what the stream now says through the lens at this layer.
  • MoE (ffn_moe_*) — the layer × expert atlas and the experts the selected token was routed to, with their weights.
  • Anything else — the tokens × features plane and each token's norm.

Watch pins are shared with the Traces section, and a pinned node later in the pass than the cursor shows "not yet run" rather than its value: reading it would be reading the future.

The reads, and the stride trap they fix​

stepper.py owns them. get_record ships a flat prefix, which is right for a value strip and wrong for anything with two axes — the first 8192 values of a [256, 6, 24] attention record are a few rows of head 0. record_array rebuilds the 4-D tensor through its strides, and there is a trap in that:

nb is recorded in the source dtype. The tracer copies strides off the live tensor and then downcasts f32 to f16, so an fp16 record's bytes are half as wide as its strides claim. Element strides are nb / 4 for fp16 and nb / itemsize for full; dividing by the stored width doubles every stride and reads a different, plausible-looking tensor. Pinned by test_llamacpp_stepper.py, alongside a permuted view, KV cropping (kq_soft_max is as wide as the allocated cache — 256 on a six-token prompt — and its masked columns are exact zeros), and the pruned tail (the last block's output on a prompt pass is one column, compared against the last column of the block before, never the first).

Two things were found on real traces rather than assumed:

  • inp_embd is in the capture set and still absent on b10362 — the embedding lookup happens before the callback sees the graph. Block 0's delta therefore reports that its input was not recorded instead of subtracting zero.
  • Every attention row sums to 1 and the upper triangle is exactly 0, which is the check that the stride reading is right.

Mixture-of-experts routing is now captured​

ffn_moe_probs, ffn_moe_topk and ffn_moe_weights joined DEFAULT_CAPTURE — the literals were checked in the installed llama.dll (b10362), the SSM set's lesson. On a dense model they match nothing and cost nothing. ffn_moe_topk is i32 indices, so the tracer now keeps i32 tensors whole (a summary of indices is meaningless, and an f16 downcast is lossy past 2048), and traces.decode reads i32. Traces recorded before this change have no router nodes and the atlas says so.

The explorer follows the cursor​

The cursor publishes traceId, modelSha, layer, stage, step, passIndex and position to the model locus. The model explorer selects that block, lights the sub-block the stage names, and shows a following the stepper strip; clicking a block or sub-block on its diagram publishes back, and the stepper jumps there. An echo of the block the cursor is already in is ignored, so the explorer re-selecting what it was just shown cannot yank the cursor to the block's start.

Other surfaces open a trace in the stepper through stepper/target.ts (openInStepper with a trace id and a step or a layer) — the trace-prompt.ts seam, for the same two-directional mount race. dash.lens gained program, tensor, attention, delta and experts (numpy, the same functions the routes call), and focus(stage=..., step=...) moves the cursor from the REPL.

Not done here​

topk fidelity is declared as an idea and not implemented — only full, fp16 and summary are written, and the enum lists exactly those. Traces are not yet overlaid on a chat turn in the interpretability pane: traces.matches_run is the guard that work will need, but the pane wiring is not built. The tracer also samples greedily and traces a raw prompt, so it reproduces a forward pass, not a chat turn.

No fitted J-lens ships yet: the artifact format, the picker and the unavailable verdict for a non-identity lens are all in place, but importing Neuronpedia's published .pt lenses (which needs a torch environment to convert once) and fitting our own are not built.

Distributed computation is designed for and not built: a trace, a grid and a fork are all job-shaped, so a peer holding the same modelSha could run one and return only the result — with traces.matches_run generalizing exactly (a result computed elsewhere may only be shown here when the build and the model hash agree). Nothing crosses the fabric today.