Module: llamacpp
The node serving its own weights. Ollama and LM Studio are applications the user
installs and runs elsewhere; this module makes the backend itself fetch a
llama-server build, keep a catalog of GGUF files, and supervise the process the
agent talks to.
Status: implemented — backend in backend/modules/llamacpp/, frontend in
packages/core/src/modules/llamacpp/, tests in backend/tests/test_llamacpp.py.
Agent tools
This module had none until now, which made the fine-tuning agent's own prompt a description of a loop it could not run: it could convert a checkpoint and then had no way to serve the result — the step immediately before the eval it is told to run.
| Tool | Does |
|---|---|
llamacpp.status | Whether a local GGUF is being served, and which |
llamacpp.list_models | Every servable GGUF — the managed directory plus whatever Ollama and LM Studio already have — with its base model where this node trained it |
llamacpp.serve | Load a GGUF into the local server |
llamacpp.stop | Stop it, freeing the memory |
llamacpp.serve refuses rather than replacing a server that is already running a
different model. llama-server holds one model at a time, so serving a second means
stopping the first — and the first may be the model the user is chatting with, which is
not a decision a tool call should take on their behalf. Asking for the model already
loaded is neither an error nor a reload; it reports alreadyServing.
wait_ready returning False (rather than raising) when a model is too large to load is
the trap on this path: reporting success there is how a caller comes to score a model
that never came up and read the zero as the model's fault. serve turns it into an
error.
Deliberately not exposed: installing or removing a build, and deleting a model. Fetching a multi-gigabyte binary or erasing weights belongs to a person looking at a disk-budget readout, not to a turn that inferred it would be helpful.
Note this inverts a rule recorded in two places. roster.py and
training both said llamacpp must never appear in an agent's
tool_groups, because a group is a tool name's prefix and that namespace held only
settings keys — true until these tools existed, wrong after. Both comments are
updated.
Why this is not just another provider entry
Every other local provider answers "which model are you running?" with a name. It
cannot tell you which file that name resolves to, what quantization it is, or how
big it is — which is why interpretability has to
reverse-engineer a path out of Ollama's blob store and LM Studio's undocumented
index just to read a tensor table. Here the node picked the file, so the path is
known rather than inferred, and the model explorer gets its highest-confidence
source for free.
The second reason is that it removes the last hard dependency on somebody else's app. A packaged node can serve a model with nothing else installed.
Three artifacts, three lifetimes
| Where it comes from | Where it lives | Who owns it | |
|---|---|---|---|
| The binary | Upstream GitHub release, downloaded on demand | $HORRIBLE_DATA_DIR/llamacpp/bin/<tag>-<variant>/ | us |
| The weights | Hugging Face, or already on disk | $HORRIBLE_DATA_DIR/llamacpp/models/, or Ollama's / LM Studio's store | mixed |
| The process | Spawned by server.py | pid + a loopback port | us |
Nothing is bundled. That is the same call as SearXNG, AssaultCube content and the DB-IP database: we support it, we fetch it on request, we ship none of it.
Asset selection is matched, not hardcoded
Release asset names have changed shape several times, so pinning exact strings
guarantees a break on some future release. select_asset matches on the tokens that
carry meaning — OS, arch, accelerator — with two rules that are the whole test suite:
- An accelerator build is never substituted for
cpu. A CUDA archive's name contains every token a CPU one does, plus one. Downloading it onto a machine with no NVIDIA driver produces a binary that fails to load its runtime, which reads like a corrupt download. - The extension is per-OS. Upstream ships
.zipfor Windows and.tar.gzeverywhere else. Matching only.zip— as this did until it was caught — reduced the whole module to a Windows feature, and reported it in upstream's voice: "release b10373 publishes no cpu build for ubuntu/x64". Both formats now match and_extractpicks the reader by extension rather than by OS, because the extension is what the bytes actually are. - The name must start with
llama-. Releases also carrycudart-llama-bin-win-cu12.4-x64.zip, the CUDA runtime redistributable, which matches every other token, is shorter than the real CUDA build, and contains no server at all.
When nothing matches, the error names the assets it did see, rather than failing blind on a release whose naming moved again.
The picker knows before you click install
select_asset used to run only after the user picked a variant and hit
Install — a release publishing no CUDA build for Linux surfaced as an error only
once the download was already "starting." GET /llamacpp/install/variants runs
the same match ahead of time, for every variant against the current release and
this OS/arch, and the pane's <select> greys out (disabled) any option that
comes back unavailable, labelled with the release tag. It reuses select_asset
rather than a second notion of "available" that could disagree with what
actually gates the download; a failed lookup (offline, GitHub down) leaves every
option enabled, since "we could not ask" must never be shown as "unavailable."
Which variant, though — the hardware probe decides
variant used to default to a flat cpu, which is correct on a machine with no
GPU and silently wrong on every machine with one. It now defaults to "auto" and
is resolved by the hardware probe, which still falls back to cpu
whenever it could not determine what the machine has — the asymmetry above still
holds, and "we could not ask" must never install a CUDA build.
The card alone does not decide the variant; the OS does too. Upstream
publishes CUDA for Windows only — there is no ubuntu-cuda-* asset in any
release, because Linux NVIDIA users are expected to build it or run the Vulkan
build. So an NVIDIA card on Linux resolves to vulkan, with a reason string that
says why. Getting this wrong is not a slow build, it is no install at all, and
it only became reachable when the default moved from cpu to auto.
test_upstream_publishes_no_linux_cuda_build pins the upstream fact, so if a
Linux CUDA asset ever appears the special case can be deleted on a failing test
rather than on a hunch.
The same probe supplies gpuLayers and threads at spawn and tokenCap for a
trace. Those request fields are nullable, and the resolution is an is None
check rather than a falsiness one: an explicit 0 is a request for pure CPU and
has to survive.
A form must send the nulls, or none of the above happens. The pane originally
initialized its own state to variant: 'cpu' and gpuLayers: 0 — plausible-looking
defaults that are in fact concrete user instructions, so variant in ("", "auto")
and gpuLayers is None never fired and the whole probe was bypassed by its primary
caller. A machine with a 4090 installed a CPU build and offloaded nothing, with
nothing in the UI to explain it. The pane now holds 'auto' and null, renders the
resolved value as the input's placeholder, and prints the matching reasons
string under the control. Three properties are load-bearing:
- An empty box is a question, a typed number is an instruction.
0in the GPU layers box still means pure CPU, and says so, rather than being silently reinterpreted as "auto". - The reading sits next to the decision. The probe's findings used to live only in the Settings page. The pane now shows the accelerator line above the Build card, with a Re-probe button, because that is where the choice is made.
- Unknown is not none. With
certain: falsethe line says the probe could not ask — the same rule the hardware settings section follows.
The offload preview — how many layers actually fit
GET /models/layers (offload.py) sums the selected GGUF's tensor directory into
per-block byte totals, and the pane draws the stack against the card's capacity with
the GPU-layers slider on top of it. Everything it needs was already on disk — the
directory carries a size and a block index per tensor — so the answer to "how many of
these fit?" was one sum away and was simply never taken.
Two of llama.cpp's own rules are load-bearing, and reversing either draws a confident picture of the wrong thing:
- The offloaded blocks are the LAST N.
llm_load_tensorssetsi_gpu_start = n_layer - n_gpu_layers. Taking the first N gives the same total (blocks are near-uniform) while marking the wrong end of the stack. - The output tensors only move once N exceeds the block count — what
-ngl 99is for. On a large-vocabulary model that is most of the file: Gemma 4 E4B keeps 2.9 GB of its 5.3 GB in the embedding and output head, so folding them into "layers" would be wrong by more than half.layerBytesandoverheadBytesare therefore separate fields, and the bar shows the jump when the slider passes the end.
The KV cache is the one modelled quantity, and it earns its place: it is computed
from attention.head_count_kv, so a GQA model's four-times-smaller cache is reported
as such rather than inflated from the query head count. It is returned per token
so the pane multiplies by the context size the user is still free to change — at
65k context on a 20B model the cache alone is 3 GB, which is the difference between
fitting and an OOM at load, and baking a context into the server's answer would make
it silently wrong the moment the field moved. It gets its own band on the bar rather
than being added to the weights, because it is VRAM the file size does not account for.
What it does not claim: compute buffers, the CUDA context and driver overhead
depend on the build and the batch size, not on the weights. Rather than a fudge factor
hidden inside the total, a flat VRAM_RESERVE_BYTES is held back and named in the
tooltip. And with no measured VRAM — a card reporting no memory size, a probe that
could not ask, or unified memory where the "VRAM" is the RAM — the ceiling is simply
not drawn and no verdict is given, because "unknown" rendering as "fits" is the exact
failure the hardware module exists to prevent.
The arithmetic lives in offload.ts (pure, unit-tested) rather than in the component,
including the Fit it suggestion, which is the largest count that still fits and is
reported as layerCount + 1 when everything does — that being the value to actually
send.
hardware.defaults() decides gpuLayers from VRAM alone, so on a 12 GB card it says
"all". The preview will show that overflowing once you raise the context, because it
knows the model and the context and the probe knows neither. The preview is right; the
default is a starting point.
The planner draws every block, not three totals
The preview above began life as three stacked widths — weights on the GPU, KV cache, weights left in RAM — with a ceiling line across them. That drawing could state both of the rules in the previous section and show neither, because a single bar has no place to put a block index.
LayerColumns.tsx draws one column per decoder block, height from the real
layerBytes[i], filled from the right. Dragging the slider then teaches the last-N
rule without a sentence: the fill travels backwards down the stack, which is exactly
what i_gpu_start = n_layer - n_gpu_layers does. The embedding and output head are a
separate, wider column set apart from the rest, so the 2.9-of-5.3 GB fact is a thing
you can see rather than a number in a paragraph, and it visibly stays empty until the
count passes the block count.
The ceiling changed shape too. offloadColumns (pure, in offload.ts, tested beside
estimateOffload) accumulates a running total in offload order — last block
first, output tensors last — and the VRAM line crosses that curve at exactly the block
where the model stops fitting. "Does it fit" becomes a position rather than a verdict.
Two honesty rules the columns inherit and one they add:
- No measured VRAM means no ceiling and no verdict, unchanged from the bar.
- A block whose bytes could not be sized still gets a tick, at a floor height. The rail is an index as well as a chart, and a missing column would shift every label after it.
- A budget larger than the whole model draws no ceiling either. Clamping the line to 100% put it across the top of the chart, which reads as "this exactly fills the card" — an assertion of a crossing that does not exist. A model that fits entirely has no crossing point, and the honest drawing of no crossing is no line, plus a sentence saying so.
estimateOffload and offloadColumns are checked against each other at every slider
position, so the legend and the picture cannot drift apart.
The drawing primitives are shared now
Four of the visuals on these tabs used to be private to one component: a sparkline
and a heat strip inside TracesSection.tsx, a progress bar in ServerPane.tsx, and
a segmented budget bar in OffloadPreview.tsx — the last of which had two more call
sites that had settled for a sentence instead. They live in packages/core/src/viz/
now, beside Primitives.tsx and DataList.tsx which set that precedent.
Three of them carry a rule that used to be a comment:
spark.ts— a gap is a gap. A pass whose record was summarized without a statistic has nothing to report, and joining the line across it draws a measurement that was never taken. A lone measured point between two gaps becomes a dot, because a one-point polyline draws nothing at all — silently.Meter—threshold: nullmeans no verdict. The "unknown VRAM draws no ceiling" rule is now in the type: there is deliberately nofitsprop, so a caller cannot assert a verdict the component would have refused to draw.ramp.tsreturns a sign and an alpha, never a colour. The oldcellColorreturnedrgb(110 190 255 / 34%), which renders identically on the light theme and slipped pastno-hex-literals.test.tsonly because that guard matches#rrggbb. Passing the ratchet is not the same as being themed. The sign becomes a data attribute and the alpha a custom property, resolved in CSS against two new per-theme tokens,--ramp-posand--ramp-neg— the only two tokens whose job is to be read as a direction rather than as a status, which is why they are not--accentand--danger: a negative activation is not an error.
HeatCanvas is deliberately not added to the hex guard's CANVAS_EXEMPT bucket.
That exemption exists because a CanvasRenderingContext2D cannot resolve a custom
property — true, but it only covers a canvas with no way to ask. This one reads its
ramp through getComputedStyle and repaints on a theme change. The cautionary case is
training/panels/MetricsPane.tsx, which passed var(--accent, #539bf5) straight into
uPlot: ctx.strokeStyle ignores an unparseable value without throwing, so that
series was drawn in uPlot's default colour on all six themes while looking, in the
source, fully themed. viz/uplot-theme.ts resolves tokens properly and that bug is
fixed.
The machine band — what the probe found, and what it could not ask
The Server section used to open with one line naming profile.primary. Three things
the hardware probe had already measured were reaching the pane and being dropped:
accelerators[], not justprimary. A machine with a discrete card and an integrated GPU showed one of them, so "why did it pick that one" had no answer on screen. Each device now gets a card with its own memory meter and ameasured/reportedmark taken fromAccelerator.exact.notes[]— the gaps. This is the hardware module's entire reason for existing and it had no renderer anywhere in the app. An absentnvidia-smiand an absent GPU produce the same empty list and are not the same fact; the backend is careful to distinguish them and the UI flattened it. They are now listed under "what the probe could not ask".defaults.reasonsfor every auto-resolved field, not the two that had tooltips.
installs[] had the same problem: always fetched, never listed. A machine that has
changed build variant a few times accumulates them — five, on the machine this was
developed against, four of them stale CUDA builds — and POST /install/remove had no
frontend caller at all. The list is now rendered whenever there is more than one, and
the Remove button says why it is disabled while the server is up rather than letting
the backend answer 409.
Finally running and ready are drawn as the two separate facts the API models them
as. Loading a large GGUF takes tens of seconds during which the process is up and
every request fails; collapsing that into one sentence made a normal load look like a
fault and gave no way to tell it from a server that is genuinely stuck.
Verification says what it actually knows
GitHub publishes a digest for release assets. When it is there, a mismatch aborts
the install and nothing is written — an unverified half-install that a later
newest_install() hands to the spawner is worse than no install. When it is absent,
the digest we computed is recorded and the install is marked verified: false,
which the pane renders as a chip. Recording a self-computed hash and calling it
verification would be theatre.
The catalog
Three origins, and the distinction is load-bearing:
managed— under our models directory. Downloadable, deletable, counted against the disk budget.ollama/lmstudio— files those apps own. Serveable, never touched.llama-servermemory-maps a GGUF read-only, so pointing it at LM Studio's copy costs nothing, and duplicating a 20 GB file to "own" it would be absurd. But a delete route that can reach into another app's store is a footgun, so the origin gates deletion — andis_managedresolves both paths, because a..segment in a caller-supplied path is exactly how a delete route becomes an arbitrary-file-delete route.extra— directories added viallamacpp.modelDirs.
mmproj-*.gguf is skipped everywhere. A multimodal repo ships its vision projector
beside the weights: a real GGUF that parses cleanly and cannot be served as a chat
model, so offering it in the picker is a guaranteed bad spawn. It is the same trap
the model explorer hits from the other direction.
The quantization label is the most common tensor type, not general.file_type:
that KV is an enum id whose meaning has shifted between llama.cpp releases, and a
mixed-quant build (most K-quant files keep some tensors at F32) is the norm. Header
facts are memoized on (path, size, mtime) — scanning a directory of 20 GB files on
every status poll would otherwise re-read all of them.
Models you trained land here too
The training module converts a finished checkpoint into a GGUF in
the managed directory, so it shows up in this catalog like any download and is
served the same way. That is the loop closing: train it here, serve it here, look
inside it here. A LoRA result is labelled an adapter, because llama-server takes
it with --lora beside a base model rather than serving it alone.
GET /models carries that provenance, not just the agent tool. Each row gets
baseModel, projectId and isAdapter, joined from training.lineage on the GGUF
path. Without it the pane — which is where a person actually picks a model — listed a
fine-tune by whatever general.name the converter wrote: the adapter of a Qwen
fine-tune appeared as a servable model called "Qwen/Qwen3-0.6B", and the merged model
beside it as "Merged". The row now says "fine-tune of <base>", the adapter is
tagged in the list and in the start picker, and a missing row reads as unknown
provenance rather than as a claim about a base.
The disk budget is pre-flight
A download is refused before a byte is written, against the size the server
declares in a HEAD. Checking as it goes means discovering the disk is full 30 GB
into a 40 GB transfer. The ceiling is llamacpp.diskBudgetGb (default 80).
Downloads write to a .part file and rename on success, so an interrupted transfer
can never be picked up by the catalog scan as a servable model — a truncated GGUF
fails at load time with an error that reads like a corrupt model.
Reading the catalogue
The model list is grouped by origin with each group's root named, because "why is
this file here" and "why can I not delete it" are the same question and origin is
the whole answer — it was previously a bare tag on a row, sorted next to nothing.
extraDirs is finally rendered, so a file from a user-configured directory says which
one.
Two things that were sentences became pictures:
- The disk budget is a segmented meter, one segment per model, hovering either
way between a segment and its row.
usedBytes of budgetBytestold you the total and never which file was responsible for it. - Quantization cost is a log-log scatter of
parametersagainstsizeBytes. Plotted against each other the two numbers stop being two numbers and become a rate — bytes per parameter — and each quantization family falls on its own line through the origin, with guides drawn at 0.5, 1, 2 and 4. Q8_0 sits at about twice Q4_K_M's slope and F16 at four times, which is readable without knowing what any of the names mean. Log-log because a catalogue spans 0.5B to 70B and a linear axis puts every small model in one pixel; a constant ratio is still a straight line after the transform, so the property being read survives it. It refuses to draw under three points — a scatter of one is a dot with axes implying a trend nobody measured.
Each row also carries a context rail against the longest context in the catalogue, so the over-context warning on the Server tab is predictable before you type a number rather than after.
The server
--jinja is passed explicitly, never inherited
It selects the model's own chat template, which is what carries its tool-call
syntax. Upstream currently defaults it on — verified on b10362, where a server
started without it reports the same template — but the default has flipped before,
--no-jinja exists, and it also reads LLAMA_ARG_JINJA from the environment. So
whether the agent can call tools would otherwise depend on which build happened to
be downloaded and what is in the user's env.
Without the model's template, llama-server accepts only "commonly used" templates
and falls back to a generic one for the rest — and that failure is silent: 200,
a perfectly fluent answer, and every tool in the schema simply never called, which
is indistinguishable from "this small model is bad at tool use". Hence the explicit
flag, and the test that pins it.
Ready is not the same as running
Loading a large GGUF takes tens of seconds, during which the process is up and every
request is refused. wait_ready polls /health; the status the pane reads
distinguishes running from ready, and a process that dies during load reports
its last log lines rather than a timeout.
Two more things that would otherwise bite
- The port is negotiated, not assumed. Hyper-V reserves ranges that swallow
ordinary dev ports on this machine, and 8080 is a popular one, so the manager
binds the default if it can and takes an ephemeral port if it cannot. This is why
_endpoint_forchecks the live manager before the saved config for this provider alone — a saved:8080would point at nothing while the real server sat elsewhere. - Spawning goes through
subprocess.Popenon a thread. Underuvicorn --reloadon Windows the loop is aSelectorEventLoop, where asyncio subprocess creation raises — the same reason the LSP and PTY managers do it this way. The app's lifespan stops the server on shutdown, or an orphan survives a reload holding gigabytes of mapped weights and a bound port.
How it was started is recorded, so it can be put back
spawn stores a SpawnSpec — model path, alias, port, context size, GPU layers,
threads, extra args — and spawn_spec hands it to anyone who has to restore the
server. There is exactly one process and one model, so anything that wants to
borrow it has to be able to give it back: evals
loads a GGUF to score it and then restores whatever was running.
Restoring from the spec rather than from defaults is the whole point. Defaults
would bring the user's server back with gpu_layers reset, extra_args gone and
context_size shrunk to 4096 — a machine that is quietly slower afterwards, with
nothing said. The same failure hardware.defaults() exists to prevent.
Chat integration: four lines, not a new dialect
llama-server speaks the OpenAI API, so this is one PROVIDERS entry with
dialect="openai" and can_spawn=True. Everything already built on that dialect —
streamed reasoning, tool-call assembly from indexed deltas, the
tool_choice="required" retry — works unchanged.
A bespoke dialect would have meant six new branches in providers.py and would
have silently lost that retry, which is gated on info.dialect == "openai".
It is deliberately not can_pull: weights are fetched by this module's catalog, not
by asking the inference server to pull them the way Ollama does.
Speculative decoding
A small draft model proposes tokens and the large target verifies a batch
of them in one pass. It is local — -md/--model-draft takes a file path, and
there is no remote-draft mode in llama.cpp, so a peer cannot supply draft tokens
however convenient that would be.
Flags are probed, never hardcoded
features.probe_flags() shells llama-server --help and reports which spelling
this build wants, cached by binary path and mtime.
Two reasons, both silent failures otherwise. Upstream renamed the tuning flags
under a --spec- prefix in 2026 (--draft-max → --spec-draft-n-max), and both
spellings are in the wild; passing the wrong one does not warn — the server exits
during load and wait_ready reports the generic "did not start", which reads as a
broken model. And --rpc is a compile-time feature whose presence in upstream's
release binaries has changed over time: b10453-cuda, which this module
downloads, does have it, so a documented constant would have been true when
written and quietly wrong later.
The probe answers in three states (hardware/probe.py's rule): present,
absent, or could-not-ask. A knob the build does not advertise is dropped from
the command line rather than swapped for the other spelling — its absence means
the build uses its own default, which works, where a wrong flag does not.
A draft must share the target's tokenizer
speculative.check_compatible compares tokenizer.ggml.model, tokenizer.ggml.pre,
vocabulary size, a hash over a stride of the token list, and the BOS/EOS ids.
It refuses rather than warning, because the failure is invisible: the draft proposes token ids, so a mismatched tokenizer collapses the acceptance rate and makes generation slower than not drafting at all, while looking like it worked. A warning on a feature whose only symptom is "mysteriously slower" gets ignored. For the same reason an unreadable header is a refusal, not a pass, and compatibility is read from headers rather than inferred from names — files sharing a name prefix routinely have different tokenizers.
llamacpp.find_drafts lists compatible candidates smallest-first, and also
reports whether the build can draft at all, which saves a doomed spawn.
Dividing the card
hardware.defaults().gpu_layers stays 0-or-999 — every caller depends on that
number, and this is the only place that needs to split a card between two models.
speculative.spec_plan does the division, reusing offload.layer_plan and
inheriting its honesty: that function reports weights and KV cache and explicitly
declines to invent a compute-buffer allowance, so headroom is a caller-supplied
parameter here too.
The draft goes on the card whole or not at all. Splitting an already-small
model across host and device costs a PCIe round trip per token on the very path
whose cheapness is the entire premise of drafting. If it will not fit alongside
anything, spec_plan offers no plan and says why, rather than guessing — a guess
here is an out-of-memory error at load.
Measuring it, because it may not help
specbench.py is the deliverable, not garnish. On a single consumer card the
draft's VRAM comes out of the target's KV cache, and drafting costs a forward pass
per batch whether the guesses land or not; the trade can genuinely be negative.
It reports p50 and p10, never a mean (variance is the defining property — a
mean hides that a code prompt got 2.4× and a prose prompt 0.8×), per prompt,
never only aggregated (an aggregate over mostly-boilerplate prompts is how you
convince yourself of a speedup you will never see), and A/B back-to-back on the
same machine, since tokens/sec is not comparable across runs on a laptop that
throttles. compare() may return a verdict of "slower" — that is a result.
The acceptance rate is included because it explains the outcome: 20% acceptance and slower is a model pairing problem, 70% and slower is a memory problem.
Per-agent provider selection
model, temperature, contextSize, maxTokens and topP have been per-agent
since the roster landed. provider and endpoint were not — which made "run the
coder on the node's llama.cpp server and leave the orchestrator on Ollama"
unexpressible, because the only per-agent knob was a model name, and a name means
nothing on a server that does not have that model.
roster.resolve_provider(config, agent_id) resolves both, per key:
agent.<id>.provider → agent.orchestrator.provider → the saved global config. An
override naming an unknown provider is ignored, not fatal — a stale settings
value must not take the agent down.
run_agent_loop is called from three places: the orchestrator
(orchestrator.py), local delegation (delegate.py), and the flow executor's Agent
node. A site left on provider_for(config.provider) does not fail — it quietly runs
that path on the global provider, which is how a delegate ends up on a different
model than the one its settings show.
Contributions to the layout shell
| Kind | ID | Notes |
|---|---|---|
| Panel | llamacpp.server | "llama.cpp" 🦙; singleton, role document — running a model, not reading one |
| Section | llamacpp.server → server | Default, key s — build + process + log |
| Section | llamacpp.server → models | Key m — catalog + downloads |
| Section | interpretability.suite → traces | The activation scrubber — a section of the interpretability suite |
| Section | interpretability.suite → lens | A trace read as words (the Trace lens tab) |
| Section | interpretability.suite → stepper | The layer debugger: a trace in execution order (see "The stepper" below) |
| Command | section.show:llamacpp.server:{traces,lens,stepper} | Kept as shims for those sections' old ids; they open the suite section |
| Command | llamacpp.open | llama.cpp: Server and models |
| Command | llamacpp.openStepper | llama.cpp: Step through a trace (/stepper) |
| Command | llamacpp.stepper.* | Step into/back/over/out, continue (both ways), toggle breakpoint, play, restart — no-ops unless the Stepper section is mounted |
| Keybinding | stepper verbs | →/← step, ↓/F10 over, ↑/Shift+F10 back over, Shift+→ out, C/Shift+C continue, B breakpoint, Space play, Home restart — scoped paneFocus == 'interpretability.suite' && paneSection == 'stepper' && !textInput, because the suite keeps the stepper mounted behind its other sections |
| Command | llamacpp.traceSelection | llama.cpp: Trace editor selection (/trace-selection) |
| Setting | llamacpp.modelDirs | Extra GGUF directories to scan |
| Setting | llamacpp.diskBudgetGb | Ceiling for the managed directory |
| Setting | llamacpp.traceBudgetGb | Ceiling for stored traces (default 2 GB) |
| Setting | llamacpp.lensCacheGb | Ceiling for the dequantized output-head cache (default 4 GB) |
One pane, five sections — the build, the weights, the traces and the lens are one workflow in a strict order (no build, no server; no weights, nothing to serve; nothing served, nothing to look inside; nothing traced, nothing to read), so splitting them would give panes of which most are usually an instruction to open another. Traces and Lens are two readings of the same object rather than two objects, which is exactly what the sections primitive is for. The pane is also tabbed into the Lab frame beside the Hub and the model explorer: find weights → serve them → look at what you served is one working position.
Provider and endpoint overrides are edited in Settings → Agent orchestrator, beside the model override they have to agree with.
Tracing what you are actually writing
llamacpp.traceSelection (/trace-selection) sends the editor's selection — or
the whole buffer when nothing is selected — into the traces section as the prompt,
then reveals that section. The prompt box was otherwise a fixed
'The capital of France is', which demonstrates the machinery and answers nothing
anyone was wondering; fed the buffer instead, the token strip becomes your own
identifiers and (with attention on) the attention map is a map over your code. It
pairs naturally with the fact that traces apply no chat template: what is
tokenized is exactly the text, which is what you want for a code fragment.
Four things it does deliberately:
- The editor is reached through its registered service,
EditorService(registry.getService('editor')), which gainedgetBufferSelectionfor this — the same seam the visualizer uses, never a deep import. - An unmounted buffer is declined, not guessed. A selection exists only in a live CodeMirror view; falling back to the persisted bytes would trace something other than what is on screen.
- An untitled buffer is reported specifically. The editor tracks the active buffer by source URI, and an unsaved one has none — so the message says "needs a saved buffer" rather than "needs an open buffer", which would read as a bug when one is plainly open. See editor.
- The origin is shown and clears on edit. The section prints "From
foo.py(selection)"; typing in the box drops the label, because a textarea that silently repopulated itself is indistinguishable from a bug.
The handoff itself is trace-prompt.ts: a pending value plus a subscription,
because revealing the section mounts it fresh (a prompt pushed first has no listener
yet) while a push into an already-open section has no mount to read it. Either
mechanism alone drops half the cases, and the read clears the value so a later visit
does not resurrect an old selection over what you were typing.
Backend surface
| Method | Path | Purpose |
|---|---|---|
GET | /api/llamacpp/status | Build, process, readiness, log tail |
POST | /api/llamacpp/install | Download + unpack a build (NDJSON progress) |
POST | /api/llamacpp/install/remove | Delete a build (refused while it runs) |
GET | /api/llamacpp/models | Catalog + disk usage + starter repos |
GET | /api/llamacpp/repo?repo= | GGUF files in a Hugging Face repo, with sizes |
POST | /api/llamacpp/models/download | Fetch one GGUF (NDJSON progress) |
POST | /api/llamacpp/models/delete | Delete a managed model (403 otherwise) |
GET | /api/llamacpp/models/layers | Per-block byte sizes + KV cost per token, for the offload preview. Catalog-only — it opens a client-supplied path, so an uncatalogued one 404s rather than becoming an arbitrary-file read |
POST | /api/llamacpp/start | Spawn on a chosen GGUF, optionally waiting for ready |
POST | /api/llamacpp/stop | Terminate |
GET | /api/llamacpp/traces | Stored traces, disk usage, and whether tracing is available at all |
POST | /api/llamacpp/traces/estimate | Pre-flight cost, from the GGUF header |
POST | /api/llamacpp/traces | Run one traced forward pass (NDJSON progress) |
GET | /api/llamacpp/traces/{id} | Manifest, record list, tokens |
GET | /api/llamacpp/traces/{id}/record/{i} | One record's values, decoded server-side (capped at _VALUE_CAP) |
GET | /api/llamacpp/traces/{id}/series | One node's statistic per forward pass, by name. Summarizes the whole record, unlike /record/{i} |
GET | /api/llamacpp/traces/{id}/tensors | The raw blob, honouring Range |
GET | /api/llamacpp/traces/{id}/record/{i}/matrix?head=&cols= | One 2-D plane read through the record's strides: tokens × features (max-abs pooled, full-resolution row norms) or one attention head's query × key matrix cropped to the KV in use |
GET | /api/llamacpp/traces/{id}/record/{i}/heads | Per-head entropy / self-weight / first-token weight for an attention record |
GET | /api/llamacpp/traces/{id}/heads | Every head of a pass scored (heads.py): previous-token, induction, duplicate, sink, self, entropy — induction/duplicate null on a prompt with no repeats |
POST | /api/llamacpp/heads/probe | Record the induction probe on a GGUF: a random word sequence twice, attention + l_out only; streams like /traces |
GET | /api/llamacpp/traces/{id}/layer/{L}/delta?passIndex= | What block L wrote into the residual stream, per token (relative change, cosine) |
GET | /api/llamacpp/traces/{id}/experts?passIndex= | Mixture-of-experts routing: per-token expert choices and weights, per-layer counts. moe: false on a dense model |
POST | /api/llamacpp/traces/{id}/fork | Re-run with some tokens replaced (NDJSON progress) |
GET | /api/llamacpp/traces/capture-sets | The named capture sets and their real ggml node patterns |
GET | /api/llamacpp/traces/{id}/lens | The layer × position readout |
GET | /api/llamacpp/traces/{id}/lens/track | One vocabulary token's logit and rank at every cell |
GET | /api/llamacpp/traces/{id}/lenses | Which lenses apply to this trace's model |
GET | /api/llamacpp/models/vocab | Search a GGUF's own vocabulary — the token picker behind a swap |
GET | /api/llamacpp/traces/catalog | The llamacpp_traces rows in app.db — filterable by model hash or parent, which the directory listing cannot do |
POST | /api/llamacpp/traces/{id}/finding | File a lens reading into a library as a note. A 200 with error set is a refusal, not a failure |
DELETE | /api/llamacpp/traces/{id} | Delete a trace |
Progress rides NDJSON on the request, not a /ws channel — the shape
/agent/pull already uses, and it keeps a multi-gigabyte transfer scoped to the
request that asked for it: navigating away cancels it rather than leaving a
broadcast running for nobody. The shared client helper is streamNdjson in
packages/core/src/api.ts.
A Hugging Face token is used when the connector is connected, and not required: the great majority of GGUF repos are public, so gating weights on an OAuth flow nobody needs would be backwards.
Browser vs desktop
No difference. The server is a backend child process and the pane is ordinary HTTP — both layouts drive the same node. The only asymmetry is the usual one: a browser layout pointed at someone else's backend manages that machine's models, which is what "the node serves its own weights" means.
Testing
backend/tests/test_llamacpp.py. The tests worth keeping are the ones pinning
failures that are otherwise silent:
--jinjapresent on every spawn, so the model's own chat template is never left to a build default or an env var.- An accelerator build is never selected for a
cpurequest, andcudart-*is never mistaken for a server. - A digest mismatch leaves no install behind.
- An asset with no published digest installs as
verified: false. deleterefuses a path outside the managed directory, including one that walks out of it via...- The disk budget refuses before writing.
- A per-agent provider override reaches resolution, an unknown one degrades to the configured provider, and an unoverridden agent is untouched.
The tracer
Everything above serves tokens. This part serves activations: a traced forward pass, captured node by node out of ggml and written to disk.
It is a separate artifact from the chat path and stays that way. Chat runs a
downloaded llama-server binary; tracing runs whatever libllama the
llama-cpp-python wheel bundled, installed by uv sync --extra llamacpp and pinned
with == rather than >=. The chat provider must not depend on the risky code, and
the two llama.cpps genuinely disagree — which is why a trace may only be overlaid on
a chat turn when both llamaBuild and modelSha match (traces.matches_run).
A pane that overlaid them anyway would be showing a forward pass that did not
produce the answer above it.
The watch window — pinned nodes, gdb-style
The scrubber could show exactly one node at a time, so the two questions an interpretability session is made of — what does this node do as the model generates? and how do these two compare? — could only be answered by clicking back and forth and remembering numbers. Pinning answers both: pinned nodes stack in a Watch card above the list, all visible at once, re-resolved for whichever pass is on screen.
A pin holds a node NAME, not a record index, and that is the whole design. A watch in a debugger is an expression re-evaluated at each stop, not a pointer to one evaluation of it. So a pin survives switching pass (each generated token is its own forward pass with its own records under the same names) and survives opening a different trace of the same model — the closest thing here to re-running a program under gdb. Pins are keyed by model, not by trace, for the same reason: "watch layer 41's residual" is a fact about the architecture, and keying by trace id would throw the list away exactly when it became useful, on the second trace.
Three states are deliberate:
- A pin naming a node this trace never captured is shown as unresolved, struck
through with "not captured in this trace" — never silently dropped. It is a real
finding: pin
kq_soft_max-17from an attention-on trace, open an attention-off one, and the gap tells you the capture set differed rather than pretending the node was never interesting. - A
summarypin shows statistics and no strip. The rule that a summary carries statistics instead of a tensor is preserved rather than special-cased away. - The cap is 8, and it says why. Each pin is a request per pass change; an
uncapped watch list turns switching pass into a request storm.
pin allon a block fills up to the cap rather than dropping the ones that fit.
The sparkline is a second route, GET /traces/{id}/series?name=…, and it exists
because the manifest cannot answer it: tracer._capture writes summary only
for fidelity="summary" records, so exactly the records that have data carry no
statistics. It summarizes the whole record where /record/{i} caps at
_VALUE_CAP — a series is a comparison across passes, and a statistic over a prefix
compared against one over a different prefix is not a comparison. This is measurable,
not theoretical: on a 9-token prompt pass l_out-41 has 23040 values, and the capped
route reports rms 0.894 against the whole record's 1.248. Where the record fits under
the cap the two agree to the last digit. A pass with nothing measured contributes
null and is drawn as a gap, never interpolated.
The node list gained filters to match — name search, layer range, and kind toggles
built from a pure node-kind.ts classifier. Two details worth keeping:
- Kind is derived on the frontend from the node name, not stamped on the record at write time, so it works on every trace already on disk rather than only on ones recorded after the change.
- Narrowing the layer range hides the layerless nodes (
inp_embd,result_norm,result_output) and leaving it untouched keeps them — mirroringTraceFilter.wantedin the tracer, which refuses to let a layer selection silently drop the two ends of the story. - Only the kinds present are offered as toggles. A Mamba trace showing an "attention" chip that matches nothing is worse than no chip.
The depth profile — the pass, without pinning anything
A TraceRecord carries seven statistics (count, min, max, mean, rms, absMax, zeroFraction) and the pane rendered exactly one of them, for pinned nodes only — so
the most basic question you can ask of a traced pass, does the residual grow with
depth?, took eight manual pins and still could not be seen at a glance. The Profile
card answers it in two views, above the record list.
Why it is a route and not a useMemo. The first cut computed it from
records[].summary, already in the browser, and cost no request. Against every trace
on disk it drew an empty chart reading 0 of 28 layers measured, and blamed the
fidelity for it. tracer._capture writes summary only for summary-fidelity
records — the ones that hold no data — so on an fp16 trace the manifest carries 170
records and zero statistics. Exactly the records with data have no statistics. That
is the same fact that forced GET /series to exist, and
GET /traces/{id}/profile?passIndex=&stat= is its sibling one axis over: every record
of one pass, reduced server-side over its whole tensor rather than the
_VALUE_CAP prefix get_record ships to a browser. Summarizing 322 records that way
is 22 MB of activations, which is why traces.summarize_array is numpy and
summarize now delegates to it — one implementation, not two that can disagree about
the number the pane prints.
The pass and the statistic are the whole request; which role is on screen is not, because that is arrangement of a response already in hand. So:
- One role against depth. A role is a record name with its block index
removed, so
kqv_out-3andkqv_out-27are the same role at two depths — which is the only way to compare them. Two naming schemes reachroleOfand only one isblk.N.: graph nodes named by llama.cpp'scb()carry the index as a-Nsuffix, and handling only the segment form left every activation name unchanged, so each depth read as its own role and nothing was profilable at all. Only roles appearing at more than one depth are offered; a node captured once is not a profile. - The pass as a fingerprint — node kind against depth as a heat grid, normalized per row, not globally. An attention score and an FFN activation are different quantities in different units, and one shared scale would render every row but the largest as uniformly blank, which reads as nothing happens there. Per-row normalization asks the only question a heat grid can honestly answer here: where along the depth is this quantity large? Several records of one kind landing on one layer (q, k and v are all attention) average rather than max, so one outlier cannot stand for the group.
The statistic selector is as much the point as the charts: rms answers "how big",
zeroFraction "how sparse", absMax "is there an outlier". Those are different
questions about the same pass, and six of the seven statistics previously had no
renderer at all. The same selector now also drives the watch sparklines, which the
series route already accepted as ?stat=.
A missing statistic is null, never 0. A summary-fidelity record can lack a
statistic outright, and zero is a measurement — reporting it would draw a value that
was never taken and drag every mean and every colour scale toward it. The route emits
null and the grid skips it; a gap is drawn as a gap. The other half of the same
rule is why the grid tests value !== null rather than truthiness: a genuine 0.0
is a reading and belongs on the chart.
A failed request is surfaced, not swallowed — an empty profile and a broken one look identical on screen otherwise, and only one of them is a bug worth seeing.
Architectures beyond the transformer
DEFAULT_CAPTURE is a transformer capture set — attn_norm, kqv_out, ffn_*.
llama.cpp also runs Mamba/Mamba2, RWKV and hybrids (Jamba, Falcon-H1, Nemotron-H),
and on those none of those names exist: tracing one recorded the residual stream and
nothing whatsoever about the mechanism, which reads as a broken tracer rather
than as an unasked-for capture. tracer.capture_for(architecture) now adds an SSM or
RWKV group, chosen by general.architecture read from the GGUF header at request
time (routes._architecture, failing soft to the transformer defaults).
Two rules:
- Union, never replacement. A hybrid interleaves attention blocks with SSM blocks, so swapping the list on an architecture match would blind the trace to half of every such model — quietly, since what remained would still look plausible.
- The names were verified, not guessed. They come from the string literals in the
installed
llama.dll(b10448):ssm_conv1dis real andssm_scanis not. A capture pattern matching nothing fails silently, which is the exact failure being fixed, so the patterns are prefixes (ssm_,time_mix_,channel_mix_) matched the same substring wayTracer.wantedalready works.gguf._BLOCK_RULESgained("ssm_", "ssm")too, or the model explorer files every Mamba tensor under "other".
Image diffusion is stable-diffusion.cpp — a different binary this module does not
run, and nothing here traces it. Diffusion language models (LLaDA, Dream) do exist
in llama.cpp, but they denoise the whole sequence iteratively, so their steps are not
passIndex-as-generated-token and the tracer's autoregressive llama_decode loop
does not express them. Both are open, not half-supported.
The SSM capture set is verified against the build's own strings and against
Tracer.wanted, but not yet against real Mamba weights — no Mamba GGUF is
present on the development machine.
Why snapshots, not a stream
Attention scores alone are fp32 [n_kv, n_tokens, n_head] — roughly 33 MB per
layer per pass at 512 tokens and 32 heads, about a gigabyte across 32 layers. A
live tensor feed is therefore a websocket carrying a gigabyte a second to a pane
that can render none of it. A run writes a snapshot and the pane scrubs it
afterwards, which also makes a trace something you can keep, compare and delete.
$HORRIBLE_DATA_DIR/llamacpp/traces/<trace_id>/
manifest.json # provenance + every record (name, layer, pass, op, dtype,
# ne[4], nb[4], offset, length, fidelity)
tokens.json # the tokens the pass actually ran on
tensors.bin # one append-only blob, records in capture order
One blob rather than a file per layer — hassault's cube-grid precedent literally.
Per-layer files give thousands of handles and still cannot address a single node
inside a layer; blob plus Range gives both, and the range handling (including the
suffix form, where bytes=-3 is the last three bytes) is copied from karaoke
along with its tests.
The ggml ABI is the top risk, and it is contained three ways
struct ggml_tensor is internal C with no stability promise, and
llama-cpp-python declares the cb_eval callback type and zero ggml accessors —
so the callback hands over a void * that nothing in the wheel can read.
ggml_abi.py closes that gap as narrowly as possible:
- Mirror as little as possible. Only the first four fields (
type,buffer,ne[4],nb[4]). Name, op and data are read through functions ggml exports — the same instinct as servingplane_orderinstead of hardcoding it. - A self-check gates every trace. On the first tensor,
nbytesis recomputed from(type, ne, nb)using ggml's own formula and compared againstggml_nbytes(), and a dimension count derived fromneagainstggml_n_dims(). A field inserted upstream shifts the layout and the two disagree immediately. On mismatch the run aborts and writes nothing — a trace that parses but is garbage is worse than no trace. - The tracer is a subprocess.
python -m backend.modules.llamacpp.tracer <spec.json>, spawned withsubprocess.Popenon a worker thread (neverasyncio.create_subprocess_exec— underuvicorn --reloadon Windows the loop cannot spawn subprocesses at all). A segfault inside a ggml callback is then an exit code the runner reports, not a dead FastAPI backend mid-turn. A non-zero exit with no error line is reported as exactly that.
Those exported functions live in ggml-base, and only there. A modern wheel
splits the build into several shared libraries, and llama.dll re-exports none of
the ggml API — nor does ggml.dll (the backend registry) or ggml-cpu.dll. Nor is
the CDLL reliably reachable from llama_cpp._ggml under a fixed attribute name (it
has been both lib and libggml). So _ggml_candidates() yields every plausible
holder — each CDLL bound on _ggml, then *ggml-base* from the wheel's lib/
directory, then the old single-library llama_cpp.llama_cpp._lib — and each is
probed for ggml_nbytes before being accepted. Picking a library by name
instead surfaces later and further away, as AbiMismatch: ggml exports none of ggml_type_name out of bind(), which reads like an upstream ABI break rather than
the wrong file being open.
Four things that fail silently if forgotten
- Flash attention must be disabled explicitly. With it on,
ggml_flash_attn_extis one fused node and the score matrix never materialises — attention capture would produce nothing, reading as a bug in the name filter rather than a missing flag. Recent llama.cpp auto-enables it, so the default is not safe;_disable_flash_attnsetsflash_attn_type(or the olderflash_attnbool, whichever this build has). - The
CFUNCTYPEinstance is held in a module global. Built inline, it is garbage-collected the moment the call that created it returns and the next graph node jumps into freed memory. - Quantized tensors are recorded as metadata only. Dequantizing in Python would
be a second implementation of ggml's block formats — wrong in one corner and
plausible everywhere else. There is no dequantizer here, and
traces.decoderefuses a quantized dtype rather than guessing. - No chat template is applied. The prompt is traced as raw text; rendering a
template ourselves would put a different token sequence through the model than
the one shown beside it. The manifest carries
chatTemplate: falseand says so.
Everything declares what it is
The module's three-state honesty applies unchanged:
- Every record carries a
fidelity—full(the tensor),fp16(a downcast of it), orsummary(statistics instead of the tensor). A summary record has no bytes by construction:TraceWriter.appendrefuses a payload for one, the record route returns emptyvalues, and the pane draws the statistics as statistics. modelShasays what it hashed. Digesting 20 GB to open a pane would be absurd, so it is the GGUF's first megabyte plus its size, and every manifest carriesmodelShaScope: "gguf-header-1mib+size". It is enough to catch "a different model" and does not pretend to be a whole-file checksum.- The estimate is labelled an estimate.
POST /traces/estimatecosts the run from the GGUF header (block_count,embedding_length,attention.head_count) before it starts, because the difference between a 200 MB trace and a 12 GB one is one checkbox and a progress bar that has already started is too late. The manifest'sblobBytesis what actually happened.
Cost control is three layers: a hard MAX_TRACE_TOKENS cap (512) regardless of what
the caller asks for, attention off by default and per-layer opt-in, and a global
byte budget (llamacpp.traceBudgetGb, default 2 GB) pruned oldest-first after every
run.
Testing
backend/tests/test_llamacpp_trace.py, and it needs no native library — bind()
is duck-typed against a stub ggml, so the ABI check is exercised as arithmetic over
ctypes structures built in Python memory. The tests pin: the self-check passing on a
consistent tensor and failing on a mutated one (both the nbytes and the
n_dims disagreement), the quantized block branch (misreading nb[0] as a stride
overstates a q5_0 row by 32×), manifest/blob round-trip including a zero-length
summary record, Range requests including the suffix and open-ended forms,
server-side decoding of both dtypes, a summary record returning no values,
oldest-first pruning, the overlay guard refusing when either the build or the model
differs, and a trace id that tries to walk out of the trace directory with ...
The lens — a trace read as words
The Traces section is a debugger: pin a node, watch its numbers. The Lens
section (packages/core/src/modules/llamacpp/lens/, backend
backend/modules/llamacpp/lens.py) asks the other question, the one
Neuronpedia's Jacobian Lens asks —
at every layer and every position, what is the model disposed to say?
The readout is unembed(norm(J_L · h)), where h is the residual stream at
(layer, position) — which the tracer already captures as l_out-L — and J_L is
a per-layer transport matrix. Anthropic's J-lens fits J_L = E[∂h_final/∂h_L]
over a corpus, which needs autograd this backend deliberately does not have. Set
J_L = I and the same machinery is the classic logit lens: no artifact, no
download, works on every GGUF on the machine. That is what ships. LensSpec
names the transport, so a fitted J is a directory of J_<layer>.npy plus a
lens.json rather than a rewrite — and the identity lens is labelled
"identity (logit lens)" everywhere so it is never mistaken for a J-space readout.
On a real 40-layer Mistral trace of The capital of France is, Paris enters
the top 4000 around layer 16, reaches rank 513 by layer 24 and rank 1 by layer 28.
That progression is the artifact; a single grid is a screenshot.
The self-check is the correctness argument
Per-architecture details are exactly the kind that yield confident wrong numbers
rather than errors: Gemma folds 1 + into its norm weights at convert time,
Gemma 2 softcaps final logits, most models tie output.weight to
token_embd.weight. So at the last layer the identity lens must reproduce the
trace's own captured result_output, and result_norm independently checks the
norm on its own — the two are separately wrong-able, and a disagreement that names
neither is a worse answer than none.
verified is therefore three-valued, the module's standing rule: true,
false (rendered as a banner, never as numbers), and unavailable — the trace
captured no logits to check against. A fitted lens always reports unavailable,
because a J-lens does not reproduce the model's output by design; reporting
true for one would mean the check was measuring nothing. It still says whether
the identity path agreed, which is what tells you the machinery is sound even
when the lens on screen cannot be validated.
The check costs nothing: it rides the grid's own streaming pass as one extra column rather than a second pass over the output head.
Reading a pruned graph
The rows are ragged, and that is not a bug in the trace. llama.cpp prunes its
graph to what the pass needs, so on a prompt pass the last block's l_out is
one column wide — only the final position's residual produces logits — while
every earlier block is the full width, and result_output is one column covering
that same final position. Two things follow, both silent when wrong:
- A narrow tensor covers the tail of the sequence, so
column_ofmaps a position to a column and returnsNonewhen there is none. A cell that was never computed renders blank; falling back to the nearest column would put the last token's readout under every earlier one and make the top of the grid look uncannily prescient. - The self-check addresses both tensors from the end. Taking column 0 of a
one-wide
result_outputagainst position 0 of the residual compares the last token's logits with the first token's activation, disagrees wildly, and reports every real trace as unverified.
The output head is never materialized
For gemma-3-12b the head is 262144 × 3840 — 2 GB in fp16. Weight chunks stream
past a matrix of every cell in the grid stacked together, so a 40 × 6 grid
costs one pass over the head rather than 240. Dequantization is llama.cpp's own
numpy implementation (the gguf package, ~110 kB over numpy); a quantization it
cannot read is named and refused, the same instinct as traces.decode
refusing a dtype it never wrote.
The dequantized fp16 head is cached under
$HORRIBLE_DATA_DIR/llamacpp/lens/<modelSha>/unembed.f16.npy when it fits
llamacpp.lensCacheGb (4 GB default) — measured on a Q6_K 131k × 5120 head, that
takes a grid from 7.9s to 2.0s. Two traps are pinned by tests: np.save appends
.npy to a path that lacks it, so a temp file named unembed.f16.part lands as
unembed.f16.part.npy and the rename after it finds nothing (a cache never
installed, plus a gigabyte of litter per grid); and a cache whose shape does not
match is deleted rather than used, which needs the memmap released first
because on Windows an open mapping keeps the file undeletable.
modelSha keys the cache, which is the same key the trace carries — two
quantizations of one model are two caches, as they must be.
Narrowing, and why it is a stride
The grid is a <table> with one row per layer and one column per token, so a
262-layer trace over a 200-token prompt is 52,400 cells — a request measured in
seconds and a picture nothing can be found in. getLensGrid has accepted layers
and positions since it was written and the pane passed neither.
The control is a stride, not a window, and that is the interesting choice. What a logit lens is for is seeing where along the depth a prediction settles. A window over layers 0–32 of a 262-layer model answers a different question: it shows a sixth of the model in full detail and hides where the answer actually appeared. A stride keeps the full span and samples it, so the knee is still visible and you can then narrow to a window once you know roughly where to look.
Three rules in narrow.ts (pure, tested):
- The last layer is always kept. It is the model's real output, and a stride that happened to skip it would draw a lens whose final row is a prediction the model never made.
- The window is applied first, then the stride samples within it. Striding first and clipping after gives a different row count depending on where the window sits, so widening it could make the picture coarser — the opposite of what the control says it does.
- Only an unnarrowed response tells us the true depth. A narrowed one reports the rows it was asked for; treating that as the depth would let the suggested stride ratchet itself down on every refresh.
A deep trace opens at a suggested stride rather than opening broken; touching the control stops it being suggested again.
The track ribbon — where a word gets decided
LensTrack carries logits[layer][position] and ranks[layer][position], and
the pane used only the ranks: a superscript in a cell and a tint behind it. The logits
were fetched on every track and thrown away.
That is the whole point of a logit lens, and it had no picture. A word does not fade in evenly across the depth — it sits near the floor for two thirds of the stack and then climbs steeply over a handful of layers. A per-cell tint cannot show a knee. A curve can, and the knee is the finding.
TrackRibbon draws both encodings on purpose. The line is the logit, the quantity
the model computes. The band beneath is the rank, log-scaled, which is what that
logit is worth against every other token — a rising logit stuck at rank 4000 has
decided nothing. The first layer at which the token is the top choice is called out as
a badge, and a token that never wins says so rather than leaving the badge absent,
which would read as a missing feature.
Swapping a token
A grid alone cannot separate the model knows this from the prompt said it.
POST /traces/{id}/fork replaces tokens and re-runs, and the two grids are
diffed cell by cell.
Three decisions carry it:
- Token ids, not text.
TraceRequest.tokenIdsbypasses tokenization entirely. Re-tokenizing an edited string can merge or split its neighbours, so the run would differ in more places than the one you changed and the diff would be unreadable. - Everything else is inherited from the parent — model, generation length, layer selection, fidelity, capture set. A fork that also changed those is not a counterfactual.
fromIdis stamped from the parent, never trusted from the caller. It is the record of what was actually replaced, and a client sending a stale one would make the fork's own provenance wrong.
Only the prompt is forkable: a generated token is an output, and replacing
one would be editing the answer rather than the question. The diff aligns by
(layer, position) rather than by array index, because a fork can differ from its
parent in either and comparing cells[2][3] to cells[2][3] of a grid with a
different layer subset silently compares different layers.
The picker searches the GGUF's own vocabulary (GET /models/vocab), not an HF
tokenizer's: interpretability/tokenizer.py's family fallback can hand back a
different generation's vocabulary while looking exact, and a swap has to name a
token the traced weights actually have. It distinguishes Italy (#16507) from
Italy (#93355), which is the distinction a swap turns on.
Capture sets
TraceRequest.capture is now reachable over HTTP, and GET /traces/capture-sets
serves the named sets with their real ggml node patterns — the plane_order
precedent: two lists of node names in two languages is one upstream rename away
from a capture set that matches nothing and fails silently.
lens is the one that matters. The lens reads the residual stream and the output
head and nothing else, so it drops the default six-nodes-per-block to one: a
40-layer, 6-token trace is megabytes rather than a gigabyte, and seconds rather
than minutes. That is what makes swapping a token and looking again a loop rather
than an errand. traces.nodes_per_layer teaches the pre-flight estimate about it,
counting only the per-block nodes — inp_embd and the output head are captured
once per pass, and charging them per layer would inflate the estimate by half.
The trace catalog — why a trace is in two places
trace_catalog.py keeps one row per trace in app.db (llamacpp_traces) while
the activations stay on disk. That is the split library_sources and
karaoke_songs already use, and it is not redundancy: a directory of manifests is
answerable only by this module's own code, whereas a table in app.db sits beside
agent_turns and traj_steps, so "which traces were taken during that agent run"
and "every fork of this trace" become one query in the database console.
Disk stays the authority. A row is a description of a directory and nothing
depends on it — delete app.db and you lose the query surface, not an activation.
Two hooks and a reconciler keep the two honest:
trace_runner.run_tracerecords the row at the one point every trace, forks included, is known to have finished.traces.delete_tracedrops it. Hooked there rather than at the routes becauseprune()also deletes — a budget eviction is the path nothing else would ever tell the catalog about.sync()runs at startup and reconciles both ways. A node that traced before this table existed has directories and no rows, and an empty catalog reads as "no traces" rather than "not indexed yet"; a manualrm -rfleaves rows claiming traces that are gone.
The model locus
packages/core/src/model-locus.ts is "which part of the model am I looking at",
the way locus.ts is "which code am I looking at": a global
{modelSha, traceId, layer, position, tokenId} that panes publish to and follow.
Click layer 15 in the lens grid and the model explorer reveals blk.15's
tensors — neither pane imports the other or knows it exists. Core rather than a
module, for the same reason locus.ts is: it is the spine several surfaces hang
off.
One deliberate difference from the code locus: the /ws mirror is inbound
only. The code locus is bidirectional because two panes in two windows publish
to it. Here every publisher that matters is either in this tab — the grid and the
explorer, which a local store notifies synchronously — or on the backend
(dash.lens.focus, the lens.focus tool). Sending our own clicks up would buy
nothing and cost a frame per mouse-move over a grid. So setModelLocus does not
send, there is deliberately no lens entry in app.py's inbound channel
ladder (only the outbound push_lens_events fan-out), and consequently there is
no self-echo to suppress — which is why this has no origin field where the code
locus needs one.
setModelLocus merges; an inbound one replaces. The grid knows the layer
and position but not which vocabulary token is pinned, so a local replace would
make every publisher restate facts it does not own and the first to forget would
silently clear a pin. A locus off the wire came from a publisher that stated the
whole thing it meant, and merging would leave a stale layer under an agent's "look
at the embedding".
Agent tools
Two groups, both loaded on demand.
llamacpp gains the trace verbs it never had: llamacpp.trace records a pass and
llamacpp.list_traces reads the catalog. trace defaults to the lens capture
set with attention off — megabytes and seconds — and full: true opts into the
whole mechanism; attention needs full, since the score matrix is the largest
thing a trace can hold.
lens is the readout: lens.grid, lens.cell, lens.track_token, lens.fork,
lens.focus, lens.layer and lens.save_finding. lens.layer is what one block
did, in numbers an explanation can quote — the share of each token's residual it
rewrote and the angle it turned, its sharpest content heads and its attention
sinks (ranked separately: a head parked on the first token is low-entropy too, so
ranking by entropy alone just re-lists the sinks), and on a mixture-of-experts model
which experts it routed tokens to. It reuses the stepper's reads, so the agent and
the pane cannot disagree about a block. lens.focus takes a stage and a
passIndex, which move the stepper's cursor — "look at block 17's attention" puts
the user's screen there. Two tools rather than a new group: the group already sits
under the per-turn tool budget, and a separate stepper.* group would be one more
load_tools round for the same question. The group name equals the tool prefix because
_group_of splits on the dot and AgentTool.group does not name the group —
the connectors rule, and the trap llamacpp itself inverted.
Three things carry the containment:
- An unverified grid is reported as unverified, never as numbers. Every payload
carries
verifiedin the same three states the pane renders, andfalse/unavailablelead with a warning rather than burying it under the cells. A model whose norm convention we get wrong produces confident wrong words, not an error, and an agent summarising those into prose is how a wrong reading becomes a claim. - A grid too large to carry is refused, with its dimensions. A tool answer is
prose in a context window, not a screen an eye skips;
MAX_TOOL_CELLSbounds it and the error names the axes so a caller narrows rather than guesses. lens.forkandlens.save_findingare side effects. A fork mmaps the weights for minutes and writes a trace that may prune an older one; a save writes to the user's library.
lens.track_token accepts text as well as an id, and does not strip it: a
leading space is what makes " Paris" a token and "Paris" not one, so stripping
deletes the very thing the error message tells you to add.
dash.lens
The REPL facade, for sweeps — "which layer does the answer first appear at,
across these forty prompts" is a loop and a max(), which is neither a click nor a
chat turn. Backend-local like dash.io, so it needs no browser attached; the one
exception is focus(), which pushes to browsers and still works with none, since
the locus is stored and the next window to attach follows it.
Saving a finding
findings.py renders a grid plus the reader's note as a note source in a library
— the browser.save precedent, and the reason it matters is that a trace is pruned
the moment the budget wants its bytes.
A note is deliberately not an artifact: the activations are not copied, because megabytes of fp16 have no business being embedded and the numbers mean nothing apart from the weights that produced them. What is written down is the reading — prompt, top token per layer, and the provenance to reproduce it (model sha, llama build, trace id, and any edits, so a fork's note says what was changed).
An unverified grid is refused rather than filed with a caveat. A caveat in a note body survives exactly as far as the first retrieval that quotes the numbers without it.
"Trace this" on someone else's surface
The module contributes a contextMenu provider for two target kinds it does not
own — editor.selection and library.chunk — so the verb reaches the text
instead of requiring the palette and a focused editor. Neither the editor nor the
library learns this module exists, which is the point of the context-menu registry
(see context menus). The editor reads the
selection from the live view rather than the persisted bytes, for the same
reason llamacpp.traceSelection does: tracing something other than what is on
screen is worse than declining.
The stepper — a forward pass stepped like source
The Traces section is a record list you pin and scrub; the Lens is a grid of words.
The Stepper is the third reading: execution order. It needs nothing new on
disk, because a trace already is an instruction stream — the tracer appends
records in the order ggml computed them, each tagged with its pass and its block.
stepper/program.ts turns the record list into a program and gives every debugger
verb a meaning:
| verb | lands on |
|---|---|
| step into / back | the next / previous graph node |
| step over | the first node of the next frame (a block of one pass; the embedding and output head are frames of their own, so stepping over the last block lands on the head rather than skipping it) |
| reverse step over | the start of this frame, then of the previous one |
| step out | the first node of the next forward pass |
| continue / reverse continue | the next node a breakpoint stops on |
| play | step over on a timer, stopping at any breakpoint crossed inside a frame |
Reverse execution is free — a snapshot has no side effects to undo — which is the one way this debugger is better than the real kind.
One classifier. A step's stage (attention, ffn, moe, residual, …) comes
from node-kind.ts, the same function the Traces filter uses. The program is built
in the browser from the record list GET /traces/{id} already returns, precisely so
no second list of ggml node names exists to drift from the first.
Breakpoints, including ones no source debugger has
- block — stops on entry to the block only, so Continue does not stop six times inside it;
- stage ("every attention node") and node (one name);
- threshold — a stage's
rmsabove a value, evaluated against the per-pass profile. A node whose statistic was not measured never fires: stopping on a guess is worse than not stopping; - lens — "stop where the lens first says ' Paris'". The stepper reads the
identity lens at the selected position for every layer of the pass (one
/lens?positions=&passIndex=&k=1call) and offers each token that is ever top-1, with the block it first appears at. Verified on Llama-3.2-3B with the prompt "The capital of France is": ' Paris' first wins at block 19, Continue stops onl_out-19, and the timeline marks blocks 19–27.
Breakpoint marks on the timeline are drawn with the same HitContext Continue
evaluates, so a mark is exactly where Continue would stop.
What each stage shows
stepper/StageView.tsx picks the view that answers the question you stepped there
to ask. Every read goes through the stride-aware routes; a summary record has no
values and says so. Reads are cached for the session (stepper/load-cache.ts, last
200), because a recorded trace never changes: stepping back to a node already seen
paints it at once instead of "Reading…". A failed read shows its last error at once
and is still retried, since an error may be transient.
- Attention (
kq_soft_max,kq) — a strip of heads (the fill is each head's weight on position 0, the attention sink), then the selected head's query × key matrix onHeatCanvas, with the selected position's row outlined and its top keys listed.kqis labelled as raw pre-softmax scores, where a query may score keys after it — causality is applied by the next node. - Residual (
l_out) — what the block wrote: the share of each token's vector it rewrote and the angle the vector turned, and what the stream now says through the lens at this layer. - MoE (
ffn_moe_*) — the layer × expert atlas and the experts the selected token was routed to, with their weights. - Anything else — the tokens × features plane and each token's norm.
Watch pins are shared with the Traces section, and a pinned node later in the pass than the cursor shows "not yet run" rather than its value: reading it would be reading the future.
The reads, and the stride trap they fix
stepper.py owns them. get_record ships a flat prefix, which is right for a value
strip and wrong for anything with two axes — the first 8192 values of a
[256, 6, 24] attention record are a few rows of head 0. record_array rebuilds
the 4-D tensor through its strides, and there is a trap in that:
nb is recorded in the source dtype. The tracer copies strides off the live
tensor and then downcasts f32 to f16, so an fp16 record's bytes are half as wide
as its strides claim. Element strides are nb / 4 for fp16 and nb / itemsize
for full; dividing by the stored width doubles every stride and reads a different,
plausible-looking tensor. Pinned by test_llamacpp_stepper.py, alongside a permuted
view, KV cropping (kq_soft_max is as wide as the allocated cache — 256 on a
six-token prompt — and its masked columns are exact zeros), and the pruned tail (the
last block's output on a prompt pass is one column, compared against the last
column of the block before, never the first).
Two things were found on real traces rather than assumed:
inp_embdis in the capture set and still absent on b10362 — the embedding lookup happens before the callback sees the graph. Block 0's delta therefore reports that its input was not recorded instead of subtracting zero.- Every attention row sums to 1 and the upper triangle is exactly 0, which is the check that the stride reading is right.
Mixture-of-experts routing is now captured
ffn_moe_probs, ffn_moe_topk and ffn_moe_weights joined DEFAULT_CAPTURE — the
literals were checked in the installed llama.dll (b10362), the SSM set's lesson.
On a dense model they match nothing and cost nothing. ffn_moe_topk is i32
indices, so the tracer now keeps i32 tensors whole (a summary of indices is
meaningless, and an f16 downcast is lossy past 2048), and traces.decode reads
i32. Traces recorded before this change have no router nodes and the atlas says so.
The explorer follows the cursor
The cursor publishes traceId, modelSha, layer, stage, step, passIndex
and position to the model locus. The model explorer
selects that block, lights the sub-block the stage names, and shows a following the
stepper strip; clicking a block or sub-block on its diagram publishes back, and the
stepper jumps there. An echo of the block the cursor is already in is ignored, so the
explorer re-selecting what it was just shown cannot yank the cursor to the block's
start.
Other surfaces open a trace in the stepper through stepper/target.ts
(openInStepper with a trace id and a step or a layer) — the trace-prompt.ts seam,
for the same two-directional mount race. dash.lens gained program, tensor,
attention, delta and experts (numpy, the same functions the routes call), and
focus(stage=..., step=...) moves the cursor from the REPL.
Not done here
topk fidelity is declared as an idea and not implemented — only full, fp16 and
summary are written, and the enum lists exactly those. Traces are not yet overlaid
on a chat turn in the interpretability pane: traces.matches_run is the guard that
work will need, but the pane wiring is not built. The tracer also samples greedily
and traces a raw prompt, so it reproduces a forward pass, not a chat turn.
No fitted J-lens ships yet: the artifact format, the picker and the
unavailable verdict for a non-identity lens are all in place, but importing
Neuronpedia's published .pt lenses (which needs a torch environment to convert
once) and fitting our own are not built.
Distributed computation is designed for and not built: a trace, a grid and a fork
are all job-shaped, so a peer holding the same modelSha could run one and return
only the result — with traces.matches_run generalizing exactly (a result computed
elsewhere may only be shown here when the build and the model hash agree). Nothing
crosses the fabric today.