WebML Phase 6: our own WGSL engine for GGUF
Status: 6.0–6.6 built. 6.4's and 6.5's exit checks pass on an Intel UHD iGPU
(see 6.4, 6.5). Real-model parity with llama.cpp holds for SmolLM2-360M
Q4_K_M, Qwen3-1.7B Q4_K_M, Qwen3-0.6B Q8_0, Llama-3.2-1B Q8_0,
DeepSeek-R1-Distill-Qwen-1.5B Q8_0 (qwen2) and Gemma-3-1B Q4_K_M. Still open from
earlier stages: the browser-provider agent round with a Hermes tool call (6.3).
6.6 is built (6.6): the qwen2, gemma3 and smollm3
architectures, the logit lens during generation and in the model explorer, and GGUF
in published Scrive pages. SmolLM3 is checked against llama.cpp on a tiny file only:
no real SmolLM3 GGUF was on this node and the Hub was out of reach. This is the plan WebML said
Phase 6 needed before any code was written. It is split into stages; each one ships
on its own and ends with a check that has to pass before the next starts. Nothing
in Phases 0–5 is replaced. The transformers.js worker stays as the engine for ONNX
models (in the app and on published Scrive pages) and for the WASM CPU fallback.
Why build an engine at all
Phases 0–5 run ONNX exports through transformers.js and ONNX Runtime Web. That works, and it limits us in four ways that patches on our side cannot fix:
| Limit today | Where it bites | What owning the forward pass gives |
|---|---|---|
Only the curated onnx-community exports run | The catalog is seven models; the node's own GGUFs (llama.cpp catalog, training's convert.py output, Ollama / LM Studio blobs) cannot run in the window | Any GGUF of a supported architecture, including a fine-tune this node just trained |
No num_logits_to_keep | Logits for every prompt position: the 2.4 GB bad_alloc, worked around with 256-token chunked prefill | The LM head runs on the last position only. The chunking exists only to bound activation memory |
| KV cache thrown away after every reply | An agent round re-prefills ≈4k tokens (system prompt + tool schemas) every time: ~50 s per round on Qwen3-0.6B | The longest common token prefix is reused across calls, so a round prefills only what changed |
| Logits come back to the CPU every step | ~600 KB readback per token for a 152k vocabulary; the step events cost a pass over the vocabulary in JS | Sampling and top-k run on the GPU. A step reads back a token id, or with topK > 0 the k alternatives |
A fifth gain comes later (6.6): the residual stream lives in our own buffers, so interpretability readouts (per-layer norms, logit lens) can be read during a normal generation.
Non-goals for Phase 6: multimodal checkpoints, training or LoRA in the browser, a CPU/WASM GGUF path (machines without WebGPU keep transformers.js), and every ggml quantization type. GGUF in published Scrive pages started as a non-goal, since it needed our engine as a bundled static asset; 6.6 built that.
Baseline to beat
From the Phase 5 measurements (2026-10-07, Intel Gen12 iGPU):
- SmolLM2-360M chat, q4f16 ONNX: ~27 tok/s, TTFT ~0.7 s
- Agent round, Qwen3-0.6B, ≈4k-token prompt: ~50 s, almost all prefill
Stage 6.4 is measured against these numbers on the same machine.
Shape of the change
One protocol, two workers. gguf.worker.ts speaks the same
protocol.ts events (progress, ready, delta, step, done, …), so the
playground, the browser provider relay and {webllm} blocks barely change.
A GGUF is addressed by its model id, gguf:<owner>/<repo>/<path/to/file.gguf>,
and WebmlEngine routes on that prefix. The plan's source field was not needed:
every consumer already passes model ids around as opaque strings, including
browser_provider.py, the manifest and webml.defaultModel. The protocol change is
additive and optional:
LoadRequest.contextLength: KV positions to allocate (default 4096, capped by the model).readygainsengine('onnx' | 'gguf'),quant(the weight type holding most of the file, e.g.Q8_0),contextLength, andthinking(the template takesenable_thinking).EngineState.readymirrors them.
Switching engines terminates the other worker. WebmlEngine takes a factory
per engine kind. Loading a model of the other kind terminates the current worker
instead of sending it unload, because terminating is the only way to release a
GPU device. Only one model is resident at a time, as today.
New files
| Path | Holds |
|---|---|
packages/webml/src/gguf/parse.ts | GGUF header reader: a TS port of backend/modules/interpretability/gguf.py, with the same bounds (_MAX_TENSORS, _MAX_STRING, …) and the same ggml type table |
packages/webml/src/gguf/source.ts | Range reads: HF resolve/ URLs, the node's file route, OPFS files |
packages/webml/src/gguf/store.ts | OPFS weight store (webml-gguf/, one file and a JSON sidecar per model id): list, delete, resumable download, quota check |
packages/webml/src/gguf/arch.ts | Metadata → ModelConfig, plus a support check that says why a file is refused |
packages/webml/src/gguf/tokenizer.ts | BPE (gpt2) and SentencePiece (llama) from tokenizer.ggml.*, with pre-tokenizers per tokenizer.ggml.pre |
packages/webml/src/gguf/template.ts | tokenizer.chat_template through @huggingface/jinja (already a transformers.js dependency) |
packages/webml/src/gguf/quant.ts | CPU reference dequantizers: the test oracle, never the runtime path |
packages/webml/src/gguf/ggml.ts | The ggml type table (block elements and bytes), copied entry for entry from gguf.py |
packages/webml/src/gpu/device.ts | The device, requested with the adapter's limits and optional features |
packages/webml/src/gpu/chunks.ts | Splitting a tensor over the binding limit into whole-row chunks |
packages/webml/src/gpu/dispatch.ts | Workgroup counts, spilling into y past the per-dimension limit |
packages/webml/src/gpu/kernels.ts | Every pipeline, composed per weight type and cached; uniform and bind-group helpers |
packages/webml/src/gguf/rope.ts | The rope angle table, built the way ggml builds its rope cache |
packages/webml/src/gguf/runtime.ts | Buffers, KV cache, the per-token command list, logits readback |
packages/webml/src/gguf/sample.ts | llama.cpp's default sampler chain; the distribution behind step events |
packages/webml/src/gguf/session.ts | One loaded model: template → tokens → runtime → sampler → streamed text; preflight() for headers |
packages/webml/src/gguf/hub.ts | A repo's GGUF files (tree API) and one file's header verdict, before download |
packages/core/src/modules/webml/panels/GgufPicker.tsx | The playground's GGUF chooser |
packages/webml/src/wgsl/*.wgsl | Kernels, imported with ?raw |
packages/webml/src/engine/gguf.worker.ts | The protocol loop around runtime.ts |
packages/webml/kernel-tests/ | Vitest browser mode (Playwright provider): each kernel on the GPU vs quant.ts / a CPU reference |
scripts/gen_webml_gguf_fixture.py | The header conformance fixture and golden dequantized blocks (gguf pip package). Token-id goldens (transformers tokenizers) come in 6.2 |
scripts/gen_webml_llama_parity.py | llama.cpp references (via llama-cpp-python): the tiny-model fixtures, and --model runs on real GGUFs |
Technical decisions
The stages assume these decisions. Each one is checked in the stage named in brackets, and a stage that disproves one stops and asks. Decisions 1 and 7 were confirmed on 2026-10-08 (see Decided).
- Weights live in OPFS, not Cache Storage [6.2]. A GGUF is one file of
0.3–2.5 GB. Cache Storage hands back a
Response, and reading a byte range out of one means reading the body. OPFScreateSyncAccessHandle(in the worker) gives positioned reads. The download streams to disk, and the load streams tensor by tensor from disk intoGPUBuffers, so the file is never in JS memory at once. Downloads resume from the.partlength the way the llama.cpp module's do.cache.ts's listing and deleting cover both stores, so "On this device" stays one list. - Read the header before downloading [6.0]. The tensor directory and
metadata sit before
data_offset, a few MB including the vocabulary. A Range fetch of the header lets the picker say "architecturephi3is not supported" or "IQ2_XSis not supported" before a 2 GB click, and size the GPU budget from real tensor sizes. - Port the backend's reader instead of adding
@huggingface/gguf[6.0]. The two readers must agree on what a file contains, and the backend one is already the house reference, with a size table that refuses to guess unknown types. The port is about 250 lines and runs against the same fixtures. - Request the adapter's limits, then chunk anyway [6.0]. Default
maxStorageBufferBindingSizeis 128 MiB. A tied 151,936 × 2,048 embedding / LM head in Q6_K is ~250 MB. The device is requested with the adapter's maxima, and any tensor over the binding limit is split by rows into several buffers; the matvec dispatches once per chunk. The probe already reports both limits. - f32 activations and accumulation first, f16 later [6.1 → 6.4]. Correctness
is established with f32 everywhere except weights.
shader-f16storage for activations and the KV cache is a 6.4 optimization, accumulation stays f32, and parity is re-checked afterwards. - Dequantize inside the kernels [6.1]. Weights stay in their ggml block
layout on the GPU (uploaded as
u32words). Decode kernels dequantize in registers; prefill tiles dequantize into workgroup memory. Dequantizing to f16 at load would double or triple VRAM, and VRAM is what an iGPU shares with the tab. - Kernel tests run in Chromium on SwiftShader [6.0]. Playwright drives the
preinstalled Chromium with
--enable-unsafe-webgpu --use-webgpu-adapter=swiftshader, so CI exercises every kernel with no GPU. It is slow, and that is acceptable: the tests use small random tensors. Dawn's node bindings are the fallback if SwiftShader turns out to miss a feature we need. - The rope variant is a property of the architecture, not the file [6.3].
convert_hf_to_gguf.pypermutes Q/K forllama, which uses llama.cpp's "normal" interleaved RoPE.qwen2andqwen3use NEOX (split halves). Getting this wrong still produces fluent-looking garbage for a few tokens, which is why the parity check runs 64 greedy tokens and not 5. Llama 3.x also shipsrope_freqs.weight(per-dimension factors), which must be applied.
The kernels
All are compute shaders. "Fused" kernels are a 6.4 change; until then each step is its own dispatch.
| Kernel | Notes |
|---|---|
embed | Dequantize one row per token id. The id is read from a GPU buffer, so the decode loop can feed sampled ids without a round-trip (6.4) |
rmsnorm | Workgroup reduction (subgroup reduction when subgroups is available) times weight. A per-head variant is Qwen3's QK-norm. With accumulate (6.6) it adds into its output: Gemma's post-norms, fused into the residual add |
matvec_<q> | Decode (M = 1), memory-bound: the hot path. One variant per quant type: F32, F16, Q8_0, Q4_0, Q5_0 (6.4), Q4_K, Q5_K, Q6_K |
matmul_<q> | Prefill (M > 1): tiled, weights dequantized into shared memory |
rope | norm and neox variants, applied in place. Angles come from a host-built table indexed by position (see rope.ts); rope_freqs factors arrive with 6.3 |
kv_store | The pass's new K and V rows (computed, normed and rotated in f32) into the cache at their positions, rounded to f16 (default) or kept f32. Layout [pos][kv_heads × head_dim], preallocated to the context budget. (Through 6.3 the projections wrote the f32 cache directly.) |
attention + attn_combine | Decode: split over 128-position chunks (flash-decoding), one workgroup per (KV head, chunk) serving the whole GQA group, then a merge per head. A sliding window (6.6) masks earlier keys; chunks wholly before it are skipped |
attn_prefill | Causal tiled flash attention, with the same optional window |
glu | silu(gate) * up, or gelu(gate) * up for Gemma (6.6), fused into one pass. (swiglu until 6.6) |
bias | Adds a projection's bias to every row: Qwen2's Q, K and V (6.6) |
| residual | Folded into the attention-output and FFN-down projections in 6.4 (accumulate flag on matvec / matmul). (add, out-of-place, is the 6.0 harness kernel) |
lm_head | matvec over the vocabulary on the last position only |
sample | llama.cpp's chain (top-k → top-p and min-p → temperature → draw with a host-supplied random number), with an exact top-k. Also writes p(chosen), entropy and the alternatives for step events |
Per-token encoding. Bind groups and uniform buffers are built once at load.
The current position and sequence length sit in one small storage buffer that
writeBuffer updates per step, so a token is one command buffer of roughly
layers × 12 dispatches, re-encoded with no allocation. Readback goes through a
ring of mapped staging buffers.
Pipelined decode (6.4). sample writes the token id into the buffer embed
reads, so step N+1 can be submitted before the host has read step N's token. The
host reads ids asynchronously to stream text and check stop conditions. On EOS, the
one speculative step is discarded and its KV slot is dropped.
Prefix reuse (6.4). The worker keeps the token ids behind the live KV cache.
A generate call tokenizes its prompt, finds the longest common prefix with those
ids, keeps the cache up to that point, and prefills only the suffix. For an agent
round, the suffix is the last tool result plus the new turn.
Supported models (the first set)
| Model (GGUF) | Arch | Quants | Why |
|---|---|---|---|
| SmolLM2-360M-Instruct | llama | Q8_0, then Q4_K_M | Small, fast to iterate on, a direct comparison with the ONNX baseline |
| Llama-3.2-1B-Instruct | llama | Q4_K_M | rope_freqs (Llama 3 rope scaling) |
| Qwen3-0.6B / 1.7B | qwen3 | Q4_K_M, Q8_0 | QK-norm, NEOX rope, head_dim ≠ embd / heads, Hermes tool calls: the agent models |
| Qwen2.5, DeepSeek-R1's Qwen distills (6.6) | qwen2 | Q4_K_M, Q8_0 | Q/K/V biases, NEOX rope |
| Gemma 3 1B (6.6) | gemma3 | Q4_K_M | Sliding-window attention on five layers in six, post-norms, √embd embedding scale, GELU gate, a 262k vocabulary |
| SmolLM3 3B (6.6) | smollm3 | Q4_K_M, Q8_0 (checked on a tiny file only) | llama with no rope on every fourth layer (NoPE), Llama 3's tokenizer |
Q4_K_M files mix Q4_K, Q5_K and Q6_K (the LM head is usually Q6_K), and the norms are F32. Supporting "Q4_K_M" therefore means all of these types.
arch.ts refuses anything else by name.
Stages
6.0 Harness and header (no inference)
gguf/parse.tsplus tests against synthetic GGUFs built in the test, and one real header captured as a fixture. Its output must matchgguf.pyon the same bytes; a Python script dumps the backend reader's view to JSON for comparison.gguf/source.ts: Range reads from HFresolve/URLs, with the HF token where the connector has one, the same way the llama.cpp catalog does.kernel-tests/running one trivial kernel (add) in Chromium on SwiftShader in CI.- Device creation with the adapter's limits. The chunking helper is tested with a deliberately low limit.
Exit: the header of a real Qwen3 Q4_K_M GGUF parses in the browser from a Range
read, with tensor bytes summing to the file size minus data_offset, and
the kernel harness is green in CI.
Built (2026-10-08):
gguf/ggml.ts,gguf/parse.ts(the reader pluslayoutProblems, which catches a truncated download before any buffer is made) andgguf/source.ts(HttpSource,blobSource,bytesSource,hfResolveUrl).gpu/device.ts,gpu/chunks.ts,gpu/dispatch.ts, and theaddkernel (wgsl/add.wgsl,gpu/add.ts).- Tests:
pnpm --filter @horrible/webml testis the node project (parser edge cases, Range handling, chunk and dispatch planning).test:kernelsruns Chromium on SwiftShader: the device gets the adapter's limits, andaddis exact against the CPU, across a dispatch that spills into y and across row chunks under a 1000-byte limit. CI is.github/workflows/webml.yml. - Header conformance. The fixture comes from llama.cpp's own writer rather than
a downloaded header: a tiny Qwen3-shaped file with Q8_0, Q4_0, Q4_K, Q6_K, F16 and
F32 tensors and every metadata value type.
parse.tsmust read it exactly asgguf.pydoes (fixture.test.ts), andbackend/tests/test_webml_gguf_fixture.pychecks thatgguf.pystill does. Regenerate it withPYTHONPATH=. uv run python scripts/gen_webml_gguf_fixture.py.
Still open:
- The real-model check is written but has not run.
WEBML_HF_HEADER=unsloth/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q4_K_M.gguf pnpm --filter @horrible/webml test:kernelsneeds a network that reaches huggingface.co. It also confirms that the Hub's CDN exposesContent-Rangeto script; if it does not, the picker passes the tree API's size instead. - The HF token for gated repos.
HttpSourcetakesheaders, but nothing in the frontend holds a token yet. That is decided in 6.2 with the picker. test_webml_gguf_fixture.pyruns with the backend suite, which has no CI workflow. Only the TS half runs in CI.
6.1 Correct, slow, llama only
quant.tsreferences for F32, F16, Q8_0, Q4_0, checked against golden vectors from theggufpip package's dequantizers.- Naive kernels:
embed,rmsnorm,matvecfor those types,rope(norm),kv_write,attn_decode,swiglu,add,lm_head. Each is checked against its CPU reference in the harness. Prefill runs as a loop of decode steps. - Logits read back to the CPU and sampled in JS with the existing
distribution(). - A dev-only page (not a panel) runs SmolLM2-360M Q8_0 from a fixed prompt.
Exit: 64 greedy tokens match llama.cpp token for token on SmolLM2-360M Q8_0,
and the top-5 probabilities at each step are within 1e-2. The reference is
llama-server's /completion with n_probs, fetched by the llama.cpp module and
run by scripts/webml/parity.py.
Built (2026-10-08):
gguf/quant.ts: matches theggufpackage's dequantizers exactly on all four types (quant.test.ts, golden blocks inquant-golden.json).gguf/arch.ts: header →ModelConfig. It refuses other architectures, other weight types, rope scaling,rope_freqs, and tensors whose shapes contradict the metadata, each with a reason.- Kernels:
embed,rmsnorm,matvec(F32 / F16 / Q8_0 / Q4_0),rope,attention,swiglu,accumulate.kernel-tests/kernels.test.tschecks each one against a CPU reference (embed exactly, the rest within 1e-5). gguf/runtime.ts:GgufRuntime.load→queueStep/forward/readLogits. Everything per-token is built at load, and a step rewrites one 16-byte uniform.- Parity with llama.cpp, without the Hub. The reference is
llama-cpp-python, which builds llama.cpp from source, rather thanllama-server.scripts/gen_webml_llama_parity.pywrites two tiny random-weightllamaGGUFs and records llama.cpp's logits at every position, with an f32 KV cache and no flash attention.kernel-tests/forward.test.tsreplays them:- All-F32 model: the same 32 greedy tokens. The worst logit differs by 5×10⁻⁷ of the row's largest, about the fixture's 6-decimal rounding.
- The same model with every tensor chunked to 1 KB bindings: the same logits.
- Q8_0 / Q4_0 / F16 model with tied embeddings: worst 0.77% and mean 0.42%, from llama.cpp quantizing activations to q8_0/f16 where the engine keeps f32. The top token agrees at 40 of 40 positions.
- The suite catches bugs. These planted bugs each fail it: llama treated as
NEOX rope, GQA heads mapped
h % group, Q4_0 nibbles swapped, attention scaled by 1/d instead of 1/√d.
Changed from the plan:
- The dev page is replaced by
kernel-tests/real-parity.test.ts. It runs a real model from a fixed prompt and checks the result instead of only showing it. There is no tokenizer until 6.2, so the prompt's token ids come from llama.cpp's tokenizer in the reference file. - The reference script is
gen_webml_llama_parity.py --model, not allama-serverdriver.
Real models (run 2026-10-08, Intel UHD Gen12 iGPU, WEBML_GPU=hardware):
the same 64 greedy tokens as llama.cpp on Qwen3-0.6B Q8_0 and
Llama-3.2-1B-Instruct Q8_0, top-5 probabilities within 2×10⁻⁶ of llama.cpp
with an f32 cache (3×10⁻⁶ after the 6.4 kernels; 1.1×10⁻³ with 6.4's default f16
cache). The reference must come from --f32: against llama.cpp's own Q8_0 run the
difference reaches 0.08 on Qwen3 and a near-tie flips on Llama at step 15, because
llama.cpp rounds activations to q8 inside quantized matmuls and its F32 run of the
same weights diverges from its Q8 run the same way.
uv run --with llama-cpp-python python scripts/gen_webml_llama_parity.py \
--f32 --model ~/models/Qwen3-0.6B-Q8_0.gguf --out ~/models/qwen3.json
WEBML_GPU=hardware WEBML_PARITY_MODEL=~/models/Qwen3-0.6B-Q8_0.gguf \
WEBML_PARITY_EXPECTED=~/models/qwen3.json \
pnpm --filter @horrible/webml exec vitest run --project kernels real-parity
On Windows, llama-cpp-python installs from its CPU wheel index
(--extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu), and Vite's
/@fs/ only serves files on the project's drive, so models on another drive go
through a junction. The summary lands in kernel-tests/.results/ (browser mode does
not forward the page's console).
SmolLM2-360M was run later in its Q4_K_M form (see 6.4), not Q8_0.
6.2 A real engine in the app
tokenizer.ts:gpt2BPE with thesmollm,llama-bpeandqwen2pre-tokenizers, plusllamaSentencePiece. A golden corpus (code, CJK, emoji, whitespace runs, special tokens) is checked againsttransformerstokenizers of the source repos. Unknowntokenizer.ggml.prevalues are refused.template.ts: chat templates withtoolsandenable_thinking, rendered the same as transformers.js renders the same template, verified on the same message fixtures.gguf.worker.tsimplements the protocol: progress per byte range,warmup(pipeline compilation plus one token), interrupt, unload, and the one-generation-at-a-time rule.store.ts(OPFS) with resumable downloads. The cache listing covers both stores.- Playground: a GGUF tab next to the catalog. Enter a repo, see its
.gguffiles with sizes (HF tree API, aslist_repo_ggufsdoes on the backend), and get the header preflight verdict before the download button enables. device.lostis handled like a worker crash, throughonCrash.
Exit: chat in the playground with SmolLM2-360M GGUF. The model shows in
webml_manifest, and a browser-provider chat turn runs on it. pnpm test
and the kernel harness are green.
Built (2026-10-08):
- Tokenizer (
tokenizer.ts), ported from llama.cpp'sllama-vocab.cpp:- byte-level BPE with the
default,gpt-2,starcoder/smollm,llama-bpe/llama3andqwen2pre-tokenizers, and SentencePiece with byte fallback; - special tokens split out longest first, and llama.cpp's end-of-generation set.
- It matches the reference ids for every test string of llama.cpp's five
vocab-only GGUFs (llama-bpe, qwen2, starcoder, gpt-2, llama-spm). Those are the
.outfiles llama.cpp's own tokenizer tests use. The tests download them from release tagb6800, pin each SHA-256, and CI caches them. smollmhas no vocab file of its own. llama.cpp gives it the starcoder regexes, and starcoder is tested.
- byte-level BPE with the
- Templates (
template.ts): tested on Qwen2's own template and on Llama 3 style BOS,toolsandenable_thinkingtemplates. ChatML is the fallback only when the vocabulary has its markers. - Sampler (
sample.ts): top-k 40, top-p 0.95, min-p 0.05, then temperature, as llama.cpp's defaults are (GGUFs carry no generation config). - Session, worker, client:
- The worker checks the header over Range before downloading anything.
- Prefill runs in batches of 32, so Stop is heard during a long prompt.
unloadwaits for a running generation instead of freeing its buffers.- A lost GPU device is reported, and the next load creates a new one.
- Store: resumable downloads, and partial files are flagged in "On this device" but never offered to the manifest.
- Tests:
- Unit: 77 in the node project, including the tokenizer vectors.
- Kernel project: 35, in Chromium on SwiftShader. The session and the real
WebmlEngine+ worker each reproduce llama.cpp's greedy continuation from a text prompt. OPFS download, resume and delete run in a real browser. - The tiny models'
t2…t63tokens are user-defined, so a prompt written as text tokenizes to exactly the reference ids.
- In the app:
pnpm devand Playwright, with huggingface.co answered by a stand-in that serves the tiny model. Find listed the file and its verdict (llama · F32 · 64 context), and Load downloaded it into OPFS and reportedready · F32 · 64 context · webgpu. A chat reply streamed with token probabilities. A Qwen3 file was refused, with the reason listed and Load disabled.
Changed from the plan:
- Tokenizer reference: llama.cpp's vocab test files, not a corpus run through
transformerstokenizers. Those need the Hub. - Picker placement: a "GGUF file from the Hub…" choice in the model picker, not a separate tab, so load, progress and ready stay one UI.
- Device loss: handled in the worker rather than through the client's
onCrash. The worker survives a lost device; only the device has to be recreated.
Still open:
- The SmolLM2 chat itself, and a
browser-provider agent turn on a GGUF model. Both need a real model from the Hub. - The manifest already lists complete
gguf:ids, and the relay passes the template's thinking flag. Neither has run against the backend loop with a real model.
6.3 K-quants and the agent models
- Q4_K, Q5_K, Q6_K (CPU references, then kernels). The
neoxrope, QK-norm,rope_freqs, separatekey_length/value_length. - Llama-3.2-1B and Qwen3 entries go into a GGUF section of
catalog.ts, withtoolFormatandthinkingas for the ONNX entries.
Exit: parity as in 6.1 for every model in the table at Q4_K_M. A Qwen3-1.7B
browser-provider agent round emits a Hermes tool call that the backend parses and
answers, and the next round completes.
Built (2026-10-08):
- K-quants: Q4_K, Q5_K and Q6_K, as CPU references (equal to the
ggufpackage's dequantizers on golden blocks) and as kernels.- The weight snippets now work in 32-value units, and a K-quant super-block is eight of them, so a K-quant row still spreads across the whole workgroup.
- Matvec and embed are checked for all seven types.
- Planted bugs are caught: Q6_K's second scale ignored, Q5_K's high bit read from the wrong plane, the packed 6-bit scales read from the wrong byte.
- Architectures (
arch.ts):qwen3(NEOX rope, per-head Q/K RMS norms, every head dimension rotated, as llama.cpp asserts) and Llama 3'srope_freqs, plus an optionalattention.scale.- The check now also refuses tensors the engine would ignore. Attention biases on
a
llamafile, for example, are refused rather than silently skipped.
- The check now also refuses tensors the engine would ignore. Attention biases on
a
- Kernels and rope: a new
headnormkernel does the Q/K norms. The rope table divides angles byrope_freqsin float32, as ggml does. - Parity with llama.cpp (
forward.test.ts). The K-quant files come from llama.cpp's own quantizer (llama_model_quantizefrom F32 sources), so their tensor-type mix is the one real Q4_K_M / Q5_K_M files have.- Qwen3, F32 (Q/K norms, NEOX, head size 32 with
embd / heads= 16): the same 32 greedy tokens, logits within 5×10⁻⁷. - Llama, Q4_K_M (Q4_K + Q6_K,
rope_freqs, tied): worst 0.39%, mean 0.25%, top token 40 of 40. - Qwen3, Q5_K_M (Q5_K + Q6_K, tied): worst 0.39%, mean 0.25%, top token 40 of 40.
- The differences come from llama.cpp rounding activations to q8_K for K-quant dot products.
- Planted bugs are caught: Q/K norms skipped,
rope_freqsignored, Qwen3 with NORM rope.
- Qwen3, F32 (Q/K norms, NEOX, head size 32 with
- Suggested repos:
GGUF_SUGGESTIONSincatalog.tslists SmolLM2-360M, Qwen3-0.6B/1.7B and Llama-3.2-1B GGUF repos with their tool format and thinking switch, shown as suggestions in the picker. The relay uses a suggestion's tool format as it uses the ONNX catalog's.
Changed from the plan: suggestions carry no sizes, unlike the ONNX catalog. The picker lists sizes live from the tree API, and the repo and file names could not be checked against the Hub from this environment. A suggested file is only preselected when the listing has it.
Real models (2026-10-08): Q4_K_M parity passes for SmolLM2-360M and Qwen3-1.7B (see 6.4). Qwen3-0.6B and Llama-3.2-1B were checked in Q8_0 (see 6.1); their Q4_K_M files were not downloaded.
Still open: a Qwen3-1.7B browser-provider agent round with a Hermes tool call
that the backend parses and answers.
6.4 Fast
In order, re-checking parity after each step:
- Tiled
matmul_<q>andattn_prefill, so prefill stops being a loop of decodes. - Prefix KV reuse across
generatecalls. - GPU
samplewith top-k readback, then pipelined decode. - Subgroup reductions; matvec tuned per quant (rows per workgroup, vectorized
u32loads). shader-f16activations and KV cache, where available.- Fused epilogues (residual add, norm into the next matvec's input).
timestamp-query, when the adapter has it, reports per-kernel time in a dev
overlay, so each step's change is measured rather than assumed.
Exit, on the Gen12 machine:
- SmolLM2-360M Q4_K_M decode > 27 tok/s (faster than its ONNX export).
- TTFT ≤ 0.7 s for a short chat prompt.
- A Qwen3-1.7B agent round after the first under 10 s, with prefix reuse. Today the 0.6B model takes ~50 s.
Which engine is the default. For a model that has both an ONNX export and a
GGUF, the GGUF engine becomes the default only if its decode is faster than
the ONNX export on the same machine. Equal speed is not enough, even with prefix
reuse. Each catalog model with both formats is measured here, and the result is
recorded in catalog.ts as that model's preferred engine. Where GGUF loses, ONNX
stays the default and GGUF is still available when picked explicitly. Models with
no ONNX export (local fine-tunes, anything from the GGUF tab) always run on GGUF.
Built (2026-10-08): all six steps. Each one was re-checked against llama.cpp on the tiny fixtures (both on SwiftShader and on the iGPU) and on the two real Q8_0 models.
- Batched prefill. The runtime builds two dispatch lists over the same
weights and caches: decode (one token: matvec, split attention) and prefill
(up to
batchtokens, default 256:matmul.wgslandattn_prefill.wgsl), plus a shared head that runs the final norm and the LM head on the last position only. Token ids live in a GPU buffer and a pass's (position, row count) in one Step uniform.forward.test.tschecks the logits after every prefix length, in passes of 5 tokens, against llama.cpp on all five fixtures. - Prefix reuse.
GgufSessionkeeps the token ids behind the live cache and prefills only what follows the longest common prefix (always feeding the last prompt token).usage.cachedTokensreports what was reused. A failed generation drops the reuse state. - GPU sampling, pipelined decode.
sample.wgslruns llama.cpp's chain (top-k 40, top-p, min-p, temperature, draw with a host random number) in one workgroup: an exact top-k by bisecting for a threshold, then a bitonic sort of the candidates. It writes the token into the buffer the next step's embed reads and a small record (id, p, entropy, alternatives) that the host reads instead of a vocabulary of logits. The session queues step N+1 before reading token N; on EOS the speculative step's cache entry is not kept.sample.test.tsholds it togguf/sample.tson seeded cases, including flat logits with thousands of candidates. - Kernel tuning. These changes had the most effect:
- Weight snippets read whole words (a funnel shift for blocks at 2-byte alignment) and unpack four values at a time, rather than one byte at a time.
- Matvec gives each lane a 32-value unit and reuses its activations across four rows, so X is not re-read per row. Because the activations are f32, X was ~4× the bytes of a Q8_0 row.
- Decode attention splits positions into 128-key chunks (flash-decoding plus a combine pass), serves every query head of a KV group from one workgroup, and reads K in 16-byte pieces shared by four lanes.
- Prefill attention keeps Q and the output in registers and stages nothing but a 4 KB block of partial scores in workgroup memory. Staging the K/V tiles there left only two workgroups per core: 9.6 GFLOP/s.
- Matmul keeps its tiles k-major as vec4s, uses an 8×4 per-thread block, and
decodes a unit's scales once (
unit_prep/unit_quad).
- f16 KV cache (
LoadOptions.kvCache, defaultf16). New K/V rows are computed, normed and rotated in f32, then stored bykv_store.wgsl, as llama.cpp does. The store rounds to nearest even itself: this iGPU'spack2x16floatrounds toward zero. It uses core WGSL packing, notshader-f16, so SwiftShader and CI run the same path. - Fused epilogues. The two residual adds per layer are a flag on the
attention-output and FFN-down projections (
accumulate). The separate kernel is gone.
Measuring: bench.test.ts (WEBML_BENCH_MODEL) times chat and two agent rounds
through GgufSession and records a per-kernel profile (GgufRuntime.profile,
timestamp queries). perf.test.ts (WEBML_PERF=1) times each kernel alone at
Qwen3-0.6B shapes. Results go to kernel-tests/.results/.
Measured on Qwen3-0.6B Q8_0, Intel UHD (Gen12, 67 GB/s streaming reads), with a 3.5k-token agent prompt. The first column is steps 1–3 on the 6.3 kernels. The 6.3 engine itself (one token at a time, no reuse) had not finished its first agent round after 80 minutes and was stopped.
| Steps 1–3, 6.3 kernels | All six steps | |
|---|---|---|
| Chat: TTFT, 20-token prompt | 554 ms | 171 ms |
| Chat: decode | 2.9 tok/s | 19.0 tok/s |
| Agent round 1 (3,668 tokens prefilled, 30 generated) | 209 s | 43.5 s |
| Agent round 2 (3,664 of 3,719 tokens reused) | 18.9 s | 3.4 s |
| Matvec (Q8_0 / Q4_K) | ~2 GB/s | 13 / 10 GB/s |
| Matmul, 256 tokens (Q8_0 / Q4_K) | 61 GFLOP/s | 187 / 193 GFLOP/s |
| Decode attention at 3.7k positions | 3.5 ms/layer | 0.83 ms (f16), 0.59 (f32) |
The ONNX baseline for an agent round on this model was ~50 s.
Exit, on the same machine (bench.test.ts, 3.5k-token agent prompt, files from
the Hub: bartowski/SmolLM2-360M-Instruct-GGUF and unsloth/Qwen3-1.7B-GGUF):
| Check | Target | Measured |
|---|---|---|
| SmolLM2-360M Q4_K_M decode | > 27 tok/s | 30.2 tok/s (25.7 at 3.7k context) |
| SmolLM2-360M chat TTFT (38-token prompt) | ≤ 0.7 s | 161 ms |
| Qwen3-1.7B Q4_K_M agent round after the first | < 10 s | 5.8 s: 3,664 of 3,721 tokens reused, 29 generated at 8 tok/s |
Both models also match llama.cpp (--f32 references, default f16 cache): the same
64 greedy tokens, top-5 probabilities within 5.7×10⁻⁴ (SmolLM2) and 1.1×10⁻³
(Qwen3-1.7B). Qwen3-1.7B's first round, a cold 3.7k-token prefill, takes 81 s.
Q5_0 joined the supported types. The "Q4_K_M" SmolLM2 file holds mostly Q5_0
(159 of 269 MB): llama.cpp's quantizer falls back to it where a row is not a multiple
of 256 values, and SmolLM2's is 960. quant.ts matches the gguf package's
dequantizer on golden blocks, and the kernel snippet is checked as the others are.
A planted high-bit slip fails the matvec and matmul tests.
Changed from the plan:
- f16 activations: not done. Only the cache is f16. Activations are a small share of the bytes now that matvec reuses them from registers.
- The dev overlay is
GgufRuntime.profileand the bench's report, not a panel.
Still open:
- The default engine per model. The rule needs ONNX and GGUF runs of the same
model on this machine. The one comparison so far is SmolLM2-360M: 30.2 tok/s on
GGUF Q4_K_M against Phase 5's 27 tok/s on ONNX q4f16. That was measured on this
iGPU but in an earlier session, with different prompts, so
catalog.tsis unchanged until both run in the bench. - Matvec is at ~20% of streaming bandwidth. The next candidates are subgroup reductions and more rows per activation load. Neither won on a first try: 8 rows per lane group was slower.
- f16 decode attention is ~40% slower than f32 on this iGPU although it reads half the bytes. Neither the conversions nor the load shapes explain it, measured separately. It stays the default for the memory (half a gigabyte at 4k context on Qwen3) and for prefill, where it is twice as fast.
- SwiftShader miscompiles a dynamically indexed private array kept live across workgroup barriers. The decode attention avoids the pattern; new kernels should too.
6.5 This node's GGUFs in the window
- A backend route in the llama.cpp module streams a servable GGUF with
Rangesupport. It serves only filesllamacpp.list_modelsalready lists, never an arbitrary path, behind the same auth as the rest of/llamacpp. - Run in this window on the llama.cpp model list and on a finished training
run's GGUF (training lineage already links them). The load uses
source: { kind: 'gguf-node' }. Same-origin bytes skip the HF download, and OPFS still caches them. - Hosted (Vercel) has no node, so the action is hidden there.
Exit: a model fine-tuned and converted on this node chats in the playground with nothing fetched from the Hub.
Built (2026-10-08):
- The route:
GET /api/llamacpp/models/file?path=…serves a catalog GGUF through Starlette'sFileResponse, which honoursRange. The boundary is/models/layers's:catalog.find_modelmust list the path, so an arbitrary file, or a path that walks out of the managed directory, gets a 404.- A lookup the catalog vouched for is kept for 60 s. A load is hundreds of Range
reads, and
find_modelrescans every model directory on each call. - CORS now exposes
Content-Range, because the desktop webview is a different origin from the backend.
- A lookup the catalog vouched for is kept for 60 s. A load is hundreds of Range
reads, and
- The id:
gguf-node:<path>, beside the Hub'sgguf:<repo>/<file>. Both run on the GGUF worker.- The worker cannot know the backend's origin, so the window passes the file's
URL with the load:
LoadRequest.url, fromWebmlEngine'ssourceUrloption. Core sets that option to the route. - There is no
.ggufcheck on the path, because Ollama's blobs have none.
- The worker cannot know the backend's origin, so the window passes the file's
URL with the load:
- The worker checks the header over
Range, then uploads the tensors straight from the route. The reads are the "download" progress the playground shows. - Run in this window appears on the llama.cpp pane's model rows and on the
training convert card's result, through the webml module's
runNodeGguf. It is offered only when:- the architecture is one the engine runs;
- the file is not a LoRA adapter;
- the app is not hosted. A hosted backend is a remote machine, so every load would pull the whole file across the network.
- Tests:
- Backend: Range reads, refused paths, the exposed header.
- Unit: id round-trips, routing, the URL handed to the worker.
- Kernel project: the real worker streams the tiny model from an HTTP URL and reproduces llama.cpp's continuation, storing nothing in OPFS.
- Exit, in the running app (
pnpm dev, Playwright driving Edge on the iGPU):- The node already held
qwen3-0-6b-sft-merged-q8_0.gguf, a Qwen3-0.6B fine-tune trained and converted here. "Run in this window" on its row loaded it in 7.9 s, in 323 Range reads from the node. - It answered at 19 tok/s, TTFT 219 ms.
- The page made no request to the Hub. The LoRA adapter's row offers no button.
- The node already held
Changed from the plan:
- No OPFS copy. The plan said OPFS would still cache node files. It does not: the node is this machine, and a second copy of a file already on its disk is waste. Loading streams from local disk over loopback in seconds.
- No
sourcefield. The id andLoadRequest.urlcarry the source, as 6.2 already decided for Hub files.
6.6 Later, each optional
- Interpretability readouts during generation: per-layer residual norms and a
logit lens into
{tokenviz}and the model explorer. - Qwen2.5 (
qwen2, QKV bias), Gemma 3 and SmolLM3 architectures. - GGUF in published Scrive pages (bundled engine asset).
Built (2026-10-08): qwen2 and gemma3. Both were tested on files already on
this node, so nothing was downloaded: DeepSeek-R1-Distill-Qwen-1.5B Q8_0 (LM Studio,
qwen2) and Gemma-3-1B-it Q4_K_M (llama.cpp's cache).
-
qwen2:attn_q.bias,attn_k.biasandattn_v.biasare added after their projections by thebiaskernel. Rope is NEOX. Thedeepseek-r1-qwenpre-tokenizer is Qwen2's, as in llama.cpp. -
gemma3, as llama.cpp'sllm_build_gemma3builds it:- The embedding rows are multiplied by √embd.
- The output projection and the FFN-down projection write a scratch row.
rmsnormwithaccumulatethen norms it into the residual (post_attention_norm,post_ffw_norm). - The gate is GELU's tanh approximation.
- Of every six layers, five see only the last
sliding_windowpositions (512) and use rope base 10000; the sixth sees everything at the file's base (10⁶).attention.sliding_window_patternandrope.freq_base_swaare honoured when present. - The attention scale is computed, not read: 1/√head_dim, or 1/√(embd/heads) for 27B, which is detected by its 62 layers.
- Final-logit soft-capping and a per-layer window pattern are refused by name.
- The KV cache is still allocated in full for windowed layers. A ring buffer would cut that memory; correctness did not need it.
-
A tokenizer bug the new check found. Gemma's vocabulary holds both
#and\ufeff#. The GGUF reader'sTextDecoderdropped a leading byte-order mark, so the two became one string and code comments tokenized differently from llama.cpp. The reader, and both decoders of generated text, now keep it. -
Tests:
- Kernels: windowed decode attention (a window inside one chunk, one starting
past whole chunks, one wider than the context so far) and windowed prefill;
GELU with gates out to ±60; the bias add; accumulating
rmsnorm; the scaled embedding. - Forward: tiny
qwen2(biases) andgemma3(six layers, a 5-key window, two rope bases) files written and run by llama.cpp, greedy and batched prefill.qwen2agrees within 1e-5.- Gemma agrees within 2e-4. llama.cpp's CPU GELU is a lookup table from f16 inputs to f16 outputs. Emulating that table in the kernel brought the gap to 3.7e-5, so the gap is the table.
- Real parity: the reference script now also records the prompt's text and llama.cpp's ids for a corpus of awkward strings. The test checks the engine's tokenizer against both, on the model's own vocabulary.
- Kernels: windowed decode attention (a window inside one chunk, one starting
past whole chunks, one wider than the context so far) and windowed prefill;
GELU with gates out to ±60; the bias add; accumulating
-
Real-model parity on the iGPU, against llama.cpp's F32 run, 64 greedy tokens each:
Model Prompt Worst top-5 Δp Mean DeepSeek-R1-Distill-Qwen-1.5B Q8_0 11 tokens 0.0013 0.00018 Gemma-3-1B Q4_K_M 17 tokens 0.0021 0.00027 Gemma-3-1B Q4_K_M 1,396 tokens: decode runs past the window 0.0018 0.00015 -
In the app: "Run in this window" on the node's DeepSeek-R1-Distill-Qwen-1.5B row loaded it in 18.5 s, with 371 Range reads. It reasoned, closed
</think>, answered and stopped on EOS: 242 tokens at 7.7 tok/s, TTFT 552 ms, no Hub requests.- Gemma 3 is not in the node's catalog, so it ran through the bench instead, which uses the same session, template and tokenizer.
- Its chat stopped at
<end_of_turn>after 106 tokens: TTFT 367 ms, decode 11.5 tok/s. - In the agent bench, round 1 (3,691 tokens) took 39.9 s and round 2 took 1.75 s with prefix reuse.
- Decode is FFN-bound, plus 24 ms per token for the 262k-row LM head.
-
The picker suggests Qwen2.5-0.5B-Instruct and Gemma-3-1B-it. Neither repo is gated.
Built (2026-10-08): the logit lens during generation. A generate with
lens: true adds, to every step event, one readout per layer at the position
being predicted:
- the residual stream's L2 norm after the layer;
- the entropy of what the LM head says from it;
- that distribution's top five tokens.
How the engine does it:
- Collecting the inputs:
lens_in.wgslruns after each layer's last residual add. It norms the pass's last row with the model's final norm into that layer's row of a[layers × embd]buffer. - Per layer, at once: the head list then runs the LM head over all the rows as
one prefill-matmul batch, so it is one pass over the head's weights rather than
one per layer.
lens_topk.wgslkeeps each row's entropy and top tokens. Onlylayers × 12words come back, copied beside the sampler's record so pipelined decode is untouched. - Switching: the dispatches are flagged and skipped unless
runtime.lensis set, so the lens costs nothing when off. Its buffers are always allocated,layers × vocabfloats: 17 MB for Qwen3-0.6B and 27 MB for Gemma-3-1B. - The self-check: the last layer's lens is the model's own output, so it must
reproduce the pass's logits (same top tokens, probabilities and entropy). This is
tested on llama, Qwen3 and Gemma 3, after a prefill pass and a decode step. It is
the same argument
backend/modules/llamacpp/lens.pymakes for its traces. - Cost, Qwen3-0.6B Q8_0 on the iGPU:
- Decode drops from 19.4 to 9.3 tok/s.
- The 28-row lens matmul over the 151,936-row head takes 51 ms per token, against 12 ms for the one-row head. It is compute-bound, so it is opt-in.
- Where it shows:
- The
webml.lenssetting turns it on in the playground. :show: lensturns it on for{webllm}.- The token strip's hover card shows a per-layer table: each layer's best token, in the accent once it is the token chosen, its probability, and a norm bar.
- A run kept as a
{tokenviz}figure carries the lens, so a published page shows it as plain markup.
- The
- In the app, a node Qwen3-0.6B fine-tune answering "the capital of France in one word" shows the expected shape. The early layers say "no" and "events" at a few percent; layer 25 says "France"; layers 26–27 say "Paris".
Built (2026-10-08): smollm3. llama.cpp's llm_build_smollm3 is its llama
graph with one change: layers where (il + 1) % 4 == 0 (3, 7, 11, …) skip rope on Q
and K, so they see positions only through the causal mask. The step is hard-coded to
4 in llama.cpp, not read from the file, and the engine does the same
(ModelConfig.noRopeStep, layerRoped). As llama.cpp asserts, every head dimension
must be rotated on the other layers. The tokenizer is Llama 3's: its pre-tokenizer
hashes to llama-bpe in llama.cpp's converter, which the engine already runs.
- Tests: a tiny five-layer
smollm3file, so the NoPE layer sits between roped ones, written and run by llama.cpp (llama-cpp-python 0.3.34). Greedy continuation, logits within 1e-5, batched prefill and the lens self-check all pass. With rope wrongly applied on layer 3, the prefill check fails at the fourth token. - Not checked on a real file. No SmolLM3 GGUF was on this node, and the Hub was
out of reach from where this was built. So there is no real-model parity run yet.
SmolLM3's chat template (
/thinkand/no_think,enable_thinking) has also not been run through the template interpreter, and the playground's picker does not suggest a SmolLM3 file until it has.
Built (2026-10-08): the lens in the model explorer. The explorer draws the window's latest lens run under Whole model and under a selected block, as a layer × token grid with a footer row saying where each prediction settles (see Interpretability).
- Recording: the app's engine (
webml/engine.ts) tees thestepevents of any generation withlens: trueinto one app-global run (webml/lens-run.ts). The playground and{webllm}are unchanged, and the grid fills while the reply is written. - Linking: rows select blocks in the diagram only when the run's layer count matches the explored model's block count. The explorer follows the node's model, and the window's is often a different one, so the grid always names its own.
Built (2026-10-08): GGUF in published Scrive pages. A published {webllm} with a
gguf:<owner>/<repo>/<file.gguf> model now runs in our engine instead of being an
ONNX-only block.
- The asset:
vite.embed.config.tsbuildsgguf.worker.ts, with the WGSL kernels, the tokenizer and the template interpreter, into one ES-module worker,backend/modules/scrive/static/webml-gguf.worker.js. It is 211 kB (56 kB gzipped), gitignored and built like the scene runtime: bypnpm --filter @horrible/webml build:embed, bypnpm devwhen it is missing, by the desktop backend bundle, and inpackages/webml's CI. - Publishing: a site with a
gguf:{webllm}gets the worker at_scrive/webml/gguf.worker.js, besideembed.html. If it is not built, the publish stops and names the command. Agguf-node:model (a file on the author's machine) is a preflight warning (node-model), and its published block says it cannot run. - The page:
embed.htmlstarts the worker as a module worker and speaks the app's worker protocol in a few lines of plain JS. The app's client and the transformers.js worker stay out of the bundle. The file is read from the Hub over Range requests and kept in the site's OPFS, so the button says the size (from the Hub's tree API) or that it is already cached. - Lens:
:show: lensreaches published pages too. Each token's hover card shows the per-layer table the app's token strip does. - Checked in headless Chromium on SwiftShader, with the Hub's responses served
from the tiny
llamaandsmollm3fixtures. The page loaded the file, answered with its token strip and a per-layer lens, and said "cached on this device" after a reload. This also fixed an older bug in the page:display: flexoverrodehidden, so the "Run it here" gate stayed on screen after loading and the message box showed before it.
Built (2026-11-14): neuron edits and activation capture for the Interp Suite's Neurons tool, a WebML take on webml-community's Neuron Studio.
- Edit site: the gated feed-forward's activation, after
glu.wgsland before the down projection reads it.neuron_edit.wgslsetsact[i] = act[i] * scale + offseton every row of the pass.- All layers' edits live in one buffer of 16-byte records,
MAX_LAYER_EDITS(64) slots per layer. - A layer with no edits skips its dispatch, as the lens's dispatches are skipped
while
lensis off. An empty recipe therefore runs exactly the old lists. GgufRuntime.setNeuronEditsreplaces the whole set.
- All layers' edits live in one buffer of 16-byte records,
- Cache:
GgufSessiondrops its KV prefix whenever the edit set changes (editsKey). A steered reply to the same prompt as the baseline must not reuse the baseline's unedited positions. - Capture:
neuron_capture.wgslcopies one position's activation, after the edits, into a layers × ffn buffer.- Only the pass holding that position writes, so it can sit in every prefill batch's list.
GgufSession.captureActivationsreplays a conversation plus a reply up to the position that predicted reply tokenindex, then reads the buffer back. Greedy replies are deterministic, so a replay under the same edits reproduces them.
- Down columns: ‖W_down[:, n]‖₂ is the norm of
W_down · e_n. That is the existing decode matvec on a one-hot input, built per layer at load and kept out of the decode list as the LM head is. Every weight type is read by its own kernel, with no CPU dequantizer, and the result is cached per neuron. - Residual medians:
GgufSession.residualMediansfeeds a fixed neutral passage a token at a time with the lens on. It takes each layer's median ‖x‖₂ (the lens readout'snorm) over positions after the first. An additive dose isdose × median_layer ÷ ‖W_down[:, n]‖. - Protocol:
GenerateRequest.neuronEdits, plus oneneuronsrequest with opscolumns,mediansandcapture.- These run in turn with generations, and the capture's array is transferred.
readynow carries the model'sshape(layers, ffn, embd, arch), and the client exposes it asWebmlEngine.neurons(op).
- Checked in
kernel-tests/neurons.test.tson SwiftShader:- An empty recipe gives the llama.cpp reference reply, and the steered reply's cache does not leak into the next unedited one.
- An ablation zeroes exactly the named neuron while earlier layers are untouched, and an offset lands at +2.5.
- Prefilled and decoded captures agree.
- Down-column norms match the dequantized f32 and Q5_K weights.
Risks
| Risk | Mitigation |
|---|---|
| iGPU limits: binding size, total VRAM shared with the tab | Chunked tensors (decision 4). The preflight budget compares the header's tensor bytes plus KV (ctx × layers × 2 × kv_heads × head_dim × bytes) with maxBufferSize and refuses with numbers |
| Tokenizer drift: one wrong pre-tokenizer regex silently changes every prompt | Golden corpus per supported tokenizer.ggml.pre, and anything else is refused |
| Numerical drift that "looks fine" | 64-token greedy parity against llama.cpp at every stage exit, re-run after every 6.4 step |
| SwiftShader lacks a feature (subgroups, f16) | Those paths fall back, so the baseline kernel is always tested. The fast paths are checked on real hardware by the parity script |
| OPFS quota or eviction | navigator.storage.persist() on the first GGUF download. Quota is checked against the file size before starting |
| Maintaining a second engine indefinitely | The scope is the supported-models table. Everything else stays on transformers.js and is refused at preflight, not half-run |
Decided
Settled on 2026-10-08, before 6.0:
- GGUF weights live in OPFS (decision 1). "On this device" lists both OPFS and Cache Storage.
- Kernel tests run in Playwright on SwiftShader (decision 7), as a browser
job in
packages/webml's CI. - Qwen2.5 moves to 6.6. The 6.3 set is SmolLM2, Llama 3.2 and Qwen3.
- GGUF replaces ONNX as a model's default only if it decodes faster (6.4). Matching ONNX speed is not enough.