Module: training
Notebook-driven neural-network training inside the dashboard: pick an environment
(a Kaggle competition, a Hugging Face dataset, a Gymnasium env — or any plugin-provided
source), get a project with its own uv-managed venv and a scaffolded .ipynb
notebook, then build and train models with live visual feedback (metrics charts, model
architecture graph, manim explainers, gym rollout frames). The agent drives the whole
flow: "i want to work on kaggle's pokemon tcg competition" resolves the competition,
creates the project, downloads the data, and opens the right panes.
Status: implemented — frontend in packages/core/src/modules/training/, backend in
backend/modules/training/. Projects + per-project venvs + the environment provider
layer, the native notebook pane on the Jupyter kernel protocol, the live metrics
pipeline + chart pane, the architecture graph pane, the manim + gym-rollout runners,
Kaggle-kernels/Colab push, the peer-fabric training ads, and the agent tool integration
that drives the whole flow from a natural-language ask.
Projects on disk
Each project is one directory under training.projectsRoot (default
~/horrible/training); the directory is the record:
<projectsRoot>/<slug>/
main.ipynb # scaffolded notebook (nbformat v4, Kaggle-compatible)
project.json # ProjectModel dump (id, name, refs, python, flags)
data/ # provider-fetched datasets
media/ # manim renders, saved frames
.venv/ # uv-managed project venv
Torch-sized dependencies live in the project venv, never the backend env. The venv
is created with uv venv --python <training.defaultPython> and bootstrapped with
ipykernel and ipywidgets (envs.KERNEL_PACKAGES) plus the local
horrible-train helper package, then the provider's scaffold requirements (pandas,
gymnasium[box2d], …). ipywidgets is there because the libraries a training
notebook runs look for it: tqdm and the Hugging Face Trainer draw a widget
progress bar when it is importable and otherwise print one frame per update — a worse
bar, plus a warning telling the user to upgrade Jupyter, which was never the problem.
The kernel packages are installed by install_stack as well, because a venv is
bootstrapped once and never revisited, so a project made before that list changed
would otherwise never catch up. All subprocess work (uv, kernels, renderers) uses blocking
Popen pumped on daemon threads — the Windows-safe pattern shared with the LSP/terminal
modules — with output lines streamed to the UI.
A directory whose project.json is missing or unreadable (e.g. a partial dir left by an
aborted bootstrap — just a .venv) is treated as not a project: it's skipped by
listing/lookup, creating the same env adopts the slug (writes project.json, reuses
the existing .venv) instead of stranding it behind a -2 suffix, and delete removes
such partial dirs (which get_project can't resolve). If a persisted notebook pane
references a project that no longer exists, the kernel-open error carries an
unknown_project code and the pane shows a "no longer exists" state with a Close
button that drops the dead pane from the saved layout — so it self-heals on the next load.
Projects another module owns
ProjectModel.owner names the module that keeps a project as working storage.
The only one today is evals,
which builds one per suite to run Hugging Face benchmarks in. Those are created
straight through create_project rather than through the create route, so — and
this is the part that bit — they have no scaffolded main.ipynb, and their venv
carries only the benchmark's requirements: no ipykernel, no horrible_train.
They are listed and marked, not hidden (the bundled-suite rule): a project
missing from the pane is a directory eating disk you can neither see nor delete. What
they do not get is authoring actions — open notebook, recipe, and both pushes are
disabled, each carrying the reason as its detail, because every one of them could
only fail. Disabled-with-a-reason rather than absent: the reason is the whole value,
and hiding them would make an inert project look identical to a healthy one. Delete
stays enabled — it is exactly the one you might want the disk back from.
The project row is a target, not a toolbar
A project used to render as one flex line holding its name, the owner badge, a
venv/data chip and five buttons (Open notebook, 🧪 Recipe, ⇪ Kaggle,
⇪ Colab, ✕) — in a 280px dock or a region strip beside the notebook. Nothing
read as primary, a push to somebody else's servers sat at the same weight as opening
the notebook, and at that width each button was a couple of characters wide.
Now the row is two lines (name + status; refs and the live progress note beneath)
and the row itself opens the notebook. Everything else is a context menu — a
declared contextMenu provider for the target kind training.project, the same
extension point file explorer and library use — reached by
right-click or by the row's ⋯ button, so the actions are discoverable without
knowing to right-click. Kaggle and Colab became a Push a copy to ▸
submenu: they are one verb with two destinations, and as sibling buttons they read
as two unrelated features.
Moving the actions out of the component took their progress lines with them — a
provider declared in the module manifest is a plain function with no component
state — so the pushing… / pushed → url / push failed line now lives in a small
module-level note store (training/activity.ts), which the pane subscribes to and
renders in the same slot as the env_progress / fetch_progress events.
Deleting a project also broadcasts project_changed now (with deleted: true).
Every other change to a project did; a delete did not, so the only list that updated
was the one belonging to whoever clicked, and a project removed through the agent or
from a second window stayed on screen.
Two failures this closed, both of the silent kind:
- A missing notebook read as a path. The open reached a bare
FileNotFoundError, whosestr()is only the path, and the pane renders that verbatim — so the whole explanation was a loneC:\…\main.ipynb. It is now aMissingNotebookErrorcarrying a sentence, and it deliberately does not use theunknown_projectcode: the project is fine, so a pane that self-heals by closing itself would be throwing away the wrong thing. - The kernel badge contradicted the error.
kernelstarts atstartingand onlyopenedor akernel_statusevent ever moved it, so a failed open left "● starting" sitting next to the message explaining why it hadn't. An error arriving beforeopenednow marks the kerneldead— and only then, since the same event also carries recoverable errors on a live session (a rejectedcellsop), where calling the kernel dead would be its own lie.
That "unreadable means not a project" rule has a sharp edge: background workers rewrite
project.json (_mark(project, venv_ready=True)) while requests read it, and a
write_text truncates before it writes. A read landing in that window saw an empty file
and the route answered 404 for a project that exists — a bug that only appeared under
load and looked like anything but itself. Both sides now go through
backend/atomic_write.py: writes replace atomically, and reads distinguish "no such
record" from "someone is replacing it right now", which on Windows is a transient
PermissionError rather than a missing file. Pinned by a concurrent read/write test in
test_training_projects.py.
Background work started by a route (fetch, venv bootstrap, dep install) runs on
tracked daemon threads: routes._spawn registers each one and routes.join_workers
waits for them. Nothing in the product blocks on it — it exists so tests can wait on the
work itself rather than on a wall clock, which is what a fixed-deadline poll silently
degrades into on a loaded machine.
Environment providers
A provider is a thin synchronous adapter behind one uniform contract
(backend/modules/training/providers/base.py), mirroring the database module's driver
layer: search, resolve, fetch (into data/, with progress), and scaffold
(starter notebook cells + venv requirements). Client libraries import lazily inside
methods; short calls run via asyncio.to_thread, fetches on daemon threads.
| Provider | Kinds | Fetch | Scaffold |
|---|---|---|---|
kaggle | competition, dataset | download + unzip into data/ (creds: training.kaggle.* settings → ~/.kaggle/kaggle.json) | pandas data-loading starter |
huggingface | dataset | lazy — the notebook calls datasets.load_dataset(id); snapshot into data/ on request | load_dataset starter |
gymnasium | env | nothing to fetch (bundled static catalog; no gym import in the backend env) | RL rollout loop streaming frames + episode rewards |
Plugins add providers: a backend plugin calls host.add_training_provider(provider)
(see backend plugin SDK); get_provider()
falls back to the plugin registry, so a new environment type is one file or one plugin
call.
The horrible-train helper
backend/modules/training/helper/ is a zero-dependency pip package installed into every
project venv. Training code (notebook cells or scripts) calls:
import horrible_train as ht
ht.run("baseline") # name the run (auto-created on first log)
ht.log(step=i, loss=0.42) # → live metrics chart pane
ht.frame(env.render()) # → live rollout pane
ht.watch(model, example=x) # → architecture graph pane (torch.fx, tree fallback)
Each call prints one @@HORRIBLE@@{json} sentinel line to stdout; the backend
strips these from visible output and fans them out as training channel events — the
same line-protocol trick the visualizer uses for Pygame frames, so it works identically
under a kernel, a script runner, or manim.
The recipe surface
A fine-tune is a couple of dozen numbers, and until this landed the only place to
put them was a notebook cell written from memory. recipes.py makes them a
schema: a curated catalog of fields, each with a type, a default and a
one-line description, rendered as a form by training.recipe (non-singleton,
params {projectId}) and materialized into main.ipynb.
There is no new execution path. materialize() returns notebook cells. They
run on the same kernel in the same venv, their ht.log() output is
indistinguishable from hand-written code's, and the same notebook is what the
Kaggle and Colab push already send. The recipe itself lives in recipe.json so
the form round-trips; the cells are yours the moment they land and are never
read back. A project with no recipe.json yet starts on its own dataset: the
Hugging Face dataset it was created from fills the dataset field, rather than
asking again for the one thing the project already knew.
The typed dataset ref is looked at too
Shape detection (datasets) used to run only
through the dataset picker — which covers registered datasets, while the default
path puts the id in the free-text field. So the most ordinary flow there is —
create a project from a Hub dataset, open its recipe, write cells — reached the
generated code with nothing having ever looked at the data: columnMap empty,
textField still text, no reshape emitted. On a question/solution reasoning
set that is KeyError: 'text' inside trl, minutes in, after the base model has
downloaded.
It also means dataset_id is checked rather than trusted. It names a dataset in
the datasets registry, not a Hub ref — but recipe.set offers a
model two parameters that both look like "the dataset", so a Hub id lands in the
wrong one. A dangling id is worse than none: resolve_dataset marks the dataset
missing, and detection used to skip a recipe that had one (a registered set
carries its own map, so nothing re-derives it). One wrong field became a run that
died on KeyError: 'text'. Now recipe.set rejects an id that resolves to
nothing, naming the difference, and only a resolvable id suppresses detection.
recipes.ensure_shape(recipe) peeks five real rows of a typed ref (cached, 5-minute
TTL, best-effort — an unreachable Hub leaves the recipe untouched, because being
offline is not evidence about a dataset's shape), and runs at both seams that
matter: rendering the form, and writing the cells. The second is not redundant —
recipe.apply and recipe.run are agent tools that may be the first thing ever to
touch a recipe. It returns the verdict either way, so GET .../recipe carries a
datasetShape the pane renders under the dataset field, and adopts it only into
a blank column map: a hand correction is the whole reason the map is editable and
must never be re-guessed over. A registered id is not second-guessed at all. A regenerate replaces the previous block (found by a
horrible_recipe cell-metadata flag, with the header line as a fallback) rather
than appending — two SFTConfig definitions in one notebook is a bug where the
second silently wins, discovered after a wasted run.
Frameworks: the recipe-backend layer
The recipe began as one framework and one task, hardcoded end to end: the field
catalog named SFTConfig, the probe imported trl, the emitter wrote SFTTrainer.
That was the right first shape and it does not extend — DPO is not a variant of
SFTConfig, unsloth loads a model differently, and pre-training is not a notebook
cell at all.
backends/base.py is the seam. A RecipeBackend declares which tasks it can train,
which knobs each takes, and what code those knobs become; everything around it — the
form, the persistence, the notebook write, the sweep — is written once against that
protocol. Four ship:
| backend | tasks |
|---|---|
trl | sft, cpt, dpo, kto, reward, grpo |
unsloth | sft, dpo — 4-bit QLoRA on fused kernels, NVIDIA only |
torchtitan | pretrain |
nanotron | pretrain |
A plugin adds more with host.add_recipe_backend(backend).
Unsloth reuses trl's field catalog rather than copying it — it ends in the same
SFTTrainer, and a second copy would be a second place to fix the
warmup_ratio/warmup_steps unit trap. What it adds is the loader, and three
things there are each a real bug if missed: the model is passed as an object
(passing the name loads a second unoptimised copy), LoRA comes from
get_peft_model and not a peft_config= argument (passing both applies the
adapter twice), and max_seq_length is set at load time because RoPE scaling is
chosen there.
Pre-training is a different execution model and says so. A titan run is
torchrun --nproc_per_node=N -m torchtitan.train --job.config_file config.toml —
a multi-process launch reading a config file, with no object whose constructor takes
a learning rate. Those backends emit a config file plus a launch cell and run
through the script runner, single-node in v1. This node already advertises GPUs
over the peer fabric with no remote execution engine behind it; shipping --nnodes 1
and naming the gap is better than a launcher that appears to support a cluster.
check(task, profile) reads the hardware probe and honours its
three states — found a card, looked and found none, and could not ask. Saying
"no NVIDIA GPU" when nvidia-smi merely was not on PATH is the failure that module
exists to prevent.
Installing the stack
envs.install_stack is the step that never existed. bootstrap installs
ipykernel and the horrible-train helper and nothing else, which is why the
recipe form spent its whole life reporting "trl and peft are not installed", the
generated code was emitted unvalidated, and the LSP had no framework to index.
Autocomplete coverage was never a completion-engine problem; it was an empty venv.
The torch wheel is chosen from the hardware probe by card and OS together, the
same rule hardware._variant_for follows for llama.cpp builds — and the reason is
returned to the pane, because a CPU build landing on a machine whose owner knows
they have a card needs an explanation rather than a shrug. NVIDIA on Windows
needs the CUDA index (notebook.env.PYTORCH_CUDA_INDEX, shared so the two venvs
cannot disagree): PyPI's Windows torch wheels are CPU-only. This used to claim the
default wheel "already targets" the card, which put +cpu on an RTX 4080 and only
surfaced three cells later as trl's "Your setup doesn't support bf16/gpu". When
the probe could not ask, a GPU-capable wheel is installed rather than the CPU
one — on Windows that again means the CUDA index.
A venv that already has a CPU build is replaced, not kept: uv pip install torch counts any installed torch as satisfying the request, so without
--reinstall-package re-running the install reports success and changes nothing.
The build is read from torch/version.py, because the wheel's METADATA and
dist-info name both say plain 2.14.0 for 2.14.0+cpu.
Validated against what is actually installed
The expensive failure this exists to prevent: trl renamed
SFTConfig.max_seq_length to max_length, so a recipe emitting the wrong one
dies with a TypeError several minutes in, after the dataset has downloaded. So
fields carry aliases, and introspect() asks the project venv — one
subprocess, cached on the venv's mtime — which names its installed classes
actually accept, via dataclasses.fields() on the real SFTConfig/LoraConfig.
Four states, and the last two are not the same thing:
| Status | Meaning | Emitted? |
|---|---|---|
ok | the installed class accepts this name | yes |
renamed | it accepts an alias instead | yes, under the alias, with a comment |
unsupported | we asked, and it accepts neither | no — dropped, and named as dropped in the header |
unvalidated | we never got to ask (no venv, no trl) | yes, with a banner saying it wasn't checked |
Two fields make the rule concrete. max_length/max_seq_length are aliases —
same meaning, different spellings across versions. warmup_ratio and
warmup_steps are deliberately not, even though transformers 5 dropped the
first and kept the second: they carry different units, so aliasing them would
send 0.03 — three percent of training — into a field meaning "three hundredths
of a step", which is no warmup at all and raises nothing anywhere. Whichever one
the installed version accepts is the one that renders.
The form admits it is not the whole API. TrainingArguments has 112 fields
on transformers 5.15; this catalog has 24. Introspection.extra counts the
difference and the header says "24 of 118 knobs — edit the generated cell for the
rest", because a form that looks complete is a form you trust to be complete.
Parameter docs come from the node's own versioned doc index (the trl and
peft crawler seeds): GET /training/recipe/docs returns a link per config class
carrying the indexed version and the installedMismatch annotation, so the link
is to docs for a version you can compare against rather than to whatever is
current. An empty index yields no links rather than a guessed URL.
The base model is searched and checked, not just typed
basemodels.py is the one field whose value is copied verbatim into
from_pretrained(...), so every way of being wrong about it used to surface minutes
into a run, from inside transformers, after the dataset had downloaded.
GET /training/models/searchlists Hugging Face text-generation models, most downloaded first, and the pane's picker fills the field from it — the id is copied rather than remembered. A…-GGUFrepo is cautioned, not hidden: it is a real model, just the served copy, and a row missing with no explanation is worse.basemodels.checkjudges whatever is in the box and lands in the form's warnings. It is advisory and never blocks: a recipe that refused to generate because the Hub was unreachable would be worse than a cell that fails honestly.
Four judgements it makes, each a failure seen for real:
qwen3:0.6bis an Ollama tag, not a Hub id. It is the name this app shows everywhere else, so it is the obvious thing to type — and it reachedfrom_pretrainedunchanged. The warning names the search term to look for.- A bare name is not malformed.
gpt2andbert-base-uncasedare canonical Hub ids with no owner, so demandingowner/namewould reject ids that work. The Hub decides, not a regex. - "Not found" is not a licence problem. The Hub answers a missing repo and a
private one the same way, and its own 404 text mentions "private or gated" — so
matching on the word
gatedtold every typo to go and accept a licence. The not-found branch is checked first and worded to cover both. - A GGUF repo, an encoder, or a repo with no loadable weights gets the reason
rather than a stack trace: those are serving artifacts and non-causal models, and
SFTTrainercannot start from any of them.
Offline, every one of these yields no warning at all. Being unable to ask is not evidence that a model does not exist.
Holding a split out
evalSplit names the split scored during training (eval_strategy /eval_steps /
per_device_eval_batch_size), and empty means no evaluation — the honest
default, because most Hub datasets have no held-out split and inventing one fails at
load_dataset after the download. Without it a run reports only training loss, which
falls whether the model is learning or memorising: the one number that cannot tell
those apart.
Two details:
- The held-out split is loaded through the same
dataset_callas the training data — same ref, same loader, same local-file handling — so an eval set cannot be read differently from the data it is held out of. - An
eval_strategywith no split is overridden tono, not emitted: the trainer rejects it on construction, which is exactly the "option that emits a config the trainer refuses" this catalog exists to avoid. The generated cell says# no eval split is setrather than carrying a setting that does not describe the run.
eval_strategy and evaluation_strategy are aliases (renamed in transformers
4.46) — unlike warmup_ratio/warmup_steps, which carry different units — so the
probe emits whichever spelling the installed version accepts.
Two knobs a first fine-tune needs
max_steps and dataset_num_proc, both absent and both felt immediately:
max_stepsstops a run after N optimizer steps whatever the epochs say (-1runs the epochs). Without it, capping a trial run meant expressing "fifty steps" as a fraction of an epoch and doing the arithmetic against the dataset's length.dataset_num_procis the tokenization pass before training starts — single-core by default, which is two minutes of a 15,000-row chat dataset before a step runs.
Both validate against the installed class like every other field, so a version that
drops one drops it from the form rather than emitting a TypeError.
report_to is first-class, and local metrics stay authoritative
The tracker picker (none / tensorboard / wandb / mlflow / trackio) sets
SFTConfig.report_to. It is genuinely additive: every generated recipe
installs ht.callback() — a TrainerCallback in the helper package that mirrors
each logged scalar into the metrics pane — so the chart works offline, works with
report_to=["none"], and keeps working when a tracker's key expires mid-run.
"None" means no third party, never no metrics.
Credentials live in the trackers connector (Fernet-encrypted, W&B key and
MLflow tracking URI, both secret because a tracking URI routinely embeds one),
never in settings — GET /api/settings hands the whole bag to the browser. This
is the one connector whose id names no agent-tool group, because "log this run
to W&B" is a property of a recipe you wrote, not something the agent should do on
your behalf.
They reach training code through the environment at spawn: SessionConfig.env
(new, on notebook_core) is merged over the backend's own by the kernel manager,
and the script runner does the same. That is what keeps the key out of the
notebook, out of project.json, and out of every HTTP response. env_for() is
scoped to the trackers the recipe selected, so connecting the tile does not by
itself start shipping every run somewhere.
Ablation sweeps
sweeps.py runs the same recipe N times with one thing changed. Before it, "what
does learning rate actually do here" meant editing a number, running, screenshotting
the chart, editing it back, and remembering — which is not an experiment, it is a
recollection.
A sweep is the recipe plus axes: a field and the values to try, expanded as a
grid (every combination), a zip (axes step together, for knobs that must move in
lockstep like rank and alpha), or a seeded random sample.
Four things do the work:
- Points are processes, not kernel cells. A kernel is single and serial. Each
point becomes a
.pyfromrecipes.materialize_script— a flattening of the samematerialize()the notebook uses, so a sweep and a manual run cannot disagree about what a recipe means — and runs through the script runner. - Each run's
config_jsonis its resolved values dict, which is the entire mechanism behindlocaltrack.compare_runs. See localtrack. - One at a time by default. Two fine-tunes on one GPU is an out-of-memory error at an unpredictable point in the second, not double the throughput.
- Refusals are explained. An axis naming a field the selected task does not have
produces N identical runs and a comparison showing no effect — which reads as
"this knob does nothing".
validate()refuses it and says which fields exist.
MAX_POINTS is 64: a four-axis grid of five values each is 625 runs, and the cap
turns "I did not think about that" into an error rather than a machine that is busy
until Thursday.
Runs have an end now
A sweep is the first thing in this module that needed a run to finish. Everything
before it watched one run at a time, so "still running" was indistinguishable from
"finished and nothing else started" — metrics._mirror_for closed the previous run
only when a new one began, which for the last run of a session is never. Twelve
points made that visible immediately.
ht.finish(status) is the helper's terminal signal, emitted from ht.callback()'s
on_train_end, and the script runner closes any run its process opened when the
process exits (finished / failed on a stop / crashed otherwise). Both call
metrics.finish_run, which is idempotent — the common case is two calls for one
run, and the second must not overwrite the real status with the process's.
Closing the loop: checkpoint → GGUF
convert.py turns a finished checkpoint into a GGUF in the llama.cpp module's
managed model directory — the one its catalog scans, its disk budget counts and
its delete route may touch — so a model you trained is one this node serves. See
llamacpp.
- The converter is not in the release binaries.
convert_hf_to_gguf.pylives in the llama.cpp repo, so it is fetched on demand at the tag of the installed build (a converter from a different release can write a file the server reads slightly wrong), cached under the data dir with its sha256 recorded asverified: false— GitHub publishes no digest for a source archive. - And it is not one file. The script imports a sibling
conversion/package (one module per architecture) and puts the repo'sgguf-pyahead of the PyPIgguf, so the tag's source archive is fetched and exactly those parts — both converter scripts,conversion/,gguf-py/— are extracted into a staging directory that is renamed into place with aconverter.jsonmarker only once it is complete. Fetching the lone script, as this once did, failed every conversion withNo module named 'conversion'; a cached directory without the marker (that old layout) is replaced. - A LoRA checkpoint is not a model. Told apart by
adapter_config.jsonand sent toconvert_lora_to_gguf.pywith the base model read from that file; the result is labelled an adapter, sincellama-serverloads it with--lorabeside a base model rather than serving it alone. The base goes to--base-model-idwhen it is a Hub id and--baseonly when it is a local directory —--baseis a path, so the Hub id every adapter records made the converter look for a directory calledQwen/Qwen3-0.6Band exit. - So a LoRA recipe also saves a merged model. Nothing on this node serves an
adapter beside its base, so the default recipe used to end at a file that could
not be served. The generated save cell now also runs
merge_and_unload()into<output_dir>/merged(weights and tokenizer), which the checkpoint list offers as a whole model and the converter turns into a servable GGUF. - It runs in the project venv, which is where torch lives. Missing converter
deps (
gguf,sentencepiece,protobuf) are installed throughenvs.install—uv pip install --python <venv>, because a uv-created venv has no pip in it at all andpython -m pipfails on every project this module makes. - Output types are the ones the converter itself writes (
f16/bf16/f32/q8_0).q4_k_mis a separatellama-quantizestep and is deliberately not offered, because offering it and silently producing f16 would be worse.
The conversion records where the file came from
lineage.py writes one row per produced GGUF into app.db, keyed by its path.
Before it, a file called my-project-checkpoint-1200-f16.gguf in the llama.cpp
catalog told you a project id and nothing else — not the base model it was fine-tuned
from, not the recipe that produced it. "Score my fine-tune against its base" needs
that first one, and it was something you had to remember by hand.
The GGUF path is the join key, which is what keeps this a provenance row rather
than a workflow engine: RunTarget.model_path already names a file and
_output_path already writes a deterministic name, so everything downstream keys off
the path already. One writer — convert.py, at the one moment every field is
known. Several readers: GET /api/evals/targets labels a target "fine-tune of
<base>" and carries baseModel, and llamacpp.list_models reports the same.
Two details that decide whether it works:
- The base model falls back to the recipe.
run_conversion'sbase_modelargument is only required for a LoRA (an adapter is meaningless without one), so for a full fine-tune it is empty and the recipe is the only place the base is recorded. Reading the argument alone would leave every full fine-tune with no base — that is, with the feature not working in the common case. - The recipe is snapshotted, not referenced.
recipe.jsonkeeps changing as you tune the next run, and a row saying "trained with lr=2e-4" has to keep meaning that after you try 1e-4.
Writing the row can never fail a conversion: lineage.record swallows, because the
GGUF is on disk and servable whether or not the bookkeeping lands.
The notebook pane & kernel sessions
training.notebook is a native cell UI (CodeMirror cells, non-singleton, opened
with params {projectId, notebook}) speaking the Jupyter kernel protocol — no iframed
JupyterLab. The kernel engine itself is domain-neutral and shared: it lives in
backend/notebook_core/ (KernelSession, nbformat IO, the manager's ws event handling)
and is also consumed by the generic notebook module. training/kernels.py
subclasses it with the training specifics — sessions keyed {projectId}:{path}, spawned
from the project venv, with the stdout sentinel scan. The backend side runs one
TrainingKernelSession per open notebook:
- The kernel spawns from the project venv's python (
-m ipykernel_launcher) viajupyter_client's sync API on daemon threads (event-loop-agnostic: works under uvicorn--reloadon Windows). No kernelspec files — an in-memory spec pins the venv. - Sessions are process-global: closing the pane (or the tab) leaves training running; reopening reattaches and replays the current notebook state.
- The in-memory nbformat doc is authoritative — UI/agent edits mutate it, iopub
outputs append to it, and it's flushed atomically to the
.ipynb(debounced 2 s + after each completed cell), so the on-disk notebook stays Kaggle-compatible truth. - Executions run serially through a queue; stream output passes through the sentinel
splitter, so
horrible_traintelemetry never appears as cell text. !pip install x/%pip install xinstall into the project venv through uv (a uv venv has no pip), via the kernel extension every shared-engine kernel loads — see Installing packages.- Interrupt rides the platform interrupt (Win32 event / SIGINT). Windows caveat:
it lands between statements — training loops stop in milliseconds, but a single
long blocking C call (one
time.sleep(600)) defers it until that call returns; Restart is the hard stop.
A finished run still holds the GPU. The kernel is process-global and outlives the pane by design (that is what lets a six-hour fine-tune survive closing the tab), so the model, optimizer states and cache stay resident until the kernel is restarted or shut down — around 10 GB after a small LoRA run. Restart is the button; there is deliberately no automatic teardown, because "the run finished" and "you are done with the session" are different facts and only one of them is knowable from here.
Cells also get a language server — completion, hover and diagnostics, with names
resolving across cells (see
Notebook cells get the same language server).
This is the one place it matters most: the server resolves its interpreter by walking
up from the file to the nearest .venv, which for a training notebook is the project
venv — so a recipe completes against the transformers/trl/peft it will actually
run on, not against the dashboard's own environment. That is also why the pane resolves
Project.root from GET /api/training/projects/{id} and joins it to the relative
notebook path before attaching: a relative path would point the server somewhere else
entirely, and it would resolve something rather than fail.
Kernel slice of the training ws channel: client sends open, run_cell, run_all,
cells (ops: insert/edit/delete/move), interrupt, restart, shutdown, watch_run;
server sends opened, kernel_status, execution_state, output (raw nbformat output
dicts), cells_changed, run_backfill, and error.
Cells: the keyboard, and markdown
Jupyter's contract, and both notebook panes share it (notebook/CellEditor.tsx,
notebook/advance.ts):
| key | code cell | markdown cell |
|---|---|---|
Ctrl/Cmd+Enter | run, stay put | render, stay put |
Shift+Enter | run, focus the next cell | render, focus the next cell |
Shift+Enter on the last cell | run, append a code cell and land in it | same |
| double-click | — | edit |
Three things this replaced. The two keys were bound to the same callback, so
run-and-advance did not exist and there was no way to walk a notebook without the
mouse. The training pane never rendered markdown — a prose cell was a raw
editor — and its Shift+Enter sent that cell to the kernel, which refused it;
the pane's error banner then read no code cell <id>, which sounds like the
notebook lost the cell you were looking at. And the backend collapsed two failures
into that one sentence, so it now says either no cell <id> in this notebook or
cell <id> is markdown, not code — nothing to run. (cells.run_cell /
notebook.run_cell answer the second case immediately too; a markdown cell never
gets a run state, so the tool used to wait out its full 120s timeout and then
report unknown.)
Advancing into a markdown cell opens it for editing rather than leaving it rendered. A rendered cell has no editor to hold the caret, so the walk would stop there — and the alternative is Jupyter's command mode, which needs a cell-selection model the pane does not have.
Focus requests are tracked by position, not by cell id: a cell appended at the
end carries a temporary id until cells_changed answers with the real one, and
that swap remounts the row.
Visualizations
All four visual panes feed off the same training channel; the notebook (or a script
run) drives them through the horrible_train helper's sentinel events:
-
Metrics (
training.metrics, uPlot) — one streaming chart per metric of the selected run.metrics.pykeeps a per-run ring buffer (training.metrics.bufferPoints, default 5000) so a chart opened mid-run backfills viawatch_run, and coalesces the wire to ≤20 events/s per run while keeping every point server-side.And they persist now. They used to live only in that ring buffer, so they died with the backend process: you could fine-tune for six hours, restart the app, and have nothing but a checkpoint. Meanwhile localtrack — the module whose entire job is keeping exactly this — had one producer (the evals sweep) and four default charts (
train/loss,eval/loss,eval/accuracy,train/learning_rate) that nothing had ever written. Those are Hugging FaceTrainerkey names andht.callback()emits precisely them, so connecting the two fills the default workspace on the first run.Three decisions about where that hook goes:
- In
metrics.record_event, not inhorrible_train. The helper's contract is zero dependencies and a stdout sentinel protocol; giving it an HTTP client breaks the Kaggle and Colab push, which run it on someone else's machine. - Not through
report_to. That is a Hugging Face field with known integration names — an unknown one is a runtime error, not a no-op.ht.callback()is already additive and installed regardless ofreport_to, which is exactly why it is the right carrier.localtrackis deliberately not added torecipes.TRACKERS. - Every point, not the coalesced wire. The wire drops intermediates at 20/s because a chart cannot draw faster; a store has no such excuse, and a loss curve missing 90% of its points is not the curve.
A run's tracked config is its recipe (
recipes.run_config, flat becausecompare_runsdiffs keys): a sweep point declares exactly what it varies, and a run started from the notebook — the common case — used to record{source, projectId}and nothing else, so those were the runs a comparison could say nothing about. Thesweeptag still means "a sweep declared this" and is kept apart from the fallback.The localtrack project is the training project's id, so each project keeps its own run history — deliberately unlike the evals sweep, which files everything under one hardcoded
evalsproject. The mirroring itself islocaltrack.mirror.RunMirror, shared with that sweep so the swallow-everything posture cannot drift between two copies.One honest limit: the helper emits no "finished" event and kernel lifecycle lives in shared
notebook_core, so a run is marked finished when a new run supersedes it for the same project, or at backend shutdown. A run whose process merely died staysrunninguntil the next one starts — visible in the pane, and better than guessing from a timeout. The four panes below are declaredembedded: they are companion strips of the notebook they ride on, reached with their region keys or theTraining: Open …commands (which reveal the strip rather than opening a pane).training.metricsis deliberately not — it is seeded as the Training workspace's own center pane, so it is a destination in a way the others are not. See Embedded views and sections. - In
-
Architecture (
training.modelgraph, dagre SVG) — the layer graph:torch.fxtopology (with tensor shapes when anexample=input is given), or thenamed_modulestree when the model isn't traceable. Withweights=True, per-layer gradient norms color-map the nodes live during training (model_statsevents). It is scoped to one project, taken from the host notebook's params, because two notebooks open at once otherwise overwrite each other's graph.Two things make it fill itself rather than sitting on an empty state:
ht.callback()publishes the architecture it is handed. TheTrainerpassesmodel=to every callback hook and the callback used to drop it, so a generated recipe could run to completion with the pane still reading "callhorrible_train.watch(model)" — naming a call the recipe does not make and, on the trl path, could not make (the trainer is given the model's name, so no model object exists in the notebook namespace before it is built).graph=Trueis the default because the fallback is a walk ofnamed_modules();weights=Falseis, because that one norms every parameter on everylog(). Both are arguments:ht.callback(graph=False),ht.callback(weights=True).The last graph is cached per project (
metrics.graph_backfill) and replayed to a pane that asks (watch_graphon thetrainingchannel, thewatch_runpattern).model_graphis emitted exactly once, at the top oftrainer.train(), and the strip starts closed — so without this the only way to ever see a graph was to have had the strip open at that instant. A run emits metrics continuously, so the chart pane can join late; an architecture does not. -
Rollout (
training.rollout) — the latestframeevent as an<img>, rAF-throttled with an fps readout; gym envs stream viahorrible_train.frame(env.render()). -
Manim (
training.manim) — plays the mp4 a render produces (manim_statusprogress lines +manim_done {url}), served by the media route.
Runners
Two subprocess runners live outside the kernel (both Popen-on-thread, sentinel-parsed, process-global so they survive pane/tab closes):
-
Script runner (
runners/script_runner.py) —python -u <script>in the project venv;ht.log()/ht.frame()in the script are indistinguishable from notebook cells. Driven byPOST/GET/DELETE /projects/{id}/runsand byPOST /projects/{id}/recipe/run.Every run also writes its output to
runs/logs/<script>-<runId>.logunder the project. Until it did, that output existed only asrun_outputbroadcasts: whoever had a pane open saw it and nobody else ever could — so a headless run, which is the only kind an agent can start, was observable asstate: runningand then, at best,returncode: 1. A crash was a number. And because the registry lives in memory, a backend restart (a--reloadon any file underbackend/, tests included) erased even that: the run list came back empty, which is indistinguishable from a run that finished cleanly.GET /projects/{id}/runs/logsreads the directory, so it answers for runs this backend has no memory of, andGET /runs?tail=Ncarries the last N lines inline. The agent tool istraining.run_output. -
Manim runner (
runners/manim_runner.py) —python -m manim renderagainst a scene file undermedia/scenes/; auto-installs manim into the venv on first use; locates the newest mp4 and serves it via the traversal-guarded media route.
Cloud push
A PushTarget driver layer (push/, same shape as the provider layer) sends a
project's main.ipynb to cloud compute:
- Kaggle kernels (
push/kaggle_push.py) — generateskernel-metadata.json(competition/dataset sources drawn from the project's Kaggle refs,enable_gpu), thenkernels_push;statuspollskernels_status. Competition submissions stay manual (kaggle competitions submit …in the terminal pane). - Google Colab (
push/colab_push.py+google_auth.py) — Colab has no API, so the notebook uploads to Drive with the Colab mimetype; the file id is remembered (.push-colab.json) so re-pushes update in place. OAuth uses the user's own client (settingstraining.google.clientId/clientSecret), manual-code flow, refresh token in.data/training_google_token.json— never returned to the client.
Both pushers are also used by notebook publishing, through
keyword-only options whose defaults are the training behaviour, so the training routes
call them unchanged: KagglePush.push(..., private=, enable_gpu=) (a publish can be
public and does not spend GPU quota) and ColabPush.push(..., anyone_with_link=), where
None leaves a file's Drive sharing alone. kaggle_push.has_credentials() answers
"could a push authenticate" without importing the kaggle package, and
ColabPush.delete removes the pushed Drive file and forgets it.
Learn: the notebook explains itself
The notebook's right strip opens on Learn (training.learn, LearnPane.tsx),
because the point of doing AI research in this app rather than in a bare notebook
is that it teaches while you work. The other strips (Metrics, Architecture, Manim,
Rollout) show what is happening; Learn says what it means. Three parts, each
grounded in something real rather than generated:
- This cell. The cell you are in (
learn/focus.ts: the notebook reports focus perprojectId:notebook, the same key asopenSession) is matched by identifier against two sources. Knobs —learning_rate,lora_r,max_lengthand their aliases — come fromGET /api/training/learn/glossary, which isrecipes.glossary(): the recipe form's ownhelptext, one entry per knob across every backend. It is served, never copied into TypeScript, so the strip and the form cannot explain a knob two ways. Concepts — the classes and calls a knob is passed to (SFTTrainer,get_peft_model,loss.backward(), mixed precision, cross-entropy) — have no form row, so they are written inlearn/terms.ts, each saying why it matters rather than restating the API. Matching uses word boundaries:TrainerinsideSFTTraineris not the generic training-loop concept. - This run.
learn/diagnose.tsreads the live curves (the samemetricsevents the Metrics strip draws) with plain rules: waiting for a curve, loss falling, loss flat, a spike against the recent median, overfitting (eval loss rising three evaluations running while train loss falls), an exploded gradient norm, and NaN — which stops every other read, since nothing after it means anything. Every diagnosis states the rule that fired, with its numbers, so the strip teaches how to read a loss curve instead of issuing verdicts. - Go deeper. Reveal Curves or Architecture, open the Model explorer, or Ask
the agent — which opens the chat with an explanation request for this cell
drafted, not sent (
draftInChatin the agent module), so the user reads and edits the question before a turn is spent on it.
Peer fabric (removed)
A training.peers strip once advertised a node as "offering GPU / seeking help"
over the peer fabric (fabric.py, specs.py, /training/fabric/*,
training.fabric.* settings). It was advertising with no remote execution behind
it, its buttons did not reflect the saved mode after a reload, and a disconnected
peer's ad lingered until restart — so it was removed rather than fixed. Collaboration
on training still rides the ordinary paths: shared panes and
agent.ask_peer.
Agent integration (the flagship)
Typing "i want to work on kaggle's pokemon tcg competition" into the ask bar drives the whole flow. Two tool groups make this work, both progressively disclosed (loaded on demand or by keyword) so they don't bloat the model's tool budget:
-
training— backend tools (agent_tools.py, registered into the backend-sdk registry viaregister_agent_tools(), called fromapp.py). Server-resolved, so they run with no tab attached:training.search_environments,resolve_environment,list_projects,project_status(read-only);create_project,fetch_data,install_deps,start_run,stop_run,push,render_manim(side-effecting, gated by the agent permission engine with aspecifierTemplatelike{provider}:{ref}).Plus the two that close the loop:
training.list_checkpoints(read-only — checkpoint names are not guessable, so this is the call before the next one) andtraining.convert, which turns a checkpoint into a GGUF in llama.cpp's managed directory.convertdrains the conversion's NDJSON stream and reports only the outcome: the route streams it so a pane can show a progress log, but an agent has nowhere to put hundreds of progress lines and would pay for every one in context. It takes minutes and is deliberately not backgroundable the waystart_runis — there is no id to poll afterwards, the answer is the output path. -
training— frontend tools (agentToolson thetraining.notebookpanel). Full cell CRUD + execute against the pane's session store:cells.list_cells,read_cell,kernel_status(read-only);insert_cell,edit_cell,delete_cell,run_cell(waits ≤120s, returns outputs/traceback),run_all,interrupt,restart.These were
notebook.*until they collided with the reactive notebook, which declares the same seven verbs against a different store with a different session argument (path, notprojectId). The frontend resolves a tool call by name withfindand this module registers first, so these won and the reactive notebook's were unreachable — silently, decided by registration order inmain.tsx. They are grouped by name prefix, so moving them totraining.*both ends the collision and files them with the rest of this module. AduplicateToolNamescheck in the manifest now logs any recurrence, and stored permission rules naming the old tools are migrated on load (permission_store.rename_in_rule) — anotebook.run_cellgrant was a grant over this notebook, and must follow the tool rather than the name.
The orchestrator changes are generic (backend/modules/agent/orchestrator.py): AgentTool
gained an optional group field (backend/sdk/types.py) — None keeps a plugin tool in
the always-loaded core (unchanged), a set group moves it into the progressive-disclosure
pool. _GROUP_KEYWORDS preloads training on "kaggle/train/dataset/gym/…" and notebook
on "notebook/cell/kernel/…". Now that the cell verbs are training.*, this flow needs
only the one group — but a prompt naming both notebooks preloads both, and with the cell
tools counted in training that is over TOOL_BUDGET; see the note below.
open_pane now accepts optional params, so the agent opens the notebook pane bound to a
project (open_pane("training.notebook", {params: {projectId, notebook}})).
Flow: keyword-preload training → search_environments → create_project (approval
gate, specifier provider:ref) → fetch_data → open_pane(training.notebook, params) →
show("Training Metrics") → cells.insert_cell/run_cell to build the
starter. Verified end to end against a live Ollama model: the ask reaches
training.create_project, the gate fires with the right specifier, and on approval the
project + scaffolded notebook land on disk.
Resolved: training was split
The group had grown to 34 tools — and the count was easy to miss, because only 14 of
them are declared in this module's agent_tools.py. The other twenty were the ten
recipe/sweep verbs in recipe_tools.py and the ten notebook cell verbs registered by the
frontend (packages/core/src/modules/training/agentTools.ts). A tool's group is its
name prefix (_group_of) and does not care which side of the wire declared it, so all
thirty-four landed in one group.
With a ~9-tool core that left no room for evals (10) beside it: the loop this module's
own agent prompt prescribes — start a run, read its metrics, score the result — did not
fit under TOOL_BUDGET, and the overflow was dropped. Loudly in the log, invisibly
everywhere else.
The split is by job, since a group is the unit load_tools hands out:
| group | tools | what |
|---|---|---|
training | 14 | projects, venvs, fetch, runs, checkpoints, convert, push |
recipe | 7 | author a recipe, install the stack, run and stop a sweep |
cells | 10 | read/edit/run the open training notebook's cells |
wandb | 3 | import someone else's W&B runs into localtrack |
cells, not notebook: that name belongs to the reactive notebook module, and these two
collided under it once already. Both old spellings (notebook.* and training.*) are
carried in permission_store._RULE_RENAMES, listed rather than chained — rename_in_rule
does one lookup, so a rule left to reach cells.run_cell by way of training.run_cell
would stop at a name that no longer exists, and a grant pointing at nothing fails closed
and re-asks the user for a decision they already made.
Every phase of the loop now fits over the core at the 38 floor — pick data (25), run a job
(30), edit the notebook (33), convert and serve (30), judge it (26) — and the whole thing
at once is 73, which a hosted model carries in one round. test_trainer_tool_budget.py
asserts this by counting the groups, frontend tools included, rather than trusting a
number written in a comment; that is how the group reached 34 unnoticed.
Backend surface
/api/training (backend/modules/training/routes.py):
| Route | Purpose |
|---|---|
GET /providers, GET /providers/{p}/search?q=&kind=, POST /providers/{p}/resolve | provider catalog + discovery |
GET/POST /projects, GET/DELETE /projects/{id} | project store (create resolves the ref, scaffolds main.ipynb, kicks off the venv bootstrap) |
POST /projects/{id}/fetch (202) | dataset download on a worker thread |
POST /projects/{id}/deps (202) | uv pip install extra packages into the venv |
GET/PUT /projects/{id}/notebook?path= | whole-document notebook read/replace (path-traversal-guarded) |
POST/GET/DELETE /projects/{id}/runs, GET/DELETE /runs/{id} | script runs (start/list/stop) |
POST /projects/{id}/manim (202), GET /projects/{id}/media/{path} | manim render + traversal-guarded media serving |
GET/PUT /projects/{id}/recipe, POST /projects/{id}/recipe/apply | the recipe form's schema + validation, and writing its cells into main.ipynb |
GET /recipe/docs | doc links per config class from the versioned index |
GET /projects/{id}/checkpoints, POST /projects/{id}/convert | checkpoints to convert, and the conversion itself (NDJSON progress) |
GET /push/targets, POST/GET /projects/{id}/push/{target}[/status] | Kaggle/Colab push + status |
POST /google/auth/start, POST /google/auth/complete, GET/DELETE /google/... | Colab Drive OAuth (server-held token) |
GET /fabric/ads, POST /fabric/advertise | peer training ads (list / set advertise mode) |
Long-running work reports over the shared /ws socket on the training channel
(every connection is subscribed on connect): env_progress / fetch_progress
{projectId, line, pct?}, project_changed (a full ProjectModel), the kernel-protocol
events, plus metrics / model_graph / model_stats / frame / run_output /
run_state / manim_status / manim_done.
Opening a training pane without a project
training.notebook and training.recipe are params-bound ({projectId, notebook}),
but most ways of opening them — the Start menu, the palette, a restored layout —
carry no params. There is exactly one project someone means when they open the
training notebook without saying which, and it is the one they were in last, so
last-project.ts records it (localStorage, following layout/recents.ts — a
per-viewer convenience, and deliberately not a SettingDecl, which would put "Last
project" on the settings page as something to configure). Both panes fall back to it
when opened with no params, resolved once per mount so the remembered id changing
under an open pane cannot swap the notebook out from under you.
A remembered project that no longer exists forgets itself and shows the picker, rather than the "no longer exists" dead-pane message. That message is right for a pane the user deliberately opened against that project and wrong for a fallback: nobody asked for this project, the value is not one they can see or clear, and without the self-heal the pane would greet them with the same dead project on every open, forever.
A params-bound pane opened cold — from the start menu or the palette — renders the projects picker itself rather than a sentence. It used to say
"open me from the Training projects pane", which named something the reader could
not open: consolidation made training.projects embedded, i.e. a section of
Explorer. Picking a project calls openTrainingNotebook, whose
canReuse retargets that same instance, so the pane you opened becomes the
notebook instead of leaving an empty one behind. training.recipe and the notebook
module's notebook.editor do the same with their own pickers.
The fine-tuning agent
trainer is the agent for this loop — pick it in the chat. The roster spec permits
training, evals, localtrack, editor and files, and preloads only training, so reading a recipe costs a
load_tools rather than schema space on every turn.
llamacpp now appears in that list, and used not to. The original reasoning was
right: a group is a tool name's prefix (_group_of), the llamacpp.* namespace held
only settings keys (llamacpp.diskBudgetGb), and listing it would have silently
permitted nothing at all — the same trap the connectors rule names when it says a
connector id must equal its tool prefix.
That stopped being true the moment llama.cpp grew real agent tools.
Without the group this agent could convert a checkpoint and then had no way to serve
it, which is the step immediately before the eval its own prompt tells it to run. It
is permitted but not preloaded — this flow already sits close to TOOL_BUDGET.
hardware still does not appear, for the original reason: settings keys only.
Contributions to the Layout Shell
| Contribution | Value |
|---|---|
Panel training.projects | singleton hub, embedded: the Projects section of Explorer and the left region strip of training.notebook. Provider search → create project, project list with live venv/data status; a row opens its notebook and carries the rest in a training.project context menu |
Panel training.recipe | non-singleton fine-tuning recipe form, params {projectId}, opened from a project row (openTrainingRecipe); typed schema + validation chips, tracker picker, checkpoint→GGUF card |
Panel training.notebook ("Training Notebook") | non-singleton notebook pane (center), params {projectId, notebook}, labelled with the notebook's file name via a title param, titled apart from the notebook module's own "Notebook Editor", opened via openTrainingNotebook() so it focuses/reuses rather than splitting (see windowing); CodeMirror cells, run/interrupt/restart toolbar, nbformat output renderers (stream+ANSI, results, images, HTML, tracebacks) |
Widget training.metrics | live metric charts (uPlot, one per metric of the selected run; ring-buffer backfill via watch_run, ≤20 wire events/s per run with every point kept server-side) |
Widget training.modelgraph | model architecture graph (dagre SVG; fx/module-tree, grad-norm heatmap) |
Widget training.rollout | live gym/render frame stream with fps readout |
Widget training.manim | manim render viewer (progress log + mp4 player) |
Widget training.learn | the teaching strip: this cell's knobs and concepts, rule-based reads of the run, go-deeper links — see Learn |
Regions on training.notebook | learn (open by default)/metrics/architecture/manim/rollout on the right strip, projects on the left |
| Commands | training.open, training.openRecipe, training.openMetrics, training.openModelGraph, training.openRollout, training.openManim, training.openLearn |
| Recipe pane hand-off | after a conversion, Serve this starts a local llama-server on the new GGUF and Score it opens Evals on its Run section — those panes used to sit side by side with nothing connecting them |
| Project row actions | open notebook, 🧪 recipe, ⇪ push to Kaggle, ⇪ push to Colab, delete |
Connector trackers | W&B API key + MLflow tracking URI, Fernet-encrypted. The one connector contributing no agent tools, so its id names no tool group |
| Settings | training.projectsRoot, training.defaultPython, training.kaggle.username, training.kaggle.key, training.hf.token, training.google.clientId, training.google.clientSecret, training.metrics.bufferPoints, training.manim.quality |
Credentials note: settings persist in the plaintext .data/settings.json store — same
v1 posture as database connection credentials (encryption is a tracked follow-up).
Browser vs desktop
| Capability | Browser | Desktop |
|---|---|---|
| Project/provider management | ✅ | ✅ |
| Venv + kernel subprocesses | ✅ (backend-side) | ✅ (backend-side) |
Everything runs on the backend node; both layouts are equivalent.