Skip to main content

Module: training

Notebook-driven neural-network training inside the dashboard: pick an environment (a Kaggle competition, a Hugging Face dataset, a Gymnasium env — or any plugin-provided source), get a project with its own uv-managed venv and a scaffolded .ipynb notebook, then build and train models with live visual feedback (metrics charts, model architecture graph, manim explainers, gym rollout frames). The agent drives the whole flow: "i want to work on kaggle's pokemon tcg competition" resolves the competition, creates the project, downloads the data, and opens the right panes.

Status: implemented — frontend in packages/core/src/modules/training/, backend in backend/modules/training/. Projects + per-project venvs + the environment provider layer, the native notebook pane on the Jupyter kernel protocol, the live metrics pipeline + chart pane, the architecture graph pane, the manim + gym-rollout runners, Kaggle-kernels/Colab push, the peer-fabric training ads, and the agent tool integration that drives the whole flow from a natural-language ask.

Projects on disk​

Each project is one directory under training.projectsRoot (default ~/horrible/training); the directory is the record:

<projectsRoot>/<slug>/
main.ipynb # scaffolded notebook (nbformat v4, Kaggle-compatible)
project.json # ProjectModel dump (id, name, refs, python, flags)
data/ # provider-fetched datasets
media/ # manim renders, saved frames
.venv/ # uv-managed project venv

Torch-sized dependencies live in the project venv, never the backend env. The venv is created with uv venv --python <training.defaultPython> and bootstrapped with ipykernel and ipywidgets (envs.KERNEL_PACKAGES) plus the local horrible-train helper package, then the provider's scaffold requirements (pandas, gymnasium[box2d], …). ipywidgets is there because the libraries a training notebook runs look for it: tqdm and the Hugging Face Trainer draw a widget progress bar when it is importable and otherwise print one frame per update — a worse bar, plus a warning telling the user to upgrade Jupyter, which was never the problem. The kernel packages are installed by install_stack as well, because a venv is bootstrapped once and never revisited, so a project made before that list changed would otherwise never catch up. All subprocess work (uv, kernels, renderers) uses blocking Popen pumped on daemon threads — the Windows-safe pattern shared with the LSP/terminal modules — with output lines streamed to the UI.

A directory whose project.json is missing or unreadable (e.g. a partial dir left by an aborted bootstrap — just a .venv) is treated as not a project: it's skipped by listing/lookup, creating the same env adopts the slug (writes project.json, reuses the existing .venv) instead of stranding it behind a -2 suffix, and delete removes such partial dirs (which get_project can't resolve). If a persisted notebook pane references a project that no longer exists, the kernel-open error carries an unknown_project code and the pane shows a "no longer exists" state with a Close button that drops the dead pane from the saved layout — so it self-heals on the next load.

Projects another module owns​

ProjectModel.owner names the module that keeps a project as working storage. The only one today is evals, which builds one per suite to run Hugging Face benchmarks in. Those are created straight through create_project rather than through the create route, so — and this is the part that bit — they have no scaffolded main.ipynb, and their venv carries only the benchmark's requirements: no ipykernel, no horrible_train.

They are listed and marked, not hidden (the bundled-suite rule): a project missing from the pane is a directory eating disk you can neither see nor delete. What they do not get is authoring actions — open notebook, recipe, and both pushes are disabled, each carrying the reason as its detail, because every one of them could only fail. Disabled-with-a-reason rather than absent: the reason is the whole value, and hiding them would make an inert project look identical to a healthy one. Delete stays enabled — it is exactly the one you might want the disk back from.

The project row is a target, not a toolbar​

A project used to render as one flex line holding its name, the owner badge, a venv/data chip and five buttons (Open notebook, 🧪 Recipe, ⇪ Kaggle, ⇪ Colab, ✕) — in a 280px dock or a region strip beside the notebook. Nothing read as primary, a push to somebody else's servers sat at the same weight as opening the notebook, and at that width each button was a couple of characters wide.

Now the row is two lines (name + status; refs and the live progress note beneath) and the row itself opens the notebook. Everything else is a context menu — a declared contextMenu provider for the target kind training.project, the same extension point file explorer and library use — reached by right-click or by the row's ⋯ button, so the actions are discoverable without knowing to right-click. Kaggle and Colab became a Push a copy to ▸ submenu: they are one verb with two destinations, and as sibling buttons they read as two unrelated features.

Moving the actions out of the component took their progress lines with them — a provider declared in the module manifest is a plain function with no component state — so the pushing… / pushed → url / push failed line now lives in a small module-level note store (training/activity.ts), which the pane subscribes to and renders in the same slot as the env_progress / fetch_progress events.

Deleting a project also broadcasts project_changed now (with deleted: true). Every other change to a project did; a delete did not, so the only list that updated was the one belonging to whoever clicked, and a project removed through the agent or from a second window stayed on screen.

Two failures this closed, both of the silent kind:

  • A missing notebook read as a path. The open reached a bare FileNotFoundError, whose str() is only the path, and the pane renders that verbatim — so the whole explanation was a lone C:\…\main.ipynb. It is now a MissingNotebookError carrying a sentence, and it deliberately does not use the unknown_project code: the project is fine, so a pane that self-heals by closing itself would be throwing away the wrong thing.
  • The kernel badge contradicted the error. kernel starts at starting and only opened or a kernel_status event ever moved it, so a failed open left "● starting" sitting next to the message explaining why it hadn't. An error arriving before opened now marks the kernel dead — and only then, since the same event also carries recoverable errors on a live session (a rejected cells op), where calling the kernel dead would be its own lie.

That "unreadable means not a project" rule has a sharp edge: background workers rewrite project.json (_mark(project, venv_ready=True)) while requests read it, and a write_text truncates before it writes. A read landing in that window saw an empty file and the route answered 404 for a project that exists — a bug that only appeared under load and looked like anything but itself. Both sides now go through backend/atomic_write.py: writes replace atomically, and reads distinguish "no such record" from "someone is replacing it right now", which on Windows is a transient PermissionError rather than a missing file. Pinned by a concurrent read/write test in test_training_projects.py.

Background work started by a route (fetch, venv bootstrap, dep install) runs on tracked daemon threads: routes._spawn registers each one and routes.join_workers waits for them. Nothing in the product blocks on it — it exists so tests can wait on the work itself rather than on a wall clock, which is what a fixed-deadline poll silently degrades into on a loaded machine.

Environment providers​

A provider is a thin synchronous adapter behind one uniform contract (backend/modules/training/providers/base.py), mirroring the database module's driver layer: search, resolve, fetch (into data/, with progress), and scaffold (starter notebook cells + venv requirements). Client libraries import lazily inside methods; short calls run via asyncio.to_thread, fetches on daemon threads.

ProviderKindsFetchScaffold
kagglecompetition, datasetdownload + unzip into data/ (creds: training.kaggle.* settings → ~/.kaggle/kaggle.json)pandas data-loading starter
huggingfacedatasetlazy — the notebook calls datasets.load_dataset(id); snapshot into data/ on requestload_dataset starter
gymnasiumenvnothing to fetch (bundled static catalog; no gym import in the backend env)RL rollout loop streaming frames + episode rewards

Plugins add providers: a backend plugin calls host.add_training_provider(provider) (see backend plugin SDK); get_provider() falls back to the plugin registry, so a new environment type is one file or one plugin call.

The horrible-train helper​

backend/modules/training/helper/ is a zero-dependency pip package installed into every project venv. Training code (notebook cells or scripts) calls:

import horrible_train as ht
ht.run("baseline") # name the run (auto-created on first log)
ht.log(step=i, loss=0.42) # → live metrics chart pane
ht.frame(env.render()) # → live rollout pane
ht.watch(model, example=x) # → architecture graph pane (torch.fx, tree fallback)

Each call prints one @@HORRIBLE@@{json} sentinel line to stdout; the backend strips these from visible output and fans them out as training channel events — the same line-protocol trick the visualizer uses for Pygame frames, so it works identically under a kernel, a script runner, or manim.

The recipe surface​

A fine-tune is a couple of dozen numbers, and until this landed the only place to put them was a notebook cell written from memory. recipes.py makes them a schema: a curated catalog of fields, each with a type, a default and a one-line description, rendered as a form by training.recipe (non-singleton, params {projectId}) and materialized into main.ipynb.

There is no new execution path. materialize() returns notebook cells. They run on the same kernel in the same venv, their ht.log() output is indistinguishable from hand-written code's, and the same notebook is what the Kaggle and Colab push already send. The recipe itself lives in recipe.json so the form round-trips; the cells are yours the moment they land and are never read back. A project with no recipe.json yet starts on its own dataset: the Hugging Face dataset it was created from fills the dataset field, rather than asking again for the one thing the project already knew.

The typed dataset ref is looked at too​

Shape detection (datasets) used to run only through the dataset picker — which covers registered datasets, while the default path puts the id in the free-text field. So the most ordinary flow there is — create a project from a Hub dataset, open its recipe, write cells — reached the generated code with nothing having ever looked at the data: columnMap empty, textField still text, no reshape emitted. On a question/solution reasoning set that is KeyError: 'text' inside trl, minutes in, after the base model has downloaded.

It also means dataset_id is checked rather than trusted. It names a dataset in the datasets registry, not a Hub ref — but recipe.set offers a model two parameters that both look like "the dataset", so a Hub id lands in the wrong one. A dangling id is worse than none: resolve_dataset marks the dataset missing, and detection used to skip a recipe that had one (a registered set carries its own map, so nothing re-derives it). One wrong field became a run that died on KeyError: 'text'. Now recipe.set rejects an id that resolves to nothing, naming the difference, and only a resolvable id suppresses detection.

recipes.ensure_shape(recipe) peeks five real rows of a typed ref (cached, 5-minute TTL, best-effort — an unreachable Hub leaves the recipe untouched, because being offline is not evidence about a dataset's shape), and runs at both seams that matter: rendering the form, and writing the cells. The second is not redundant — recipe.apply and recipe.run are agent tools that may be the first thing ever to touch a recipe. It returns the verdict either way, so GET .../recipe carries a datasetShape the pane renders under the dataset field, and adopts it only into a blank column map: a hand correction is the whole reason the map is editable and must never be re-guessed over. A registered id is not second-guessed at all. A regenerate replaces the previous block (found by a horrible_recipe cell-metadata flag, with the header line as a fallback) rather than appending — two SFTConfig definitions in one notebook is a bug where the second silently wins, discovered after a wasted run.

Frameworks: the recipe-backend layer​

The recipe began as one framework and one task, hardcoded end to end: the field catalog named SFTConfig, the probe imported trl, the emitter wrote SFTTrainer. That was the right first shape and it does not extend — DPO is not a variant of SFTConfig, unsloth loads a model differently, and pre-training is not a notebook cell at all.

backends/base.py is the seam. A RecipeBackend declares which tasks it can train, which knobs each takes, and what code those knobs become; everything around it — the form, the persistence, the notebook write, the sweep — is written once against that protocol. Four ship:

backendtasks
trlsft, cpt, dpo, kto, reward, grpo
unslothsft, dpo — 4-bit QLoRA on fused kernels, NVIDIA only
torchtitanpretrain
nanotronpretrain

A plugin adds more with host.add_recipe_backend(backend).

Unsloth reuses trl's field catalog rather than copying it — it ends in the same SFTTrainer, and a second copy would be a second place to fix the warmup_ratio/warmup_steps unit trap. What it adds is the loader, and three things there are each a real bug if missed: the model is passed as an object (passing the name loads a second unoptimised copy), LoRA comes from get_peft_model and not a peft_config= argument (passing both applies the adapter twice), and max_seq_length is set at load time because RoPE scaling is chosen there.

Pre-training is a different execution model and says so. A titan run is torchrun --nproc_per_node=N -m torchtitan.train --job.config_file config.toml — a multi-process launch reading a config file, with no object whose constructor takes a learning rate. Those backends emit a config file plus a launch cell and run through the script runner, single-node in v1. This node already advertises GPUs over the peer fabric with no remote execution engine behind it; shipping --nnodes 1 and naming the gap is better than a launcher that appears to support a cluster.

check(task, profile) reads the hardware probe and honours its three states — found a card, looked and found none, and could not ask. Saying "no NVIDIA GPU" when nvidia-smi merely was not on PATH is the failure that module exists to prevent.

Installing the stack​

envs.install_stack is the step that never existed. bootstrap installs ipykernel and the horrible-train helper and nothing else, which is why the recipe form spent its whole life reporting "trl and peft are not installed", the generated code was emitted unvalidated, and the LSP had no framework to index. Autocomplete coverage was never a completion-engine problem; it was an empty venv.

The torch wheel is chosen from the hardware probe by card and OS together, the same rule hardware._variant_for follows for llama.cpp builds — and the reason is returned to the pane, because a CPU build landing on a machine whose owner knows they have a card needs an explanation rather than a shrug. NVIDIA on Windows needs the CUDA index (notebook.env.PYTORCH_CUDA_INDEX, shared so the two venvs cannot disagree): PyPI's Windows torch wheels are CPU-only. This used to claim the default wheel "already targets" the card, which put +cpu on an RTX 4080 and only surfaced three cells later as trl's "Your setup doesn't support bf16/gpu". When the probe could not ask, a GPU-capable wheel is installed rather than the CPU one — on Windows that again means the CUDA index.

A venv that already has a CPU build is replaced, not kept: uv pip install torch counts any installed torch as satisfying the request, so without --reinstall-package re-running the install reports success and changes nothing. The build is read from torch/version.py, because the wheel's METADATA and dist-info name both say plain 2.14.0 for 2.14.0+cpu.

Validated against what is actually installed​

The expensive failure this exists to prevent: trl renamed SFTConfig.max_seq_length to max_length, so a recipe emitting the wrong one dies with a TypeError several minutes in, after the dataset has downloaded. So fields carry aliases, and introspect() asks the project venv — one subprocess, cached on the venv's mtime — which names its installed classes actually accept, via dataclasses.fields() on the real SFTConfig/LoraConfig.

Four states, and the last two are not the same thing:

StatusMeaningEmitted?
okthe installed class accepts this nameyes
renamedit accepts an alias insteadyes, under the alias, with a comment
unsupportedwe asked, and it accepts neitherno — dropped, and named as dropped in the header
unvalidatedwe never got to ask (no venv, no trl)yes, with a banner saying it wasn't checked

Two fields make the rule concrete. max_length/max_seq_length are aliases — same meaning, different spellings across versions. warmup_ratio and warmup_steps are deliberately not, even though transformers 5 dropped the first and kept the second: they carry different units, so aliasing them would send 0.03 — three percent of training — into a field meaning "three hundredths of a step", which is no warmup at all and raises nothing anywhere. Whichever one the installed version accepts is the one that renders.

The form admits it is not the whole API. TrainingArguments has 112 fields on transformers 5.15; this catalog has 24. Introspection.extra counts the difference and the header says "24 of 118 knobs — edit the generated cell for the rest", because a form that looks complete is a form you trust to be complete.

Parameter docs come from the node's own versioned doc index (the trl and peft crawler seeds): GET /training/recipe/docs returns a link per config class carrying the indexed version and the installedMismatch annotation, so the link is to docs for a version you can compare against rather than to whatever is current. An empty index yields no links rather than a guessed URL.

The base model is searched and checked, not just typed​

basemodels.py is the one field whose value is copied verbatim into from_pretrained(...), so every way of being wrong about it used to surface minutes into a run, from inside transformers, after the dataset had downloaded.

  • GET /training/models/search lists Hugging Face text-generation models, most downloaded first, and the pane's picker fills the field from it — the id is copied rather than remembered. A …-GGUF repo is cautioned, not hidden: it is a real model, just the served copy, and a row missing with no explanation is worse.
  • basemodels.check judges whatever is in the box and lands in the form's warnings. It is advisory and never blocks: a recipe that refused to generate because the Hub was unreachable would be worse than a cell that fails honestly.

Four judgements it makes, each a failure seen for real:

  • qwen3:0.6b is an Ollama tag, not a Hub id. It is the name this app shows everywhere else, so it is the obvious thing to type — and it reached from_pretrained unchanged. The warning names the search term to look for.
  • A bare name is not malformed. gpt2 and bert-base-uncased are canonical Hub ids with no owner, so demanding owner/name would reject ids that work. The Hub decides, not a regex.
  • "Not found" is not a licence problem. The Hub answers a missing repo and a private one the same way, and its own 404 text mentions "private or gated" — so matching on the word gated told every typo to go and accept a licence. The not-found branch is checked first and worded to cover both.
  • A GGUF repo, an encoder, or a repo with no loadable weights gets the reason rather than a stack trace: those are serving artifacts and non-causal models, and SFTTrainer cannot start from any of them.

Offline, every one of these yields no warning at all. Being unable to ask is not evidence that a model does not exist.

Holding a split out​

evalSplit names the split scored during training (eval_strategy /eval_steps / per_device_eval_batch_size), and empty means no evaluation — the honest default, because most Hub datasets have no held-out split and inventing one fails at load_dataset after the download. Without it a run reports only training loss, which falls whether the model is learning or memorising: the one number that cannot tell those apart.

Two details:

  • The held-out split is loaded through the same dataset_call as the training data — same ref, same loader, same local-file handling — so an eval set cannot be read differently from the data it is held out of.
  • An eval_strategy with no split is overridden to no, not emitted: the trainer rejects it on construction, which is exactly the "option that emits a config the trainer refuses" this catalog exists to avoid. The generated cell says # no eval split is set rather than carrying a setting that does not describe the run.

eval_strategy and evaluation_strategy are aliases (renamed in transformers 4.46) — unlike warmup_ratio/warmup_steps, which carry different units — so the probe emits whichever spelling the installed version accepts.

Two knobs a first fine-tune needs​

max_steps and dataset_num_proc, both absent and both felt immediately:

  • max_steps stops a run after N optimizer steps whatever the epochs say (-1 runs the epochs). Without it, capping a trial run meant expressing "fifty steps" as a fraction of an epoch and doing the arithmetic against the dataset's length.
  • dataset_num_proc is the tokenization pass before training starts — single-core by default, which is two minutes of a 15,000-row chat dataset before a step runs.

Both validate against the installed class like every other field, so a version that drops one drops it from the form rather than emitting a TypeError.

report_to is first-class, and local metrics stay authoritative​

The tracker picker (none / tensorboard / wandb / mlflow / trackio) sets SFTConfig.report_to. It is genuinely additive: every generated recipe installs ht.callback() — a TrainerCallback in the helper package that mirrors each logged scalar into the metrics pane — so the chart works offline, works with report_to=["none"], and keeps working when a tracker's key expires mid-run. "None" means no third party, never no metrics.

Credentials live in the trackers connector (Fernet-encrypted, W&B key and MLflow tracking URI, both secret because a tracking URI routinely embeds one), never in settings — GET /api/settings hands the whole bag to the browser. This is the one connector whose id names no agent-tool group, because "log this run to W&B" is a property of a recipe you wrote, not something the agent should do on your behalf.

They reach training code through the environment at spawn: SessionConfig.env (new, on notebook_core) is merged over the backend's own by the kernel manager, and the script runner does the same. That is what keeps the key out of the notebook, out of project.json, and out of every HTTP response. env_for() is scoped to the trackers the recipe selected, so connecting the tile does not by itself start shipping every run somewhere.

Ablation sweeps​

sweeps.py runs the same recipe N times with one thing changed. Before it, "what does learning rate actually do here" meant editing a number, running, screenshotting the chart, editing it back, and remembering — which is not an experiment, it is a recollection.

A sweep is the recipe plus axes: a field and the values to try, expanded as a grid (every combination), a zip (axes step together, for knobs that must move in lockstep like rank and alpha), or a seeded random sample.

Four things do the work:

  • Points are processes, not kernel cells. A kernel is single and serial. Each point becomes a .py from recipes.materialize_script — a flattening of the same materialize() the notebook uses, so a sweep and a manual run cannot disagree about what a recipe means — and runs through the script runner.
  • Each run's config_json is its resolved values dict, which is the entire mechanism behind localtrack.compare_runs. See localtrack.
  • One at a time by default. Two fine-tunes on one GPU is an out-of-memory error at an unpredictable point in the second, not double the throughput.
  • Refusals are explained. An axis naming a field the selected task does not have produces N identical runs and a comparison showing no effect — which reads as "this knob does nothing". validate() refuses it and says which fields exist.

MAX_POINTS is 64: a four-axis grid of five values each is 625 runs, and the cap turns "I did not think about that" into an error rather than a machine that is busy until Thursday.

Runs have an end now​

A sweep is the first thing in this module that needed a run to finish. Everything before it watched one run at a time, so "still running" was indistinguishable from "finished and nothing else started" — metrics._mirror_for closed the previous run only when a new one began, which for the last run of a session is never. Twelve points made that visible immediately.

ht.finish(status) is the helper's terminal signal, emitted from ht.callback()'s on_train_end, and the script runner closes any run its process opened when the process exits (finished / failed on a stop / crashed otherwise). Both call metrics.finish_run, which is idempotent — the common case is two calls for one run, and the second must not overwrite the real status with the process's.

Closing the loop: checkpoint → GGUF​

convert.py turns a finished checkpoint into a GGUF in the llama.cpp module's managed model directory — the one its catalog scans, its disk budget counts and its delete route may touch — so a model you trained is one this node serves. See llamacpp.

  • The converter is not in the release binaries. convert_hf_to_gguf.py lives in the llama.cpp repo, so it is fetched on demand at the tag of the installed build (a converter from a different release can write a file the server reads slightly wrong), cached under the data dir with its sha256 recorded as verified: false — GitHub publishes no digest for a source archive.
  • And it is not one file. The script imports a sibling conversion/ package (one module per architecture) and puts the repo's gguf-py ahead of the PyPI gguf, so the tag's source archive is fetched and exactly those parts — both converter scripts, conversion/, gguf-py/ — are extracted into a staging directory that is renamed into place with a converter.json marker only once it is complete. Fetching the lone script, as this once did, failed every conversion with No module named 'conversion'; a cached directory without the marker (that old layout) is replaced.
  • A LoRA checkpoint is not a model. Told apart by adapter_config.json and sent to convert_lora_to_gguf.py with the base model read from that file; the result is labelled an adapter, since llama-server loads it with --lora beside a base model rather than serving it alone. The base goes to --base-model-id when it is a Hub id and --base only when it is a local directory — --base is a path, so the Hub id every adapter records made the converter look for a directory called Qwen/Qwen3-0.6B and exit.
  • So a LoRA recipe also saves a merged model. Nothing on this node serves an adapter beside its base, so the default recipe used to end at a file that could not be served. The generated save cell now also runs merge_and_unload() into <output_dir>/merged (weights and tokenizer), which the checkpoint list offers as a whole model and the converter turns into a servable GGUF.
  • It runs in the project venv, which is where torch lives. Missing converter deps (gguf, sentencepiece, protobuf) are installed through envs.install — uv pip install --python <venv>, because a uv-created venv has no pip in it at all and python -m pip fails on every project this module makes.
  • Output types are the ones the converter itself writes (f16/bf16/f32/ q8_0). q4_k_m is a separate llama-quantize step and is deliberately not offered, because offering it and silently producing f16 would be worse.

The conversion records where the file came from​

lineage.py writes one row per produced GGUF into app.db, keyed by its path.

Before it, a file called my-project-checkpoint-1200-f16.gguf in the llama.cpp catalog told you a project id and nothing else — not the base model it was fine-tuned from, not the recipe that produced it. "Score my fine-tune against its base" needs that first one, and it was something you had to remember by hand.

The GGUF path is the join key, which is what keeps this a provenance row rather than a workflow engine: RunTarget.model_path already names a file and _output_path already writes a deterministic name, so everything downstream keys off the path already. One writer — convert.py, at the one moment every field is known. Several readers: GET /api/evals/targets labels a target "fine-tune of <base>" and carries baseModel, and llamacpp.list_models reports the same.

Two details that decide whether it works:

  • The base model falls back to the recipe. run_conversion's base_model argument is only required for a LoRA (an adapter is meaningless without one), so for a full fine-tune it is empty and the recipe is the only place the base is recorded. Reading the argument alone would leave every full fine-tune with no base — that is, with the feature not working in the common case.
  • The recipe is snapshotted, not referenced. recipe.json keeps changing as you tune the next run, and a row saying "trained with lr=2e-4" has to keep meaning that after you try 1e-4.

Writing the row can never fail a conversion: lineage.record swallows, because the GGUF is on disk and servable whether or not the bookkeeping lands.

The notebook pane & kernel sessions​

training.notebook is a native cell UI (CodeMirror cells, non-singleton, opened with params {projectId, notebook}) speaking the Jupyter kernel protocol — no iframed JupyterLab. The kernel engine itself is domain-neutral and shared: it lives in backend/notebook_core/ (KernelSession, nbformat IO, the manager's ws event handling) and is also consumed by the generic notebook module. training/kernels.py subclasses it with the training specifics — sessions keyed {projectId}:{path}, spawned from the project venv, with the stdout sentinel scan. The backend side runs one TrainingKernelSession per open notebook:

  • The kernel spawns from the project venv's python (-m ipykernel_launcher) via jupyter_client's sync API on daemon threads (event-loop-agnostic: works under uvicorn --reload on Windows). No kernelspec files — an in-memory spec pins the venv.
  • Sessions are process-global: closing the pane (or the tab) leaves training running; reopening reattaches and replays the current notebook state.
  • The in-memory nbformat doc is authoritative — UI/agent edits mutate it, iopub outputs append to it, and it's flushed atomically to the .ipynb (debounced 2 s + after each completed cell), so the on-disk notebook stays Kaggle-compatible truth.
  • Executions run serially through a queue; stream output passes through the sentinel splitter, so horrible_train telemetry never appears as cell text.
  • !pip install x / %pip install x install into the project venv through uv (a uv venv has no pip), via the kernel extension every shared-engine kernel loads — see Installing packages.
  • Interrupt rides the platform interrupt (Win32 event / SIGINT). Windows caveat: it lands between statements — training loops stop in milliseconds, but a single long blocking C call (one time.sleep(600)) defers it until that call returns; Restart is the hard stop.

A finished run still holds the GPU. The kernel is process-global and outlives the pane by design (that is what lets a six-hour fine-tune survive closing the tab), so the model, optimizer states and cache stay resident until the kernel is restarted or shut down — around 10 GB after a small LoRA run. Restart is the button; there is deliberately no automatic teardown, because "the run finished" and "you are done with the session" are different facts and only one of them is knowable from here.

Cells also get a language server — completion, hover and diagnostics, with names resolving across cells (see Notebook cells get the same language server). This is the one place it matters most: the server resolves its interpreter by walking up from the file to the nearest .venv, which for a training notebook is the project venv — so a recipe completes against the transformers/trl/peft it will actually run on, not against the dashboard's own environment. That is also why the pane resolves Project.root from GET /api/training/projects/{id} and joins it to the relative notebook path before attaching: a relative path would point the server somewhere else entirely, and it would resolve something rather than fail.

Kernel slice of the training ws channel: client sends open, run_cell, run_all, cells (ops: insert/edit/delete/move), interrupt, restart, shutdown, watch_run; server sends opened, kernel_status, execution_state, output (raw nbformat output dicts), cells_changed, run_backfill, and error.

Cells: the keyboard, and markdown​

Jupyter's contract, and both notebook panes share it (notebook/CellEditor.tsx, notebook/advance.ts):

keycode cellmarkdown cell
Ctrl/Cmd+Enterrun, stay putrender, stay put
Shift+Enterrun, focus the next cellrender, focus the next cell
Shift+Enter on the last cellrun, append a code cell and land in itsame
double-click—edit

Three things this replaced. The two keys were bound to the same callback, so run-and-advance did not exist and there was no way to walk a notebook without the mouse. The training pane never rendered markdown — a prose cell was a raw editor — and its Shift+Enter sent that cell to the kernel, which refused it; the pane's error banner then read no code cell <id>, which sounds like the notebook lost the cell you were looking at. And the backend collapsed two failures into that one sentence, so it now says either no cell <id> in this notebook or cell <id> is markdown, not code — nothing to run. (cells.run_cell / notebook.run_cell answer the second case immediately too; a markdown cell never gets a run state, so the tool used to wait out its full 120s timeout and then report unknown.)

Advancing into a markdown cell opens it for editing rather than leaving it rendered. A rendered cell has no editor to hold the caret, so the walk would stop there — and the alternative is Jupyter's command mode, which needs a cell-selection model the pane does not have.

Focus requests are tracked by position, not by cell id: a cell appended at the end carries a temporary id until cells_changed answers with the real one, and that swap remounts the row.

Visualizations​

All four visual panes feed off the same training channel; the notebook (or a script run) drives them through the horrible_train helper's sentinel events:

  • Metrics (training.metrics, uPlot) — one streaming chart per metric of the selected run. metrics.py keeps a per-run ring buffer (training.metrics.bufferPoints, default 5000) so a chart opened mid-run backfills via watch_run, and coalesces the wire to ≤20 events/s per run while keeping every point server-side.

    And they persist now. They used to live only in that ring buffer, so they died with the backend process: you could fine-tune for six hours, restart the app, and have nothing but a checkpoint. Meanwhile localtrack — the module whose entire job is keeping exactly this — had one producer (the evals sweep) and four default charts (train/loss, eval/loss, eval/accuracy, train/learning_rate) that nothing had ever written. Those are Hugging Face Trainer key names and ht.callback() emits precisely them, so connecting the two fills the default workspace on the first run.

    Three decisions about where that hook goes:

    • In metrics.record_event, not in horrible_train. The helper's contract is zero dependencies and a stdout sentinel protocol; giving it an HTTP client breaks the Kaggle and Colab push, which run it on someone else's machine.
    • Not through report_to. That is a Hugging Face field with known integration names — an unknown one is a runtime error, not a no-op. ht.callback() is already additive and installed regardless of report_to, which is exactly why it is the right carrier. localtrack is deliberately not added to recipes.TRACKERS.
    • Every point, not the coalesced wire. The wire drops intermediates at 20/s because a chart cannot draw faster; a store has no such excuse, and a loss curve missing 90% of its points is not the curve.

    A run's tracked config is its recipe (recipes.run_config, flat because compare_runs diffs keys): a sweep point declares exactly what it varies, and a run started from the notebook — the common case — used to record {source, projectId} and nothing else, so those were the runs a comparison could say nothing about. The sweep tag still means "a sweep declared this" and is kept apart from the fallback.

    The localtrack project is the training project's id, so each project keeps its own run history — deliberately unlike the evals sweep, which files everything under one hardcoded evals project. The mirroring itself is localtrack.mirror.RunMirror, shared with that sweep so the swallow-everything posture cannot drift between two copies.

    One honest limit: the helper emits no "finished" event and kernel lifecycle lives in shared notebook_core, so a run is marked finished when a new run supersedes it for the same project, or at backend shutdown. A run whose process merely died stays running until the next one starts — visible in the pane, and better than guessing from a timeout. The four panes below are declared embedded: they are companion strips of the notebook they ride on, reached with their region keys or the Training: Open … commands (which reveal the strip rather than opening a pane). training.metrics is deliberately not — it is seeded as the Training workspace's own center pane, so it is a destination in a way the others are not. See Embedded views and sections.

  • Architecture (training.modelgraph, dagre SVG) — the layer graph: torch.fx topology (with tensor shapes when an example= input is given), or the named_modules tree when the model isn't traceable. With weights=True, per-layer gradient norms color-map the nodes live during training (model_stats events). It is scoped to one project, taken from the host notebook's params, because two notebooks open at once otherwise overwrite each other's graph.

    Two things make it fill itself rather than sitting on an empty state:

    ht.callback() publishes the architecture it is handed. The Trainer passes model= to every callback hook and the callback used to drop it, so a generated recipe could run to completion with the pane still reading "call horrible_train.watch(model)" — naming a call the recipe does not make and, on the trl path, could not make (the trainer is given the model's name, so no model object exists in the notebook namespace before it is built). graph=True is the default because the fallback is a walk of named_modules(); weights=False is, because that one norms every parameter on every log(). Both are arguments: ht.callback(graph=False), ht.callback(weights=True).

    The last graph is cached per project (metrics.graph_backfill) and replayed to a pane that asks (watch_graph on the training channel, the watch_run pattern). model_graph is emitted exactly once, at the top of trainer.train(), and the strip starts closed — so without this the only way to ever see a graph was to have had the strip open at that instant. A run emits metrics continuously, so the chart pane can join late; an architecture does not.

  • Rollout (training.rollout) — the latest frame event as an <img>, rAF-throttled with an fps readout; gym envs stream via horrible_train.frame(env.render()).

  • Manim (training.manim) — plays the mp4 a render produces (manim_status progress lines + manim_done {url}), served by the media route.

Runners​

Two subprocess runners live outside the kernel (both Popen-on-thread, sentinel-parsed, process-global so they survive pane/tab closes):

  • Script runner (runners/script_runner.py) — python -u <script> in the project venv; ht.log()/ht.frame() in the script are indistinguishable from notebook cells. Driven by POST/GET/DELETE /projects/{id}/runs and by POST /projects/{id}/recipe/run.

    Every run also writes its output to runs/logs/<script>-<runId>.log under the project. Until it did, that output existed only as run_output broadcasts: whoever had a pane open saw it and nobody else ever could — so a headless run, which is the only kind an agent can start, was observable as state: running and then, at best, returncode: 1. A crash was a number. And because the registry lives in memory, a backend restart (a --reload on any file under backend/, tests included) erased even that: the run list came back empty, which is indistinguishable from a run that finished cleanly. GET /projects/{id}/runs/logs reads the directory, so it answers for runs this backend has no memory of, and GET /runs?tail=N carries the last N lines inline. The agent tool is training.run_output.

  • Manim runner (runners/manim_runner.py) — python -m manim render against a scene file under media/scenes/; auto-installs manim into the venv on first use; locates the newest mp4 and serves it via the traversal-guarded media route.

Cloud push​

A PushTarget driver layer (push/, same shape as the provider layer) sends a project's main.ipynb to cloud compute:

  • Kaggle kernels (push/kaggle_push.py) — generates kernel-metadata.json (competition/dataset sources drawn from the project's Kaggle refs, enable_gpu), then kernels_push; status polls kernels_status. Competition submissions stay manual (kaggle competitions submit … in the terminal pane).
  • Google Colab (push/colab_push.py + google_auth.py) — Colab has no API, so the notebook uploads to Drive with the Colab mimetype; the file id is remembered (.push-colab.json) so re-pushes update in place. OAuth uses the user's own client (settings training.google.clientId/clientSecret), manual-code flow, refresh token in .data/training_google_token.json — never returned to the client.

Both pushers are also used by notebook publishing, through keyword-only options whose defaults are the training behaviour, so the training routes call them unchanged: KagglePush.push(..., private=, enable_gpu=) (a publish can be public and does not spend GPU quota) and ColabPush.push(..., anyone_with_link=), where None leaves a file's Drive sharing alone. kaggle_push.has_credentials() answers "could a push authenticate" without importing the kaggle package, and ColabPush.delete removes the pushed Drive file and forgets it.

Learn: the notebook explains itself​

The notebook's right strip opens on Learn (training.learn, LearnPane.tsx), because the point of doing AI research in this app rather than in a bare notebook is that it teaches while you work. The other strips (Metrics, Architecture, Manim, Rollout) show what is happening; Learn says what it means. Three parts, each grounded in something real rather than generated:

  • This cell. The cell you are in (learn/focus.ts: the notebook reports focus per projectId:notebook, the same key as openSession) is matched by identifier against two sources. Knobs — learning_rate, lora_r, max_length and their aliases — come from GET /api/training/learn/glossary, which is recipes.glossary(): the recipe form's own help text, one entry per knob across every backend. It is served, never copied into TypeScript, so the strip and the form cannot explain a knob two ways. Concepts — the classes and calls a knob is passed to (SFTTrainer, get_peft_model, loss.backward(), mixed precision, cross-entropy) — have no form row, so they are written in learn/terms.ts, each saying why it matters rather than restating the API. Matching uses word boundaries: Trainer inside SFTTrainer is not the generic training-loop concept.
  • This run. learn/diagnose.ts reads the live curves (the same metrics events the Metrics strip draws) with plain rules: waiting for a curve, loss falling, loss flat, a spike against the recent median, overfitting (eval loss rising three evaluations running while train loss falls), an exploded gradient norm, and NaN — which stops every other read, since nothing after it means anything. Every diagnosis states the rule that fired, with its numbers, so the strip teaches how to read a loss curve instead of issuing verdicts.
  • Go deeper. Reveal Curves or Architecture, open the Model explorer, or Ask the agent — which opens the chat with an explanation request for this cell drafted, not sent (draftInChat in the agent module), so the user reads and edits the question before a turn is spent on it.

Peer fabric (removed)​

A training.peers strip once advertised a node as "offering GPU / seeking help" over the peer fabric (fabric.py, specs.py, /training/fabric/*, training.fabric.* settings). It was advertising with no remote execution behind it, its buttons did not reflect the saved mode after a reload, and a disconnected peer's ad lingered until restart — so it was removed rather than fixed. Collaboration on training still rides the ordinary paths: shared panes and agent.ask_peer.

Agent integration (the flagship)​

Typing "i want to work on kaggle's pokemon tcg competition" into the ask bar drives the whole flow. Two tool groups make this work, both progressively disclosed (loaded on demand or by keyword) so they don't bloat the model's tool budget:

  • training — backend tools (agent_tools.py, registered into the backend-sdk registry via register_agent_tools(), called from app.py). Server-resolved, so they run with no tab attached: training.search_environments, resolve_environment, list_projects, project_status (read-only); create_project, fetch_data, install_deps, start_run, stop_run, push, render_manim (side-effecting, gated by the agent permission engine with a specifierTemplate like {provider}:{ref}).

    Plus the two that close the loop: training.list_checkpoints (read-only — checkpoint names are not guessable, so this is the call before the next one) and training.convert, which turns a checkpoint into a GGUF in llama.cpp's managed directory. convert drains the conversion's NDJSON stream and reports only the outcome: the route streams it so a pane can show a progress log, but an agent has nowhere to put hundreds of progress lines and would pay for every one in context. It takes minutes and is deliberately not backgroundable the way start_run is — there is no id to poll afterwards, the answer is the output path.

  • training — frontend tools (agentTools on the training.notebook panel). Full cell CRUD + execute against the pane's session store: cells.list_cells, read_cell, kernel_status (read-only); insert_cell, edit_cell, delete_cell, run_cell (waits ≤120s, returns outputs/traceback), run_all, interrupt, restart.

    These were notebook.* until they collided with the reactive notebook, which declares the same seven verbs against a different store with a different session argument (path, not projectId). The frontend resolves a tool call by name with find and this module registers first, so these won and the reactive notebook's were unreachable — silently, decided by registration order in main.tsx. They are grouped by name prefix, so moving them to training.* both ends the collision and files them with the rest of this module. A duplicateToolNames check in the manifest now logs any recurrence, and stored permission rules naming the old tools are migrated on load (permission_store.rename_in_rule) — a notebook.run_cell grant was a grant over this notebook, and must follow the tool rather than the name.

The orchestrator changes are generic (backend/modules/agent/orchestrator.py): AgentTool gained an optional group field (backend/sdk/types.py) — None keeps a plugin tool in the always-loaded core (unchanged), a set group moves it into the progressive-disclosure pool. _GROUP_KEYWORDS preloads training on "kaggle/train/dataset/gym/…" and notebook on "notebook/cell/kernel/…". Now that the cell verbs are training.*, this flow needs only the one group — but a prompt naming both notebooks preloads both, and with the cell tools counted in training that is over TOOL_BUDGET; see the note below. open_pane now accepts optional params, so the agent opens the notebook pane bound to a project (open_pane("training.notebook", {params: {projectId, notebook}})).

Flow: keyword-preload training → search_environments → create_project (approval gate, specifier provider:ref) → fetch_data → open_pane(training.notebook, params) → show("Training Metrics") → cells.insert_cell/run_cell to build the starter. Verified end to end against a live Ollama model: the ask reaches training.create_project, the gate fires with the right specifier, and on approval the project + scaffolded notebook land on disk.

Resolved: training was split​

The group had grown to 34 tools — and the count was easy to miss, because only 14 of them are declared in this module's agent_tools.py. The other twenty were the ten recipe/sweep verbs in recipe_tools.py and the ten notebook cell verbs registered by the frontend (packages/core/src/modules/training/agentTools.ts). A tool's group is its name prefix (_group_of) and does not care which side of the wire declared it, so all thirty-four landed in one group.

With a ~9-tool core that left no room for evals (10) beside it: the loop this module's own agent prompt prescribes — start a run, read its metrics, score the result — did not fit under TOOL_BUDGET, and the overflow was dropped. Loudly in the log, invisibly everywhere else.

The split is by job, since a group is the unit load_tools hands out:

grouptoolswhat
training14projects, venvs, fetch, runs, checkpoints, convert, push
recipe7author a recipe, install the stack, run and stop a sweep
cells10read/edit/run the open training notebook's cells
wandb3import someone else's W&B runs into localtrack

cells, not notebook: that name belongs to the reactive notebook module, and these two collided under it once already. Both old spellings (notebook.* and training.*) are carried in permission_store._RULE_RENAMES, listed rather than chained — rename_in_rule does one lookup, so a rule left to reach cells.run_cell by way of training.run_cell would stop at a name that no longer exists, and a grant pointing at nothing fails closed and re-asks the user for a decision they already made.

Every phase of the loop now fits over the core at the 38 floor — pick data (25), run a job (30), edit the notebook (33), convert and serve (30), judge it (26) — and the whole thing at once is 73, which a hosted model carries in one round. test_trainer_tool_budget.py asserts this by counting the groups, frontend tools included, rather than trusting a number written in a comment; that is how the group reached 34 unnoticed.

Backend surface​

/api/training (backend/modules/training/routes.py):

RoutePurpose
GET /providers, GET /providers/{p}/search?q=&kind=, POST /providers/{p}/resolveprovider catalog + discovery
GET/POST /projects, GET/DELETE /projects/{id}project store (create resolves the ref, scaffolds main.ipynb, kicks off the venv bootstrap)
POST /projects/{id}/fetch (202)dataset download on a worker thread
POST /projects/{id}/deps (202)uv pip install extra packages into the venv
GET/PUT /projects/{id}/notebook?path=whole-document notebook read/replace (path-traversal-guarded)
POST/GET/DELETE /projects/{id}/runs, GET/DELETE /runs/{id}script runs (start/list/stop)
POST /projects/{id}/manim (202), GET /projects/{id}/media/{path}manim render + traversal-guarded media serving
GET/PUT /projects/{id}/recipe, POST /projects/{id}/recipe/applythe recipe form's schema + validation, and writing its cells into main.ipynb
GET /recipe/docsdoc links per config class from the versioned index
GET /projects/{id}/checkpoints, POST /projects/{id}/convertcheckpoints to convert, and the conversion itself (NDJSON progress)
GET /push/targets, POST/GET /projects/{id}/push/{target}[/status]Kaggle/Colab push + status
POST /google/auth/start, POST /google/auth/complete, GET/DELETE /google/...Colab Drive OAuth (server-held token)
GET /fabric/ads, POST /fabric/advertisepeer training ads (list / set advertise mode)

Long-running work reports over the shared /ws socket on the training channel (every connection is subscribed on connect): env_progress / fetch_progress {projectId, line, pct?}, project_changed (a full ProjectModel), the kernel-protocol events, plus metrics / model_graph / model_stats / frame / run_output / run_state / manim_status / manim_done.

Opening a training pane without a project​

training.notebook and training.recipe are params-bound ({projectId, notebook}), but most ways of opening them — the Start menu, the palette, a restored layout — carry no params. There is exactly one project someone means when they open the training notebook without saying which, and it is the one they were in last, so last-project.ts records it (localStorage, following layout/recents.ts — a per-viewer convenience, and deliberately not a SettingDecl, which would put "Last project" on the settings page as something to configure). Both panes fall back to it when opened with no params, resolved once per mount so the remembered id changing under an open pane cannot swap the notebook out from under you.

A remembered project that no longer exists forgets itself and shows the picker, rather than the "no longer exists" dead-pane message. That message is right for a pane the user deliberately opened against that project and wrong for a fallback: nobody asked for this project, the value is not one they can see or clear, and without the self-heal the pane would greet them with the same dead project on every open, forever.

A params-bound pane opened cold — from the start menu or the palette — renders the projects picker itself rather than a sentence. It used to say "open me from the Training projects pane", which named something the reader could not open: consolidation made training.projects embedded, i.e. a section of Explorer. Picking a project calls openTrainingNotebook, whose canReuse retargets that same instance, so the pane you opened becomes the notebook instead of leaving an empty one behind. training.recipe and the notebook module's notebook.editor do the same with their own pickers.

The fine-tuning agent​

trainer is the agent for this loop — pick it in the chat. The roster spec permits training, evals, localtrack, editor and files, and preloads only training, so reading a recipe costs a load_tools rather than schema space on every turn.

llamacpp now appears in that list, and used not to. The original reasoning was right: a group is a tool name's prefix (_group_of), the llamacpp.* namespace held only settings keys (llamacpp.diskBudgetGb), and listing it would have silently permitted nothing at all — the same trap the connectors rule names when it says a connector id must equal its tool prefix.

That stopped being true the moment llama.cpp grew real agent tools. Without the group this agent could convert a checkpoint and then had no way to serve it, which is the step immediately before the eval its own prompt tells it to run. It is permitted but not preloaded — this flow already sits close to TOOL_BUDGET.

hardware still does not appear, for the original reason: settings keys only.

Contributions to the Layout Shell​

ContributionValue
Panel training.projectssingleton hub, embedded: the Projects section of Explorer and the left region strip of training.notebook. Provider search → create project, project list with live venv/data status; a row opens its notebook and carries the rest in a training.project context menu
Panel training.recipenon-singleton fine-tuning recipe form, params {projectId}, opened from a project row (openTrainingRecipe); typed schema + validation chips, tracker picker, checkpoint→GGUF card
Panel training.notebook ("Training Notebook")non-singleton notebook pane (center), params {projectId, notebook}, labelled with the notebook's file name via a title param, titled apart from the notebook module's own "Notebook Editor", opened via openTrainingNotebook() so it focuses/reuses rather than splitting (see windowing); CodeMirror cells, run/interrupt/restart toolbar, nbformat output renderers (stream+ANSI, results, images, HTML, tracebacks)
Widget training.metricslive metric charts (uPlot, one per metric of the selected run; ring-buffer backfill via watch_run, ≤20 wire events/s per run with every point kept server-side)
Widget training.modelgraphmodel architecture graph (dagre SVG; fx/module-tree, grad-norm heatmap)
Widget training.rolloutlive gym/render frame stream with fps readout
Widget training.manimmanim render viewer (progress log + mp4 player)
Widget training.learnthe teaching strip: this cell's knobs and concepts, rule-based reads of the run, go-deeper links — see Learn
Regions on training.notebooklearn (open by default)/metrics/architecture/manim/rollout on the right strip, projects on the left
Commandstraining.open, training.openRecipe, training.openMetrics, training.openModelGraph, training.openRollout, training.openManim, training.openLearn
Recipe pane hand-offafter a conversion, Serve this starts a local llama-server on the new GGUF and Score it opens Evals on its Run section — those panes used to sit side by side with nothing connecting them
Project row actionsopen notebook, 🧪 recipe, ⇪ push to Kaggle, ⇪ push to Colab, delete
Connector trackersW&B API key + MLflow tracking URI, Fernet-encrypted. The one connector contributing no agent tools, so its id names no tool group
Settingstraining.projectsRoot, training.defaultPython, training.kaggle.username, training.kaggle.key, training.hf.token, training.google.clientId, training.google.clientSecret, training.metrics.bufferPoints, training.manim.quality

Credentials note: settings persist in the plaintext .data/settings.json store — same v1 posture as database connection credentials (encryption is a tracked follow-up).

Browser vs desktop​

CapabilityBrowserDesktop
Project/provider management✅✅
Venv + kernel subprocesses✅ (backend-side)✅ (backend-side)

Everything runs on the backend node; both layouts are equivalent.