Module: localtrack
LocalTrack is the node's local experiment tracker — W&B/MLflow-shaped, but the
data never leaves the machine. Runs, parameters, metric time-series and artifacts
live in SQLite ($HORRIBLE_DATA_DIR/localtrack.db, WAL); the pane is a
multi-run chart workspace with server-side LTTB downsampling and live EMA smoothing.
It is the watching half of the fine-tune loop, and it has two producers:
- Training runs. Every metric a run emits through
ht.callback()is stored here, under a localtrack project named for the training project. Those keys are Hugging FaceTrainer's own (train/loss,eval/loss,eval/accuracy,train/learning_rate) — which is to say they are exactly the four default panels below, and until this producer existed nothing had ever written them. See training for why the hook sits inmetrics.record_eventrather than in the helper or inreport_to. - Eval sweeps. One run per sweep, carrying per-case pass/duration/rounds.
Both go through localtrack.mirror.RunMirror, which is best-effort by construction: a
tracking failure must never cost a training run or a sweep the results it already has.
Overview
LocalTrack is designed for zero-configuration, privacy-preserving local ML workflows. All experiment metadata, parameters, metric time-series, and artifact files are stored locally in the node's SQLite database ($HORRIBLE_DATA_DIR/localtrack.db).
Key Features
- Multi-run overlay. A collapsible run list with a per-run colour that matches that run's line in every chart. Selecting a run is a real checkbox — see The controls are controls.
- Live. A
localtrack/wschannel means a run in flight moves the charts with no refresh. See Liveness. - High-density charting.
uPlotplus server-side LTTB downsampling renders 100k+ points without lag; EMA smoothing (0.00–0.99) is applied server-side too. - Regex panel filter, falling back to substring while the expression is
incomplete — a half-typed
eval/[is a normal intermediate state, not an error. - Panel layouts persist server-side, per project. See Layouts are server-side.
- Python client SDK and HF callback:
BaseLocalTrackLogger, a batchingLocalTrackClient, andLocalTrackHFCallback(TrainerCallback). - Agent tools (
localtrack.*) for conversational run comparison.
Liveness
There was none: no channel, no polling, no interval. A sweep or a training run could write for ten minutes while the pane sat still, and the only way to see anything was to hit refresh. For a module whose whole job is watching a run happen, that was the defect.
backend/modules/localtrack/stream.py publishes on the localtrack channel:
| Event | When | Payload |
|---|---|---|
run_created | a run is created | runId, projectId, name |
run_updated | status/summary change | runId, projectId, status |
run_deleted | a run is deleted | runId |
metrics | new points ingested | runId, keys |
Four things there are decisions rather than plumbing.
Publishing happens after the commit, never before. A pane that refetches on the event would otherwise race the writer and read the pre-write state — which looks exactly like a dropped point.
metrics carries keys, not values. The pane applies its own downsample budget
and smoothing, so it must fetch its own series anyway; shipping values here would
put a second, un-downsampled copy of the data on the wire for no benefit.
Coalescing unions its keys. A tight loop emits far faster than a chart can
drink, so bursts collapse to MAX_EVENTS_PER_S (20), latest-wins, with a timer
flushing the tail so a run's final point always lands. But the coalesced payload
merges keys rather than replacing: overwrite and every key but the last in a
burst is silently never announced, leaving one panel frozen while its neighbour
updates. Measured: 30 writes in 0.25s → 6 events, all three metrics still present.
Revisions are per metric key. The store keeps metricRevisions[key], not one
counter, and each panel watches only its own key. A single global counter made every
train/loss point refetch the accuracy and learning-rate panels too — measured at
172 refetches for 40 ingested points across three panels. Scoped, the same workload
costs 43 refetches on the metric that actually changed and 3 on each idle panel.
Nothing in stream.py can raise into a caller. A tracker that fails a training run
by failing to draw a chart is worse than one that draws nothing.
Layouts are server-side
The panel arrangement was in localStorage, which is per-browser-origin: a layout
built in the browser layout was invisible in the desktop shell and vice versa, and
clearing site data reset it silently. It is now lt_layouts in the module's own DB,
behind GET|PUT /api/localtrack/projects/{id}/layout, stored as an opaque blob
the backend never interprets (the /api/flows precedent — a new panel type needs no
migration).
It is per project, not per workspace, which is why it is not in the workspace store.
null and [] are different answers and the API keeps them apart: null means
"never saved one" and yields the four defaults, [] means the user removed every
panel. The old code tested parsed.length > 0, so a deliberately cleared workspace
sprang back to four charts on every reload.
The controls are controls
Run visibility was an emoji (👁 / 👁🗨) on a <button>. Two problems, not one:
it breaks the house rule against native emoji as iconography, and it was not a
checkbox — nothing announced it as one and the space bar did nothing. It is now a
real <input type="checkbox">, visually hidden behind the run's colour swatch,
carrying an aria-label naming the run (a wrapping label's title does not
become the input's accessible name).
The hand-rolled fixed-position "new project" modal is gone too; dialogs.prompt is
themed, focus-managed and stacks correctly with everything else. Deletion of runs
and projects is wired for the first time — removeRun/removeProject existed in
the store, were exported by the hook, and were attached to no control at all.
Failures are visible
state.error was set in six places and rendered by nothing, so a failed fetch was
indistinguishable from an empty project; state.loading was tracked and unused.
Both now render — the error as a dismissible banner (a failed save is still true a
minute later, so a toast that has already faded cannot be re-read), the loading as
a 2px scan bar that leaves the previous values readable underneath.
The two declared settings — localtrack.defaultProject and
localtrack.defaultSmoothing — are read for the first time as well; the store used
to hardcode 'default' and 0.0, so changing either did nothing.
Routes are namespaced
Every route used to be registered twice: once under /api/localtrack/… and once
at a bare /api/projects, /api/runs, /api/metrics/…. Nothing consumed the bare
form (the Python client has always used the prefixed one) and it claimed three very
generic paths for one module. The aliases are gone.
Architecture
Template Notebooks (notebooks/)
LocalTrack includes pre-built Jupyter Notebook templates (.ipynb) ready to run inside the dashboard's notebook editor:
notebooks/01_localtrack_huggingface_training.ipynb: Fine-tuning models with PyTorch & Hugging Facetransformerswhile streaming telemetry throughLocalTrackHFCallback.notebooks/02_localtrack_experiment_analysis.ipynb: Fetching historical runs, Pandas DataFrame comparisons, and LTTB downsampled time-series querying across multiple models.notebooks/03_localtrack_llm_evaluations.ipynb: Evaluating LLM reasoning benchmarks, streaming running accuracy curves, and uploading summary artifacts.
Orchestrator Agent Tools (localtrack.*)
LocalTrack exports tools to the central AI orchestrator agent:
| Tool Name | Description | Side Effect |
|---|---|---|
localtrack.list_projects | List all experiment tracking projects and run counts | Read-only |
localtrack.create_project | Create a new experiment tracking project | Yes (Gated) |
localtrack.list_runs | List runs for a project with status and summaries | Read-only |
localtrack.get_run | Get full configuration, hyperparameters, and artifacts | Read-only |
localtrack.query_metrics | Fetch downsampled metric curves across multiple runs | Read-only |
localtrack.get_metric_keys | Discover all logged metric keys | Read-only |
Multi-Turn Agent Evaluation Suite
Evaluated via backend/tests/test_localtrack_agent_eval.py:
- Automatic keyword preloading for prompts mentioning
"loss curves","experiment tracking","wandb", or"localtrack". - Multi-turn tool execution inspecting projects, comparing metric series, creating projects, and retrieving artifact files.
Python Client & Hugging Face Integration
Hugging Face TrainerCallback Example
from transformers import Trainer, TrainingArguments
from backend.sdk.localtrack import LocalTrackHFCallback
# Initialize callback
callback = LocalTrackHFCallback(
project_name="my-llm-finetuning",
run_name="llama3-lora-run-1",
base_url="http://127.0.0.1:8000",
tags=["llama-3", "lora", "sft"],
)
training_args = TrainingArguments(
output_dir="./results",
learning_rate=2e-4,
per_device_train_batch_size=4,
num_train_epochs=3,
logging_steps=10,
report_to=["none"], # No need for external cloud trackers
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
callbacks=[callback],
)
trainer.train()
Custom PyTorch / Python Logging
from backend.sdk.localtrack import LocalTrackClient
logger = LocalTrackClient(base_url="http://127.0.0.1:8000")
# Initialize run
run_id = logger.init_run(
project_name="rl-experiments",
run_name="ppo-halfcheetah-v4",
config={"lr": 3e-4, "gamma": 0.99, "clip_eps": 0.2},
tags=["rl", "ppo"],
)
# In training loop
for step in range(1000):
loss, reward = train_step()
logger.log_metrics({"train/loss": loss, "eval/reward": reward}, step=step)
# Upload saved weights or config
logger.log_artifact("config.json")
logger.finish_run(status="finished")
Model Context Protocol (MCP) & Agent Skills
LocalTrack also integrates with external Weights & Biases (W&B) endpoints:
- MCP Server Catalog: Registered in
backend/modules/mcp/curated.jsonsupporting both hosted HTTP (https://mcp.withwandb.com/mcp) and localuvx wandb-mcp-server. - Agent Skills: Registered in
.claude/skills/(wandb-primary,wandb-eval-tables,wandb-autoresearch).
Core API Endpoints
GET /api/localtrack/projects: List projects with run counts and activity dates.POST /api/localtrack/projects: Create or update a project.DELETE /api/localtrack/projects/{id}: Delete a project and all associated runs.GET /api/localtrack/runs?project_id={id}: List runs for an active project.POST /api/localtrack/runs: Create an experiment run.GET /api/localtrack/runs/{id}: Get run configuration, summary stats, and hardware info.PATCH /api/localtrack/runs/{id}: Update run status (running,finished,failed) and metrics summary.POST /api/localtrack/metrics/ingest: High-throughput batch ingestion of metric logs.GET /api/localtrack/metrics/keys: Discover all logged metric keys.POST /api/localtrack/metrics/query: Fetch time-series metric data with server-side LTTB downsampling and smoothing.POST /api/localtrack/runs/{id}/artifacts: Upload artifact files (config.json,trainer_state.json).GET /api/localtrack/runs/{id}/artifacts: List run artifacts.GET /api/localtrack/runs/{id}/artifacts/{artifact_id}/download: Download an artifact.GET /api/localtrack/projects/{id}/layout: The saved panel arrangement, orpanels: nullwhen the project has never saved one.PUT /api/localtrack/projects/{id}/layout: Replace it.400ifpanelsis not a list.
Note there is no bare-path alias for any of these — see Routes are namespaced.
localtrack.compare_runs
The comparison is an agent tool, not just a route. store.compare_runs and
POST /api/localtrack/compare had both existed since the store was written, and
nothing exposed them to a turn — so an agent asked "did this help?" pulled raw metric
series with query_metrics and re-derived the comparison itself.
Worse than absent: training/recipe_tools.py instructs the model, twice and by name,
to call localtrack.compare_runs after a sweep. Following the instruction produced an
unknown-tool error. test_comparison_agent_tools.py now source-scans the instruction
strings — including the note fields handlers return, which no amount of
introspecting the tool objects would reach — and requires every <group>.<tool> they
name to exist.
The tool is thin on purpose: every judgement (varied vs shared config, last value
rather than mean, flagging incomparable runs) stays in the store, so the tool and the
pane cannot drift into two different answers to one question. A mixed comparison is
reported in a note rather than left in a field the model may not read.
Deep links into the pane
localtrack.workspace accepts {projectId, runNames} and opens on that project with
those runs selected. This is what the sweep pane's Compare button sends; it used
to call openPanel('localtrack.workspace') with no arguments at all, landing you on
whichever project was last open with the sweep you had just run nowhere in sight.
Runs are matched by name, not by id, and that is not a shortcut. A sweep point
has three different ids over its life — the script-runner process
(SweepRecord.runIds), the training run the script announces
(SweepResultEntry.metricRuns), and the localtrack run the mirror mints — and none
of them is the next one. The name is the only thing both ends agree on in advance:
sweeps.py writes point.label into the generated script, metrics.py names the
mirrored run after it (_names), and that is precisely why a point reads
learning_rate=0.0002 instead of a hex id. Matching on ids would select nothing,
silently — and an empty comparison does not look like a broken link.
Names that match nothing are reported, not dropped: a point whose run never started is a real thing to know, and a comparison quietly missing two of its six points is worse than one that says so.
The delivery mechanism is SET_PANE_PARAMS (see
windowing): openPane used to focus an
already-open pane and discard the params it was handed, so every deep link worked
only while its target pane happened to be closed.
Comparing runs
compare_runs(run_ids, metric) and the two panel types over it — table and
parcoords — are the read a sweep exists for.
Every ingredient has been in this schema since it was written: config_json per run,
metrics per (run, key, step). What was missing was a producer that put anything
comparable in the first field. The training sweep now declares one config per point,
so the table has something to diff.
Three things it does that a raw row dump does not:
- Splits varied from shared. A fine-tune has forty knobs and an ablation moves one or two; a table repeating the other thirty-eight on every row buries the answer. What was held constant is reported once, underneath.
- Takes each metric's LAST value, not its mean. "Which run ended up best" is the question; a mean over a descending loss curve mostly measures how long the run was.
- Flags mixed comparisons. Runs on different backends or tasks are still comparable — you may well want to — but the difference between them is not the axis you varied, and a table that stays silent reads as an ablation result.
The parcoords panel answers the shape question the table cannot: whether the metric
moves monotonically with a knob, whether two knobs interact, and whether the best run
is a trend or one lucky point. Colour encodes the metric, not the run, because
"which of these lines is good" is the question — a categorical palette would mean
cross-referencing a legend for every line.
Importing from Weights & Biases
wandb.import_runs pulls a W&B run's config and metric history in through the
same RunMirror, so a run from a rented box charts and compares beside the ones
trained here. The history W&B returns is sampled by default and the imported run
says so: a 500-point curve standing in for 40,000 must not be presented as the
original. See training.