Skip to main content

Module: localtrack

LocalTrack is the node's local experiment tracker — W&B/MLflow-shaped, but the data never leaves the machine. Runs, parameters, metric time-series and artifacts live in SQLite ($HORRIBLE_DATA_DIR/localtrack.db, WAL); the pane is a multi-run chart workspace with server-side LTTB downsampling and live EMA smoothing.

It is the watching half of the fine-tune loop, and it has two producers:

  • Training runs. Every metric a run emits through ht.callback() is stored here, under a localtrack project named for the training project. Those keys are Hugging Face Trainer's own (train/loss, eval/loss, eval/accuracy, train/learning_rate) — which is to say they are exactly the four default panels below, and until this producer existed nothing had ever written them. See training for why the hook sits in metrics.record_event rather than in the helper or in report_to.
  • Eval sweeps. One run per sweep, carrying per-case pass/duration/rounds.

Both go through localtrack.mirror.RunMirror, which is best-effort by construction: a tracking failure must never cost a training run or a sweep the results it already has.


Overview​

LocalTrack is designed for zero-configuration, privacy-preserving local ML workflows. All experiment metadata, parameters, metric time-series, and artifact files are stored locally in the node's SQLite database ($HORRIBLE_DATA_DIR/localtrack.db).

Key Features​

  • Multi-run overlay. A collapsible run list with a per-run colour that matches that run's line in every chart. Selecting a run is a real checkbox — see The controls are controls.
  • Live. A localtrack /ws channel means a run in flight moves the charts with no refresh. See Liveness.
  • High-density charting. uPlot plus server-side LTTB downsampling renders 100k+ points without lag; EMA smoothing (0.00–0.99) is applied server-side too.
  • Regex panel filter, falling back to substring while the expression is incomplete — a half-typed eval/[ is a normal intermediate state, not an error.
  • Panel layouts persist server-side, per project. See Layouts are server-side.
  • Python client SDK and HF callback: BaseLocalTrackLogger, a batching LocalTrackClient, and LocalTrackHFCallback(TrainerCallback).
  • Agent tools (localtrack.*) for conversational run comparison.

Liveness​

There was none: no channel, no polling, no interval. A sweep or a training run could write for ten minutes while the pane sat still, and the only way to see anything was to hit refresh. For a module whose whole job is watching a run happen, that was the defect.

backend/modules/localtrack/stream.py publishes on the localtrack channel:

EventWhenPayload
run_createda run is createdrunId, projectId, name
run_updatedstatus/summary changerunId, projectId, status
run_deleteda run is deletedrunId
metricsnew points ingestedrunId, keys

Four things there are decisions rather than plumbing.

Publishing happens after the commit, never before. A pane that refetches on the event would otherwise race the writer and read the pre-write state — which looks exactly like a dropped point.

metrics carries keys, not values. The pane applies its own downsample budget and smoothing, so it must fetch its own series anyway; shipping values here would put a second, un-downsampled copy of the data on the wire for no benefit.

Coalescing unions its keys. A tight loop emits far faster than a chart can drink, so bursts collapse to MAX_EVENTS_PER_S (20), latest-wins, with a timer flushing the tail so a run's final point always lands. But the coalesced payload merges keys rather than replacing: overwrite and every key but the last in a burst is silently never announced, leaving one panel frozen while its neighbour updates. Measured: 30 writes in 0.25s → 6 events, all three metrics still present.

Revisions are per metric key. The store keeps metricRevisions[key], not one counter, and each panel watches only its own key. A single global counter made every train/loss point refetch the accuracy and learning-rate panels too — measured at 172 refetches for 40 ingested points across three panels. Scoped, the same workload costs 43 refetches on the metric that actually changed and 3 on each idle panel.

Nothing in stream.py can raise into a caller. A tracker that fails a training run by failing to draw a chart is worse than one that draws nothing.

Layouts are server-side​

The panel arrangement was in localStorage, which is per-browser-origin: a layout built in the browser layout was invisible in the desktop shell and vice versa, and clearing site data reset it silently. It is now lt_layouts in the module's own DB, behind GET|PUT /api/localtrack/projects/{id}/layout, stored as an opaque blob the backend never interprets (the /api/flows precedent — a new panel type needs no migration).

It is per project, not per workspace, which is why it is not in the workspace store.

null and [] are different answers and the API keeps them apart: null means "never saved one" and yields the four defaults, [] means the user removed every panel. The old code tested parsed.length > 0, so a deliberately cleared workspace sprang back to four charts on every reload.

The controls are controls​

Run visibility was an emoji (👁 / 👁‍🗨) on a <button>. Two problems, not one: it breaks the house rule against native emoji as iconography, and it was not a checkbox — nothing announced it as one and the space bar did nothing. It is now a real <input type="checkbox">, visually hidden behind the run's colour swatch, carrying an aria-label naming the run (a wrapping label's title does not become the input's accessible name).

The hand-rolled fixed-position "new project" modal is gone too; dialogs.prompt is themed, focus-managed and stacks correctly with everything else. Deletion of runs and projects is wired for the first time — removeRun/removeProject existed in the store, were exported by the hook, and were attached to no control at all.

Failures are visible​

state.error was set in six places and rendered by nothing, so a failed fetch was indistinguishable from an empty project; state.loading was tracked and unused. Both now render — the error as a dismissible banner (a failed save is still true a minute later, so a toast that has already faded cannot be re-read), the loading as a 2px scan bar that leaves the previous values readable underneath.

The two declared settings — localtrack.defaultProject and localtrack.defaultSmoothing — are read for the first time as well; the store used to hardcode 'default' and 0.0, so changing either did nothing.

Routes are namespaced​

Every route used to be registered twice: once under /api/localtrack/… and once at a bare /api/projects, /api/runs, /api/metrics/…. Nothing consumed the bare form (the Python client has always used the prefixed one) and it claimed three very generic paths for one module. The aliases are gone.


Architecture​


Template Notebooks (notebooks/)​

LocalTrack includes pre-built Jupyter Notebook templates (.ipynb) ready to run inside the dashboard's notebook editor:

  1. notebooks/01_localtrack_huggingface_training.ipynb: Fine-tuning models with PyTorch & Hugging Face transformers while streaming telemetry through LocalTrackHFCallback.
  2. notebooks/02_localtrack_experiment_analysis.ipynb: Fetching historical runs, Pandas DataFrame comparisons, and LTTB downsampled time-series querying across multiple models.
  3. notebooks/03_localtrack_llm_evaluations.ipynb: Evaluating LLM reasoning benchmarks, streaming running accuracy curves, and uploading summary artifacts.

Orchestrator Agent Tools (localtrack.*)​

LocalTrack exports tools to the central AI orchestrator agent:

Tool NameDescriptionSide Effect
localtrack.list_projectsList all experiment tracking projects and run countsRead-only
localtrack.create_projectCreate a new experiment tracking projectYes (Gated)
localtrack.list_runsList runs for a project with status and summariesRead-only
localtrack.get_runGet full configuration, hyperparameters, and artifactsRead-only
localtrack.query_metricsFetch downsampled metric curves across multiple runsRead-only
localtrack.get_metric_keysDiscover all logged metric keysRead-only

Multi-Turn Agent Evaluation Suite​

Evaluated via backend/tests/test_localtrack_agent_eval.py:

  • Automatic keyword preloading for prompts mentioning "loss curves", "experiment tracking", "wandb", or "localtrack".
  • Multi-turn tool execution inspecting projects, comparing metric series, creating projects, and retrieving artifact files.

Python Client & Hugging Face Integration​

Hugging Face TrainerCallback Example​

from transformers import Trainer, TrainingArguments
from backend.sdk.localtrack import LocalTrackHFCallback

# Initialize callback
callback = LocalTrackHFCallback(
project_name="my-llm-finetuning",
run_name="llama3-lora-run-1",
base_url="http://127.0.0.1:8000",
tags=["llama-3", "lora", "sft"],
)

training_args = TrainingArguments(
output_dir="./results",
learning_rate=2e-4,
per_device_train_batch_size=4,
num_train_epochs=3,
logging_steps=10,
report_to=["none"], # No need for external cloud trackers
)

trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
callbacks=[callback],
)

trainer.train()

Custom PyTorch / Python Logging​

from backend.sdk.localtrack import LocalTrackClient

logger = LocalTrackClient(base_url="http://127.0.0.1:8000")

# Initialize run
run_id = logger.init_run(
project_name="rl-experiments",
run_name="ppo-halfcheetah-v4",
config={"lr": 3e-4, "gamma": 0.99, "clip_eps": 0.2},
tags=["rl", "ppo"],
)

# In training loop
for step in range(1000):
loss, reward = train_step()
logger.log_metrics({"train/loss": loss, "eval/reward": reward}, step=step)

# Upload saved weights or config
logger.log_artifact("config.json")
logger.finish_run(status="finished")

Model Context Protocol (MCP) & Agent Skills​

LocalTrack also integrates with external Weights & Biases (W&B) endpoints:

  • MCP Server Catalog: Registered in backend/modules/mcp/curated.json supporting both hosted HTTP (https://mcp.withwandb.com/mcp) and local uvx wandb-mcp-server.
  • Agent Skills: Registered in .claude/skills/ (wandb-primary, wandb-eval-tables, wandb-autoresearch).

Core API Endpoints​

  • GET /api/localtrack/projects: List projects with run counts and activity dates.
  • POST /api/localtrack/projects: Create or update a project.
  • DELETE /api/localtrack/projects/{id}: Delete a project and all associated runs.
  • GET /api/localtrack/runs?project_id={id}: List runs for an active project.
  • POST /api/localtrack/runs: Create an experiment run.
  • GET /api/localtrack/runs/{id}: Get run configuration, summary stats, and hardware info.
  • PATCH /api/localtrack/runs/{id}: Update run status (running, finished, failed) and metrics summary.
  • POST /api/localtrack/metrics/ingest: High-throughput batch ingestion of metric logs.
  • GET /api/localtrack/metrics/keys: Discover all logged metric keys.
  • POST /api/localtrack/metrics/query: Fetch time-series metric data with server-side LTTB downsampling and smoothing.
  • POST /api/localtrack/runs/{id}/artifacts: Upload artifact files (config.json, trainer_state.json).
  • GET /api/localtrack/runs/{id}/artifacts: List run artifacts.
  • GET /api/localtrack/runs/{id}/artifacts/{artifact_id}/download: Download an artifact.
  • GET /api/localtrack/projects/{id}/layout: The saved panel arrangement, or panels: null when the project has never saved one.
  • PUT /api/localtrack/projects/{id}/layout: Replace it. 400 if panels is not a list.

Note there is no bare-path alias for any of these — see Routes are namespaced.

localtrack.compare_runs​

The comparison is an agent tool, not just a route. store.compare_runs and POST /api/localtrack/compare had both existed since the store was written, and nothing exposed them to a turn — so an agent asked "did this help?" pulled raw metric series with query_metrics and re-derived the comparison itself.

Worse than absent: training/recipe_tools.py instructs the model, twice and by name, to call localtrack.compare_runs after a sweep. Following the instruction produced an unknown-tool error. test_comparison_agent_tools.py now source-scans the instruction strings — including the note fields handlers return, which no amount of introspecting the tool objects would reach — and requires every <group>.<tool> they name to exist.

The tool is thin on purpose: every judgement (varied vs shared config, last value rather than mean, flagging incomparable runs) stays in the store, so the tool and the pane cannot drift into two different answers to one question. A mixed comparison is reported in a note rather than left in a field the model may not read.

localtrack.workspace accepts {projectId, runNames} and opens on that project with those runs selected. This is what the sweep pane's Compare button sends; it used to call openPanel('localtrack.workspace') with no arguments at all, landing you on whichever project was last open with the sweep you had just run nowhere in sight.

Runs are matched by name, not by id, and that is not a shortcut. A sweep point has three different ids over its life — the script-runner process (SweepRecord.runIds), the training run the script announces (SweepResultEntry.metricRuns), and the localtrack run the mirror mints — and none of them is the next one. The name is the only thing both ends agree on in advance: sweeps.py writes point.label into the generated script, metrics.py names the mirrored run after it (_names), and that is precisely why a point reads learning_rate=0.0002 instead of a hex id. Matching on ids would select nothing, silently — and an empty comparison does not look like a broken link.

Names that match nothing are reported, not dropped: a point whose run never started is a real thing to know, and a comparison quietly missing two of its six points is worse than one that says so.

The delivery mechanism is SET_PANE_PARAMS (see windowing): openPane used to focus an already-open pane and discard the params it was handed, so every deep link worked only while its target pane happened to be closed.

Comparing runs​

compare_runs(run_ids, metric) and the two panel types over it — table and parcoords — are the read a sweep exists for.

Every ingredient has been in this schema since it was written: config_json per run, metrics per (run, key, step). What was missing was a producer that put anything comparable in the first field. The training sweep now declares one config per point, so the table has something to diff.

Three things it does that a raw row dump does not:

  • Splits varied from shared. A fine-tune has forty knobs and an ablation moves one or two; a table repeating the other thirty-eight on every row buries the answer. What was held constant is reported once, underneath.
  • Takes each metric's LAST value, not its mean. "Which run ended up best" is the question; a mean over a descending loss curve mostly measures how long the run was.
  • Flags mixed comparisons. Runs on different backends or tasks are still comparable — you may well want to — but the difference between them is not the axis you varied, and a table that stays silent reads as an ablation result.

The parcoords panel answers the shape question the table cannot: whether the metric moves monotonically with a knob, whether two knobs interact, and whether the best run is a trend or one lucky point. Colour encodes the metric, not the run, because "which of these lines is good" is the question — a categorical palette would mean cross-referencing a legend for every line.

Importing from Weights & Biases​

wandb.import_runs pulls a W&B run's config and metric history in through the same RunMirror, so a run from a rented box charts and compares beside the ones trained here. The history W&B returns is sampled by default and the imported run says so: a 500-point curve standing in for 40,000 must not be presented as the original. See training.