Skip to main content

Module: datasets

The material a fine-tune is made of, as a first-class object. Find training data, look at what is actually in it, register it so a run is reproducible, and build new datasets out of old ones.

Status: implemented — backend in backend/modules/datasets/, frontend in packages/core/src/modules/datasets/.

Why this exists​

A recipe used to hold a dataset name: RecipePane rendered a bare text input and recipes._dataset_call() guessed from the file extension whether it was a Hub id or a path. Meanwhile four separate places in this app already produced or inspected training material —

  • the Hugging Face peek layer in evals (columns and rows without downloading),
  • the Hub browser in lab,
  • evals.export's SFT .jsonl of the cases a model got wrong,
  • trajectories.export's graded agent runs,

— and none of them was reachable from the recipe form. The last two are the sharp end of it: "fine-tune on what the model got wrong" was fully implemented at both ends and joined by a path you had to type from memory.

So a dataset here is a registered thing: a source, a ref, a split, a detected format, and a column map. Registering one is what makes a rerun a rerun, and what lets a sweep's twelve points provably eat the same rows in the same shape.

Sources​

sources.py defines the DatasetSource protocol — deliberately the same shape as the training module's EnvironmentProvider and the database module's driver layer, so there is one contract to learn. Five ship:

idwhat
hubHugging Face, peeked through the datasets-server without downloading a byte
local.jsonl / .json / .csv / .parquet under $HORRIBLE_DATA_DIR/datasets and every training project's data/ directory
exportswhat evals.export and trajectories.export already write
kaggledelegates to the training module's existing Kaggle provider
librarya knowledge library's chunks, as rows

local spans two kinds of root, because a provider fetch does not land in the data dir: _start_fetch writes into <project.root>/data, and project.root is under training.projectsRoot (default ~/horrible/training). A file there is addressed with a project:<project id>/ prefix; files in the data dir keep bare refs, so refs saved when there was only one root still resolve. Prefixes are matched before the bare root, so a prefixed ref can never be shadowed by a same-named directory inside the data dir — and a path segment cannot contain : on Windows anyway. This is also what makes the kaggle source's advice real: Kaggle has no row-preview endpoint (a dataset is a zip of arbitrary files), so peek refuses rather than downloading gigabytes behind a preview button, and points you at fetch-then-peek instead.

library: the direction the graph was missing​

Research files material into a library — research.start files its report, browser.save files a page, arxiv.download files a paper — and the library chunks and embeds it. Then it stopped. The sources were hub / local / exports, so a corpus you spent an afternoon assembling could be searched by the agent and never trained on; research and training met only through a human copying text out of one and into the other.

A library is a vector table, not a file, so LibrarySource is not a _FileSource:

  • search lists libraries, not rows — one ref per library, described by what is in it (3 chunks from 2 sources). The count is the description: a library of four sources and one of four hundred are different propositions.
  • peek reads chunks straight out of the store. No file is written to answer a look.
  • locate materializes the library to $HORRIBLE_DATA_DIR/datasets/library-<name>.jsonl and returns that path, because local: true makes a recipe emit load_dataset('json', data_files=…) and that needs a real file.

locate doing work is deliberate. The alternative is an Export button: one more thing to forget, and one more state to be stale in. The file is rewritten whenever the library holds more chunks than the file holds lines — so a re-run after adding sources trains on the new ones — and not rewritten when they agree, because a recipe render calls locate and re-serialising a large library each time would make opening the form feel broken. Writes land via .part+rename, the llama.cpp download rule: a half-written jsonl is a file load_dataset will happily read the truncated half of.

Rows are one per chunk, not per source: chunks are what the library stores, they are already sized for a context window, and one "source" here may be a whole book. Each row carries source_id/title/url, because a chunk with no provenance is not reviewable — and being able to go and read where a row came from is the entire argument for training on your own corpus.

A plugin adds more with host.add_dataset_source(source) — see the backend plugin SDK. Built-ins win id conflicts, matching training.providers and search.providers.

_read_rows is stdlib for jsonl/json/csv on purpose: peeking is what you do before deciding to train, and making it depend on a project venv that may not have datasets installed yet would put the check behind the thing it is meant to check. Parquet needs pyarrow and says so.

Format detection​

formats.py is the piece that replaces the text_field guess. A fine-tuning dataset is not one shape but a handful of conventions that look alike from a distance and are incompatible up close:

formatshape
chatml{messages: [{role, content}]}
sharegpt{conversations: [{from, value}]}
alpaca{instruction, input?, output}
prompt_completiona prompt column and a completion column — trl's own standard pair, and how nearly every reasoning dataset is spelled
preference{prompt, chosen, rejected} — the only thing DPO/KTO/reward can train
raw_text{text}

Three things it insists on:

It is a verdict with evidence, never a coercion. Detection.reason is a sentence — "conversations holds from/value pairs in 3 of 3 rows" — rendered verbatim in the pane. A wrong verdict without its evidence is indistinguishable from a right one, and the wrong one costs a GPU hour.

Preference is checked first. A preference set also has a prompt column and would otherwise read as Alpaca — and read that way, DPO data trains an SFT objective with nothing anywhere saying so.

ChatML and ShareGPT are told apart by the keys inside the list, which no column-name check can see. Get it wrong and apply_chat_template silently produces empty turns.

Which completion column is chosen is measured, not ranked. prompt_completion covers question/solution (LIMO), problem/solution (OpenR1-Math) and prompt/completion alike — but a reasoning dataset routinely carries two completion-shaped columns, and they mean very different things. GAIR/LIMO has solution, the entire chain of thought, and answer, which is the string "25". Taking the first name off a fixed list would train the model to emit the final number and drop the reasoning: the exact opposite of what post-training on a reasoning dataset is for, with no error anywhere and a loss curve that looks fine. So when several candidates are present the choice comes from measuring the sampled rows, and the measurement goes in the reason — "solution averages 17121 characters against answer at 3" — where it can be overruled on sight.

Its reshape is a rename, never a concatenation: trl trains a prompt/completion pair natively and masks the prompt out of the loss, and flattening both into one text column would teach the model to predict the question as well.

Until this shape existed, a question/solution dataset matched nothing — Alpaca demands the literal column instruction — so it reported unknown, no column map was inferred, and the recipe emitted a config with no dataset_text_field. The run died on KeyError: 'text' minutes in, after the base model had loaded, naming neither the dataset nor the column.

adapt(detection, task) then answers whether a task can eat a shape, and the refusal is the feature: trl will happily accept a dataset with the wrong objective's columns and train on whatever it finds.

Token statistics​

tokens.py answers the one question that is otherwise completely silent: how many examples would max_length truncate? Set it to 1024 against a dataset whose median example is 3000 tokens and two thirds of every answer is discarded. Nothing errors. Loss goes down. The model learns the first third of answers.

It reuses interpretability.tokenizer.Counter rather than counting again — that class already resolves a repo from a model name, falls back to a chars/4 estimate, and knows whether the answer is exact. An estimate is reported as one.

The builder​

builder.py is this app's answer to Unsloth Studio's "Data Recipes": a pipeline of typed steps (load → filter → template → dedupe → sample → synthesize → split → write), because the useful training set is almost never the one you downloaded.

Validate, then preview, then build — the split that makes a node editor usable rather than a slower way to write the same code:

  • validate() answers "is this pipeline coherent" without touching data. A filter on a column no upstream step produces is caught in milliseconds. That check earns the pass its keep: such a pipeline yields zero rows and blames the data.
  • preview() runs the whole thing over the first N rows and reports per-step row counts. A filter that drops everything produces empty output either way; the counts say which step did it.
  • start_build() runs it for real, detached under a semaphore — not on the shared task queue, which is serial and would park every library ingest behind a build (the karaoke download and evals.sweep precedent).

Steps are a closed vocabulary, not arbitrary code. A map step taking a lambda would be more powerful and would also make a pipeline an arbitrary-code document that an agent can author and a plugin can install. Anything the steps cannot express belongs in the notebook, where it is visible.

Synthetic rows​

The synthesize step has two engines: a local model, through the same roster.resolve_provider every agent turn uses, and NVIDIA Nemo Data Designer when the nvidia connector is connected.

Every generated row carries synthetic: true and the model that made it. A synthetic set that cannot be told from a collected one is how a benchmark ends up contaminated by its own training data — and by the time that shows as a suspiciously high score, the provenance is gone.

Panes​

idrolewhat
datasets.browserdocumentSources / Library / Inspect
datasets.pickerwidget, embeddedmounted as a region on training.recipe

The picker is embedded rather than a destination because "which dataset, in which shape" is a question you answer while looking at the recipe, and a tab switch is where the shape check gets skipped.

Agent tools (group datasets)​

search, peek, splits, adapt, register, list, token_stats, build, build_status.

peek and adapt are the two that matter: they are how an agent stops guessing a text_field the way the form used to make a person guess one.

The registry​

registry.py keeps saved datasets in app.db, beside model_lineage and the eval tables — so "which sweeps trained on the dataset I built from last month's eval failures" is a join in the database console rather than a feature request.

Rows are keyed by a fingerprint over the definition (source, ref, config, split, format, column map), not over the rows. Hashing the rows would mean downloading them, which is what the peek layer exists to avoid; and the question a recipe needs answered is "was this the same dataset specification" — which is exactly what changes when someone edits a column map.

Editing a registered dataset re-fingerprints it, deliberately: a column map edit changes what a rerun trains on, so it must change the identity.