Skip to main content

Module: evals — measuring tool calling, and the fine-tune loop

Status: implemented, and it has already found things (see The first thing it measured). Both runners — the in-node one for tool calling and the project-venv one for Hugging Face benchmarks — the graders, the store, the sweep orchestration, the SFT exporter, the REST surface, the agent tools and the evals.hub pane are all in — and a failing case now links straight to what the model was given.

The module exists because of a gap that had been open for a long time: progressive disclosure and TOOL_BUDGET = 38 are mitigations for a small model drowning in tools, and nobody had ever measured whether they help. There was no evaluation surface anywhere in the app — not for tool calling, not for an agent, not for a fine-tune — and no path from "this model fails these tools" to a fine-tune that fixes it.

The design decision the module stands on​

A case runs through the real run_agent_loop — the same function a chat turn and a flow's Agent node run through — with a stubbed browser on the end of it.

There is no second tool-calling loop and no hand-built prompt. What is under test is not "can this model emit JSON", it is can it pick the right tool out of the catalog this app actually shows it, and that catalog is assembled by _select_tools from progressive disclosure, the group guides, the preload heuristics and the budget. A harness that built its own tool list would be measuring a catalog nobody ever ships.

The seam​

_call_frontend_tool registers a future in conn.pending[callId] and then awaits conn.send_json. So an object whose send_json resolves that future is a complete stand-in for a browser — no monkeypatching of orchestrator internals, and nothing to keep in sync as the orchestrator changes.

The connection is still the browser stand-in the loop needs — it owns pending, pending_approvals and the tool manifest, and it is what answers the gate. What it no longer does is decide the tool result: that moved to the hook, for the reason below.

EvalConnection is now an alias: the class itself lives at backend/modules/agent/offline_conn.py as OfflineConnection, because agentpedia's fork needs the identical browser stand-in and importing this module's internals would break the rule that modules do not reach into each other. One proven copy, two callers.

Tools are simulated, never executed​

A case supplies fixtures: what each tool returns. Running the real tool would make a suite destructive (files.delete is in the catalog), unrepeatable (the result would depend on what happened to be open), and slow. What is graded is the choice; the fixture is what lets a multi-step case continue past it.

A tool with no fixture returns a bland {"ok": true} rather than an error — otherwise the model's next move is a reaction to a broken tool, and the case ends up measuring error handling.

The connection alone did not guarantee any of that, and for a while this module believed it did. A tool call can act through three legs and only the first passes through a connection: browser tools are relayed, but backend tools (agent.delegate, agent.ask_peer) and backend-plugin tools are resolved server-side and never touch one. A case whose model reached for agent.delegate really did delegate. It went unnoticed because the shipped suites happen to exercise UI tools — which is the worst shape a destructive bug can take in a measurement harness: it fires only on the runs where a model happens to make that choice, so the models that trigger it are exactly the ones whose scores you cannot trust.

A run now passes run_agent_loop's simulate hook (see agentpedia), which stands in for all three legs at the single point after the permission gate where a call would otherwise act. The meta tools — list_tool_groups, load_tools, use_skill — deliberately still run for real: what they do is shape the catalog, and progressive disclosure is precisely what this module exists to measure.

That also makes the hook, not the connection, the graded record: it is the one place that sees every call whichever leg it would have taken. gated still comes from the connection, because the gate fires before the hook does.

The permission mode is forced​

A gated tool prompts, and a prompt in a headless run either hangs or is denied. Either way the case would be scoring the local machine's saved permission rules rather than the model. So a run forces AUTONOMOUS, auto-approves, and records that a gate fired. An eval whose result depends on who ran it is not an eval.

Grades: why there are six and not one​

A single "did it call the right tool" check is wrong in both directions at once. It fails a model that called the right tool with a harmless extra argument, and it passes a model that fired a tool at a question that wanted an answer. Both are the normal behaviour of a small model.

GradePasses when
exactname and arguments identical
name_onlythe right tool; arguments unchecked
subsetevery expected argument present and equal, extras tolerated (default)
sequencethe expected calls in order, as a subsequence — a model that looked something up first was careful, not wrong
no_callno tool at all
judgean LLM grades the answer against a rubric — declared, not selectable

judge is rejected at validation. The judge needs a provider and graders.py is deliberately pure, so nothing routes it: a case selecting it fails every time it runs, and the failure reads as the model getting it wrong — the one misreading this module exists to prevent. The frontend's picker never offered it; Expect.grade now refuses it too, closing the two paths that bypass the picker (an agent-authored case and a hand-edited .jsonl). It stays in the Grade vocabulary because a result row from an older run can carry it, and the grader keeps its refusing branch as the backstop under the validator.

no_call is first-class. The dominant failure of a 2B model with 38 tools in front of it is reaching for one when the question wanted an answer, and a suite of positives only would score that model highly while it made the app unusable. Around a third of the starter cases are negatives for exactly this reason.

Arguments are compared after a light normalisation — whitespace, case, and 1 / 1.0 / "1" — because a model that formatted an argument differently picked the right tool and the right argument, and being strict there measures JSON formatting rather than tool selection. Two deliberate exceptions: True is not 1 (bool is a subclass of int in Python and True == 1.0, so booleans are tagged before the number branch), and an explicit null is not a missing key.

Exposure is a property of the case​

expose is on the case, not the run, because "can this model pick open_pane out of 38 tools" and "out of 6" are different questions and a suite is allowed to ask both. It is also what makes results comparable: if exposure came from whatever the node's settings happened to be, two runs of the same suite would not be measuring the same thing.

ModeThe model sees
progressivethe shipped path — a small core plus list_tool_groups/load_tools, preload as a head start
allevery group at once, still capped by TOOL_BUDGET. The comparison that says whether disclosure earns its keep
explicitonly the named groups, and it cannot load its way out — for isolating one capability

What the budget cut is recorded​

A tool the budget keeps off the list is indistinguishable, from the verdict alone, from a tool the model chose not to call. So each result carries tools_dropped: the tools in the case's scope that _select_tools cut, sorted. Found live, and the reason it exists: an all-mode case expecting scratch.open failed because 38 slots filled with backend tools first — browser-pushed tools come last in the dynamic list, so they are the first casualties — and the row said tools_offered: 38, tools_dropped: []. The field had been declared, stored and rendered for a long time; nothing ever filled it.

"Dropped" means cut from the largest catalog the case reached:

Modetools_dropped
allthe cut of the one list the run started with
progressivethe cut of the final selection. The catalog grows as the model calls load_tools, so a tool in a group it loaded, which the budget then refused to show it, was dropped just as surely as one cut up front. A group the model never loaded is out of scope, not dropped — that is disclosure working, not the budget
explicitalways empty: that mode is uncapped by design

One recomputation after the loop is enough, because the selection only grows by appending groups and the cap slices from the tail: a tool past the budget in an early round is still past it in the last.

When a failing case's expected tool is among them, the result's detail leads with "expected tool … was dropped by the tool budget and never offered to the model", and the Results pane says the same in place of the generic budget line. That failure measured the catalog, not the model, and the red verdict beside it says the opposite. A pass is left alone — a case that succeeded without the tool was not held back by its absence.

Bundled suites are data, not code​

The starter cases began life as Python constructors in a seed.py, copied into every new suite. That was wrong, and for a reason worth naming: it gave the module two authoring formats — .jsonl for you, Python for us — and the one we used was neither reviewable as data nor editable in the app that exists to edit it. It is the same mistake hassault explicitly refuses when it keeps its maps as hd_*.json brush files rather than committed .cgz binaries.

A bundled suite is now a plain .jsonl in backend/modules/evals/suites/, in exactly the format you author in, and it resolves beside your own — the mapsource.py rule: bundled content is an addition, never a prerequisite, and the two catalogs cannot shadow each other. A bundled id is prefixed bundled:; a user id is 12 hex characters from uuid4, so the two spaces cannot collide however you name yours.

They are read-only. Writing would edit a file inside the repo — you would lose it on the next pull and it would show as a dirty working tree in the meantime — so a write returns 409 and names the way through: fork it. fork_suite copies the cases rather than the file, so a fork is normalised through the same parser everything else uses and a suite with a broken line does not carry it forward.

A bundled id arrives from a URL, so bundled:../../secrets must not become a path. path_for resolves through a catalog of known slugs rather than joining the id onto the directory — the same reasoning that makes is_managed gate deletion in the llama.cpp module.

Creating a suite no longer seeds anything: a new suite is empty and yours. The starter is there to fork, read, or run as-is.

What is in it​

suites/starter.jsonl — the failures a small model running this dashboard has, which are not the ones a generic tool-calling benchmark covers: the layout floor, negatives (including one preloaded with the tempting group), discovery through load_tools, multi-step cases where the second argument can only come from the first tool's result, and a context case where the answer is already in the workspace snapshot. Blank lines and # comments are skipped by the loader, so the file keeps its section headers and a long prose header explaining itself.

It names real tools. If a tool is renamed the cases fail loudly, which is correct — a suite that silently kept grading against a tool nobody ships would be worse than no suite.

suites/ml-expert.jsonl grades the knowledge base's ML Expert agent against its contract, not its ML knowledge. It checks that the agent:

  • searches the user's material first;
  • cites [n] with links;
  • writes math in LaTeX;
  • says plainly when the material is silent, and labels what came from elsewhere;
  • does the catalog chores (save, mark done, course progress, import) with the right tool.
  • answers the question instead of acting unasked. Found live on kimi-k3: told to "offer to save" a source, it saved it and replied "Done — I saved…" with no answer. mlx-answers-instead-of-saving gives kb.save a fixture so an unasked save succeeds, and the judge reads the reply that follows.

It has three negatives. Every kb.* result is a fixture of the shape kb.search really returns, so it needs no imported content. The judge sees only the prompt and the answer, so each judge rubric restates what the fixture passages said. A test checks that every tool the suite names is registered and inside the agent's tool groups.

Which agent answers​

A case is graded against one agent's prompt and tool groups. RunTarget.agent_id picks the roster agent for a target. When a target names none, the suite's own agent_id applies; a bundled suite declares it in bundled.AGENT_FOR, and ml-expert runs as ml_expert. Otherwise the main agent answers. An agent the roster doesn't know fails the run, not every case. A non-main agent is part of the run's default label (model · ml_expert), so the same model answering as two agents gets two columns. A fork does not inherit the suite's agent: set agent_id on the target.

The first thing it measured​

Two results from the first sweeps this module ever ran, against llama-3.2-3b-instruct on the starter suite. Both are the module justifying itself, in opposite directions.

It failed the case author before it failed the model​

The first run scored 5/12, and three of the failures were the suite being wrong:

FAIL layout-open-terminal expected open_pane; called show(target=Terminal)
FAIL layout-open-named-pane expected open_pane(id=settings); called show(target=settings)
FAIL multistep-uses-the-tool-result called close_pane(id=e1) but expected close_pane(instanceId=e1)

The layout core's own show description says, in as many words, that you do not need open_pane to reach something and that show is the verb for any "show/open/go to X" request. And close_pane takes id, not instanceId. The model had done exactly the right thing three times and been marked wrong for it.

Corrected, the same model on the same suite scores 8/12. The lesson is worth stating as a rule, because it will happen to everyone who writes a case: an expectation that contradicts a tool's own description grades the case author, not the model. Read the tool declaration before writing what you expect of it.

Progressive disclosure did not work for this model​

The four remaining failures are real, and they cluster:

CaseTools offeredWhat happened
discovery-github-repos12called nothing at all
discovery-library-search12show(target="notes about vector databases"), then files.list(path="library/notes")
negative-greeting12called show twice, over 9 rounds, at "hey, are you there?"
negative-ambiguous-reference30show(target="People") at "close it"

Both discovery cases failed — the two whose capability was reachable only by calling load_tools. This model never called it. It has, in practice, no access to the library or GitHub at all, no matter how well it calls the tools it can already see. Note the second row especially: rather than loading the group, it tried to reach the library by guessing at the filesystem.

That is precisely the question the module was built to ask — progressive disclosure and TOOL_BUDGET were mitigations nobody had measured — and the answer for a 3B model is that the mitigation has a floor it does not clear. Whether the fix is a better load_tools description, keyword preloading, or a fine-tune is now a question with a number attached to it.

Storage: a suite is source, a result is data​

Cases live in $HORRIBLE_DATA_DIR/evals/suites/<id>.jsonl — you edit them in the editor, diff them and commit them. Runs and results live in app.db (eval_suites, eval_runs, eval_results), so the database console's built-in app connection and dash can query them with no new API: "which cases does every model fail" is a GROUP BY, not a feature request.

A malformed suite is still listable and openable — the case count is taken leniently — because one bad line must not take out the suite list and leave no way to reach the file and fix it. The parse error, with its line number, is reported by GET /suites/{id}/cases on a 200, since the pane needs to render the suite with the error rather than show a toast over nothing.

A duplicate case id is refused outright: results are keyed by (run, case), so the second one would overwrite the first and the suite would silently be one case shorter than it looks.

Sweeps​

A sweep runs detached under its own semaphore, not on the shared task queue. That queue is serial, and parking every library ingest behind a 200-case × 6-model sweep would make the eval module something you avoid running — the same reasoning as karaoke downloads. The limit is per target, not per case: most local setups have one model server, so more concurrency would not finish sooner, it would just make every per-case duration a measurement of queueing.

Progress broadcasts on the evals /ws channel rather than being replied to a request. A sweep outlives the pane that started it, so closing and reopening the pane does not lose the run, and the agent's tools see the same state with no pane open.

Two failures that are deliberately distinguished:

  • A target that cannot be resolved fails as a run, not as N failed cases. "Your endpoint is wrong" and "this model gets everything wrong" look identical on a scoreboard otherwise.
  • A provider error mid-sweep is a failed row carrying the error, not a dead sweep: one unreachable model must not abandon a sweep across five others. An errored case is forced to passed=False whatever the grader made of an empty call list — most obviously for no_call, which an exception would otherwise score as a pass.

Stopping one. A sweep is detached, so "what is running" cannot be answered from the pane's own state — the Run section asks the node (GET /sweeps) and offers Stop on each. The task is cancelled rather than a flag being set for the loop to notice: the time is spent inside a provider call, so a flag would not be read until the case in flight finished, which on a stuck endpoint is the timeout and the whole reason to press stop. Cancellation then has to close the rows out by hand — a CancelledError is raised inside whichever await is in flight and by the time it reaches the sweep's own task the gather has discarded which children were running, so run ids are handed up as each row is created. Unfinished runs become cancelled, not failed: "you stopped it" and "it broke" are different facts and the scoreboard reads a failed run as a signal about the model. A target that finished before the cancel landed keeps its verdict.

Aggregates go to localtrack (one run per sweep) so the existing charts do the plotting, into a project derived from the suite rather than one evals bucket for everything — localtrack charts a project's runs against each other, so pouring a tool-calling suite and an MMLU benchmark into one project plots two unrelated pass rates on the same axis. Wrapped in swallows throughout: localtrack is a reporting destination and a tracking failure must never cost a run the results already in app.db. The mirroring itself is now localtrack.mirror.RunMirror, shared with the training module — the four moves and the swallow-everything posture are identical and only the metric names differ, and a second copy of "wrapped in swallows throughout" is exactly the kind of duplication where one side later gains a guard and the other does not.

Sweeping a local GGUF: the server is borrowed, then given back​

GET /api/evals/targets offers the node's configured agent provider and every GGUF in the llama.cpp catalog, managed origin first — because after a fine-tune the file you want to score is the one this node just wrote. It used to offer llama.cpp installs: builds, emitted with an empty model, so the one target that mattered on the machine was the one you could not pick.

A target therefore names a file, not just a model id (RunTarget.model_path). llama-server holds one model at a time — spawn refuses while a server is running — so "score my fine-tune against its base" cannot be two models answering in parallel; it is one, then the other, with a load in between. evals/llama_target.py does that borrowing, and three rules there are bugs if dropped:

  • One llama.cpp target at a time. Targets otherwise run concurrently, which for every other provider is free parallelism. Here two of them would race over a single process and each would score the other's weights. The lock is held for the whole target, not just the load.
  • Put back what was evicted, from the recorded SpawnSpec rather than from defaults. The evicted server may be the user's chat provider, and restoring from defaults would quietly reset gpu_layers, drop extra_args and shrink context_size — leaving their machine slower with nothing said.
  • A load that fails, fails the run. wait_ready returns False rather than raising when a model is too large to load, and a target that scored zero because no server ever came up must not sit on the scoreboard next to a model that genuinely got everything wrong.

A GGUF the server already has loaded is used in place: no stop, no reload, and no restore afterwards, because nothing was taken. The endpoint is read after the spawn — the port is chosen then, and _resolve_target now asks _endpoint_for rather than falling back to :8080, which is nothing but a closed port whenever llama-server took an ephemeral one.

A target this node trained also carries its provenance. GET /targets joins the model lineage table in one query for the whole catalog (a lookup per row would be a query per file on every page load), so a converted fine-tune reads "— fine-tune of meta-llama/Llama-3.2-3B" and exposes baseModel / projectId alongside it. That is the whole of "score my fine-tune against its base": the base is a field rather than something you have to remember. Absent for every model the user merely downloaded, which is the normal case and renders as no provenance rather than as a guess.

Each target also carries its architecture from the GGUF header. That is shown in the picker, never filtered on: it is what marks the nomic-bert embedder or a TTS model living in the same directory, while a header that could not be parsed is no evidence at all — hiding a model on that basis would present a guess as a measurement.

A sweep needs a browser attached​

The frontend tool catalog only exists on a live connection — panes push their agentTools onto the socket and there is no server-side copy. A sweep with nothing connected would grade every model against backend tools only and score them all at zero on anything UI-shaped, so it refuses with that reason instead. When several windows are connected, the richest manifest wins, so a second window still registering its panes cannot hand the sweep a shorter catalog than the one you are looking at.

Peer targets​

A RunTarget may name a node. The Run section picks one for the whole sweep ("run on"), offering only peers advertising the inference capability — a lease on a peer with no llama.cpp of its own would be granted and then fail every case, which on a scoreboard is indistinguishable from a model that gets everything wrong.

The run row records the node it ran on, and shows it only when that was somebody else's machine. Recorded, not inferred: a peer target reaches its lender through a local tunnel port, so the endpoint on the row reads 127.0.0.1 for a remote run exactly as it does for a local one. The sweep then takes a compute lease on that peer and points the target at the tunnel endpoint — everything downstream is unchanged, because a borrowed endpoint is just an endpoint. That is why this is plumbing rather than a second runner.

The point is not remoteness. _llama_lock and _target_semaphore exist because this node has one llama.cpp process to fight over, and that constraint does not apply to somebody else's machine. So a peer target holds neither: N peers means N processes, which is the entire speedup. Holding the semaphore for a peer target would serialize away the whole reason for distributing a sweep.

Two rules carry over unchanged from llama_target.serving:

  • A lease that cannot be granted fails the run, not the cases. "Your friend said no" and "this model gets everything wrong" must not look alike on a scoreboard — the same reasoning as an unloadable GGUF.
  • The lease is released in a finally. A sweep that dies must not leave a friend's GPU reserved until the lease expires on its own.

Running on a friend's agent​

A peer target borrows a friend's GPU; grading, the catalog and the harness stay here. The Run section's On a friend's agent block does something different: it hands the cases to the friend's node, which runs them through its own run_agent_loop — its skills, its MCP servers, its model — and reports each verdict back. That is the only way to ask "how does their setup do on my suite", and it is also why none of it can be verified from here. backend/modules/evals/fabric.py owns the wire:

Two gates, because it spends somebody else's tokens​

  1. evals.acceptRemoteSuites, default off. The handler is registered anyway, so an offer to a node with it off comes back as a reasoned refusal on the run row (declined: … evals.acceptRemoteSuites is off) — silence would read as a network fault. The capability is advertised with no live detail on purpose: "accepting" changes after the handshake, and a stale badge would be worse than a refusal.
  2. A person accepts each offer. The offer shows the case count and a cost floor — the case text at ~4 characters a token, priced through the node's own pricing table, free on a local model and price unknown rather than $0 when there is no price. It is a floor and says so: every round also resends the system prompt and the catalog. network.remoteAgentMode is deliberately not reused; it answers a different question, and "read-only" means nothing for an N-case suite.

session.info.trusted comes first on every handler, and every message about an offer is checked against the node it was offered to — a friend who knows a run id cannot report results into it.

What a friend's node will not run​

Only cases run_case executes in-process with simulate, so no tool acts on their machine: every call gets the offerer's fixture. hf_benchmark cases (a venv, and for code_exec model-written code) and judge grades (a second model call nobody estimated, against a judge the offerer names) are refused on the runner — the offerer leaves them out too and says how many, but that is a courtesy, not the gate.

Honesty about what came back​

  • The friend's node reports fingerprint.compute() and its model name. The envelope is signed, which proves their node said it, not that it is true, and the hash covers skills and MCP tools — not model weights. A friend reporting qwen3:8b could be running a fine-tune.
  • So the run is stored with attestation = 'peer'. It ranks beside local runs, never merged into them: the ranking row says peer-reported, and Compare's harness banner says the catalog was reported — even when the hashes match, which then means only "they reported the same catalog".
  • case_hash is computed here, from this node's case, never taken from the reply: the question asked is the offerer's to vouch for.
  • A reply of done with cases missing is recorded as failed, naming how many never reported; a run that answered half a suite must not rank beside one that answered it all.

What leaves the friend's node is bounded and scrubbed with the trajectories redactor (outbound.py): the model's answer is clipped to 4,000 characters, call arguments redacted, turn_id dropped. Each progress message is kept under 256 KiB, because one frame over 1 MiB closes the dialing side's whole link (see trajectories).

An open offer lives in memory on both nodes. A restart of the offerer closes its open peer runs as failed with that reason, rather than leaving them running forever.

Contributions to the layout shell​

KindIdNotes
Panelevals.hub"Evals"; singleton, role document
Sectionevals.hub → suitesDefault, key s
Sectionevals.hub → runKey r
Sectionevals.hub → resultsKey e
Sectionevals.hub → compareKey c
Commandevals.openEvals: Open

One pane with four sections rather than four panes: Suites, Run, Results and Compare are four views of the same object, and splitting them would mean four openers and four copies of "which suite are we talking about".

The Suites section carries a case editor — id, prompt, grade, exposure, expected calls — plus Fork, Edit and Delete. It is deliberately shallow: anything richer (fixtures, history, workspace context) is a JSON field, because building a form for arbitrary tool arguments would be building a worse editor than the one already in the app. A no_call case hides the expected-calls field entirely rather than leaving it there inviting input that would be ignored.

None of the three authoring paths is privileged: the form, the agent's evals.addCase, and opening the .jsonl in the code editor all write the same file, which is what keeps a suite reviewable and diffable.

The rows are a shared component, not four hand-drawn lists​

Every section of this pane is a list of records with an identity, some figures and a verdict — cases, sweep targets, runs, results — and all four had independently converged on the same undesigned shape: one font weight throughout, colour carried on the whole perimeter, and numbers set as prose. They also referenced theme tokens that do not exist (--bg-primary, --text-primary, --border-dim, --ok), so every one of them silently fell through to a hardcoded fallback and ignored the theme entirely.

They now compose DataList / DataRow / PickRow from packages/core/src/DataList.tsx, which splits a row into three separable jobs:

  • Identity (hd-row-title) — uppercase, heavy tracking. What the row is.
  • Telemetry (hd-row-meta) — monospace, muted, divided by hairlines. Every id, count, duration and percentage. Never the title's treatment, so a row scans as label-then-figures at a glance.
  • Verdict (data-kind) — a 2px rail down the leading edge and a matching corner notch. An accent edge, not an accent perimeter: the row stays a row instead of becoming a card.

The type contract is the design. There is no color or style prop — a caller that could pick a colour directly is a caller that will eventually pick one that means nothing — and meta takes an array rather than a pre-joined string, because the hairline dividers cannot be drawn between cells that were concatenated first.

Three details that are decisions rather than styling:

  • The Models picker is a grid of tiles, not a column of checkboxes. Those items are peers being chosen between for one sweep, and the stacked-label form is exactly the shape that reads as an unstyled form. PickRow keeps the native <input type="checkbox"> — hidden, but still doing the focus, announcement and space-bar work — and draws the square beside it. Replacing the input with a role="checkbox" div is the version that looks the same and loses the keyboard.
  • A case row is kind="info" with no mark. A case has no verdict: it has not been run. A tick there would claim something that was never tested.
  • A run has three states, not two. runKind returns info while a sweep is still going — drawing it as a failure is how a sweep at 3/50 comes to look broken thirty seconds after it was started.

The entrance stagger is capped (STAGGER_CAP, 12). The delay is per item, so without a cap a suite with two hundred results is still arriving four seconds after it rendered. RollingNumber is seeded at its final value with a timeout that snaps to it: requestAnimationFrame does not fire in a backgrounded tab, and a counter reading 0 for a figure that is 62 is worse than no animation at all.

Backend surface​

backend/modules/evals/, mounted at /api/evals.

RoutePurpose
GET /api/evals/suitesEvery suite, with case counts
POST /api/evals/suitesCreate one, empty
POST /api/evals/suites/{id}/forkCopy a suite into a new one you own
GET /api/evals/suites/{id}/casesThe parsed cases, or a line-numbered parse error on a 200
PUT /api/evals/suites/{id}/casesRewrite the file (the case editor and the authoring tools). 409 for a bundled one
DELETE /api/evals/suites/{id}Forget a suite. The file survives unless remove_file=true. 409 for a bundled one
GET /api/evals/runsRecent runs, optionally by suite
GET /api/evals/runs/{id}One run and every case result
POST /api/evals/runsStart a sweep. Returns immediately
GET /api/evals/sweepsWhat is running on this node right now, with target labels
DELETE /api/evals/sweeps/{key}Stop a sweep. Targets it already finished keep their results
GET /api/evals/leaderboardA suite's runs: ranking, case matrix, universal failures
GET /api/evals/leaderboard/diffWhat one run fixed and broke relative to another
GET /api/evals/datasets/splitsThe (config, split) pairs a Hub dataset offers
POST /api/evals/datasets/peekColumns and a few real rows, without downloading it
POST /api/evals/datasets/compare-previewWhat a case would compare, for one row, plus everything wrong with it
GET /api/evals/datasets/presetsKnown-good benchmark blocks for datasets whose answer column is a trap
POST /api/evals/exportsTurn a run's results into an SFT dataset on disk
GET /api/evals/exports/previewThe first few examples, without writing a file
GET /api/evals/targetsModels this node could sweep — the agent provider plus every catalog GGUF — so the launcher is a picker not a form
GET /api/evals/remote/friendsConnected friends whose node understands suite offers
POST /api/evals/remote/offerOffer a suite to a friend's agent; 409 with the reason when refused
DELETE /api/evals/remote/runs/{id}Withdraw an offer, pending or running
GET /api/evals/remote/offersOffers friends sent this node, and whether it is accepting
POST /api/evals/remote/offers/{id}/acceptAccept one — needs a browser attached, like a sweep
POST /api/evals/remote/offers/{id}/declineDecline one; the friend's run row records it
POST /api/evals/remote/offers/{id}/stopStop a friend's suite running on this node

Agent tools​

Group evals — name prefix and declared group= identical, since an omitted group= charges the schema to every round of every agent.

ToolPurpose
evals.listSuitesWhat suites exist, with case counts, source and readOnly
evals.createSuiteA new, empty suite to add cases to
evals.forkCopy a suite — how a bundled one becomes editable
evals.addCaseAppend one case. Refuses a duplicate id
evals.removeCaseDelete one case by id
evals.runStart a sweep; returns immediately
evals.exportTurn a run's results into a fine-tuning dataset
evals.resultsRecent runs, or one run's failing cases and why each failed

Everything that writes is side_effect=True and so goes through the permission gate — a read-only tool skips it entirely, which would mean an agent editing your suites with no prompt.

evals.results returns failures only — a scoreboard of 200 passing rows is the one thing an agent's context does not need.

Why the agent may author cases​

An earlier version withheld the authoring tools on the grounds that the thing under test should not write its own exam. That was too broad. Drafting twenty cases from a set of tool descriptions is exactly the tedious work worth delegating; what matters is that a case is a claim about correct behaviour and that you review the claim — and since a suite is a .jsonl in your data dir, reviewing means reading a diff.

Two guards make that safe rather than merely convenient, both aimed at the failure mode an authoring agent actually has, which is enthusiasm rather than malice:

  • A bundled suite cannot be written to at all. The refusal names fork so the model can proceed instead of retrying.
  • addCase refuses a duplicate id rather than replacing the case. Results are keyed by (run, case), so silently overwriting a case somebody wrote is the one unrecoverable thing an authoring tool can do.

listSuites reports source and readOnly for the same reason: it saves the model a failed write and a wasted round. addCase also accepts the case as a JSON string as well as an object, because small models routinely stringify an object-typed argument and a retry round costs more than a json.loads.

Runner B: benchmarks in a project venv​

Evaluation splits into things that must run inside the node and things that must run in a Python venv, and pretending either covers both is where this would go wrong. Runner A measures this node's tool calling and has to run in-process. Runner B (runner_project.py) runs anything needing datasets, evaluate or an lm-eval-style harness — torch-class dependencies that must never become core deps — inside a training project's venv, installed through envs.install (uv pip install --python) because a uv venv has no pip at all.

Both write to the same eval_runs / eval_results tables, so a suite that mixes tool-calling cases with a benchmark still adds up to one scoreboard. A case routes to Runner B by carrying a benchmark block; the venv is prepared once per sweep and only if a benchmark is actually present, so a pure tool-calling sweep never waits for datasets to resolve. If the environment cannot be prepared, the benchmark cases fail with that reason and the tool-calling half of the suite still runs — losing it because uv is missing would be a poor trade.

That project is working storage, and says so​

ensure_project goes straight to projects.create_project, not through the training create route — so it gets a directory and a venv, but no scaffolded main.ipynb (the route is what asks a provider for starter cells), and the venv holds only requirements_for(cases): no ipykernel, no horrible_train.

That is correct for a benchmark runner and wrong for a notebook. Since the project lands under the training projects root, the training pane listed it as an ordinary project with an Open notebook button whose only possible outcome was an error — and the error it produced was a bare FileNotFoundError, whose str() is only the path, rendered beside a kernel badge still reading "starting".

So these projects carry ProjectModel.owner = "evals". The training pane marks them rather than hiding them — the same call the bundled suites above make — and disables the authoring actions with the reason in their tooltip. Delete stays enabled: a project quietly missing from the list is a directory eating disk that you can neither see nor remove. Projects created before owner existed are stamped on the next sweep, so the dead button does not survive for anyone who already ran one.

A script, not a notebook​

The design note said "a scaffolded notebook". A script is what shipped, and the reason is worth recording rather than quietly changing: a sweep is a batch job, and the notebook path is built around an interactive kernel session tied to a websocket connection (TrainingKernelManager._open takes a conn). Driving that headlessly to run twelve rows would mean owning a session lifecycle for no gain.

The readability the notebook was chosen for is kept. The script lands in the project as evals/<case-id>.py — one file per case, so a suite of five benchmarks leaves five readable scripts rather than one overwritten four times — and it is an ordinary file you can open, edit and run yourself. envs' blocking Popen on a thread is reused as-is, already the Windows-safe spawn pattern.

Three properties the generated harness has to have, and they pull against each other: it must be readable (it is how you learn what the libraries do), it must have no dependency it does not need (the model is called over plain urllib, not httpx or the openai client, so a benchmark needing only datasets installs only datasets; evaluate is imported lazily and only for metrics the harness does not implement itself), and it must be machine-readable on the way out — it prints one JSON object on a sentinel line, because parsing a score out of prose is how a harness starts reporting numbers nobody can reproduce.

The job is embedded as a JSON literal rather than interpolated field by field: a dataset id or a template containing a quote would otherwise end the string and produce a script that fails to parse.

Answer extraction, and the 0.000 that was not the model's fault​

The first real benchmark — six rows of GSM8K against a local 3B — scored 0.000. The model was fine. GSM8K stores the worked solution in its answer column and marks the final answer with a separator:

Janet sells 16 - 3 - 4 = 9 duck eggs a day.
She makes 9 * 2 = $18 every day at the farmer's market.
#### 18

Grading against that column as-is asks whether the model reproduced the dataset's own prose, which no model does. So a benchmark carries target_regex and prediction_regex — applied to the reference and the reply respectively, first capturing group wins. With a #### pattern and a last-figure pattern, the same model on the same rows scored 1.000.

A pattern that does not match leaves the value untouched rather than returning empty: degrading to "compare the whole thing" is recoverable and visible in the samples, whereas silently comparing against an empty string scores zero and looks like a model failure. The samples record both the raw reply and what was actually compared, for the same reason — a regex that matched the wrong thing is otherwise invisible, and it is the single most likely cause of a surprising zero.

normalise also strips thousands separators and a trailing currency symbol, so $70,000. and 70000 are the same answer. No dataset agrees on which to store.

Unknown fields are an error​

Pydantic ignores an unknown key by default, which for hand-authored JSONL is the worst available behaviour: target_regexp instead of target_regex silently does nothing, the run scores zero, and the obvious conclusion is that the feature is broken rather than that the key is misspelled.

This was not hypothetical — it is how the regexes above were lost the first time. A case written against a newer schema than the running backend had two fields dropped on the way in, and the only symptom was a benchmark scoring 0.000 again, for a completely different reason. Every case model now sets extra="forbid", so an unknown field is a 422 naming the field. The cost is that an older backend refuses a suite written for a newer one; that is the right trade, since a loud refusal beats a silent misgrade and SuiteFormatError already reports the line number.

Authoring a benchmark, without authoring the zero​

The case editor's benchmark half exists to stop you writing the case that scores zero. Every benchmark this module got wrong was wrong the same way — the template named a column the dataset does not have, or the answer column held the worked solution rather than the answer — and both look exactly like a bad model until ten minutes of run time have been spent.

So the form is built around the dataset rather than around the schema.

Peek. first-rows from Hugging Face's datasets-server returns the columns and a few real rows without downloading a byte. The answer-column field becomes a picker, the template gets +{column} buttons, and the config the Hub resolves is adopted — gsm8k has no default config, and omitting it fails in a way that reads as "the dataset is broken". A gated or unreachable dataset degrades to free-text fields with a note rather than blocking the form.

Presets. GSM8K, MMLU, HellaSwag and TruthfulQA ship as known-good blocks, because each has a trap: GSM8K's #### separator, MMLU and HellaSwag storing the answer as an index into a list column. A preset is cheaper than a paragraph nobody reads at the moment they need it, and backend/modules/evals/presets.py is that paragraph anyway for the ones that are not covered.

The comparison preview. The point of the whole form: for a real row, it shows the rendered prompt, the raw reference, what each side reduces to after extraction and normalisation, and whether that row would score. Below it, every problem it can see — a template naming a missing column, a regex that did not match, or the one that matters most:

⚠ no target_regex, so the whole answer column is the reference — and it looks like the worked solution rather than the answer. Try target_regex = ####\s*(.+)

That warning is the module's own history. The first real benchmark scored 0.000 with no regex at all, so there was no failed match to complain about — an earlier version of this check missed it for exactly that reason, and it now fires on the shape of the reference instead. It stays quiet when the answer column really does hold just the answer, because a warning people learn to ignore is worse than none.

The preview cannot flatter the case​

extract and normalise are imported from the harness, not reimplemented. They are real functions in harness.py and are inlined into the generated script via inspect.getsource, so the preview, the script, and the score all run the same code. A TypeScript reimplementation in the form would have been the obvious way to make the preview instant, and the thing it could quietly disagree about is what the score means — which is the one disagreement this module cannot afford. The preview is a backend call for that reason alone.

Validation before the run, not during it​

Save case refuses a benchmark with no dataset, no answer column, or a prompt template containing no {column} at all — the last because every row would then get the identical prompt, which is a ten-minute run that tells you nothing. Switching a case between tool_call and hf_benchmark keeps both drafts, and saving clears the half that does not apply so a stale benchmark block can never ride along on a tool-calling case.

The fine-tune flywheel​

export.py turns results into a supervised fine-tuning dataset — the half that makes a failing scoreboard actionable. Everything downstream already exists: a training project, the recipe form's SFTConfig/LoraConfig, convert.py → GGUF into llama.cpp's managed dir, serve, re-run the identical suite, compare.

It exports the ideal trajectory, not the model's​

The obvious design — record what the model did and train on it — is wrong in both directions. For a case the model failed, its trajectory is the mistake you are trying to remove. For one it passed, its trajectory and the ideal one are the same thing anyway. So an example is built from the case: the prompt it was given, the tools it was shown, and the call expect says was correct.

That has a consequence worth stating plainly: a case is a claim about correct behaviour, and exporting turns that claim into training data. A wrong expectation does not merely mis-score a model — it teaches one. This is why "read the tool declaration before writing what you expect of it" matters beyond the scoreboard.

Two modes, and they are different jobs. correct exports what a run got right — distillation: run a strong model, train a small one on it. repair synthesises the ideal trajectory for what it got wrong — targeted correction. both is the usual choice once there is a reference run to draw answers from.

The trajectory includes the tool results the fixtures would have produced. An example that stops at the call teaches a model to call and never to use what came back, which is what a model looping on one tool looks like.

The two traps, as code​

One system message. The runner assembles several (the agent prompt, the skills catalog, the group guides) because that is the production shape and the provider seam flattens them. A chat template does not — a strict Jinja template raises on a second system message — so _flatten_system merges them before anything is written.

Nothing is pre-rendered. Examples are written as messages + tools, never as a rendered string. The tokenizer's own apply_chat_template(messages, tools=…) is what the serving path uses, and rendering here would bake in this model's template and silently mistrain anything else — the exact failure --jinja exists to avoid at serving time.

What it refuses to invent​

A no_call case expects prose, and the case does not contain the prose. So repairing one needs a reference_run_id — a stronger model's run of the same suite, whose answer is borrowed. Without one those cases are reported as skipped, never invented: a fabricated answer is training data that teaches a model to say something nobody checked. A case this model already answers correctly needs no stronger model; its own answer is reused.

Every skip names the case and the reason. An exporter that quietly drops a third of a suite produces a dataset whose coverage nobody can account for.

Exports land in $HORRIBLE_DATA_DIR/evals/exports/. A relative out path is resolved and then checked to be inside that directory — an export name arrives in an HTTP body, and joinpath would happily follow ../...

Compare: three questions, not one​

A per-run pass rate is a number you cannot act on, so the Compare section answers three different questions. Answering only the first is what makes a leaderboard decorative.

Who is ahead. The ranking, with a bar rather than a bare percentage — the gap between 42% and 67% is the thing being read, and a column of numbers does not show it. Average rounds rides alongside, because a model that passes by taking four rounds where another took one has not passed the same way.

What changed. The fine-tune question, and deliberately not derived from the totals: a run that went 8/12 → 9/12 by fixing three and breaking two is a different event from one that fixed one and broke none, and the totals cannot tell them apart. So the diff names the cases in each group — four groups, not two, because two of them are not about the model at all. A case whose content hash changed is changed, and a case where either run hit a provider error is errored. Both would otherwise land in fixed/broke and read as the model getting better or worse. That second one is not hypothetical: the first two-run sweep after this shipped had LM Studio return a 500 on two cases, and the diff called one of them a regression until errored existed. The corrected reading — no fixes, no regressions, two errors — is what a reproducibility run at temperature 0 should say.

What nothing passes. The one this module keeps needing. Three separate times a case here was wrong rather than the model — an expectation contradicting the tool's own description, a GSM8K reference containing the whole worked solution — and the signature is every run failing the same case. It gets its own callout saying so, because a universal failure is a prompt to go and read the case, not to go and get a better model. It needs at least two runs before it fires: one run failing a case is not evidence of anything.

The case matrix renders three states, not two. A run that did not attempt a case has not failed it, and drawing both as a cross would invent regressions out of a case_ids filter.

Comparability is not free​

Two runs are only comparable if they asked the same questions, and two things break that invisibly if you compare on totals:

Different case sets. A run started with a case filter attempted fewer cases; 5/5 beats 8/12 on percentage and means nothing. Every comparison is computed over the intersection of cases both runs attempted, and the pane says how many were left out rather than quietly dropping them.

The suite changed underneath. This is the subtle one, and it is a mistake this module has already made. Case ids survive an edit, so layout-open-terminal before and after someone corrected its expectation is the same id and a different question. Comparing them shows a model "fixing" a case that was actually fixed by editing the exam.

So a result now carries a case_hash — a hash of everything that decides whether an answer is right (prompt, history, context, exposure, expectation, fixtures, benchmark block), excluding note and tags because rewording why a case exists does not change what it asks. It is the spec_id precedent from the hitbox module: a content hash cannot be forgotten the way a hand-maintained revision can. Where two runs disagree about a case's hash, the diff reports it as changed rather than as a fix or a regression, and the matrix marks the row.

Rows written before the column existed have an empty hash, which is reported as "cannot tell" — never as agreement. The column is added to existing databases with an explicit ALTER, because CREATE TABLE IF NOT EXISTS does nothing to a table that already exists and skipping that is how a new field silently never appears on anyone's machine but the one it was written on.

The harness changed underneath — the same bug one level up​

case_hash pins that two runs asked the same questions. It says nothing about whether they were offered the same tools, and two things rewrite the catalog without touching a single case:

  • every enabled skill rides every turn — its description sits in the catalog the model chooses from, and use_skill pastes its body into the turn;
  • every connected MCP server contributes a whole tool group (mcp-<id>.*).

Toggle a skill or start a server between two runs of one suite and Compare reports fixes and regressions that were yours. So a run now carries a harness_hash plus the harness_json behind it, computed once per sweep in evals/fingerprint.py from the skills store and the live MCP manager, and Compare draws a banner over the whole diff naming what moved (skill X changed, MCP tool mcp-y.z was added).

Three properties do the work, and each is a way it could have been wrong:

Content, not names. A skill whose body was rewritten is a different harness even though the catalog lists the same name, because that body reaches the model. Same for an MCP tool whose schema changed under a stable name.

Run level, not case level. Every case in a run saw the same harness, so it lives on eval_runs and surfaces as a banner — not as a fifth column repeating one fact on every row.

Empty means "cannot tell", never "the same". A run recorded before the column existed has no hash, and neither does one whose harness could not be read (the whole computation is wrapped — losing a twenty-minute sweep because a skill file would not open is a bad trade). Both read as unknown, drawn distinctly from "these differ". Added to existing databases with an explicit ALTER, for the same reason case_hash was.

MCP servers and skills are variables of the experiment, not furniture: an eval sweep measures a model with a given tool set. The trainer agent's prompt already covers eval sweeps, conversion and serving, so it is the agent to pick in the chat for this work rather than a persona invented for one pane.

Why this reads eval_results and not localtrack​

The design note said "cross-run comparison over localtrack". It reads the results tables instead, and the reason is a real limit rather than a preference: localtrack stores metrics keyed by step, which can express "this sweep averaged 0.67" but not "model A passes discovery-github-repos and model B does not". The per-case verdict is the entire point of the matrix and the diff. localtrack keeps its job — the time-series across many sweeps, where the shape of a fine-tune's progress shows up.

What it found: three bugs, and none of them the model​

The first sweep of the starter suite against a local llama-3.2-3b scored 6/13, with every positive tool-calling case failing and every negative passing. The obvious reading — "this model cannot use progressive disclosure" — was wrong three times over. The investigation is worth recording because each layer hid the one below it.

1. no_call scored silence as success​

Every one of the 13 cases had returned an empty answer. The six that "passed" were no_call cases, and the grade only asked whether a tool had been called. A model that says nothing does not call a tool, so silence scored as a clean negative — and six confident passes made the seven failures look like a specific weakness rather than a total failure.

no_call now requires an actual answer. Saying nothing is not answering, and a grade that cannot tell the difference converts a dead provider into a respectable scoreboard.

2. A provider error was swallowed mid-stream​

With the empty answers visible, the next question was why. The telemetry chokepoint had the outbound request recorded: HTTP 200, 4899 bytes of response — the model had answered. The body was:

event: error
data: {"error":{"message":"Engine protocol predict stream returned an error:
{\"code\":500,\"message\":\"The model produced output that does not match
the expected peg-native format\"}"}}

LM Studio's constrained decoder had rejected the model's tool-call output, and it reports that inside an already-200 SSE stream. raise_for_status has long since passed by then, and the openai-dialect stream loop did:

choice = (json.loads(data).get("choices") or [{}])[0]

A frame carrying error instead of choices became an empty choice. The turn returned a blank answer with no tool calls — indistinguishable from a model that chose to say nothing, which is the worst shape a failure can take in a harness whose entire job is to attribute behaviour.

providers.py now raises ProviderStreamError on such a frame. This was never an evals bug: it silently degraded every chat turn against any OpenAI-dialect provider that reports errors this way.

3. load_tools threw away the model's answer​

With errors surfacing, the real finding appeared: the model does call load_tools. It had been calling it all along. Measured directly, it returned:

{"groups": "[\"github\"]"}
{"groups": "['files', 'editor']"}

An array parameter handed back as a string — the second not even valid JSON. Small models do this constantly. The dispatcher did:

if isinstance(requested, str):
requested = [requested]

which made the requested group name literally ["github"], matched nothing, and returned loaded: [] with the whole thing under unknown. The model asked for the right group, was told it got nothing, and the capability never appeared.

_coerce_group_list now reads any of the shapes a model actually produces — a real list, a JSON array in a string, a Python-style list with single quotes, a bracketed comma list, a bare name. _coerce_args does the same job one level up for the arguments payload as a whole; this slipped past it because the payload was valid JSON, it just had a string where an array belonged.

After the fix a discovery case reports groups loaded: ['files'] — the first time progressive disclosure has been observed working on this model.

What is left, and is not ours​

LM Studio's constrained decoder still 500s intermittently on this model, and it is prompt-dependent: the identical tool set and question succeed on one phrasing and fail on another. That is an engine-side limitation. What changed is that it is now reported as a provider error on the result row rather than scored as a model that declined to act.

The lesson generalises past this bug. Every layer here failed safely — an empty answer, an empty choice, an empty group list — and safe failures compose into a plausible number. A harness has to be built so that the absence of a thing is distinguishable from the thing being absent on purpose.

From a failing case to what the model was given​

A result row's turn_id is a button. It opens agentpedia — the interpretability suite's Runs section — with {turnId} (openAgentpediaTurn) and steps straight into that turn: the exact prompt, tool schemas and rounds that produced the case.

It used to be inert text — printed next to a pane that is a turn stepper keyed on precisely that id and took no params, so the workflow was to read a hex string off the screen and hunt the Runs list for it. The concern that stopped it being a link was that the recorder keeps a live in-memory ring, which a finished sweep outlives. That turned out to be only half the picture: recorder._persist writes every turn through to agent_turns in app.db unconditionally, and GET /api/agentpedia/turns/{id} reads the ring then the table for exactly this reason. A turn the store never kept 404s and the stepper says "No captured turn <id>", which is a better answer than a dead string.

The leaderboard is reachable from a turn​

evals.leaderboard and evals.leaderboard_diff wrap leaderboard.build / .diff, which until now only the pane could ask. The diff is the one that matters: the trainer agent's prompt tells it to answer "did the fine-tune help?" with an eval sweep rather than a remembered number, and a pass-rate difference cannot tell a real improvement from an even trade — 8/12 to 9/12 by fixing three and breaking two is a different event from fixing one and breaking none.

leaderboard carries the third question with it: the cases every run fails. That is nearly always a broken case rather than four bad models, and it has already caught three of them.

Where a sweep's metrics land​

sweep.localtrack_project_for(suite_id) derives the localtrack project, backend-side, so both callers get it. Only one of them used to derive anything: the browser sent a slug of the suite's name, and evals.run — the agent tool, and the one an agent reaches for to answer "did this fine-tune help?" — sent nothing at all. _LocalTrack("") builds a RunMirror(""), which is inactive by construction and swallows every metric without error, so an agent-started sweep filled in the scoreboard and charted nothing anywhere.

Keyed on the id, not the display name: a suite rename must not silently move its history into a second project, leaving the comparison view with two halves of one story and no way to tell they belong together. The browser no longer sends a rival slug for the same reason. An explicit localtrack_project still wins.

Browser vs desktop​

Identical. The sweep runs on the backend and the pane is ordinary React; nothing here touches a platform capability.

See also​

  • Agent tools — progressive disclosure, groups, TOOL_BUDGET
  • Agent chat — the orchestrator loop a case runs through
  • localtrack — where sweep aggregates land
  • training — projects, venvs, recipes, GGUF conversion
  • llama.cpp — serving a converted fine-tune to re-run a suite against

Grading beyond string match​

Two grades landed together, both because the existing ones could only compare tool calls and exact strings.

judge is routed now​

It had been declared-but-unrouted since the module was written, and Expect rejected it at authoring time for a good reason: a grade nothing scores fails every run, and the failure reads as the model being wrong — the one thing this module exists not to do. evals/judge.py routes it, so the case is legal now. The rule was never "judge is bad"; it was "a grade nothing scores is".

Three rules keep a judge from being worse than no grade:

  • The judge model is recorded on the result. Two runs graded by different judges are not comparable, and a leaderboard that does not say so presents them as if they were.
  • The rubric is the case's; the prompt is ours. The case says what to check; the scaffolding — answer with a verdict, quote the evidence, ignore length — is fixed. A case that could rewrite the whole prompt could quietly ask for something other than grading.
  • A judge that fails is not a fail. A provider error, a truncated reply, an unparseable verdict: each is an error, never passed=False. Scoring a zero because the judge could not be reached is exactly the "the case was wrong and the model got the blame" failure.

Any exception from the judge is caught, not only JudgeError, and becomes an error on that one case. An unexpected one (live, an ImportError in judge.py) used to escape run_case and kill the whole sweep. A sweep that dies anyway closes its unfinished runs as failed with the error, the same way a cancel closes them as cancelled, so no row reads running forever.

What the judge reads. All the assistant text of the turn, in order, not just the final round: text written alongside a tool call streams to the user too. Found live: kimi-k3 explained EAGLE-3 in the round where it also searched, then closed with "the explanation above stands", and a judge shown only that closing line failed an answer the user had received in full. The case's history is excluded.

Which model judges. The main agent's, resolved the way a chat turn resolves it: the provider from roster.resolve_provider, the model from _orchestrator_model. Taking the model from the saved global config instead paired one server with another server's model. A case can still pin expect.judge_model. The suggested "this node" sweep target (GET /api/evals/targets) resolves the same way.

It is routed in runner_agent, not in graders.py — that module is pure and synchronous by design, and giving it a provider client would make every grade a potential network call.

code_exec scores code by running it​

Two correct solutions to one problem share almost no characters, so every string metric scores code at zero. That is why HumanEval and MBPP were absent from the presets: not the datasets, the scorer. code_exec runs the model's completion against the dataset's own tests.

Gated behind HORRIBLE_ENABLE_EVAL_CODE_EXEC=1, and the gate is the honest part. A subprocess with a timeout and a scrubbed environment stops a runaway loop and an accidental print(os.environ); it does not stop a determined program. This is the games module's caveat repeated: subprocess-isolated is not container-grade. It runs model-written code about a programming puzzle, on your own machine, at your explicit request — a reasonable risk to take deliberately and an unreasonable one to take by default. It is refused at job-generation time, where the mistake is cheap, rather than letting every row fail and score a zero that looks like a verdict.

Three details that are each a wrong score if missed:

  • Markdown fences are stripped. Models return ```python blocks; running the fence is a SyntaxError graded as a wrong answer.
  • The stub is prepended when the completion does not redefine the function. HumanEval's format is a completion — the model gets a signature and returns a body. Running the body alone is a NameError.
  • A timeout is a fail, not an error. An infinite loop is a wrong answer to a programming problem; calling it infrastructure trouble would hide a real result.

Nine presets​

presets.py now covers GSM8K, MMLU, HellaSwag, TruthfulQA, HumanEval, MBPP, ARC Challenge, GPQA Diamond and IFEval. Each carries a why naming the trap it exists for — MBPP spells its tests test_list as a list of asserts with no signature stub; ARC's answer is a letter while its choices are a nested structure; GPQA is hard enough that a low score is expected rather than a broken harness.

Training on what the model got wrong​

evals.export has always written an SFT .jsonl of failed cases. It is now reachable: the datasets module's exports source surfaces that directory, so registering the export and pointing a recipe at it is two clicks rather than a path typed from memory. See the AI-research pipeline.