Skip to main content

Module: agent chat cockpit

The core "agentic era" surface: converse with agents/LLMs, watch their tool calls stream, and manage sessions.

Status: first slice implemented — frontend in packages/core/src/modules/agent/, backend in backend/modules/agent/. What exists today:

  • Onboarding on the home view (see layout-shell.mdx): detects a local Ollama at http://localhost:11434 (override: HORRIBLE_OLLAMA_URL), offers a model picker defaulting to gemma4:e2b, pulls models with streamed progress (POST /api/agent/pull), and persists the choice (PUT /api/agent/config → $HORRIBLE_DATA_DIR/agent-config.json).
  • One-shot ask: the legacy POST /api/agent/chat (NDJSON passthrough of Ollama's /api/generate) still exists; the home ask bar no longer uses it.
  • Orchestrator (layout control) — the home ask bar now drives the agent orchestrator: a backend tool-calling loop that arranges the user's workspace. Like the chat widget, it renders the model's streamed reasoning (a collapsible "Reasoning" disclosure) and streams answer tokens live, not just the final answer — so a turn shows the thinking instead of looking silent if the final content is sparse. See "The orchestrator" below.
  • Conversational chat widget (agent.chat, command agent.openChat): a pane with a multi-turn transcript, a per-turn action log, and the model's reply. Each turn replays prior user/assistant turns to the backend as history (the backend stays stateless per turn). It drives the same orchestrator, so it reasons about the live layout and widget contents (via the read tools) and acts on them. See ChatWidget.tsx. The pane is laid out as a sessions rail on the left (see Sessions), a slim header (rail toggle, the conversation's title, the trace toggle), the transcript as one centred column of at most 760px (assistant replies are flat prose, not bubbles; the user's turns keep a soft tinted bubble), and a composer card: the input, and under it the mode chip, the model picker and Send. An empty transcript greets with the 3D avatar (Avatar3D, now in packages/core) cycling through its mood animations, above the starter prompts; it is not drawn once a conversation has turns. The agent.avatarAnimation setting (default on) turns it off. The transcript persists as named sessions and supports slash commands (both below). A running turn draws an activity timeline and each chat has a mode — Plan, Agent or Auto (see Activity timeline & modes).
  • Inline autosuggest source (POST /api/agent/complete): a short non-streaming fill-in completion the editor's opt-in ghost-text autosuggest calls.

Per-widget agentTools + getAgentContext widget interop ship (see agent tools & permissions). Remaining design: a fuller chat/inspector cockpit.

Readiness: four states, not one​

AgentStatus carries two independent booleans — configured (has a provider and model been chosen?) and reachable (is that provider answering right now?) — and two more failure modes live outside the payload: the backend not answering at all, and the status not having resolved yet.

Those are four different problems with four different remedies, and the pane used to render every one of them as the same grey input placeholder, Agent not ready. That is the inversion of this product's own rule that a capability check reports what it found, what it looked for and failed to find, and what it could not ask.

AgentReadiness (modules/agent/AgentReadiness.tsx) maps the status to one banner above the input:

StateSaysOffers
unresolvedChecking the agentnothing — it is not a verdict
backend unreachableBackend not answeringRetry
configured: falseNo model selectedChoose a model → Settings
reachable: falseCan't reach <provider>Retry · Change provider
readynothing—

The endpoint appears only in the reachable: false case, as telemetry in mono: nothing was tried yet when no model is configured, so naming an endpoint there would be inventing a cause. planFor is exported and tested (__tests__/readiness.test.ts) because the distinction between those states is the whole point of the component, and collapsing them does not error — it just tells everyone the least useful true thing.

An empty transcript carries three clickable opening prompts rather than one grey sentence, each exercising a different half of what makes this agent unlike a chat box: layout control, pane reading, buffer editing. Deleting a conversation goes through a confirm that names the conversation — there is no undo behind it, and the control sits beside "New chat".

Sessions (persistence)​

Chat transcripts persist server-side as named sessions — a small chat backend module (backend/modules/chat/) that mirrors the workspace store: file-backed JSON at $HORRIBLE_DATA_DIR/chat-sessions.json, CRUD under /api/chat/sessions (list returns metadata only; GET /{id} returns the full transcript; PUT /{id} is a partial upsert of title/messages/mode; plus /active and DELETE). Only a save that carries messages bumps updated: it means conversation activity, and the sessions rail orders and buckets by it, so a rename or a mode switch must not move an old chat to the top. The chat widget reads the active session the first time it talks to an agent, lazily creates a session titled from the first prompt, and auto-saves each completed turn.

The sessions rail (SessionsRail.tsx) lists them. Its head holds the agent picker (populated from GET /api/agent/roster; just the agent's name when the roster has one agent or the chat is pinned to one) and New chat; under that, a title search, then the conversations newest first, bucketed Today / Yesterday / Previous 7 days / Previous 30 days / Older by local calendar day (session-groups.ts — updated is epoch seconds). Each row shows its age (3h, 2d); hovering or focusing it reveals rename (inline; Enter saves through PUT /{id} with a title, Escape cancels) and delete (always confirmed, naming the conversation, since there is no undo). The running conversation carries a pulsing marker and cannot be deleted mid-turn.

The rail sits inline when the chat is at least 720px wide; the header's toggle then shows or hides it, and that choice is remembered per viewer (localStorage agent.sessionsRail). Narrower — the agent docks on the right at 420px by default — it starts hidden and the toggle slides it over the chat with a scrim; picking or starting a conversation closes it. A compact chat (the interp Ask strip) has no rail; /clear and /agent still work there.

Sessions are per agent (agent_id on the session, ?agent= on the list endpoint), so each agent keeps its own conversations and active pointer. See Specialized agents & delegation. Frontend client: packages/core/src/modules/agent/sessions.ts.

Each message is saved with its orchestrator turn_id. An assistant message also saves sub_turns, the sub-agents it delegated to. The chat creates the id itself (newTurnId(), passed to askAgent as turnId), so the id is on the transcript before the first frame arrives. It is the join key into interpretability and trajectories. The toolbar's trace toggle slides out a trace drawer beside the transcript. The drawer shows the trajectories Agent view: the prompt by source, a round-by-round diff, the steps and the sub-agents. It can be resized, popped out into its own window, and docked back. Each reply's trace chip opens the drawer pinned to that turn. Messages saved before this carry an empty turn_id and have nothing to trace.

The transcript lives outside the component​

packages/core/src/modules/agent/chat-state.ts holds the transcript, the in-flight turn, the draft and the active session id in a small module-level store keyed by agent id; ChatWidget reads it through useSyncExternalStore and owns none of it.

That is not tidiness, it is the fix for "the chat resets when I prompt". The shell unmounts this pane routinely — an inactive tab is unmounted rather than hidden (Unmount is not close), and the agent's own layout tools (place_pane moving it to another group, say) move it. With the conversation in useState, each of those blanked it: the pane came back showing the starter prompts, re-read the node, and got the last saved state — which for a turn still streaming is the conversation as it was before the question. So the most reliable way to lose a conversation was to ask the agent to open something, which is the one thing this agent is for.

Two consequences follow, and both are the point. A remount is invisible — same transcript, same draft, no refetch and no flash of the empty state. And a turn survives its pane: askAgent's streaming callbacks write to the store, not into a dead component's setState, so tokens that arrived while the pane was unmounted are there when it comes back rather than silently dropped.

The mount effect therefore loads only when that agent has not been read yet (loaded), and loadSessions will not overwrite a transcript it already holds for the active session — a reload landing mid-stream would otherwise truncate the answer back to the previous turn. State is per app session and deliberately not persisted: the node is still the durable store, this is only what makes the pane stateless.

A failed read is never rendered as an empty transcript. loadSessions records loading / ok / failed rather than catching into a blank pane, because empty is indistinguishable from deleted — and it was worse than cosmetic: with no activeId, the next message ensureSession sent created a second session, orphaning the real history rather than merely hiding it. So ensureSession re-reads before creating anything when the last read failed, the restore retries once on its own (boot races the backend), and the pane shows what happened with a Retry instead of a chat that looks wiped.

A model picker (ModelPicker) sits in the same bar, listing every reachable provider's models — so a hosted provider appears there the moment a key is saved, and switching models does not mean leaving the conversation for the settings page.

It is a searchable combobox, not a <select>. One OpenRouter key puts several hundred models in this list (465 on a typical node), and a native select can only be walked: its type-ahead matches a prefix, so the queries people actually have — every :free model, everything with qwen in it — are unreachable by typing. The filter is a substring match over the provider label and the model id together, word by word and ANDed, so free narrows to the free tier, lm studio to one provider, and free qwen to the intersection. Arrow keys and Enter pick; the panel is position: fixed and measured off the trigger (flipping above it when, as in the composer, there is no room below), because the composer sits inside a pane that clips its overflow — and portaled to its pane window's body (usePaneWindow(), so it opens in a pop-out too), because inside a pane box fixed alone is not enough (see the desktop shell). The settings page's ModelCombobox does the same. It writes the provider and the model together: they are two settings (agent.<id>.provider, agent.<id>.model), and the settings page edits them as two controls, but here one option carries both, because a model name means nothing on a server that does not have it — minimax/minimax-m3:free selected while the provider still says Ollama is not half-configured, it is broken, and it fails mid-turn rather than at the click. The blank choice clears both back to the configured model. A saved override the live list does not contain (a provider that went down, an id typed on the settings page) stays selectable, so the control never silently renders as something other than what will run.

Slash commands​

/-prefixed inputs run locally in the chat widget (no model turn) and render as ephemeral system output that is never persisted or replayed to the model (packages/core/src/modules/agent/slash.ts): /tools (the agent's live tool catalog, fetched via the list_tools→tools WS round-trip), /help, /panes (open panes), /clear (new session), /agent (list the roster agents or switch the pane to one: /agent dba), /llm (query, set, or reset orchestrator LLM hyperparameter overrides), /plan, /act, /auto and /mode <name> (switch the chat's mode — with text after them they switch and send it, so /plan add a dark-mode toggle is one step; Agent mode is /act because /agent is the roster command), and honest placeholders /mcp and /skills (those subsystems don't exist yet). Typing / shows a suggestion list.

Activity timeline & modes​

A turn renders as an ordered list of steps (turn-steps.ts, pure reducers fed by the WS events; drawn by ActivityTimeline.tsx):

  • Thoughts groups — consecutive reasoning and tool calls fold into one collapsible THOUGHTS group (open while the turn runs), headed by its call count and duration. Each call is a row: status glyph, a label from tool-labels.ts, a one-line argument summary, and its duration; clicking it shows the arguments and a result preview. A row waiting on the permission gate shows a lock.
  • Notes — prose the model writes before a tool call becomes a paragraph between groups; the answer event stays the authoritative final text, which renders as markdown.
  • Live line — "Reasoning…", "Running Read file…", "Waiting for your answer" or "Needs approval", with an elapsed counter (a setInterval, so it ticks in a backgrounded pane); afterwards "Worked for 1m 12s".
  • More ↓ — the log follows new output only while you are at the bottom; scroll up and a jump button appears instead.

Three backend tools feed cards instead of rows (CARD_TOOLS; a row is shown only if the call fails):

ToolWhat the chat draws
update_todos({items})A TASKS checklist pinned to the turn (and compact above the composer while busy).
ask_user({questions})QuestionCard: up to 4 questions, 2–6 options each, multi-select, an "Other" box, Skip. The turn waits for the answer (question_response); a cancelled turn resolves it as dismissed. Nobody to ask (a remote peer's turn) → an error result, never a hang.
submit_plan({plan})PlanCard — Plan mode only. Ends the turn; Approve & run switches the chat to Agent and sends "Approved. Carry out the plan above."

The interaction tools are always-on core tools; a delegated sub-agent does not get them.

Modes (modes.ts, ModePicker.tsx) map onto the permission modes: Plan = plan, Agent = default, Auto = autonomous. The chip at the left of the composer picks one (Shift+Tab cycles); the composer's top border and placeholder follow it. The mode is per chat — saved on the session (mode), sent with every ask — and a chat that never chose one follows agent.permissions.mode from Settings (acceptEdits shows as Agent). Precedence on the backend: conn.force_mode (remote peers) > the chat's mode > the roster agent's permissionMode > the global setting.

In Plan mode the backend drops every side-effecting tool, and every DESKTOP_TOOLS verb (PLAN_MODE_BLOCKED), from the offered set (_gate still refuses them if called), adds submit_plan, and appends a plan-mode addendum to the system prompt. Pane open/close stays available: it is how the agent shows things.

Saved turns carry steps, todos, plan {text, status}, started_at and ended_at (ChatMessage in backend/modules/chat/models.py), so a reloaded chat redraws its timeline. Chats saved before the timeline render from their old reasoning / actions fields (legacySteps).

The orchestrator​

The Agent is the app's orchestrator: a backend-resident brain that drives the UI through tools. Per [[agent-orchestrator-architecture]] the brain runs on the backend (so agent logic/history stays server-side) while the tools execute in the frontend; the two halves talk over the shared /ws socket on the agent channel — the turn is bidirectional and stateful, so running it on the socket (not HTTP) avoids correlating an HTTP request with the right browser.

End-to-end walkthrough​

The numbered phases below trace one turn through the code; the diagram above is the same flow visually.

  1. Manifest (once per connection). Before any turn, the frontend pushes its capability manifest — the serialized agent-exposed commands and per-pane agentTools, handlers stripped (manifest.ts, initAgentManifestSync) — on socket open and on every registry change. The backend stores it on the WsConnection and forgets it on disconnect (hence the re-push on reconnect). _tools_for later merges it with the static LAYOUT_TOOLS into the tool list the model sees each round.
  2. Ask. The chat widget (or home ask bar) calls askAgent(prompt, cb, history). askAgent attaches the user's focused-pane snapshot (readActiveAgentContext — currently the active editor buffer) and sends ask {turnId, prompt, history, context} on the agent channel.
  3. Turn setup (backend). handle_agent_message routes the ask to a detached run_agent_turn task — it must run detached (asyncio.create_task) because it awaits tool_results that arrive on the same receive loop; awaiting inline would deadlock. The task assembles the messages (SYSTEM_PROMPT + sanitized history + the focused-buffer system message from _active_editor_message + the user prompt), resolves the tool list (_tools_for), the model (_orchestrator_model — the override or the configured model), and the sampling temperature (_tool_temperature).
  4. The loop (≤ agent.maxRounds). Each round streams the provider via providers.chat_stream, which first flattens the system tier (see below); the model's reasoning (thinking/reasoning_content) and answer content token-deltas relay live as reasoning/token events the chat widget and home ask bar render as they arrive. A round ends with either tool_calls or a final answer.

The step budget, and how a turn that runs out ends​

A turn may spend agent.maxRounds rounds of tool calls (MAX_ROUNDS, 24, when the setting is 0). It is a guard against a model that never stops calling tools, not a work limit — but at its old value of 8 it was acting as one, invisibly. In the recorded turns the counts tapered off to a single turn at 7, then 32 sat at exactly 8: the shape of a budget being hit rather than of work finishing.

Two things make that shape a bug rather than a threshold doing its job:

  • Discovery used to be charged as work. Under progressive disclosure the model pays a round for each load_tools call before it can touch the task — one observed turn spent rounds 3 and 5 loading groups — so a third of a short budget could go on reading the menu. A round whose calls were all meta tools is now free, up to META_ROUND_ALLOWANCE (6). Bounded rather than free, because a model that loops loading groups for ever must still stop.
  • Running out threw the work away. The loop returned a literal "(stopped after too many steps)", so a turn that had read six files and was ready to report ended with a sentence about the machinery. Exhaustion now spends one last tool-less round (_final_answer): the model is told the budget is gone and asked to answer with what it has. Offering no tools is what makes it terminate. If that round returns nothing, or fails, the turn still says plainly that it stopped early and how many steps it used — an honest stop beats an invented answer, and both beat silence.

Passing no tools is a real request, so providers now omits tools entirely when the list is empty rather than sending [], which several OpenAI-compatible servers reject outright.

  1. Tool execution + gating. For each tool_call, _gate decides: read-only and layout verbs pass straight through; side-effecting tools are evaluated against the permission mode + rules, prompting the user via approval_request/approval_response on an ASK (and persisting a rule on "always allow"). An allowed call is relayed to the browser as tool_call, where executeTool runs it against the registry / layoutController seam / executeDynamicTool and returns a tool_result; a denied call returns an error to the model instead. Either way the result is appended to messages (provider-formatted by tool_result_message) and the loop continues so the model can react to it.
  2. Completion. When a round returns no tool calls, the backend sends the authoritative answer then done (or error on an HTTP failure). If the model narrated an action without emitting a call, the loop first grants one nudged retry — on every dialect, with tool_choice:"required" added where the provider supports it — before giving up and answering. The widget renders the reply plus the per-turn action log, and a completed turn auto-saves to its session.

The system tier is several messages until the wire​

run_agent_turn assembles the system tier as separate messages: the spec's system prompt, then the skill catalog, then the guides for the groups the turn starts with. That split is deliberate — the interpretability recorder tells the blocks apart by position and content marker (CATALOG_MARKER) to attribute each one's per-turn token cost, and one glued string would make that impossible. The forced-tool nudge adds a fourth, mid-conversation.

Many models' Jinja chat templates reject both shapes:

Error: Jinja Exception: System message must be at the beginning.

That is a 500 from the engine, not a worse answer — the turn is lost outright. So the split survives right up to the wire and providers.normalize_system_messages flattens it inside chat/chat_stream, the one chokepoint every dialect and every call site passes through:

  • leading system messages are joined with a blank line, order preserved;
  • a later one becomes a user message, because its position is the point — a nudge merged into the preamble would arrive before the failure it answers.

It runs on every dialect, not just the strict ones: the flattened form is semantically identical, and branching would let a template's strictness decide whether the nudge was delivered at all. Because the recorder captures in the orchestrator, before the provider call, it still sees the unflattened blocks.

The tool-calling loop itself is factored into a reusable run_agent_loop(conn, messages, tools, …, emit) → str (orchestrator.py): run_agent_turn assembles the messages/tools and supplies an emit that sends agent-channel reasoning/token, while the flow canvas's Agent nodes reuse the same loop with a flow-channel emit — so chat and flow share one gated tool-calling implementation.

Loop (backend/modules/agent/orchestrator.py): on an ask, a detached task streams the user's local Ollama /api/chat (stream:true) with tools=LAYOUT_TOOLS via providers.chat_stream. Each round's reasoning (thinking/reasoning_content) and answer content token-deltas are relayed live to the chat widget (reasoning/token events) as they arrive. If the model returns tool_calls, each is relayed to the browser and the loop awaits the result before continuing; otherwise it sends the final (authoritative) answer. Capped by the step budget. Ollama gets think:true best-effort — models without a thinking mode reject it (400) and the round transparently retries without, still streaming content.

Tool-calling reliability. Turns decode greedily (temperature 0, via chat_stream's new temperature arg → Ollama options / OpenAI payload) — at higher temperatures small local models narrate an action ("I'll call …") instead of emitting the structured call. The temperature is overridable through the settings store (agent.orchestrator.temperature). Other model hyperparameters can be overridden using setting keys agent.orchestrator.contextSize (context window limit, maps to Ollama's num_ctx), agent.orchestrator.maxTokens (maps to OpenAI's max_tokens / Ollama's num_predict), and agent.orchestrator.topP (Top P nucleus sampling). When the model still answers in prose that reads like an unemitted call, the loop gives it one nudged retry: a system message saying it described an action without emitting the call, plus tool_choice:"required" where the dialect has one. This runs on every dialect. It used to be gated on openai, which meant the repair never fired for Ollama — the default provider, and so never for the small local models that need it most; tool_choice is extra leverage, not the mechanism. The loop never forces a tool unconditionally: a plain conversational reply must stay a reply.

Malformed arguments are reported, not silently emptied. A tool call's arguments arrive as a dict (Ollama) or a JSON string (OpenAI, and some Ollama models), and the streaming paths accumulate that string from deltas — so a truncated payload is a real failure mode. _coerce_args used to swallow anything unparseable into {}, which meant the call ran with no arguments at all (close_pane with no instanceId, files.delete with no path) and the model saw a plain failure with no hint its JSON was to blame. It now returns an error alongside the args, carried on ToolCall.arg_error, and _dispatch_call refuses the call — nothing is relayed to the browser — returning a message the model can actually act on. An empty payload is still a valid no-arg call.

Unknown tool names get a suggestion. A name the relay can't resolve comes back with didYouMean — up to three near matches over the layout verbs plus the advertised manifest (tool-names.ts, pure and unit-tested). It matches normalized names (openPane → open_pane), dropped words (list_panes → list_open_panes), small typos, and — as a last resort — other tools in the same namespace, which is what rescues an invented verb like files.remove. For weak emitters, prefer a larger model (e.g. gemma4:12b) for the orchestrator: the agent.orchestrator.model setting overrides only the orchestrator's model — leave it on the configured agent model to reuse it, so chat/autosuggest can stay on a smaller, faster model while the tool-calling loop runs a stronger one. All knobs live in the Agent orchestrator section of the Settings page (a custom section — OrchestratorSettings — because the model is a dropdown of the selected provider's live models from /agent/status, not a static enum); the blank "Configured agent model" choice clears the override. These hyperparameters can also be inspected/configured directly in chat using the /llm slash command, or modified by the agent itself using the gated agent.setHyperparameters tool. The ghost-text /agent/complete path likewise samples at a low fixed temperature so completions are stable, not creative.

Channel protocol ({channel:'agent', event, data:{turnId, …}}):

Directioneventdata
client→servermanifest{tools: SerializedTool[]}
client→serverask{turnId, prompt, history?, context?, agentId?, sessionId?, mode?}
server→clienttool_call{turnId, callId, name, args}
client→servertool_result{turnId, callId, ok, result?, error?}
server→clientapproval_request{turnId, approvalId, tool, specifier, mode, callId, args}
client→serverapproval_response{approvalId, decision, rule?}
client→serverlist_tools{} — request the live tool catalog
client→servercancel{turnId} — Stop; only the asking socket may
server→clienttools{tools: [{name, description, source}]}
server→clientreasoning{turnId, agentId, delta} — streamed thinking
server→clienttoken{turnId, agentId, delta} — streamed answer
server→clientdelegate_token{turnId, agentId, delta} — sub-agent's stream
server→clientdelegate_start{turnId, subTurnId, agentId} — a sub-turn began
server→clientdelegate_done{turnId, subTurnId, agentId, ok} — …and ended
server→clienttool_start{turnId, agentId, callId, name, args, round} — every call, before dispatch
server→clienttool_end{turnId, agentId, callId, ok, denied, preview, durationMs}
server→clientquestion{turnId, questionId, questions} — ask_user
client→serverquestion_response{questionId, answers | null} — null = skipped
server→clienttodos{turnId, items} — update_todos
server→clientplan{turnId, text} — submit_plan
server→clientanswer{turnId, agentId, text} — final, authoritative
server→clientdone / error{turnId, agentId, …}

agentId selects (and tags) the roster agent driving the turn — omitted it means main, so pre-roster clients keep working. See Specialized agents & delegation.

sessionId is the persisted chat session the turn belongs to. It only labels the trace (gen_ai.conversation.id, a thread in Opik — see otel); the backend stays stateless per turn, and history remains the conversation.

The frontend pushes the capability manifest (manifest) on connect, on every reconnect, and whenever the registry changes (packages/core/src/modules/agent/manifest.ts, initAgentManifestSync): the serialized agent-exposed commands and per-widget/panel agentTools (handlers stripped). The backend stores it on the WsConnection, groups the pushed tools by prefix, and exposes them via progressive disclosure (see Hierarchical tools & progressive disclosure): each turn the model sees the layout/peer core + meta-tools + whatever groups are active, and load_tools injects more as needed. Relayed calls that aren't layout verbs are dispatched on the frontend via executeDynamicTool. Side-effecting calls pass through the permission gate (_gate in the orchestrator) before relay: read-only tools pass straight through, denied calls return an error to the model, and an ASK decision prompts the user via approval_request/approval_response (the in-chat approval UI is the next slice) — see agent tools & permissions.

Specialized agents & delegation​

The orchestrator is one of several roster agents — fully separate loops, each with its own system prompt, tool-group scope, per-agent settings, and chat sessions. Built-ins (backend/modules/agent/roster.py):

idscope (tool_groups)preloadeddefault mode
mainunrestricted (keyword preloading, delegate)by prompt keywordssession mode
coderfiles, editor, terminal, code, symbolseditor, files, symbolsacceptEdits
dbadatabase, symbolsdatabasesession mode
researcherbrowser, search, library, github, google, research, arxiv, recordsresearch, library, arxivsession mode
intakerecords, browser, research, libraryrecords, researchsession mode
trainertraining, recipe, cells, evals, localtrack, datasets, llamacpp, wandb, editor, filestrainingsession mode
interpinterp, lens, llamacpp, agentpedia, kb, arxivinterp, lenssession mode

interp is the interpretability suite's guide: it is the agent the suite's Ask strip talks to, and its interp group is declared by the suite pane in the frontend (interp.state, interp.attention, interp.goto).

Two of researcher's groups are load-bearing for the panes it works beside, and both were dead buttons before: the Review pane (records — it could display a proposal but not file one), and the search panel (search — the pane's own results were unreachable to the agent beside it). A pane whose agent lacks the matching group is the failure mode to watch for.

trainer shows the other half of that trap. hardware is pointedly not in its scope, even though the training panes read a hardware probe: a group is a tool name's prefix (_group_of), that namespace holds settings keys rather than tools, so naming it would permit nothing at all and say nothing about it. Check that a group you list has tools under it, not just a module with that name.

Its scope is long because the loop is long, and because training was split into four. That group had reached 34 tools — a third of them registered by the frontend, which is why every count in a comment said 14 — and with a ~9-tool core there was no room for evals beside it: the loop this agent's own prompt prescribes did not fit under TOOL_BUDGET, and the overflow was dropped with nothing but a log line to show for it. Only training is preloaded; the rest cost a load_tools round rather than schema space on every turn.

datasets was missing entirely, which is the same dead-button trap one level in: the agent that fills in a recipe's dataset_id could not search, peek at or token-stat a dataset, and can_delegate is False, so it had no way to ask anything that could.

There is deliberately no crm agent. It existed to enrich contacts and deals from the open web, which is what researcher already does now that it carries records alongside browser/search/github/google — so it was a persona rather than a capability, and its prompt hardcoded a contacts/deals model the records substrate never required. See records.

Mechanics (run_agent_turn(agent_id=…)):

  • A specialized agent skips keyword preloading — its active_groups start at the spec's preload_groups, and list_tool_groups/load_tools only see its allowed groups (no forgiveness auto-load outside them). The peer tools and agent.delegate are core only for specs that declare them (main).
  • The picker chooses the agent. The chat opens as main; choosing another from the picker is remembered (in localStorage, keyed by the desktop's workspace row) so it survives a reload. (Workspace presets used to declare a persona each — Data Ops as the dba, Data Entry as intake — until they were removed with the tiling frame.)
  • Per-agent settings agent.<id>.provider/endpoint/model/temperature/contextSize/maxTokens/topP/permissionMode fall back to the agent.orchestrator.* keys, then defaults (roster.agent_setting). The permission-mode override threads through the gate; the user's explicit allow/ask/deny rules still apply unchanged.
  • Per-agent provider (roster.resolve_provider). provider/endpoint used to be global while model was per-agent, which made "run the coder on the node's llama.cpp server and leave the orchestrator on Ollama" unexpressible: a model name means nothing on a server that does not have that model. An override naming an unknown provider is ignored rather than fatal, so a stale settings value cannot take the agent down. It must be resolved at all three run_agent_loop call sites — the orchestrator, delegate.py, and the flow executor's Agent node — because a site left on provider_for(config.provider) does not fail, it quietly runs on the global provider.
  • Per-agent sessions: ChatSession.agent_id (default main) + an active_by_agent map and a GET /api/chat/sessions?agent=<id> filter — all additive, so pre-roster chat-sessions.json files load unchanged.
  • GET /api/agent/roster lists the roster (id, name, description, scope) for the chat widget's agent picker; plugins add agents with host.add_agent(AgentSpec(…)) (see backend plugin SDK).

Delegation — agent.delegate(agentId, prompt) (backend-static, side-effecting, specifier {agentId}) runs a whole sub-turn of the target agent on the same browser connection, so unlike agent.ask_peer's tool-less remote turns the specialist can actuate — its tool calls relay to the same frontend and pass the same permission gate under its own mode. The sub-turn gets a fresh turn id (its stream stays out of the parent chat's turn-scoped subscription); its answer deltas surface as delegate_token events on the parent turn, and the final answer returns to the caller as an ordinary tool result. One level deep by design: specialists don't see the delegate tool (spec.can_delegate), and a DELEGATE_TIMEOUT_S (300 s) bounds the sub-turn.

Agent-to-agent (talking to a peer's agent)​

When this node is connected to peers (see Module: network and Distributed peer fabric), the orchestrator gains two backend-static tools, merged into the catalog by _tools_for and resolved in _run_backend_tool against the process-global PeerHub (no browser handler):

  • list_peers() — the connected peer nodes (node_id, name, capabilities). Read-only.
  • agent.ask_peer(peerId, prompt) — ask another user's agent and get its answer back as an ordinary tool result. Side-effecting (specifier {peerId}), so it passes the local permission gate first.

The network module's Agent Relay widget (network.relay) is the direct UI over this: it posts to POST /api/network/ask-peer (the same agent_bridge.ask_peer), so you can ask a peer's agent without going through your own model. See Module: network.

The cross-peer boundary is the security crux:

  • A remote turn runs behind a RemoteAgentConn (no browser) under a forced mode = network.remoteAgentMode (default plan → read-only). It is given no actuating tools, so a remote agent answers from the model but cannot drive your panes, files, or terminal. Its gate never blocks on an absent human — an ASK decision becomes a deny.
  • Admission requires network.allowRemoteAgent and a trusted peer.
  • Fan-out is bounded: an origin_chain cycle guard (reject if this node is already in the chain), a MAX_PEER_HOPS cap, and a PEER_AGENT_TIMEOUT_S request timeout.

Peer-wire message types (on /peer-ws, not the agent channel): agent_request → agent_result (caller awaits the reply via PeerHub.request).

Focused-pane context. Pane snapshots are normally read on demand (the model calls get_pane_context — pull, not push). As a convenience the frontend also attaches the user's focused pane snapshot to every ask as context: the editor's BufferView marks its instance active on focus/load (and clears it on unmount) through core's agent-context ambient slot (setActiveContextInstance/readActiveAgentContext), and askAgent reads it. When that snapshot is an addressable editor buffer, the backend injects it as a system message just before the user turn (_active_editor_message) handing the model the open code up front, so "alter/refactor/fix this code" modifies the open buffer and writes it back with editor.proposeEdit(uri=…) — no list_open_panes + get_pane_context discovery first (a multi-step dance weak local models often skip). Unsaved scratch buffers (no uri) are excluded.

This buffer rides on every turn too, and was long the one context injection with no cap at all — a single large file could dwarf everything else in a small model's window. agent.activeBufferBudget (24000 chars, 0 = unlimited) now clips it in the frontend. The clip sets a truncated flag, and honouring that flag is safety-critical rather than cosmetic: the normal message tells the model to write the complete buffer back via editor.proposeEdit, so doing that from a clipped copy would silently delete everything past the cut. A truncated snapshot therefore flips the instruction — the model is told the code is cut off and must re-read the file before any whole-buffer write. Pinned by test_agent_orchestrator.py::test_truncated_buffer_forbids_writing_the_whole_file_back.

Workspace context. Alongside the focused pane, askAgent attaches context.panes — a snapshot of every visible pane that exposes one (readVisibleAgentContexts: each non-minimized window's active tab, topmost first; background tabs are unmounted, so their state isn't readable anyway). The backend formats it as a compact index just before the user turn (_workspace_context_message). This is what lets the agent see the desk it sits at: the researcher already knows which row the Review pane has open, and the intake agent which document is on screen, without spending a list_open_panes + get_pane_context round on it. It rides on every turn, so it is budgeted — agent.workspaceContext (on/off), agent.workspaceContextPanes (6) and agent.workspaceContextBudget (1500 chars per pane, clipping oversized fields rather than the whole object, so the JSON stays parseable and the note points the model at get_pane_context for the full read).

The frontend executes each relayed tool_call against the registry and the registry.layoutController seam and the desktop controller (installed by the shell, layout/install.ts), then replies. This runs in an always-on relay handler (initAgentRelay in packages/core/src/modules/agent/orchestrator-client.ts, registered at boot) rather than inside a chat turn — so the same relay serves chat turns and flow canvas runs (Tool nodes + flow Agent nodes). askAgent keeps only a turn-scoped listener for its per-turn action log. The executor itself lives in packages/core/src/modules/agent/tool-exec.ts (executeTool) — the shared relay surface the Python REPL's dash SDK drives too, so they all stay in lockstep.

LAYOUT_TOOLS (app-level verbs over the workbench — every pane is a tab in a group, in the centre split, an edge dock or a popout, and regions are per-pane strips; the model discovers ids via the read tools, so the catalog stays frontend-owned). Reads: list_available_panes (views with role, sections + regions), list_open_panes (instances with where each lives), get_layout (the centre tree with group ids, the docks — the orientation read), get_pane_context(instanceId). Panes: open_pane(id) (where its role puts it, or focus a single-instance view), close_pane(id), focus_pane(instanceId). Arrangement: place_pane(instanceId, to, group?) (center, dock-left/-right/ -bottom, split-left/-right/-above/-below, or group), pane_state(instanceId, state) (maximize/restore/hide/show/present/ unpresent), arrange_panes(style) (columns/rows/grid/tabs/even). Regions: toggle_region(instanceId, position, open?), set_region_view(instanceId, viewId). Two panes side by side: place the second one split-right. The floating desktop's open_window, window_state and arrange_windows are still answered by the relay, mapped onto these, so old transcripts and REPL scripts keep working; they are no longer in the schema. (The tiling verbs — split_area, join_area, resize_area, move_pane, fullscreen_area, open_tool_in_dock, toggle_dock, dock_window — and the workspace verbs went with the tiling frame.)

Appearance verbs are a separate, lazily-loaded desktop group (desktop.set_backdrop, desktop.set_theme, desktop.configure_taskbar) rather than part of the layout core. Restyling the desktop is a rare request and every always-on tool is paid for on every turn by every agent — the same reasoning that cut the core 34 → 11 below. The group loads on demand or on a theme/wallpaper/taskbar keyword, and get_layout carries the backdrop and theme catalogs so those verbs have ids the model cannot guess without two more always-on list_* schemas.

The layout verbs are layout-only and ungated (like open_pane), and they execute through the same controller functions the user's gestures and keybindings use — one set of operations, three callers (see windowing). Each mutation triggers the store's autosave, so an agent's snap persists exactly like a user's drag.

The /ws handler (backend/app.py) is now a bidirectional router: one inbound receive loop dispatches by channel (agent → the orchestrator) while a telemetry push task sends outbound; WsConnection (backend/modules/ws.py) serializes sends and tracks pending tool-call futures.

The capability manifest, getAgentContext() state-reads, widget interop, and the Claude Code–style permission system gating side effects all ship now (specified in agent tools & permissions) — the shared surface the editor, terminal, and file-explorer modules build on. The chat widget now streams reasoning + answer tokens live (chat_stream), persists transcripts as sessions, and supports slash commands. Next slices: parameterized commands-as-tools and a fuller chat/inspector cockpit.

Multi-turn durability​

Two things made a long conversation degrade in ways nothing surfaced.

The transcript was replayed whole. agent.orchestrator.contextSize defaults to the provider default, so overflow never errored — the provider truncates from the front of the message list, which is exactly where the system prompt and the group guides sit. A five-turn conversation on a small local model would quietly stop following its own instructions, and the only symptom was worse answers. compactHistory (modules/agent/history.ts) now keeps the last MAX_HISTORY_TURNS verbatim, caps any single oversized message (one pasted stack trace can outweigh the rest of the conversation), and replaces what falls out with one marker message. The marker is a user message — _history_messages keeps only user/assistant text, and an assistant announcing its own amnesia reads as something the assistant said — and it tells the model to ask rather than leaving a hole it will confidently fill. The transcript draws the same seam (agent-compacted), because silent compaction is indistinguishable from an agent that forgot.

Loaded tool groups reset every turn. active_groups was rebuilt from prompt keywords each turn, so the model spent a load_tools round to reach files in turn one, used it, and in turn two the tools were simply gone — a follow-up like "now delete it" arrived with nothing to delete it with. _carried / _remember_groups keep them for the session, keyed on the socket (weakly, so a reload starts clean) and per agent_id. Empty history resets the carry — that is the widget saying "new chat", an exact signal needing no extra protocol. The carry is bounded to MAX_CARRIED_GROUPS and trims oldest-first, because every carried group costs schema bytes on every later turn and the whole point of progressive disclosure is staying under the 40-tool reasoning cliff. A carry never widens a scoped agent's reach: it can only re-activate a group spec.tool_groups already allows.

Providers & HTTP endpoints​

A provider is either a local server reached over HTTP on this machine, or a hosted API reached over the internet with a key. backend/modules/agent/providers.py is the source of truth; it collapses every one of them onto one of three dialects:

ProviderDialectDefault endpointPullSpawnKey
Ollamaollamahttp://localhost:11434yesno—
LM Studioopenaihttp://localhost:1234nono—
llama.cppopenaihttp://127.0.0.1:8080noyes—
vLLMopenaihttp://localhost:8001noyes—
Unslothopenaihttp://127.0.0.1:8888nonoopt
OpenAIlitellm—nonoyes
Anthropiclitellm—nonoyes
Google Geminilitellm—nonoyes
OpenRouterlitellm—nonoyes
In-browserbrowser— (the app window)nono—

browser is the one provider with no server: the model runs on the GPU of an open app window (WebGPU) and each round is relayed over that window's /ws socket. It is reachable only while a window reports WebGPU, lists only models that window has cached, and refuses callers with no window (evals, games). See WebML.

llamacpp is the node's own server — the backend downloads the llama-server binary, picks the GGUF and supervises the process, which is why it is the one provider whose model path is known rather than inferred. See llamacpp. It is an openai-dialect entry rather than a new dialect on purpose: a bespoke one would have meant six new branches here and would have silently lost the tool_choice="required" retry, which is gated on info.dialect == "openai".

The dialect decides the wire format and the request paths — it is not the online/offline switch, hosted is. "OpenAI-compatible" refers to LM Studio's, llama.cpp's and vLLM's wire format; it does not mean any request leaves the machine. Everything on the ollama and openai dialects runs offline against a localhost server; only the litellm dialect calls out.

unsloth is Unsloth Desktop's local server. It is another openai-dialect entry, and the one local server that checks a key: its /v1 routes want Authorization: Bearer sk-unsloth-… even on loopback, unless the user turned on its keyless mode. That is what ProviderInfo.optional_api_key means. api_key_for and auth_headers send a key when one is saved (secrets.db under unsloth, then UNSLOTH_API_KEY), and the key routes accept it, so it gets a field in Settings → API keys. It is deliberately not hosted: the probe still runs without a key and a 401 shows as unreachable, rather than "no key" being taken to mean unusable. Models are loaded in Unsloth's own window (it has no load route). Thinking is turned off with a top-level enable_thinking: false, and the context window comes from its llama-server-style /v1/props. The key itself is created in Unsloth → Settings → API.

Hosted providers and their API keys​

Four things make a hosted provider work, and each exists because the obvious alternative fails silently:

  • The key is never a setting. GET /api/settings hands the whole settings bag to the browser and to every plugin, so a credential there is a credential given away. Keys are Fernet-encrypted in secrets.db under the provider kind (the same store connectors use) and written through PUT /agent/providers/<kind>/key, which returns only has_api_key. Nothing reads a key back out, which is why the settings field is write-only: it shows a key as saved, never as its value.
  • A key already in the environment counts. litellm reads OPENAI_API_KEY, OPENROUTER_API_KEY and friends on its own, so api_key_for falls back to the variable. Without that check a key that works would be reported as missing, and the user would be sent to fix something that isn't broken. DELETE says so too: it can return has_api_key: true, because an exported variable is not ours to delete.
  • "Reachable" means "we hold a key". list_models raises MissingApiKey (an httpx.HTTPError, which is what the probe already catches) for a keyless hosted provider. Before this, a keyless OpenAI reported itself reachable with a list of models, so onboarding cheerfully offered a provider that failed on the first turn.
  • The model id needs a routing prefix. qualify_model adds ProviderInfo.model_prefix — openrouter/ for OpenRouter, gemini/ for Google AI Studio, nothing for OpenAI and Anthropic, whose ids are unambiguous. A bare minimax/minimax-m3:free is not a model litellm can place, and a bare gemini-2.5-pro routes to the wrong vendor. Applied at the call, so the id stored in settings stays the one the provider's own dashboard shows.

OpenRouter's model list is fetched live from its public catalog (https://openrouter.ai/api/v1/models, no key needed to read) rather than pinned in code: its range — and which entries are :free — changes weekly, so a hardcoded list would be wrong within the month. Ids come back tool-capable first, because a model without tool support does not merely do worse in the orchestrator loop, it never calls a tool at all — which reads as the agent ignoring you rather than as a model that cannot do this. The fetch is cached for 10 minutes and carries its own 10s timeout (the probe client's 2s budget is sized for a loopback port, not for a few hundred KB from across the internet), and a failed fetch falls back to static_models rather than raising: the key decides usability, and an empty dropdown for a working provider is the worse failure.

Google Gemini's model list is fetched live too, from AI Studio (generativelanguage.googleapis.com/v1beta/models) by _gemini_listing. Unlike OpenRouter's catalog it needs the key, which goes in the x-goog-api-key header and never in the query string, where proxy and access logs would keep it. Only models whose supportedGenerationMethods include generateContent are kept, which drops embedders, aqa and the Live-only models. A name filter then drops models that answer in speech, images or music (TTS, Nano Banana, Lyria), because the listing has no output-modality field and any of them would fail a chat turn. The list is cached for 10 minutes like the OpenRouter catalog. A rejected key (400/401/403) raises, so Gemini reads as unreachable rather than showing a dropdown that cannot make a call. Any other failure falls back to static_models (the -latest aliases plus 2.5 Pro/Flash) without caching it.

OpenAI and Anthropic have no listing we fetch, so they show static_models: a starting point for the dropdown, not the provider's full range. Every model field in the UI stays free text.

Free models are listed without a key. The catalog is public, and onboarding is the moment someone has no key yet and wants to know what they could run for nothing, so free_models(client, info) never checks for one and /agent/status reports free_models on every provider (empty for providers without a catalog). It says nothing about usability — a free OpenRouter model still needs a key to call, and reachable still means "we hold one". An entry is free when both pricing.prompt and pricing.completion parse to exactly zero, or its id ends in :free; the auto-router is priced -1 and is not. The free list shares the catalog cache with the model list, so a keyed provider costs no second fetch. Onboarding picks a free model by default when switching to a provider, lists free ids first and tags them.

Every model field is a substring-searchable combobox. The onboarding field was a <datalist> and the settings page's model override a <select>, and both effectively match from the start of the id — so with several hundred OpenRouter ids, typing free found nothing. Both are now ModelCombobox, filtering through matchesModelQuery, the same match the Agent pane's ModelPicker uses: every word anywhere in the id. A free model also matches the word free, which finds zero-priced ids without the suffix. The field still accepts an id that is not listed (Enter uses what was typed), and it commits on pick, Enter or blur rather than per keystroke, because on the settings page each commit is a settings write and a half-typed id is a broken agent until the typing stops.

A remote listing is cached and warmed; a local one never is. NVIDIA NIM's /v1/models (and any vllm/peer pointed at another host) is fetched across the internet, so list_models gives it its own REMOTE_TIMEOUT (10s) rather than the probe client's 2s budget — that budget is sized for a loopback port, and _probe swallows httpx.HTTPError, so inheriting it made a slow network indistinguishable from a provider with no models. The answer is cached under (kind, endpoint) for the same 10 minutes as the OpenRouter catalog, keyed by the endpoint too because a vLLM can be repointed and a list cached under the bare kind would then describe a machine nobody is talking to. A local server's list is deliberately never cached: ollama pull changes it, and a stale list is a worse lie than a slow one — loopback is decided by is_loopback_endpoint, for which an empty endpoint (a peer with no lease) counts as local because it is unreachable either way.

Every provider is probed in parallel, so the slowest one sets the response time, and the probe client's connect timeout is 0.75s (the overall budget stays 2s). That is the number that matters on Windows, where a connect to a closed loopback port does not fail fast — the OS retries the refused SYN for about two seconds — so each provider that simply is not running (Ollama, llama.cpp, vLLM on most machines) used to hold the whole response for the full budget. A live local server accepts in about a millisecond, or ~250ms through localhost when the IPv6 attempt is refused first. Remote listings are unaffected: they pass their own explicit timeout=. On the frontend, getAgentStatus() shares one in-flight request between callers (several components ask within a few ms of each other at boot); nothing is cached past it.

Those remote listings are warmed at startup: the app lifespan calls agent.routes.start_warm_model_lists(), which probes only the remote providers in the background so the first model dropdown opens filled instead of waiting on a round trip. It is never awaited and every failure is swallowed — a warm is not worth a second of boot, and an unwarmed provider is simply fetched on demand as before. GET /agent/status?agent_id=<id> reports the provider and model that roster agent's turns resolve to (roster.resolve_provider, so agent.<id>.provider and agent.orchestrator.provider count), not the saved config's. The chat judges readiness by it: judged by the config, an agent overridden onto a hosted provider showed "can't reach LM Studio" and refused to send whenever the local server was down, though its turns would have run.

GET /agent/status?refresh=true drops both caches first, and the refresh button in the model picker's panel — beside the filter field — is its only caller. It is a button rather than a fetch-on-open because a refresh on open would put that download in front of every use of the dropdown to serve the one time in a hundred the list actually changed, and the TTL is otherwise invisible from the UI: enabling a model on the vendor's dashboard and not finding it here reads as the model not existing. While it is in flight the button spins and the count line says so; a failure says Refresh failed — showing the last list rather than being swallowed, because a refresh that silently did nothing is indistinguishable from a vendor with no new models. A refreshed status outranks the one the pane handed the picker (own ?? status), until the pane itself refetches — a new object from the pane is in turn newer than our copy, so the effect drops ours and defers to it again.

A strict provider's tool names are translated at the wire, not renamed at the source. NVIDIA NIM validates function names against OpenAI's documented grammar (a-zA-Z0-9_-) and rejects the whole request if one fails — 400 Validation: Function at index 7 has an invalid name: "agent.ask_peer", returned in 281ms before any inference, so every turn failed identically. Every tool in this codebase is namespaced with a dot, and the orchestrator groups tools by that prefix (a connector's id must equal its tool prefix), so the names cannot be changed at the source. sanitize_tool_names rewrites them for ProviderInfo.strict_tool_names providers and restore_tool_names undoes it on the way back. Three things it must do or it half-works: rewrite the history too (an assistant turn already in the transcript carries tool_calls[].function.name, so round two fails where round one passed); restore the name on what the model called (otherwise the orchestrator is handed library_search, matches no registered tool, and reports the model hallucinating a tool it was in fact given); and disambiguate collisions rather than merge them — library.search and library_search both sanitize to the same string, and merging would route one tool's call to the other, a wrong action taken confidently.

A provider's error body survives to the message. raise_for_status produces the status line and a link to MDN, so a provider that said exactly what was wrong is reported as an unexplained 400 — diagnosing the NIM failure above meant digging the response body out of the telemetry ring, which is not something a user can do. raise_for_status_with_body puts the (truncated — some providers echo the whole request back) body in the exception message instead.

A rate limit is waited out, briefly. A 429 Too Many Requests or 503 means "not now", not "no", and nothing used to retry it. An eval sweep against NVIDIA NIM's free tier lost 8 of 15 cases to 429s, and a chat turn that hit the limit simply ended in an error. chat and chat_stream are wrapped in _retry_rate_limits, beneath traced_chat so that one span covers every attempt. It covers all dialects:

  • httpx's HTTPStatusError and litellm's exceptions, which carry status_code;
  • up to 3 retries, honouring Retry-After (seconds or an HTTP date), otherwise exponential backoff with jitter (about 1, 2, 4 s);
  • at most 30 s of waiting in total (RETRY_BUDGET_S). A Retry-After asking for more than is left raises at once instead: the user hears about it now rather than after a silent minute.

A stream is retried only while nothing has reached the caller. A 429 is refused before the first token, but a stream that emitted text and then failed cannot be replayed without showing the same words twice. Cancellation is never swallowed, so a stopped turn stops waiting immediately. Each retry is logged at INFO with the provider, the status and the wait.

orjson is a hard dependency because litellm's OpenRouter transformation imports it; without it every OpenRouter call fails as an APIConnectionError naming a missing module, which reads as a network fault.

A hosted model that never answers ends the turn. litellm's default request timeout is litellm.request_timeout = 6000 s, and nothing passed a smaller one, so a free OpenRouter model sitting in its queue left the chat on its … bubble for up to 100 minutes with the input disabled. Every litellm call now carries timeout (HOSTED_FIRST_CHUNK_S, 120 s), and the streamed paths run through _watch_stream, an explicit watchdog: no first chunk within 120 s (connect included) or a 90 s gap (HOSTED_IDLE_S) once streaming raises StreamStalled — a TimeoutError, so the turn's catch-all reports it as error + done. The transport's read timeout alone is not enough: OpenRouter keeps a queued request's connection warm with SSE comment lines, which reset a read deadline without ever yielding a chunk.

The chat can stop a turn. While a turn is in flight the send button is a Stop button: it aborts askAgent's signal, which sends cancel and resolves at once; the backend cancels the task (_turns_by_id, owner-checked — a turn id is not a secret on a shared socket hub) and still emits done for any other listener. The abort handle lives at module level beside the transcript, so a remounted pane can stop a turn it did not start.

Keys are entered in two places, both writing the same route: the Model provider API keys section of the Settings page (ApiKeysSettings), and inline in onboarding — where an unreachable hosted provider offers a key field in the same slot an unreachable local one offers "install and start it". A provider with a key then appears in the per-agent provider override dropdown like any local one, and the model override dropdown lists that provider's models (not the configured provider's — an agent pointed at OpenRouter used to be offered Ollama's model names, every one of which means nothing on that server, and the failure only showed up mid-turn).

Operationollama dialectopenai dialectUsed by
List models / probeGET /api/tagsGET /v1/modelslist_models — /agent/status
Tool-calling round (streamed)POST /api/chat (stream:true, think)POST /v1/chat/completions (stream:true)chat_stream — the orchestrator loop
One-shot streamed answerPOST /api/generatePOST /v1/chat/completions (SSE)generate_stream — legacy /agent/chat
Short fill-in completionPOST /api/generate (stream:false)POST /v1/chat/completions (stream:false)generate — /agent/complete (autosuggest)
Pull a modelPOST /api/pull— (unsupported)/agent/pull

Ollama's default URL can be overridden with HORRIBLE_OLLAMA_URL; a backend-spawned vLLM advertises its own port, and so does a spawned llama-server — for that one alone the live manager is checked before the saved endpoint, because it takes an ephemeral port when the default is occupied and a saved :8080 would point at nothing. All of these outbound calls go through instrumented_client(), so they surface in the observability I/O stream with full request↔response bodies — the streamed tool-calling round, answer, and pull responses all via tee_stream. See observability.md.

Hierarchical tools & progressive disclosure​

Local reasoning models (e.g. gemma4:e2b under Ollama) have a strict tool-count ceiling: list 40+ tool definitions and the model silently skips its thinking phase and streams the final answer from the first token; with ≤39 it reasons normally. The whole catalog (layout + peer + every module/plugin tool) is well past that, so tools are exposed hierarchically and injected on demand rather than flattened and pruned.

Tools belong to groups by name prefix (files, editor, terminal, visualizer, database, records, network, …; the dotless layout verbs are the layout group). Each turn the model sees only:

  • the core — the layout verbs (see below), the peer tools (list_peers, agent.ask_peer), and two meta-tools: list_tool_groups() (discover groups
    • counts) and load_tools(groups[]) (enable a group); plus
  • use_skill(name), but only when a skill is enabled — a third meta-tool that is absent (costing nothing) on an install with no skills; plus
  • the tools of every active group.

Skills are the same idea one tier up, and they reuse this machinery rather than paralleling it: a one-line description per skill rides every turn (that is the trigger), the instructions arrive only when use_skill asks, and a skill's allowed-tools are resolved to groups and activated in that same call — so the instructions never name tools the model cannot see. See Skills.

The layout verbs split by cost, because they were the largest always-on block and most agents never rearrange anything. The four read verbs (list_available_panes, list_open_panes, get_layout, get_pane_context) plus show stay in every agent's core — any agent may need to see what the user is looking at, and show is how it puts something there. The arrangement verbs are loaded on demand via load_tools("layout").

Including for main. They used to stay in main's core on the reasoning that driving the shell is its job — but that made the loosely prompted agent the most starved one, carrying ~1,455 tokens of geometry schemas on turns that only wanted to read a file. show covers "open/go to X" in a single call, and the narrow layout keyword list preloads the arrangement verbs on the turns that genuinely rearrange things. A spec that names layout in its tool_groups still opts back into all of them up front. layout is shell control rather than a capability, so it stays loadable for every agent regardless of scope — scoping only decides whether it costs schema space up front. Two traps worth knowing: _all_dynamic_tools dedupes against the core, and that dedupe base must be spec-aware or the arrangement verbs are swallowed and load_tools("layout") silently returns nothing; and the layout keyword list is deliberately narrow (pane, window, snap, … but never open/close), since a keyword that fires on most turns would give back exactly what the split saves.

run_agent_turn recomputes the presented list each round from active_groups, so a group loaded this round is injected into the next model call (true dynamic injection — nothing is ever dropped, just not-yet-loaded). active_groups is:

  • seeded by a keyword preload (_preload_groups): a prompt mentioning a file, shell command, etc. auto-activates that group so common asks stay single-round;
  • grown when the model calls load_tools; and
  • grown forgivingly — if the model calls a known tool from a group it never loaded, _dispatch_call activates the group and runs the call anyway.

A TOOL_BUDGET = 38 cap remains only as a backstop (core is always kept first). The /tools command (list_tools→tools) still reports the full catalog, labeled by group, independent of what any single turn has loaded.

The cap is resolved per agent, not global: tool_budget_for(agent_id) returns 38 for a model running on this machine and TOOL_BUDGET_HOSTED = 96 for one reached through a hosted API. The 38 ceiling is a measured property of small local models, not of tool calling, and applying it to a frontier model bought nothing while costing a load_tools round per group past the third.

The signal is ProviderInfo.hosted — an API reached with a key, versus a server on this machine — and deliberately not the model name: parsing a parameter count works for llama-3.2-3b-instruct and not at all for gpt-5, so a name-based rule would have left every hosted model on the floor, which is the case it exists for. Anything unidentifiable resolves to the floor. Being wrong low costs a round trip; being wrong high costs a model that silently stops reasoning.

Watch the core, not the cap

The cap was 44 — deliberately above the ≤39 ceiling, so that a legitimate training+notebook turn would not be truncated. A backstop above the cliff protects nothing, though: it lets a turn through in exactly the state the cliff warns about. What actually keeps a turn safe is a small always-on core, and once the arrangement verbs left it that same flagship flow costs 11 + 12 + 9 = 32, so the cap now sits at 38, under the cliff, without truncating real work.

Truncation is also easy to miss, which is why it now logs at ERROR and names the tools it dropped (plus a dropped list in the stats the interpretability pane reads). A silently-dropped tool looks exactly like a model that chose not to use one.

The core has come down in two steps, both measured with the interpretability recorder's own counter:

toolsalways-on schema
Originally34—
After grouping research.*/arxiv.* (see the danger note in agent-tools)26—
After moving the arrangement verbs out11-68% (15,299 → 4,867 chars)

The system prompt halved alongside it (3,017 → 1,375 chars), because the geometry rules it used to carry moved into the layout group's guide.

Contributions to the layout shell​

  • Panels: chat.conversation (main chat view), chat.sessions (session list), chat.inspector (tool-call/trace detail, opened on demand) — each in a window.
  • Commands: chat.new, chat.focusInput, chat.openSession, chat.toggleInspector, chat.stopGeneration.
  • Default keybindings: declared for new-session and focus-input; bound through the shell keybinding service.
  • Dashboard widgets: chat.recentSessions (see dashboard.md).
  • Region strip: the browser docks on the right (browser.view, key w). The agent's browser.open opens it there, beside the turn that asked. See Docked in the agent chat.

Backend surface​

backend/modules/chat/ — session CRUD over HTTP; streaming (tokens, tool-call events, status) over the shared WebSocket on chat.* channels. Agent execution lives entirely in the backend; the frontend only renders the stream and sends user input. Pydantic models define the event schema.

Browser vs desktop​

The conversation experience is identical — it's all backend traffic over the shared socket. Differences are notification/summon ergonomics only:

ConcernBrowserDesktop
Completion notificationsWeb Notifications (notifications.system capability, tab must allow it)OS notification, click focuses window
Quick summonnoneglobal shortcut raises the window and runs chat.focusInput
Long-running agentskeep tab open (no background work in the page; the backend keeps running regardless)window can be closed to tray; stream resumes on reopen

Both layouts must tolerate socket reconnects mid-generation: the backend is the source of truth for session state, and the panel re-syncs on reconnect.