Module: agent chat cockpit
The core "agentic era" surface: converse with agents/LLMs, watch their tool calls stream, and manage sessions.
Status: first slice implemented — frontend in
packages/core/src/modules/agent/, backend in backend/modules/agent/. What
exists today:
- Onboarding on the home view (see layout-shell.mdx): detects a local
Ollama at
http://localhost:11434(override:HORRIBLE_OLLAMA_URL), offers a model picker defaulting togemma4:e2b, pulls models with streamed progress (POST /api/agent/pull), and persists the choice (PUT /api/agent/config→$HORRIBLE_DATA_DIR/agent-config.json). - One-shot ask: the legacy
POST /api/agent/chat(NDJSON passthrough of Ollama's/api/generate) still exists; the home ask bar no longer uses it. - Orchestrator (layout control) — the home ask bar now drives the
agent orchestrator: a backend tool-calling loop that arranges the user's
workspace. Like the chat widget, it renders the model's streamed reasoning
(a collapsible "Reasoning" disclosure) and streams answer tokens live, not just
the final answer — so a turn shows the thinking instead of looking silent if the
final
contentis sparse. See "The orchestrator" below. - Conversational chat widget (
agent.chat, commandagent.openChat): a pane with a multi-turn transcript, a per-turn action log, and the model's reply. Each turn replays prior user/assistant turns to the backend ashistory(the backend stays stateless per turn). It drives the same orchestrator, so it reasons about the live layout and widget contents (via the read tools) and acts on them. SeeChatWidget.tsx. The pane is laid out as a sessions rail on the left (see Sessions), a slim header (rail toggle, the conversation's title, the trace toggle), the transcript as one centred column of at most 760px (assistant replies are flat prose, not bubbles; the user's turns keep a soft tinted bubble), and a composer card: the input, and under it the mode chip, the model picker and Send. An empty transcript greets with the 3D avatar (Avatar3D, now inpackages/core) cycling through its mood animations, above the starter prompts; it is not drawn once a conversation has turns. Theagent.avatarAnimationsetting (default on) turns it off. The transcript persists as named sessions and supports slash commands (both below). A running turn draws an activity timeline and each chat has a mode — Plan, Agent or Auto (see Activity timeline & modes). - Inline autosuggest source (
POST /api/agent/complete): a short non-streaming fill-in completion the editor's opt-in ghost-text autosuggest calls.
Per-widget agentTools + getAgentContext widget interop ship (see
agent tools & permissions). Remaining design:
a fuller chat/inspector cockpit.
Readiness: four states, not one
AgentStatus carries two independent booleans — configured (has a provider and
model been chosen?) and reachable (is that provider answering right now?) — and
two more failure modes live outside the payload: the backend not answering at all,
and the status not having resolved yet.
Those are four different problems with four different remedies, and the pane used
to render every one of them as the same grey input placeholder, Agent not ready.
That is the inversion of this product's own rule that a capability check reports
what it found, what it looked for and failed to find, and what it could not ask.
AgentReadiness (modules/agent/AgentReadiness.tsx) maps the status to one
banner above the input:
| State | Says | Offers |
|---|---|---|
| unresolved | Checking the agent | nothing — it is not a verdict |
| backend unreachable | Backend not answering | Retry |
configured: false | No model selected | Choose a model → Settings |
reachable: false | Can't reach <provider> | Retry · Change provider |
| ready | nothing | — |
The endpoint appears only in the reachable: false case, as telemetry in mono:
nothing was tried yet when no model is configured, so naming an endpoint there
would be inventing a cause. planFor is exported and tested
(__tests__/readiness.test.ts) because the distinction between those states is
the whole point of the component, and collapsing them does not error — it just
tells everyone the least useful true thing.
An empty transcript carries three clickable opening prompts rather than one grey sentence, each exercising a different half of what makes this agent unlike a chat box: layout control, pane reading, buffer editing. Deleting a conversation goes through a confirm that names the conversation — there is no undo behind it, and the control sits beside "New chat".
Sessions (persistence)
Chat transcripts persist server-side as named sessions — a small chat backend
module (backend/modules/chat/) that mirrors the workspace store: file-backed JSON
at $HORRIBLE_DATA_DIR/chat-sessions.json, CRUD under /api/chat/sessions (list
returns metadata only; GET /{id} returns the full transcript; PUT /{id} is a
partial upsert of title/messages/mode; plus /active and DELETE). Only a
save that carries messages bumps updated: it means conversation activity, and
the sessions rail orders and buckets by it, so a rename or a mode switch must not
move an old chat to the top. The chat widget
reads the active session the first time it talks to an agent, lazily creates a
session titled from the first prompt, and auto-saves each completed turn.
The sessions rail (SessionsRail.tsx) lists them. Its head holds the agent
picker (populated from GET /api/agent/roster; just the agent's name when the
roster has one agent or the chat is pinned to one) and New chat; under that, a
title search, then the conversations newest first, bucketed Today / Yesterday /
Previous 7 days / Previous 30 days / Older by local calendar day
(session-groups.ts — updated is epoch seconds). Each row shows its age (3h,
2d); hovering or focusing it reveals rename (inline; Enter saves through
PUT /{id} with a title, Escape cancels) and delete (always confirmed,
naming the conversation, since there is no undo). The running conversation carries
a pulsing marker and cannot be deleted mid-turn.
The rail sits inline when the chat is at least 720px wide; the header's toggle
then shows or hides it, and that choice is remembered per viewer
(localStorage agent.sessionsRail). Narrower — the agent docks on the right at
420px by default — it starts hidden and the toggle slides it over the chat with
a scrim; picking or starting a conversation closes it. A compact chat (the interp
Ask strip) has no rail; /clear and /agent still work there.
Sessions are per agent (agent_id on the session, ?agent= on the list
endpoint), so each agent keeps its own conversations and active pointer. See
Specialized agents & delegation.
Frontend client: packages/core/src/modules/agent/sessions.ts.
Each message is saved with its orchestrator turn_id. An assistant message also
saves sub_turns, the sub-agents it delegated to. The chat creates the id itself
(newTurnId(), passed to askAgent as turnId), so the id is on the transcript
before the first frame arrives. It is the join key into interpretability and
trajectories. The toolbar's trace toggle slides out a trace drawer beside the
transcript. The drawer shows the trajectories
Agent view: the prompt by source, a
round-by-round diff, the steps and the sub-agents. It can be resized, popped out
into its own window, and docked back. Each reply's trace chip opens the drawer
pinned to that turn.
Messages saved before this carry an empty turn_id and have nothing to trace.
The transcript lives outside the component
packages/core/src/modules/agent/chat-state.ts holds the transcript, the in-flight
turn, the draft and the active session id in a small module-level store keyed by
agent id; ChatWidget reads it through useSyncExternalStore and owns none of it.
That is not tidiness, it is the fix for "the chat resets when I prompt". The shell
unmounts this pane routinely — an inactive tab is unmounted rather than hidden
(Unmount is not close),
and the agent's own layout tools (place_pane moving it to another group,
say) move it. With the conversation in useState, each of those blanked it: the
pane came back showing the starter prompts, re-read the node, and got the last
saved state — which for a turn still streaming is the conversation as it was before
the question. So the most reliable way to lose a conversation was to ask the agent to
open something, which is the one thing this agent is for.
Two consequences follow, and both are the point. A remount is invisible — same
transcript, same draft, no refetch and no flash of the empty state. And a turn
survives its pane: askAgent's streaming callbacks write to the store, not into
a dead component's setState, so tokens that arrived while the pane was unmounted are
there when it comes back rather than silently dropped.
The mount effect therefore loads only when that agent has not been read yet
(loaded), and loadSessions will not overwrite a transcript it already holds for
the active session — a reload landing mid-stream would otherwise truncate the answer
back to the previous turn. State is per app session and deliberately not persisted:
the node is still the durable store, this is only what makes the pane stateless.
A failed read is never rendered as an empty transcript. loadSessions records
loading / ok / failed rather than catching into a blank pane, because empty is
indistinguishable from deleted — and it was worse than cosmetic: with no activeId,
the next message ensureSession sent created a second session, orphaning the real
history rather than merely hiding it. So ensureSession re-reads before creating
anything when the last read failed, the restore retries once on its own (boot races
the backend), and the pane shows what happened with a Retry instead of a chat that
looks wiped.
A model picker (ModelPicker) sits in the same bar, listing every reachable
provider's models — so a hosted provider appears there the moment a key is saved, and
switching models does not mean leaving the conversation for the settings page.
It is a searchable combobox, not a <select>. One OpenRouter key puts several
hundred models in this list (465 on a typical node), and a native select can only be
walked: its type-ahead matches a prefix, so the queries people actually have — every
:free model, everything with qwen in it — are unreachable by typing. The filter is
a substring match over the provider label and the model id together, word by word
and ANDed, so free narrows to the free tier, lm studio to one provider, and
free qwen to the intersection. Arrow keys and Enter pick; the panel is
position: fixed and measured off the trigger (flipping above it when, as in the
composer, there is no room below), because the composer sits inside a
pane that clips its overflow — and portaled to its pane window's body
(usePaneWindow(), so it opens in a pop-out too), because inside a pane box fixed
alone is not enough (see
the desktop shell). The settings
page's ModelCombobox does the same. It writes the provider and the model together: they are two settings
(agent.<id>.provider, agent.<id>.model), and the settings page edits them as two
controls, but here one option carries both, because a model name means nothing on a
server that does not have it — minimax/minimax-m3:free selected while the provider
still says Ollama is not half-configured, it is broken, and it fails mid-turn rather
than at the click. The blank choice clears both back to the configured model. A saved
override the live list does not contain (a provider that went down, an id typed on the
settings page) stays selectable, so the control never silently renders as something
other than what will run.
Slash commands
/-prefixed inputs run locally in the chat widget (no model turn) and render as
ephemeral system output that is never persisted or replayed to the model
(packages/core/src/modules/agent/slash.ts): /tools (the agent's live tool
catalog, fetched via the list_tools→tools WS round-trip), /help, /panes
(open panes), /clear (new session), /agent (list the roster agents or switch
the pane to one: /agent dba), /llm (query, set, or reset orchestrator LLM
hyperparameter overrides), /plan, /act, /auto and /mode <name> (switch the
chat's mode — with text after them they switch and send it, so
/plan add a dark-mode toggle is one step; Agent mode is /act because /agent
is the roster command), and honest placeholders /mcp and /skills
(those subsystems don't exist yet). Typing / shows a suggestion list.
Activity timeline & modes
A turn renders as an ordered list of steps (turn-steps.ts, pure reducers
fed by the WS events; drawn by ActivityTimeline.tsx):
- Thoughts groups — consecutive reasoning and tool calls fold into one
collapsible
THOUGHTSgroup (open while the turn runs), headed by its call count and duration. Each call is a row: status glyph, a label fromtool-labels.ts, a one-line argument summary, and its duration; clicking it shows the arguments and a result preview. A row waiting on the permission gate shows a lock. - Notes — prose the model writes before a tool call becomes a paragraph
between groups; the
answerevent stays the authoritative final text, which renders as markdown. - Live line — "Reasoning…", "Running Read file…", "Waiting for your
answer" or "Needs approval", with an elapsed counter (a
setInterval, so it ticks in a backgrounded pane); afterwards "Worked for 1m 12s". - More ↓ — the log follows new output only while you are at the bottom; scroll up and a jump button appears instead.
Three backend tools feed cards instead of rows (CARD_TOOLS; a row is shown only
if the call fails):
| Tool | What the chat draws |
|---|---|
update_todos({items}) | A TASKS checklist pinned to the turn (and compact above the composer while busy). |
ask_user({questions}) | QuestionCard: up to 4 questions, 2–6 options each, multi-select, an "Other" box, Skip. The turn waits for the answer (question_response); a cancelled turn resolves it as dismissed. Nobody to ask (a remote peer's turn) → an error result, never a hang. |
submit_plan({plan}) | PlanCard — Plan mode only. Ends the turn; Approve & run switches the chat to Agent and sends "Approved. Carry out the plan above." |
The interaction tools are always-on core tools; a delegated sub-agent does not get them.
Modes (modes.ts, ModePicker.tsx) map onto the
permission modes: Plan = plan,
Agent = default, Auto = autonomous. The chip at the left of the
composer picks one (Shift+Tab cycles); the composer's top border and placeholder
follow it. The mode is per chat — saved on the session (mode), sent with
every ask — and a chat that never chose one follows
agent.permissions.mode from Settings (acceptEdits shows as Agent).
Precedence on the backend: conn.force_mode (remote peers) > the chat's mode >
the roster agent's permissionMode > the global setting.
In Plan mode the backend drops every side-effecting tool, and every
DESKTOP_TOOLS verb (PLAN_MODE_BLOCKED), from the offered set (_gate still
refuses them if called), adds submit_plan, and appends a plan-mode addendum to
the system prompt. Pane open/close stays available: it is how the agent shows
things.
Saved turns carry steps, todos, plan {text, status}, started_at and
ended_at (ChatMessage in backend/modules/chat/models.py), so a reloaded
chat redraws its timeline. Chats saved before the timeline render from their old
reasoning / actions fields (legacySteps).
The orchestrator
The Agent is the app's orchestrator: a backend-resident brain that drives the
UI through tools. Per [[agent-orchestrator-architecture]] the brain runs on the
backend (so agent logic/history stays server-side) while the tools execute in the
frontend; the two halves talk over the shared /ws socket on the agent
channel — the turn is bidirectional and stateful, so running it on the socket
(not HTTP) avoids correlating an HTTP request with the right browser.
End-to-end walkthrough
The numbered phases below trace one turn through the code; the diagram above is the same flow visually.
- Manifest (once per connection). Before any turn, the frontend pushes its
capability manifest — the serialized agent-exposed commands and per-pane
agentTools, handlers stripped (manifest.ts,initAgentManifestSync) — on socket open and on every registry change. The backend stores it on theWsConnectionand forgets it on disconnect (hence the re-push on reconnect)._tools_forlater merges it with the staticLAYOUT_TOOLSinto the tool list the model sees each round. - Ask. The chat widget (or home ask bar) calls
askAgent(prompt, cb, history).askAgentattaches the user's focused-pane snapshot (readActiveAgentContext— currently the active editor buffer) and sendsask {turnId, prompt, history, context}on theagentchannel. - Turn setup (backend).
handle_agent_messageroutes theaskto a detachedrun_agent_turntask — it must run detached (asyncio.create_task) because it awaitstool_results that arrive on the same receive loop; awaiting inline would deadlock. The task assembles the messages (SYSTEM_PROMPT+ sanitizedhistory+ the focused-buffer system message from_active_editor_message+ the user prompt), resolves the tool list (_tools_for), the model (_orchestrator_model— the override or the configured model), and the sampling temperature (_tool_temperature). - The loop (≤
agent.maxRounds). Each round streams the provider viaproviders.chat_stream, which first flattens the system tier (see below); the model's reasoning (thinking/reasoning_content) and answercontenttoken-deltas relay live asreasoning/tokenevents the chat widget and home ask bar render as they arrive. A round ends with eithertool_callsor a final answer.
The step budget, and how a turn that runs out ends
A turn may spend agent.maxRounds rounds of tool calls (MAX_ROUNDS, 24, when the
setting is 0). It is a guard against a model that never stops calling tools, not
a work limit — but at its old value of 8 it was acting as one, invisibly. In the
recorded turns the counts tapered off to a single turn at 7, then 32 sat at exactly
8: the shape of a budget being hit rather than of work finishing.
Two things make that shape a bug rather than a threshold doing its job:
- Discovery used to be charged as work. Under progressive disclosure the model
pays a round for each
load_toolscall before it can touch the task — one observed turn spent rounds 3 and 5 loading groups — so a third of a short budget could go on reading the menu. A round whose calls were all meta tools is now free, up toMETA_ROUND_ALLOWANCE(6). Bounded rather than free, because a model that loops loading groups for ever must still stop. - Running out threw the work away. The loop returned a literal
"(stopped after too many steps)", so a turn that had read six files and was ready to report ended with a sentence about the machinery. Exhaustion now spends one last tool-less round (_final_answer): the model is told the budget is gone and asked to answer with what it has. Offering no tools is what makes it terminate. If that round returns nothing, or fails, the turn still says plainly that it stopped early and how many steps it used — an honest stop beats an invented answer, and both beat silence.
Passing no tools is a real request, so providers now omits tools entirely when
the list is empty rather than sending [], which several OpenAI-compatible servers
reject outright.
- Tool execution + gating. For each
tool_call,_gatedecides: read-only and layout verbs pass straight through; side-effecting tools are evaluated against the permission mode + rules, prompting the user viaapproval_request/approval_responseon anASK(and persisting a rule on "always allow"). An allowed call is relayed to the browser astool_call, whereexecuteToolruns it against the registry /layoutControllerseam /executeDynamicTooland returns atool_result; a denied call returns an error to the model instead. Either way the result is appended tomessages(provider-formatted bytool_result_message) and the loop continues so the model can react to it. - Completion. When a round returns no tool calls, the backend sends the
authoritative
answerthendone(orerroron an HTTP failure). If the model narrated an action without emitting a call, the loop first grants one nudged retry — on every dialect, withtool_choice:"required"added where the provider supports it — before giving up and answering. The widget renders the reply plus the per-turn action log, and a completed turn auto-saves to its session.
The system tier is several messages until the wire
run_agent_turn assembles the system tier as separate messages: the spec's system
prompt, then the skill catalog, then the guides for the groups the turn
starts with. That split is deliberate — the
interpretability recorder tells the blocks apart by position
and content marker (CATALOG_MARKER) to attribute each one's per-turn token cost, and
one glued string would make that impossible. The forced-tool nudge adds a fourth,
mid-conversation.
Many models' Jinja chat templates reject both shapes:
Error: Jinja Exception: System message must be at the beginning.
That is a 500 from the engine, not a worse answer — the turn is lost outright. So
the split survives right up to the wire and providers.normalize_system_messages
flattens it inside chat/chat_stream, the one chokepoint every dialect and every
call site passes through:
- leading system messages are joined with a blank line, order preserved;
- a later one becomes a
usermessage, because its position is the point — a nudge merged into the preamble would arrive before the failure it answers.
It runs on every dialect, not just the strict ones: the flattened form is semantically identical, and branching would let a template's strictness decide whether the nudge was delivered at all. Because the recorder captures in the orchestrator, before the provider call, it still sees the unflattened blocks.
The tool-calling loop itself is factored into a reusable
run_agent_loop(conn, messages, tools, …, emit) → str (orchestrator.py):
run_agent_turn assembles the messages/tools and supplies an emit that sends
agent-channel reasoning/token, while the flow canvas's
Agent nodes reuse the same loop with a flow-channel emit — so chat and flow
share one gated tool-calling implementation.
Loop (backend/modules/agent/orchestrator.py): on an ask, a detached task
streams the user's local Ollama /api/chat (stream:true) with
tools=LAYOUT_TOOLS via providers.chat_stream. Each round's reasoning
(thinking/reasoning_content) and answer content token-deltas are relayed live
to the chat widget (reasoning/token events) as they arrive. If the model returns
tool_calls, each is relayed to the browser and the loop awaits the result before
continuing; otherwise it sends the final (authoritative) answer. Capped by
the step budget. Ollama gets think:true best-effort — models without a thinking mode
reject it (400) and the round transparently retries without, still streaming content.
Tool-calling reliability. Turns decode greedily (temperature 0, via
chat_stream's new temperature arg → Ollama options / OpenAI payload) — at
higher temperatures small local models narrate an action ("I'll call …") instead
of emitting the structured call. The temperature is overridable through the settings
store (agent.orchestrator.temperature). Other model hyperparameters can be overridden
using setting keys agent.orchestrator.contextSize (context window limit, maps to Ollama's
num_ctx), agent.orchestrator.maxTokens (maps to OpenAI's max_tokens / Ollama's num_predict),
and agent.orchestrator.topP (Top P nucleus sampling). When the model still answers in prose that
reads like an unemitted call, the loop gives it one nudged retry: a system message
saying it described an action without emitting the call, plus
tool_choice:"required" where the dialect has one. This runs on every dialect.
It used to be gated on openai, which meant the repair never fired for Ollama —
the default provider, and so never for the small local models that need it most;
tool_choice is extra leverage, not the mechanism. The loop never forces a tool
unconditionally: a plain conversational reply must stay a reply.
Malformed arguments are reported, not silently emptied. A tool call's arguments
arrive as a dict (Ollama) or a JSON string (OpenAI, and some Ollama models), and the
streaming paths accumulate that string from deltas — so a truncated payload is a real
failure mode. _coerce_args used to swallow anything unparseable into {}, which
meant the call ran with no arguments at all (close_pane with no instanceId,
files.delete with no path) and the model saw a plain failure with no hint its JSON
was to blame. It now returns an error alongside the args, carried on
ToolCall.arg_error, and _dispatch_call refuses the call — nothing is relayed to
the browser — returning a message the model can actually act on. An empty payload
is still a valid no-arg call.
Unknown tool names get a suggestion. A name the relay can't resolve comes back
with didYouMean — up to three near matches over the layout verbs plus the advertised
manifest (tool-names.ts, pure and unit-tested). It matches normalized names
(openPane → open_pane), dropped words (list_panes → list_open_panes), small
typos, and — as a last resort — other tools in the same namespace, which is what
rescues an invented verb like files.remove. For weak emitters, prefer a larger model (e.g. gemma4:12b) for the
orchestrator: the agent.orchestrator.model setting overrides only the
orchestrator's model — leave it on the configured agent model to reuse it, so
chat/autosuggest can stay on a smaller, faster model while the tool-calling loop runs
a stronger one. All knobs live in the Agent orchestrator section of the Settings
page (a custom section — OrchestratorSettings — because the model is a dropdown of
the selected provider's live models from /agent/status, not a static enum); the blank
"Configured agent model" choice clears the override. These hyperparameters can also
be inspected/configured directly in chat using the /llm slash command, or modified
by the agent itself using the gated agent.setHyperparameters tool. The ghost-text
/agent/complete path likewise samples at a low fixed temperature so completions
are stable, not creative.
Channel protocol ({channel:'agent', event, data:{turnId, …}}):
| Direction | event | data |
|---|---|---|
| client→server | manifest | {tools: SerializedTool[]} |
| client→server | ask | {turnId, prompt, history?, context?, agentId?, sessionId?, mode?} |
| server→client | tool_call | {turnId, callId, name, args} |
| client→server | tool_result | {turnId, callId, ok, result?, error?} |
| server→client | approval_request | {turnId, approvalId, tool, specifier, mode, callId, args} |
| client→server | approval_response | {approvalId, decision, rule?} |
| client→server | list_tools | {} — request the live tool catalog |
| client→server | cancel | {turnId} — Stop; only the asking socket may |
| server→client | tools | {tools: [{name, description, source}]} |
| server→client | reasoning | {turnId, agentId, delta} — streamed thinking |
| server→client | token | {turnId, agentId, delta} — streamed answer |
| server→client | delegate_token | {turnId, agentId, delta} — sub-agent's stream |
| server→client | delegate_start | {turnId, subTurnId, agentId} — a sub-turn began |
| server→client | delegate_done | {turnId, subTurnId, agentId, ok} — …and ended |
| server→client | tool_start | {turnId, agentId, callId, name, args, round} — every call, before dispatch |
| server→client | tool_end | {turnId, agentId, callId, ok, denied, preview, durationMs} |
| server→client | question | {turnId, questionId, questions} — ask_user |
| client→server | question_response | {questionId, answers | null} — null = skipped |
| server→client | todos | {turnId, items} — update_todos |
| server→client | plan | {turnId, text} — submit_plan |
| server→client | answer | {turnId, agentId, text} — final, authoritative |
| server→client | done / error | {turnId, agentId, …} |
agentId selects (and tags) the roster agent driving the turn — omitted it
means main, so pre-roster clients keep working. See
Specialized agents & delegation.
sessionId is the persisted chat session the turn belongs to. It only labels the
trace (gen_ai.conversation.id, a thread in Opik — see otel); the
backend stays stateless per turn, and history remains the conversation.
The frontend pushes the capability manifest (manifest) on connect, on every
reconnect, and whenever the registry changes
(packages/core/src/modules/agent/manifest.ts, initAgentManifestSync): the
serialized agent-exposed commands and per-widget/panel agentTools (handlers
stripped). The backend stores it on the WsConnection, groups the pushed tools by
prefix, and exposes them via progressive disclosure (see Hierarchical tools
& progressive disclosure): each turn
the model sees the layout/peer core + meta-tools + whatever groups are active, and
load_tools injects more as needed. Relayed calls that aren't layout verbs are
dispatched on the frontend via executeDynamicTool. Side-effecting calls pass through the permission gate
(_gate in the orchestrator) before relay: read-only tools pass straight through,
denied calls return an error to the model, and an ASK decision prompts the user
via approval_request/approval_response (the in-chat approval UI is the next
slice) — see agent tools & permissions.
Specialized agents & delegation
The orchestrator is one of several roster agents — fully separate loops, each
with its own system prompt, tool-group scope, per-agent settings, and chat
sessions. Built-ins (backend/modules/agent/roster.py):
| id | scope (tool_groups) | preloaded | default mode |
|---|---|---|---|
main | unrestricted (keyword preloading, delegate) | by prompt keywords | session mode |
coder | files, editor, terminal, code, symbols | editor, files, symbols | acceptEdits |
dba | database, symbols | database | session mode |
researcher | browser, search, library, github, google, research, arxiv, records | research, library, arxiv | session mode |
intake | records, browser, research, library | records, research | session mode |
trainer | training, recipe, cells, evals, localtrack, datasets, llamacpp, wandb, editor, files | training | session mode |
interp | interp, lens, llamacpp, agentpedia, kb, arxiv | interp, lens | session mode |
interp is the interpretability suite's guide: it is the agent the suite's Ask strip talks to, and its interp group is declared by the suite pane in the frontend (interp.state, interp.attention, interp.goto).
Two of researcher's groups are load-bearing for the panes it works beside, and
both were dead buttons before: the Review pane (records — it could display a
proposal but not file one), and the search panel (search — the pane's own results
were unreachable to the agent beside it). A pane whose agent lacks the matching group
is the failure mode to watch for.
trainer shows the other half of that trap. hardware is pointedly not in its
scope, even though the training panes read a hardware probe: a group is a tool name's prefix
(_group_of), that namespace holds settings keys rather than tools, so naming it
would permit nothing at all and say nothing about it. Check that a group you list has
tools under it, not just a module with that name.
Its scope is long because the loop is long, and because training was
split into four. That group had reached 34
tools — a third of them registered by the frontend, which is why every count in a
comment said 14 — and with a ~9-tool core there was no room for evals beside it: the
loop this agent's own prompt prescribes did not fit under TOOL_BUDGET, and the
overflow was dropped with nothing but a log line to show for it. Only training is
preloaded; the rest cost a load_tools round rather than schema space on every turn.
datasets was missing entirely, which is the same dead-button trap one level in: the
agent that fills in a recipe's dataset_id could not search, peek at or token-stat a
dataset, and can_delegate is False, so it had no way to ask anything that could.
There is deliberately no crm agent. It existed to enrich contacts and deals
from the open web, which is what researcher already does now that it carries
records alongside browser/search/github/google — so it was a persona rather than
a capability, and its prompt hardcoded a contacts/deals model the records substrate
never required. See records.
Mechanics (run_agent_turn(agent_id=…)):
- A specialized agent skips keyword preloading — its
active_groupsstart at the spec'spreload_groups, andlist_tool_groups/load_toolsonly see its allowed groups (no forgiveness auto-load outside them). The peer tools andagent.delegateare core only for specs that declare them (main). - The picker chooses the agent. The chat opens as
main; choosing another from the picker is remembered (inlocalStorage, keyed by the desktop's workspace row) so it survives a reload. (Workspace presets used to declare a persona each — Data Ops as thedba, Data Entry asintake— until they were removed with the tiling frame.) - Per-agent settings
agent.<id>.provider/endpoint/model/temperature/contextSize/maxTokens/topP/permissionModefall back to theagent.orchestrator.*keys, then defaults (roster.agent_setting). The permission-mode override threads through the gate; the user's explicit allow/ask/deny rules still apply unchanged. - Per-agent provider (
roster.resolve_provider).provider/endpointused to be global whilemodelwas per-agent, which made "run the coder on the node's llama.cpp server and leave the orchestrator on Ollama" unexpressible: a model name means nothing on a server that does not have that model. An override naming an unknown provider is ignored rather than fatal, so a stale settings value cannot take the agent down. It must be resolved at all threerun_agent_loopcall sites — the orchestrator,delegate.py, and the flow executor's Agent node — because a site left onprovider_for(config.provider)does not fail, it quietly runs on the global provider. - Per-agent sessions:
ChatSession.agent_id(defaultmain) + anactive_by_agentmap and aGET /api/chat/sessions?agent=<id>filter — all additive, so pre-rosterchat-sessions.jsonfiles load unchanged. GET /api/agent/rosterlists the roster (id, name, description, scope) for the chat widget's agent picker; plugins add agents withhost.add_agent(AgentSpec(…))(see backend plugin SDK).
Delegation — agent.delegate(agentId, prompt) (backend-static, side-effecting,
specifier {agentId}) runs a whole sub-turn of the target agent on the same
browser connection, so unlike agent.ask_peer's tool-less remote turns the
specialist can actuate — its tool calls relay to the same frontend and pass the
same permission gate under its own mode. The sub-turn gets a fresh turn id (its
stream stays out of the parent chat's turn-scoped subscription); its answer deltas
surface as delegate_token events on the parent turn, and the final answer
returns to the caller as an ordinary tool result. One level deep by design:
specialists don't see the delegate tool (spec.can_delegate), and a
DELEGATE_TIMEOUT_S (300 s) bounds the sub-turn.
Agent-to-agent (talking to a peer's agent)
When this node is connected to peers (see Module: network and
Distributed peer fabric), the orchestrator gains
two backend-static tools, merged into the catalog by _tools_for and resolved
in _run_backend_tool against the process-global PeerHub (no browser handler):
list_peers()— the connected peer nodes (node_id, name, capabilities). Read-only.agent.ask_peer(peerId, prompt)— ask another user's agent and get its answer back as an ordinary tool result. Side-effecting (specifier{peerId}), so it passes the local permission gate first.
The network module's Agent Relay widget (network.relay) is the direct UI over
this: it posts to POST /api/network/ask-peer (the same agent_bridge.ask_peer),
so you can ask a peer's agent without going through your own model. See
Module: network.
The cross-peer boundary is the security crux:
- A remote turn runs behind a
RemoteAgentConn(no browser) under a forced mode =network.remoteAgentMode(defaultplan→ read-only). It is given no actuating tools, so a remote agent answers from the model but cannot drive your panes, files, or terminal. Its gate never blocks on an absent human — anASKdecision becomes a deny. - Admission requires
network.allowRemoteAgentand a trusted peer. - Fan-out is bounded: an
origin_chaincycle guard (reject if this node is already in the chain), aMAX_PEER_HOPScap, and aPEER_AGENT_TIMEOUT_Srequest timeout.
Peer-wire message types (on /peer-ws, not the agent channel):
agent_request → agent_result (caller awaits the reply via PeerHub.request).
Focused-pane context. Pane snapshots are normally read on demand (the model
calls get_pane_context — pull, not push). As a convenience the frontend also
attaches the user's focused pane snapshot to every ask as context: the
editor's BufferView marks its instance active on focus/load (and clears it on
unmount) through core's agent-context ambient slot
(setActiveContextInstance/readActiveAgentContext), and askAgent reads it. When
that snapshot is an addressable editor buffer, the backend injects it as a system
message just before the user turn (_active_editor_message) handing the model the
open code up front, so "alter/refactor/fix this code" modifies the open buffer and
writes it back with editor.proposeEdit(uri=…) — no list_open_panes +
get_pane_context discovery first (a multi-step dance weak local models often skip).
Unsaved scratch buffers (no uri) are excluded.
This buffer rides on every turn too, and was long the one context injection with no
cap at all — a single large file could dwarf everything else in a small model's
window. agent.activeBufferBudget (24000 chars, 0 = unlimited) now clips it in the
frontend. The clip sets a truncated flag, and honouring that flag is safety-critical
rather than cosmetic: the normal message tells the model to write the complete
buffer back via editor.proposeEdit, so doing that from a clipped copy would silently
delete everything past the cut. A truncated snapshot therefore flips the instruction —
the model is told the code is cut off and must re-read the file before any whole-buffer
write. Pinned by
test_agent_orchestrator.py::test_truncated_buffer_forbids_writing_the_whole_file_back.
Workspace context. Alongside the focused pane, askAgent attaches
context.panes — a snapshot of every visible pane that exposes one
(readVisibleAgentContexts: each non-minimized window's active tab, topmost first;
background tabs are unmounted, so their state isn't readable anyway). The backend
formats it as a compact index just before the user turn
(_workspace_context_message). This is what lets the agent see the desk it sits at:
the researcher already knows which row the Review pane has open,
and the intake agent which document is on screen, without spending a
list_open_panes +
get_pane_context round on it. It rides on every turn, so it is budgeted —
agent.workspaceContext (on/off), agent.workspaceContextPanes (6) and
agent.workspaceContextBudget (1500 chars per pane, clipping oversized fields
rather than the whole object, so the JSON stays parseable and the note points the
model at get_pane_context for the full read).
The frontend executes each relayed tool_call against the registry and the
registry.layoutController seam and the desktop controller (installed by the shell,
layout/install.ts), then replies. This
runs in an always-on relay handler (initAgentRelay in
packages/core/src/modules/agent/orchestrator-client.ts, registered at boot) rather
than inside a chat turn — so the same relay serves chat turns and
flow canvas runs (Tool nodes + flow Agent nodes). askAgent
keeps only a turn-scoped listener for its per-turn action log. The executor itself
lives in packages/core/src/modules/agent/tool-exec.ts (executeTool) — the
shared relay surface the Python REPL's dash SDK drives too, so
they all stay in lockstep.
LAYOUT_TOOLS (app-level verbs over the workbench — every pane is a tab in a
group, in the centre split, an edge dock or a popout, and regions are
per-pane strips; the model discovers ids via the read tools, so the catalog stays
frontend-owned). Reads: list_available_panes (views with role, sections +
regions), list_open_panes (instances with where each lives), get_layout (the
centre tree with group ids, the docks — the orientation read),
get_pane_context(instanceId). Panes: open_pane(id) (where its role puts it, or
focus a single-instance view), close_pane(id), focus_pane(instanceId).
Arrangement: place_pane(instanceId, to, group?) (center, dock-left/-right/
-bottom, split-left/-right/-above/-below, or group),
pane_state(instanceId, state) (maximize/restore/hide/show/present/
unpresent), arrange_panes(style) (columns/rows/grid/tabs/even).
Regions: toggle_region(instanceId, position, open?),
set_region_view(instanceId, viewId). Two panes side by side: place the second one
split-right. The floating desktop's open_window, window_state and
arrange_windows are still answered by the relay, mapped onto these, so old
transcripts and REPL scripts keep working; they are no longer in the schema. (The
tiling verbs — split_area, join_area, resize_area, move_pane,
fullscreen_area, open_tool_in_dock, toggle_dock, dock_window — and the
workspace verbs went with the tiling frame.)
Appearance verbs are a separate, lazily-loaded desktop group
(desktop.set_backdrop, desktop.set_theme, desktop.configure_taskbar) rather
than part of the layout core. Restyling the
desktop is a rare request and every always-on tool is paid for on every turn by every
agent — the same reasoning that cut the core 34 → 11 below. The group loads on demand
or on a theme/wallpaper/taskbar keyword, and get_layout carries the backdrop and
theme catalogs so those verbs have ids the model cannot guess without two more
always-on list_* schemas.
The layout verbs are layout-only and ungated (like open_pane), and they
execute through the same controller functions the user's gestures and keybindings
use — one set of operations, three callers (see
windowing). Each mutation triggers the store's
autosave, so an agent's snap persists exactly like a user's drag.
The /ws handler (backend/app.py) is now a bidirectional router: one inbound
receive loop dispatches by channel (agent → the orchestrator) while a telemetry
push task sends outbound; WsConnection (backend/modules/ws.py) serializes
sends and tracks pending tool-call futures.
The capability manifest, getAgentContext() state-reads, widget interop, and the
Claude Code–style permission system gating side effects all ship now (specified in
agent tools & permissions) — the shared surface
the editor, terminal, and file-explorer modules build on. The chat widget now
streams reasoning + answer tokens live (chat_stream), persists transcripts as
sessions, and supports slash commands. Next slices: parameterized
commands-as-tools and a fuller chat/inspector cockpit.
Multi-turn durability
Two things made a long conversation degrade in ways nothing surfaced.
The transcript was replayed whole. agent.orchestrator.contextSize defaults to
the provider default, so overflow never errored — the provider truncates from the
front of the message list, which is exactly where the system prompt and the
group guides sit. A five-turn conversation on a small local model would quietly
stop following its own instructions, and the only symptom was worse answers.
compactHistory (modules/agent/history.ts) now keeps the last
MAX_HISTORY_TURNS verbatim, caps any single oversized message (one pasted stack
trace can outweigh the rest of the conversation), and replaces what falls out with
one marker message. The marker is a user message — _history_messages keeps
only user/assistant text, and an assistant announcing its own amnesia reads as
something the assistant said — and it tells the model to ask rather than leaving a
hole it will confidently fill. The transcript draws the same seam
(agent-compacted), because silent compaction is indistinguishable from an agent
that forgot.
Loaded tool groups reset every turn. active_groups was rebuilt from prompt
keywords each turn, so the model spent a load_tools round to reach files in turn
one, used it, and in turn two the tools were simply gone — a follow-up like "now
delete it" arrived with nothing to delete it with. _carried / _remember_groups
keep them for the session, keyed on the socket (weakly, so a reload starts clean) and
per agent_id. Empty history resets the carry — that is the widget saying "new
chat", an exact signal needing no extra protocol. The carry is bounded to
MAX_CARRIED_GROUPS and trims oldest-first, because every carried group costs schema
bytes on every later turn and the whole point of progressive disclosure is staying
under the 40-tool reasoning cliff. A carry never widens a scoped agent's reach: it
can only re-activate a group spec.tool_groups already allows.
Providers & HTTP endpoints
A provider is either a local server reached over HTTP on this machine, or a
hosted API reached over the internet with a key. backend/modules/agent/providers.py
is the source of truth; it collapses every one of them onto one of three dialects:
| Provider | Dialect | Default endpoint | Pull | Spawn | Key |
|---|---|---|---|---|---|
| Ollama | ollama | http://localhost:11434 | yes | no | — |
| LM Studio | openai | http://localhost:1234 | no | no | — |
| llama.cpp | openai | http://127.0.0.1:8080 | no | yes | — |
| vLLM | openai | http://localhost:8001 | no | yes | — |
| Unsloth | openai | http://127.0.0.1:8888 | no | no | opt |
| OpenAI | litellm | — | no | no | yes |
| Anthropic | litellm | — | no | no | yes |
| Google Gemini | litellm | — | no | no | yes |
| OpenRouter | litellm | — | no | no | yes |
| In-browser | browser | — (the app window) | no | no | — |
browser is the one provider with no server: the model runs on the GPU of an open
app window (WebGPU) and each round is relayed over that window's /ws socket. It is
reachable only while a window reports WebGPU, lists only models that window has
cached, and refuses callers with no window (evals, games). See
WebML.
llamacpp is the node's own server — the backend downloads the llama-server
binary, picks the GGUF and supervises the process, which is why it is the one
provider whose model path is known rather than inferred. See
llamacpp. It is an openai-dialect entry rather than a new
dialect on purpose: a bespoke one would have meant six new branches here and would
have silently lost the tool_choice="required" retry, which is gated on
info.dialect == "openai".
The dialect decides the wire format and the request paths — it is not the
online/offline switch, hosted is. "OpenAI-compatible" refers to LM Studio's,
llama.cpp's and vLLM's wire format; it does not mean any request leaves the machine.
Everything on the ollama and openai dialects runs offline against a localhost
server; only the litellm dialect calls out.
unsloth is Unsloth Desktop's local server. It is another openai-dialect entry,
and the one local server that checks a key: its /v1 routes want
Authorization: Bearer sk-unsloth-… even on loopback, unless the user turned on its
keyless mode. That is what ProviderInfo.optional_api_key means. api_key_for and
auth_headers send a key when one is saved (secrets.db under unsloth, then
UNSLOTH_API_KEY), and the key routes accept it, so it gets a field in Settings →
API keys. It is deliberately not hosted: the probe still runs without a key and
a 401 shows as unreachable, rather than "no key" being taken to mean unusable. Models
are loaded in Unsloth's own window (it has no load route). Thinking is turned off
with a top-level enable_thinking: false, and the context window comes from its
llama-server-style /v1/props. The key itself is created in Unsloth → Settings → API.
Hosted providers and their API keys
Four things make a hosted provider work, and each exists because the obvious alternative fails silently:
- The key is never a setting.
GET /api/settingshands the whole settings bag to the browser and to every plugin, so a credential there is a credential given away. Keys are Fernet-encrypted insecrets.dbunder the provider kind (the same store connectors use) and written throughPUT /agent/providers/<kind>/key, which returns onlyhas_api_key. Nothing reads a key back out, which is why the settings field is write-only: it shows a key as saved, never as its value. - A key already in the environment counts. litellm reads
OPENAI_API_KEY,OPENROUTER_API_KEYand friends on its own, soapi_key_forfalls back to the variable. Without that check a key that works would be reported as missing, and the user would be sent to fix something that isn't broken.DELETEsays so too: it can returnhas_api_key: true, because an exported variable is not ours to delete. - "Reachable" means "we hold a key".
list_modelsraisesMissingApiKey(anhttpx.HTTPError, which is what the probe already catches) for a keyless hosted provider. Before this, a keyless OpenAI reported itself reachable with a list of models, so onboarding cheerfully offered a provider that failed on the first turn. - The model id needs a routing prefix.
qualify_modeladdsProviderInfo.model_prefix—openrouter/for OpenRouter,gemini/for Google AI Studio, nothing for OpenAI and Anthropic, whose ids are unambiguous. A bareminimax/minimax-m3:freeis not a model litellm can place, and a baregemini-2.5-proroutes to the wrong vendor. Applied at the call, so the id stored in settings stays the one the provider's own dashboard shows.
OpenRouter's model list is fetched live from its public catalog
(https://openrouter.ai/api/v1/models, no key needed to read) rather than pinned in
code: its range — and which entries are :free — changes weekly, so a hardcoded list
would be wrong within the month. Ids come back tool-capable first, because a model
without tool support does not merely do worse in the orchestrator loop, it never calls
a tool at all — which reads as the agent ignoring you rather than as a model that
cannot do this. The fetch is cached for 10 minutes and carries its own 10s timeout
(the probe client's 2s budget is sized for a loopback port, not for a few hundred KB
from across the internet), and a failed fetch falls back to static_models rather
than raising: the key decides usability, and an empty dropdown for a working provider
is the worse failure.
Google Gemini's model list is fetched live too, from AI Studio
(generativelanguage.googleapis.com/v1beta/models) by _gemini_listing. Unlike
OpenRouter's catalog it needs the key, which goes in the x-goog-api-key header and
never in the query string, where proxy and access logs would keep it. Only models whose
supportedGenerationMethods include generateContent are kept, which drops embedders,
aqa and the Live-only models. A name filter then drops models that answer in speech,
images or music (TTS, Nano Banana, Lyria), because the listing has no output-modality
field and any of them would fail a chat turn. The list is cached for 10 minutes like
the OpenRouter catalog. A rejected key (400/401/403) raises, so Gemini reads as
unreachable rather than showing a dropdown that cannot make a call. Any other failure
falls back to static_models (the -latest aliases plus 2.5 Pro/Flash) without
caching it.
OpenAI and Anthropic have no listing we fetch, so they show static_models: a
starting point for the dropdown, not the provider's full range. Every model field in
the UI stays free text.
Free models are listed without a key. The catalog is public, and onboarding is the
moment someone has no key yet and wants to know what they could run for nothing, so
free_models(client, info) never checks for one and /agent/status reports
free_models on every provider (empty for providers without a catalog). It says
nothing about usability — a free OpenRouter model still needs a key to call, and
reachable still means "we hold one". An entry is free when both pricing.prompt and
pricing.completion parse to exactly zero, or its id ends in :free; the auto-router
is priced -1 and is not. The free list shares the catalog cache with the model list,
so a keyed provider costs no second fetch. Onboarding picks a free model by default
when switching to a provider, lists free ids first and tags them.
Every model field is a substring-searchable combobox. The onboarding field was a
<datalist> and the settings page's model override a <select>, and both effectively
match from the start of the id — so with several hundred OpenRouter ids, typing free
found nothing. Both are now ModelCombobox, filtering through matchesModelQuery, the
same match the Agent pane's ModelPicker uses: every word anywhere in the id. A free
model also matches the word free, which finds zero-priced ids without the suffix.
The field still accepts an id that is not listed (Enter uses what was typed), and it
commits on pick, Enter or blur rather than per keystroke, because on the settings page
each commit is a settings write and a half-typed id is a broken agent until the typing
stops.
A remote listing is cached and warmed; a local one never is. NVIDIA NIM's
/v1/models (and any vllm/peer pointed at another host) is fetched across the
internet, so list_models gives it its own REMOTE_TIMEOUT (10s) rather than the
probe client's 2s budget — that budget is sized for a loopback port, and _probe
swallows httpx.HTTPError, so inheriting it made a slow network indistinguishable
from a provider with no models. The answer is cached under (kind, endpoint) for
the same 10 minutes as the OpenRouter catalog, keyed by the endpoint too because a
vLLM can be repointed and a list cached under the bare kind would then describe a
machine nobody is talking to. A local server's list is deliberately never cached:
ollama pull changes it, and a stale list is a worse lie than a slow one — loopback
is decided by is_loopback_endpoint, for which an empty endpoint (a peer with no
lease) counts as local because it is unreachable either way.
Every provider is probed in parallel, so the slowest one sets the response time, and
the probe client's connect timeout is 0.75s (the overall budget stays 2s). That is
the number that matters on Windows, where a connect to a closed loopback port does
not fail fast — the OS retries the refused SYN for about two seconds — so each
provider that simply is not running (Ollama, llama.cpp, vLLM on most machines) used
to hold the whole response for the full budget. A live local server accepts in about
a millisecond, or ~250ms through localhost when the IPv6 attempt is refused first.
Remote listings are unaffected: they pass their own explicit timeout=. On the
frontend, getAgentStatus() shares one in-flight request between callers (several
components ask within a few ms of each other at boot); nothing is cached past it.
Those remote listings are warmed at startup: the app lifespan calls
agent.routes.start_warm_model_lists(), which probes only the remote providers in
the background so the first model dropdown opens filled instead of waiting on a round
trip. It is never awaited and every failure is swallowed — a warm is not worth a
second of boot, and an unwarmed provider is simply fetched on demand as before.
GET /agent/status?agent_id=<id> reports the provider and model that roster agent's turns resolve to (roster.resolve_provider, so agent.<id>.provider and agent.orchestrator.provider count), not the saved config's. The chat judges readiness by it: judged by the config, an agent overridden onto a hosted provider showed "can't reach LM Studio" and refused to send whenever the local server was down, though its turns would have run.
GET /agent/status?refresh=true drops both caches first, and the refresh button
in the model picker's panel — beside the filter field — is its only caller. It is a
button rather than a fetch-on-open because a refresh on open would put that download
in front of every use of the dropdown to serve the one time in a hundred the list
actually changed, and the TTL is otherwise invisible from the UI: enabling a model on
the vendor's dashboard and not finding it here reads as the model not existing. While
it is in flight the button spins and the count line says so; a failure says
Refresh failed — showing the last list rather than being swallowed, because a
refresh that silently did nothing is indistinguishable from a vendor with no new
models. A refreshed status outranks the one the pane handed the picker (own ?? status), until the pane itself refetches — a new object from the pane is in turn
newer than our copy, so the effect drops ours and defers to it again.
A strict provider's tool names are translated at the wire, not renamed at the
source. NVIDIA NIM validates function names against OpenAI's documented grammar
(a-zA-Z0-9_-) and rejects the whole request if one fails — 400 Validation: Function at index 7 has an invalid name: "agent.ask_peer", returned in 281ms before
any inference, so every turn failed identically. Every tool in this codebase is
namespaced with a dot, and the orchestrator groups tools by that prefix (a connector's
id must equal its tool prefix), so the names cannot be changed at the source.
sanitize_tool_names rewrites them for ProviderInfo.strict_tool_names providers and
restore_tool_names undoes it on the way back. Three things it must do or it
half-works: rewrite the history too (an assistant turn already in the transcript
carries tool_calls[].function.name, so round two fails where round one passed);
restore the name on what the model called (otherwise the orchestrator is handed
library_search, matches no registered tool, and reports the model hallucinating a
tool it was in fact given); and disambiguate collisions rather than merge them —
library.search and library_search both sanitize to the same string, and merging
would route one tool's call to the other, a wrong action taken confidently.
A provider's error body survives to the message. raise_for_status produces the
status line and a link to MDN, so a provider that said exactly what was wrong is
reported as an unexplained 400 — diagnosing the NIM failure above meant digging the
response body out of the telemetry ring, which is not something a user can do.
raise_for_status_with_body puts the (truncated — some providers echo the whole
request back) body in the exception message instead.
A rate limit is waited out, briefly. A 429 Too Many Requests or 503 means "not
now", not "no", and nothing used to retry it. An eval sweep against NVIDIA NIM's free
tier lost 8 of 15 cases to 429s, and a chat turn that hit the limit simply ended in an
error. chat and chat_stream are wrapped in _retry_rate_limits, beneath
traced_chat so that one span covers every attempt. It covers all dialects:
- httpx's
HTTPStatusErrorand litellm's exceptions, which carrystatus_code; - up to 3 retries, honouring
Retry-After(seconds or an HTTP date), otherwise exponential backoff with jitter (about 1, 2, 4 s); - at most 30 s of waiting in total (
RETRY_BUDGET_S). ARetry-Afterasking for more than is left raises at once instead: the user hears about it now rather than after a silent minute.
A stream is retried only while nothing has reached the caller. A 429 is refused before the first token, but a stream that emitted text and then failed cannot be replayed without showing the same words twice. Cancellation is never swallowed, so a stopped turn stops waiting immediately. Each retry is logged at INFO with the provider, the status and the wait.
orjson is a hard dependency because litellm's OpenRouter transformation imports it;
without it every OpenRouter call fails as an APIConnectionError naming a missing
module, which reads as a network fault.
A hosted model that never answers ends the turn. litellm's default request timeout
is litellm.request_timeout = 6000 s, and nothing passed a smaller one, so a free
OpenRouter model sitting in its queue left the chat on its … bubble for up to 100
minutes with the input disabled. Every litellm call now carries timeout
(HOSTED_FIRST_CHUNK_S, 120 s), and the streamed paths run through _watch_stream, an
explicit watchdog: no first chunk within 120 s (connect included) or a 90 s gap
(HOSTED_IDLE_S) once streaming raises StreamStalled — a TimeoutError, so the
turn's catch-all reports it as error + done. The transport's read timeout alone is
not enough: OpenRouter keeps a queued request's connection warm with SSE comment lines,
which reset a read deadline without ever yielding a chunk.
The chat can stop a turn. While a turn is in flight the send button is a Stop
button: it aborts askAgent's signal, which sends cancel and resolves at once; the
backend cancels the task (_turns_by_id, owner-checked — a turn id is not a secret on
a shared socket hub) and still emits done for any other listener. The abort handle
lives at module level beside the transcript, so a remounted pane can stop a turn it
did not start.
Keys are entered in two places, both writing the same route: the Model provider API
keys section of the Settings page (ApiKeysSettings), and inline in onboarding —
where an unreachable hosted provider offers a key field in the same slot an
unreachable local one offers "install and start it". A provider with a key then
appears in the per-agent provider override dropdown like any local one, and the
model override dropdown lists that provider's models (not the configured
provider's — an agent pointed at OpenRouter used to be offered Ollama's model names,
every one of which means nothing on that server, and the failure only showed up
mid-turn).
| Operation | ollama dialect | openai dialect | Used by |
|---|---|---|---|
| List models / probe | GET /api/tags | GET /v1/models | list_models — /agent/status |
| Tool-calling round (streamed) | POST /api/chat (stream:true, think) | POST /v1/chat/completions (stream:true) | chat_stream — the orchestrator loop |
| One-shot streamed answer | POST /api/generate | POST /v1/chat/completions (SSE) | generate_stream — legacy /agent/chat |
| Short fill-in completion | POST /api/generate (stream:false) | POST /v1/chat/completions (stream:false) | generate — /agent/complete (autosuggest) |
| Pull a model | POST /api/pull | — (unsupported) | /agent/pull |
Ollama's default URL can be overridden with HORRIBLE_OLLAMA_URL; a backend-spawned
vLLM advertises its own port, and so does a spawned llama-server — for that one
alone the live manager is checked before the saved endpoint, because it takes an
ephemeral port when the default is occupied and a saved :8080 would point at
nothing. All of these outbound calls go through
instrumented_client(), so they surface in the observability I/O stream with full
request↔response bodies — the streamed tool-calling round, answer, and pull
responses all via tee_stream. See
observability.md.
Hierarchical tools & progressive disclosure
Local reasoning models (e.g. gemma4:e2b under Ollama) have a strict tool-count
ceiling: list 40+ tool definitions and the model silently skips its thinking
phase and streams the final answer from the first token; with ≤39 it reasons
normally. The whole catalog (layout + peer + every module/plugin tool) is well past
that, so tools are exposed hierarchically and injected on demand rather than
flattened and pruned.
Tools belong to groups by name prefix (files, editor, terminal,
visualizer, database, records, network, …; the dotless layout verbs are the
layout group). Each turn the model sees only:
- the core — the layout verbs (see below), the peer tools (
list_peers,agent.ask_peer), and two meta-tools:list_tool_groups()(discover groups- counts) and
load_tools(groups[])(enable a group); plus
- counts) and
use_skill(name), but only when a skill is enabled — a third meta-tool that is absent (costing nothing) on an install with no skills; plus- the tools of every active group.
Skills are the same idea one tier up, and they reuse this machinery rather than
paralleling it: a one-line description per skill rides every turn (that is the
trigger), the instructions arrive only when use_skill asks, and a skill's
allowed-tools are resolved to groups and activated in that same call — so the
instructions never name tools the model cannot see. See Skills.
The layout verbs split by cost, because they were the largest always-on block
and most agents never rearrange anything. The four read verbs
(list_available_panes, list_open_panes, get_layout, get_pane_context) plus
show
stay in every agent's core — any agent may need to see what the user is looking at, and
show is how it puts something there. The arrangement verbs are loaded on demand
via load_tools("layout").
Including for main. They used to stay in main's core on the reasoning that
driving the shell is its job — but that made the loosely prompted agent the most
starved one, carrying ~1,455 tokens of geometry schemas on turns that only wanted to
read a file. show covers "open/go to X" in a single call, and the narrow layout
keyword list preloads the arrangement verbs on the turns that genuinely rearrange
things. A spec that names layout in its tool_groups still opts back into all of
them up front.
layout is shell control rather than a
capability, so it stays loadable for every agent regardless of scope — scoping
only decides whether it costs schema space up front. Two traps worth knowing:
_all_dynamic_tools dedupes against the core, and that dedupe base must be
spec-aware or the arrangement verbs are swallowed and load_tools("layout")
silently returns nothing; and the layout keyword list is deliberately narrow
(pane, window, snap, … but never open/close), since a keyword that fires
on most turns would give back exactly what the split saves.
run_agent_turn recomputes the presented list each round from active_groups,
so a group loaded this round is injected into the next model call (true dynamic
injection — nothing is ever dropped, just not-yet-loaded). active_groups is:
- seeded by a keyword preload (
_preload_groups): a prompt mentioning a file, shell command, etc. auto-activates that group so common asks stay single-round; - grown when the model calls
load_tools; and - grown forgivingly — if the model calls a known tool from a group it never
loaded,
_dispatch_callactivates the group and runs the call anyway.
A TOOL_BUDGET = 38 cap remains only as a backstop (core is always kept first). The
/tools command (list_tools→tools) still reports the full catalog, labeled
by group, independent of what any single turn has loaded.
The cap is resolved per agent, not global: tool_budget_for(agent_id) returns 38
for a model running on this machine and TOOL_BUDGET_HOSTED = 96 for one reached
through a hosted API. The 38 ceiling is a measured property of small local models,
not of tool calling, and applying it to a frontier model bought nothing while costing
a load_tools round per group past the third.
The signal is ProviderInfo.hosted — an API reached with a key, versus a server on
this machine — and deliberately not the model name: parsing a parameter count works
for llama-3.2-3b-instruct and not at all for gpt-5, so a name-based rule would
have left every hosted model on the floor, which is the case it exists for. Anything
unidentifiable resolves to the floor. Being wrong low costs a round trip; being wrong
high costs a model that silently stops reasoning.
The cap was 44 — deliberately above the ≤39 ceiling, so that a legitimate
training+notebook turn would not be truncated. A backstop above the cliff protects
nothing, though: it lets a turn through in exactly the state the cliff warns about.
What actually keeps a turn safe is a small always-on core, and once the arrangement
verbs left it that same flagship flow costs 11 + 12 + 9 = 32, so the cap now sits at
38, under the cliff, without truncating real work.
Truncation is also easy to miss, which is why it now logs at ERROR and names the
tools it dropped (plus a dropped list in the stats the
interpretability pane reads). A silently-dropped tool looks
exactly like a model that chose not to use one.
The core has come down in two steps, both measured with the interpretability recorder's own counter:
| tools | always-on schema | |
|---|---|---|
| Originally | 34 | — |
After grouping research.*/arxiv.* (see the danger note in agent-tools) | 26 | — |
| After moving the arrangement verbs out | 11 | -68% (15,299 → 4,867 chars) |
The system prompt halved alongside it (3,017 → 1,375 chars), because the geometry
rules it used to carry moved into the layout group's guide.
Contributions to the layout shell
- Panels:
chat.conversation(main chat view),chat.sessions(session list),chat.inspector(tool-call/trace detail, opened on demand) — each in a window. - Commands:
chat.new,chat.focusInput,chat.openSession,chat.toggleInspector,chat.stopGeneration. - Default keybindings: declared for new-session and focus-input; bound through the shell keybinding service.
- Dashboard widgets:
chat.recentSessions(see dashboard.md). - Region strip: the browser docks on the right (
browser.view, keyw). The agent'sbrowser.openopens it there, beside the turn that asked. See Docked in the agent chat.
Backend surface
backend/modules/chat/ — session CRUD over HTTP; streaming (tokens, tool-call
events, status) over the shared WebSocket on chat.* channels. Agent execution
lives entirely in the backend; the frontend only renders the stream and sends
user input. Pydantic models define the event schema.
Browser vs desktop
The conversation experience is identical — it's all backend traffic over the shared socket. Differences are notification/summon ergonomics only:
| Concern | Browser | Desktop |
|---|---|---|
| Completion notifications | Web Notifications (notifications.system capability, tab must allow it) | OS notification, click focuses window |
| Quick summon | none | global shortcut raises the window and runs chat.focusInput |
| Long-running agents | keep tab open (no background work in the page; the backend keeps running regardless) | window can be closed to tray; stream resumes on reopen |
Both layouts must tolerate socket reconnects mid-generation: the backend is the source of truth for session state, and the panel re-syncs on reconnect.