Module: observability
Observe the app's data flow — frontend↔backend↔external — the way Docker Desktop shows a container's network I/O. The compact "Data flow" widget is part of the seeded Dashboard workspace; the full I/O panel is opened on demand.
Status: implemented — frontend in
packages/core/src/modules/observability/, instrumentation in
backend/modules/telemetry/ plus packages/core/src/{telemetry,ws}.ts.
The instrumentation, not the widget, is the point
A useful observability view can't be per-module logging — it instruments the chokepoints once so every module's traffic shows up for free (like the Docker daemon already seeing all container I/O). Five sources feed one stream:
client— every frontend→backend round-trip, recorded in the API client'srequest<T>(packages/core/src/api.ts).inbound— every request the backend receives, via an ASGI middleware (backend/modules/telemetry/instrument.py).outbound— the backend's calls to external services (Clubhouse, Ollama), viainstrumented_client()wrapping httpx; agent and clubhouse use it. This is the part the frontend can't see on its own.ws— individual frames of the multiplexed/wssocket (agent, collab, network, lobby, terminal…), captured at the two chokepoints: outbound atWsConnection.send_json(via a telemetry-free observer hook the app registers), inbound in the/wsreceive loop. Thetelemetrychannel itself is skipped so the push stream doesn't observe its own output. The frame's JSON payload is the body; the direction (send/recv) is the method.browser— every request the embedded Chromium makes, at the_routeegress hook plus Chromium's own request/response events (browser). Distinct fromoutbound: those are calls the app chose to make, these are made by pages the user visits, which is both the largest volume and the least trusted. Carries two extra fields —resource_type(document/script/image/xhr/…) andverdict(allowed/blocked), so a request the SSRF guard aborted stays visible instead of vanishing. Full mode only; an iframe's requests are made by the user's own browser and never reach the backend.
So a single user action reads end to end: client GET /agent/status →
inbound /api/agent/status → outbound GET …/api/tags (Ollama).
Which turn an event belongs to
An event recorded while an agent turn is running also carries turn_id and round,
stamped from a contextvar run_agent_loop sets (telemetry/turn.py) rather than
passed by the caller — recording happens in httpx's response hook and the /ws pump,
layers that have no business knowing what an agent turn is. Both fields are None for
the vast majority of events, which is the honest answer: most I/O this node does is
nobody's turn. Together with the
interpretability capture — which records what the model was
shown — this is what lets a request be put beside the context that produced it
instead of being matched up by timestamp.
Detail is captured raw — this is a local Wireshark
The panel is the app's built-in protocol analyzer, so it deliberately hides
nothing at the application layer: every event carries metadata (method, target,
status, duration, byte counts) plus the full bodies, unredacted, for the
inspector. Sensitive bodies (Clubhouse phone numbers, SMS codes) are shown
verbatim. The only general limit is size: bodies
are truncated to a configurable cap (observability.maxBodyChars, default 16 KB)
and never read past a 1 MB hard ceiling. Outbound URLs are recorded
scheme+host+path only (query/fragment stripped). Treat the in-memory buffer as
sensitive — it is a local introspection tool, and the data never leaves the
machine.
Credential headers are the exception to "raw"
Header values matching SENSITIVE_HEADER_PARTS are blanked to ***, everywhere,
before the event enters the ring (Recorder.record, and amend for the streamed
case). The header name is kept — "this request was authenticated" is the
difference between a 401 you misconfigured and a 401 you were refused, and that is
the part worth debugging; the value never is.
This is deliberately not symmetric with bodies, and it was learned the hard way. A
provider API key was found sitting in GET /api/telemetry/recent in full: the
outbound hook captured auth_headers()' bearer on every chat round, the panel
fetches that route, so the credential reached the browser — and on a LAN-bound node
(pnpm dev:lan, which the peer fabric needs) it reached the same network the token
exists to keep out. "It's only local" is not an argument for a buffer an inspector
reads over HTTP. Inbound headers had a partial version of this already, scoped to the
OTLP paths; the argument was never specific to those paths, and outbound is where the
real credentials are.
Redaction happens at the recorder, not the route, because the ring is read three
ways — recent(), the live /ws subscribers, and the drain that persists it — and a
route-level filter would leave the value in the other two. It is also why
capture_headers does not redact: doing it at every capture site means the one site
somebody adds next year is the one that leaks. The durable copy was already clean
(store._clean_headers blanks on write), so this closed the gap between the two
copies rather than opening a new mechanism — both now call the same
telemetry/redact.py, which is its own import-free module so the ring can use it
without importing the store.
The other exception: password bodies
Paths matching _REDACT_BODY_PREFIXES (instrument.py) — currently the
HorribleAssault/games native email+password sign-in, /api/games/auth/local/*
on the node and /auth/local/* on the game server — are recorded without
either body. They carry a plaintext password up and a freshly-minted session
token back, and the node proxies the browser's sign-in to the game server, so the
same credential crosses the inbound middleware, the outbound httpx hook, and the
frontend client recorder. Capturing it would make this panel the leak; "it's only
local" is not an argument, because a local buffer is exactly what an inspector
reads.
This is a redaction list, not a skip list: the event still appears with its method, path, status and timing, which is what you actually need to debug a sign-in. It is keyed on path prefix in all three seams rather than on a flag at the call site, so a new auth route inherits the protection by where it lives instead of by someone remembering to annotate it. See the games module for the flow itself.
The list also carries the OTLP receiver (/api/otel/v1/). Its bodies are protobuf
batches of other programs' spans, already stored in full in otel_spans. Its
credential headers used to be blanked here specifically — that is now unconditional
and lives in the recorder, as described above, so _redacts_body governs only bodies.
See OpenTelemetry.
The outbound request hook is also where traceparent is added when
otel.propagateHttp is on — loopback and LAN targets only, never a hosted provider.
Response bodies are captured on client events and on all outbound calls — so
the request↔response pair to the LLM reads end to end — truncated like request
bodies:
- Non-streaming outbound responses (e.g. the agent's
stream:false/api/chattool-calling round) are read in the response hook; httpx caches the read, so the caller's.json()still works. - Streaming outbound responses (Ollama chat/pull NDJSON, OpenAI SSE — detected
by
content-type) can't be read in the hook without consuming the stream. The hook records the event with no body yet, thentee_stream— wrapping the two streaming call sites (providers.generate_stream,routes._proxy_ndjson) — passes each line through to the caller while accumulating a capped copy, andrecorder.amendfills the body into the same event once the stream finishes (capped at the 1 MB ceiling so a long generation can't grow unbounded).
Inbound response bodies aren't captured (the client event for the same
round-trip shows the payload). Capture and truncation live at the chokepoints
(instrument.py, api.ts).
Transport
The backend keeps a 500-event ring buffer (recorder.py) and streams events to
the frontend over the shared /ws socket on the telemetry channel — the
first real use of that socket. On connect it replays the recent backlog, then
pushes live. GET /api/telemetry/recent exposes the same backlog for polling.
The frontend store (telemetry.ts) merges client events with the streamed
inbound/outbound events into one capped list, upserted by (source, id)
so an amended event (a filled-in streaming body, re-emitted under the same id)
replaces its row in place rather than duplicating it.
Persistence for agent turns
The ring forgets, and that is right for a live view and useless for a postmortem: by
the time anyone opens yesterday's agent run, the wire that produced it is gone. So the
turn-stamped slice is also written to telemetry_events in app.db
(backend/modules/telemetry/store.py), where it joins onto traj_runs.turn_id and
agent_turns.turn_id. Everything else stays ring-only.
- Only events with a
turn_idare kept. A node's background traffic is not part of any agent postmortem, and a row that joins to nothing is noise in a table whose only purpose is the join. It also keeps retention nearly a non-problem. - The key is
(boot_id, ev_id).IoEvent.idis anitertools.count(1)that restarts every process, so it is not durable alone. With a per-processboot_idprefixed,INSERT OR REPLACEmakes anamend— deliberately re-emitted under the same id — idempotent by construction. - A batched subscriber writes, never
record().record()is on the hot path of every instrumented request, and a synchronous sqlite write there would put file I/O in the middle of a chat turn to serve a debugging feature (drain.py, flushing on 200 events or 1 s). It starts in the app lifespan, before any turn can run. - Drops are counted, not prevented.
Recorder._notifydiscards on a full queue. Backpressure onrecord()would slow the app to protect a postmortem, so instead a gap inev_idis counted and reported byGET /api/telemetry/statsasdropped— a visible number rather than a table that silently claims to be complete. - Shutdown flushes what is in hand, synchronously, queue included. Two traps, each
of which made the flush look implemented while doing nothing. Every
awaitinside a task that is already being cancelled raisesCancelledErroragain, so anawait asyncio.to_thread(persist)there never reaches the database. And the batch is not all we hold: the newest events — the in-flight turn's — are usually still in the queue, not yet batched, so the handler drains the queue before writing.
Headers are blanked by name, bodies are opt-in
Header names containing any of SENSITIVE_HEADER_PARTS (authorization, auth,
token, key, secret, password, cookie, credential) are stored as *** —
blanked rather than dropped, since "this request carried credentials" is itself worth
knowing. A substring match because header names are not a closed vocabulary
(x-api-key, x-goog-api-key, x-amz-security-token), and over-blanking costs a line
of detail where under-blanking writes a live credential to disk.
trajectories.store.redact() is not used for this, and cannot be. It matches
SECRET_KEY_SUFFIXES, a dotted settings-key vocabulary (github.token,
foo.apiKey). It matches neither Authorization nor x-api-key — so reusing it, as
first planned, would have written every credential header to disk in full.
Bodies are opaque text no name match can help, so they are persisted only with
telemetry.persistBodies on (default off), capped at 4 KB. The live inspector keeps
full bodies either way.
| Setting | Default | Effect |
|---|---|---|
telemetry.retentionDays | 7 | Delete stored events older than this; 0 disables |
telemetry.retentionEvents | 200000 | Hard cap, pruned oldest-first; 0 disables |
telemetry.persistBodies | false | Also store request/response bodies, capped at 4 KB |
Retention is enforced from the drain every 60th flush rather than on a timer — the
only moment the table is known to have just grown. Deleting a trajectory run also
deletes its turn's stored events (trajectories.store.delete_run), which would
otherwise sit unreachable until they aged out.
Muting a source
The toolbar's source control is a row of toggles, one per source, not a
dropdown. It used to be a single select — "All sources", or exactly one — which
could not express the state anybody actually wants: everything except ws. A
busy websocket produces frames faster than anything else in the app, so the choice
was between a list buried in them and a list with nothing else in it.
Each chip is the same badge the rows carry, so what you are muting is recognisable as the thing you saw too much of. Muting hides rows; nothing stops being captured — unmuting brings the history back rather than starting a new one.
The mute is stored in observability.mutedSources (comma-separated, editable by
hand in Settings) rather than in component state, for two reasons: muting ws is
a standing preference about a source that never stops being noisy, and the
dashboard widget reads the same setting. A mute that applied only to the panel
would leave the surface with the least room — five rows — still full of frames.
An unknown name in that setting is dropped rather than kept: a typo that mutes nothing is better than one that sits in the list looking as though it did something.
Contributions to the layout shell
- Regions:
observability.iohostsobservability.logsas its bottom region strip — the⊡ Inspectortoggle in the pane's region controls opens the full Wireshark-style view without leaving the tool. - Widgets/tools:
observability.io("Data flow", roletool) — compact summary (call/error counts + last few, expandable the same way). Openable anywhere from the palette or the Start menu. - Panels:
observability.logs(singleton, rolewidget,embedded— the expanded inspector behindobservability.io's bottom strip, reached by revealing that region rather than by opening a pane of its own; see Embedded views and sections) — a Wireshark-style master-detail: a packet-list table (time, source badge, method, target, status, ms, size) over an inspector for the selected row. The inspector shows General metadata, then request/response headers and bodies (for awsframe, a single "Frame payload"); bodies are JSON pretty-printed and syntax-highlighted with a Raw⇄Pretty toggle and Copy. A toolbar filter narrows rows live (client-side): a free-text query over method+target+body, a source selector (all / client / inbound / outbound / ws), and an errors only toggle; the count readsshown / totalwhile a filter is active. A Clear button empties the buffer. Accessible only via the toggle strip on the Data flow widget (use the⊡ Inspectortoggle) — not openable as a standalone command. - Commands:
observability.open(opens the Data flow widget — the primary). - Settings:
observability.recentCount(number, default 5) — how many recent calls the Data flow widget lists; read live viauseSetting.observability.mutedSources(string, default empty) — comma-separated sources hidden from both the panel and the widget, written by the panel's own source toggles; andobservability.maxBodyChars(number, default 16384) — how many characters of each captured request/response body (andwsframe) to keep for the inspector. UnlikerecentCount, this one is consumed on the backend (the truncation happens at capture, ininstrument.py, which reads the persisted value via the settings module'sget_value); it's still hard-capped at 1 MB. See settings.md.
Backend surface
backend/modules/telemetry/ — GET /api/telemetry/recent (backlog), the /ws
telemetry stream, GET /api/telemetry/turn/{turn_id}?round= (stored events for one
turn) and GET /api/telemetry/stats (written / dropped / stored). A run-scoped
view of the same rows is GET /api/trajectories/runs/{run_id}/io. The telemetry
endpoints are excluded from recording to avoid feedback noise.
Browser vs desktop
Identical. The store also captures real client-side round-trip latency, which reflects wherever the backend lives (local or remote).
Agent context
Both the widget and the panel expose a getAgentContext() snapshot (total calls,
error count, and the most recent calls with source/method/target/status/duration),
so the agent can reason about live traffic — e.g. "did that request fail?". See
agent tools.
Not yet
Client-side streaming calls (the agent chat/pull streams bypass request<T>, so
they show as outbound on the backend but not as client events), inbound response
bodies (see above), and persistence across reloads for anything outside an agent turn
(only turn-stamped events are stored — see Persistence for agent turns). The
panel filter is panel-local state, so it resets when the pane is unmounted.