Skip to main content

Module: observability

Observe the app's data flow — frontend↔backend↔external — the way Docker Desktop shows a container's network I/O. The compact "Data flow" widget is part of the seeded Dashboard workspace; the full I/O panel is opened on demand.

Status: implemented — frontend in packages/core/src/modules/observability/, instrumentation in backend/modules/telemetry/ plus packages/core/src/{telemetry,ws}.ts.

The instrumentation, not the widget, is the point​

A useful observability view can't be per-module logging — it instruments the chokepoints once so every module's traffic shows up for free (like the Docker daemon already seeing all container I/O). Five sources feed one stream:

  • client — every frontend→backend round-trip, recorded in the API client's request<T> (packages/core/src/api.ts).
  • inbound — every request the backend receives, via an ASGI middleware (backend/modules/telemetry/instrument.py).
  • outbound — the backend's calls to external services (Clubhouse, Ollama), via instrumented_client() wrapping httpx; agent and clubhouse use it. This is the part the frontend can't see on its own.
  • ws — individual frames of the multiplexed /ws socket (agent, collab, network, lobby, terminal…), captured at the two chokepoints: outbound at WsConnection.send_json (via a telemetry-free observer hook the app registers), inbound in the /ws receive loop. The telemetry channel itself is skipped so the push stream doesn't observe its own output. The frame's JSON payload is the body; the direction (send/recv) is the method.
  • browser — every request the embedded Chromium makes, at the _route egress hook plus Chromium's own request/response events (browser). Distinct from outbound: those are calls the app chose to make, these are made by pages the user visits, which is both the largest volume and the least trusted. Carries two extra fields — resource_type (document/script/image/xhr/…) and verdict (allowed/blocked), so a request the SSRF guard aborted stays visible instead of vanishing. Full mode only; an iframe's requests are made by the user's own browser and never reach the backend.

So a single user action reads end to end: client GET /agent/status → inbound /api/agent/status → outbound GET …/api/tags (Ollama).

Which turn an event belongs to​

An event recorded while an agent turn is running also carries turn_id and round, stamped from a contextvar run_agent_loop sets (telemetry/turn.py) rather than passed by the caller — recording happens in httpx's response hook and the /ws pump, layers that have no business knowing what an agent turn is. Both fields are None for the vast majority of events, which is the honest answer: most I/O this node does is nobody's turn. Together with the interpretability capture — which records what the model was shown — this is what lets a request be put beside the context that produced it instead of being matched up by timestamp.

Detail is captured raw — this is a local Wireshark​

The panel is the app's built-in protocol analyzer, so it deliberately hides nothing at the application layer: every event carries metadata (method, target, status, duration, byte counts) plus the full bodies, unredacted, for the inspector. Sensitive bodies (Clubhouse phone numbers, SMS codes) are shown verbatim. The only general limit is size: bodies are truncated to a configurable cap (observability.maxBodyChars, default 16 KB) and never read past a 1 MB hard ceiling. Outbound URLs are recorded scheme+host+path only (query/fragment stripped). Treat the in-memory buffer as sensitive — it is a local introspection tool, and the data never leaves the machine.

Credential headers are the exception to "raw"​

Header values matching SENSITIVE_HEADER_PARTS are blanked to ***, everywhere, before the event enters the ring (Recorder.record, and amend for the streamed case). The header name is kept — "this request was authenticated" is the difference between a 401 you misconfigured and a 401 you were refused, and that is the part worth debugging; the value never is.

This is deliberately not symmetric with bodies, and it was learned the hard way. A provider API key was found sitting in GET /api/telemetry/recent in full: the outbound hook captured auth_headers()' bearer on every chat round, the panel fetches that route, so the credential reached the browser — and on a LAN-bound node (pnpm dev:lan, which the peer fabric needs) it reached the same network the token exists to keep out. "It's only local" is not an argument for a buffer an inspector reads over HTTP. Inbound headers had a partial version of this already, scoped to the OTLP paths; the argument was never specific to those paths, and outbound is where the real credentials are.

Redaction happens at the recorder, not the route, because the ring is read three ways — recent(), the live /ws subscribers, and the drain that persists it — and a route-level filter would leave the value in the other two. It is also why capture_headers does not redact: doing it at every capture site means the one site somebody adds next year is the one that leaks. The durable copy was already clean (store._clean_headers blanks on write), so this closed the gap between the two copies rather than opening a new mechanism — both now call the same telemetry/redact.py, which is its own import-free module so the ring can use it without importing the store.

The other exception: password bodies​

Paths matching _REDACT_BODY_PREFIXES (instrument.py) — currently the HorribleAssault/games native email+password sign-in, /api/games/auth/local/* on the node and /auth/local/* on the game server — are recorded without either body. They carry a plaintext password up and a freshly-minted session token back, and the node proxies the browser's sign-in to the game server, so the same credential crosses the inbound middleware, the outbound httpx hook, and the frontend client recorder. Capturing it would make this panel the leak; "it's only local" is not an argument, because a local buffer is exactly what an inspector reads.

This is a redaction list, not a skip list: the event still appears with its method, path, status and timing, which is what you actually need to debug a sign-in. It is keyed on path prefix in all three seams rather than on a flag at the call site, so a new auth route inherits the protection by where it lives instead of by someone remembering to annotate it. See the games module for the flow itself.

The list also carries the OTLP receiver (/api/otel/v1/). Its bodies are protobuf batches of other programs' spans, already stored in full in otel_spans. Its credential headers used to be blanked here specifically — that is now unconditional and lives in the recorder, as described above, so _redacts_body governs only bodies. See OpenTelemetry.

The outbound request hook is also where traceparent is added when otel.propagateHttp is on — loopback and LAN targets only, never a hosted provider.

Response bodies are captured on client events and on all outbound calls — so the request↔response pair to the LLM reads end to end — truncated like request bodies:

  • Non-streaming outbound responses (e.g. the agent's stream:false /api/chat tool-calling round) are read in the response hook; httpx caches the read, so the caller's .json() still works.
  • Streaming outbound responses (Ollama chat/pull NDJSON, OpenAI SSE — detected by content-type) can't be read in the hook without consuming the stream. The hook records the event with no body yet, then tee_stream — wrapping the two streaming call sites (providers.generate_stream, routes._proxy_ndjson) — passes each line through to the caller while accumulating a capped copy, and recorder.amend fills the body into the same event once the stream finishes (capped at the 1 MB ceiling so a long generation can't grow unbounded).

Inbound response bodies aren't captured (the client event for the same round-trip shows the payload). Capture and truncation live at the chokepoints (instrument.py, api.ts).

Transport​

The backend keeps a 500-event ring buffer (recorder.py) and streams events to the frontend over the shared /ws socket on the telemetry channel — the first real use of that socket. On connect it replays the recent backlog, then pushes live. GET /api/telemetry/recent exposes the same backlog for polling. The frontend store (telemetry.ts) merges client events with the streamed inbound/outbound events into one capped list, upserted by (source, id) so an amended event (a filled-in streaming body, re-emitted under the same id) replaces its row in place rather than duplicating it.

Persistence for agent turns​

The ring forgets, and that is right for a live view and useless for a postmortem: by the time anyone opens yesterday's agent run, the wire that produced it is gone. So the turn-stamped slice is also written to telemetry_events in app.db (backend/modules/telemetry/store.py), where it joins onto traj_runs.turn_id and agent_turns.turn_id. Everything else stays ring-only.

  • Only events with a turn_id are kept. A node's background traffic is not part of any agent postmortem, and a row that joins to nothing is noise in a table whose only purpose is the join. It also keeps retention nearly a non-problem.
  • The key is (boot_id, ev_id). IoEvent.id is an itertools.count(1) that restarts every process, so it is not durable alone. With a per-process boot_id prefixed, INSERT OR REPLACE makes an amend — deliberately re-emitted under the same id — idempotent by construction.
  • A batched subscriber writes, never record(). record() is on the hot path of every instrumented request, and a synchronous sqlite write there would put file I/O in the middle of a chat turn to serve a debugging feature (drain.py, flushing on 200 events or 1 s). It starts in the app lifespan, before any turn can run.
  • Drops are counted, not prevented. Recorder._notify discards on a full queue. Backpressure on record() would slow the app to protect a postmortem, so instead a gap in ev_id is counted and reported by GET /api/telemetry/stats as dropped — a visible number rather than a table that silently claims to be complete.
  • Shutdown flushes what is in hand, synchronously, queue included. Two traps, each of which made the flush look implemented while doing nothing. Every await inside a task that is already being cancelled raises CancelledError again, so an await asyncio.to_thread(persist) there never reaches the database. And the batch is not all we hold: the newest events — the in-flight turn's — are usually still in the queue, not yet batched, so the handler drains the queue before writing.

Headers are blanked by name, bodies are opt-in​

Header names containing any of SENSITIVE_HEADER_PARTS (authorization, auth, token, key, secret, password, cookie, credential) are stored as *** — blanked rather than dropped, since "this request carried credentials" is itself worth knowing. A substring match because header names are not a closed vocabulary (x-api-key, x-goog-api-key, x-amz-security-token), and over-blanking costs a line of detail where under-blanking writes a live credential to disk.

trajectories.store.redact() is not used for this, and cannot be. It matches SECRET_KEY_SUFFIXES, a dotted settings-key vocabulary (github.token, foo.apiKey). It matches neither Authorization nor x-api-key — so reusing it, as first planned, would have written every credential header to disk in full.

Bodies are opaque text no name match can help, so they are persisted only with telemetry.persistBodies on (default off), capped at 4 KB. The live inspector keeps full bodies either way.

SettingDefaultEffect
telemetry.retentionDays7Delete stored events older than this; 0 disables
telemetry.retentionEvents200000Hard cap, pruned oldest-first; 0 disables
telemetry.persistBodiesfalseAlso store request/response bodies, capped at 4 KB

Retention is enforced from the drain every 60th flush rather than on a timer — the only moment the table is known to have just grown. Deleting a trajectory run also deletes its turn's stored events (trajectories.store.delete_run), which would otherwise sit unreachable until they aged out.

Muting a source​

The toolbar's source control is a row of toggles, one per source, not a dropdown. It used to be a single select — "All sources", or exactly one — which could not express the state anybody actually wants: everything except ws. A busy websocket produces frames faster than anything else in the app, so the choice was between a list buried in them and a list with nothing else in it.

Each chip is the same badge the rows carry, so what you are muting is recognisable as the thing you saw too much of. Muting hides rows; nothing stops being captured — unmuting brings the history back rather than starting a new one.

The mute is stored in observability.mutedSources (comma-separated, editable by hand in Settings) rather than in component state, for two reasons: muting ws is a standing preference about a source that never stops being noisy, and the dashboard widget reads the same setting. A mute that applied only to the panel would leave the surface with the least room — five rows — still full of frames.

An unknown name in that setting is dropped rather than kept: a typo that mutes nothing is better than one that sits in the list looking as though it did something.

Contributions to the layout shell​

  • Regions: observability.io hosts observability.logs as its bottom region strip — the ⊡ Inspector toggle in the pane's region controls opens the full Wireshark-style view without leaving the tool.
  • Widgets/tools: observability.io ("Data flow", role tool) — compact summary (call/error counts + last few, expandable the same way). Openable anywhere from the palette or the Start menu.
  • Panels: observability.logs (singleton, role widget, embedded — the expanded inspector behind observability.io's bottom strip, reached by revealing that region rather than by opening a pane of its own; see Embedded views and sections) — a Wireshark-style master-detail: a packet-list table (time, source badge, method, target, status, ms, size) over an inspector for the selected row. The inspector shows General metadata, then request/response headers and bodies (for a ws frame, a single "Frame payload"); bodies are JSON pretty-printed and syntax-highlighted with a Raw⇄Pretty toggle and Copy. A toolbar filter narrows rows live (client-side): a free-text query over method+target+body, a source selector (all / client / inbound / outbound / ws), and an errors only toggle; the count reads shown / total while a filter is active. A Clear button empties the buffer. Accessible only via the toggle strip on the Data flow widget (use the ⊡ Inspector toggle) — not openable as a standalone command.
  • Commands: observability.open (opens the Data flow widget — the primary).
  • Settings: observability.recentCount (number, default 5) — how many recent calls the Data flow widget lists; read live via useSetting. observability.mutedSources (string, default empty) — comma-separated sources hidden from both the panel and the widget, written by the panel's own source toggles; and observability.maxBodyChars (number, default 16384) — how many characters of each captured request/response body (and ws frame) to keep for the inspector. Unlike recentCount, this one is consumed on the backend (the truncation happens at capture, in instrument.py, which reads the persisted value via the settings module's get_value); it's still hard-capped at 1 MB. See settings.md.

Backend surface​

backend/modules/telemetry/ — GET /api/telemetry/recent (backlog), the /ws telemetry stream, GET /api/telemetry/turn/{turn_id}?round= (stored events for one turn) and GET /api/telemetry/stats (written / dropped / stored). A run-scoped view of the same rows is GET /api/trajectories/runs/{run_id}/io. The telemetry endpoints are excluded from recording to avoid feedback noise.

Browser vs desktop​

Identical. The store also captures real client-side round-trip latency, which reflects wherever the backend lives (local or remote).

Agent context​

Both the widget and the panel expose a getAgentContext() snapshot (total calls, error count, and the most recent calls with source/method/target/status/duration), so the agent can reason about live traffic — e.g. "did that request fail?". See agent tools.

Not yet​

Client-side streaming calls (the agent chat/pull streams bypass request<T>, so they show as outbound on the backend but not as client events), inbound response bodies (see above), and persistence across reloads for anything outside an agent turn (only turn-stamped events are stored — see Persistence for agent turns). The panel filter is panel-local state, so it resets when the pane is unmounted.