Module: research
Collect, read, and (soon) synthesize research material. The research module is the home of the artifact store viewers and the flows that fill it: capture any URL as a self-contained HTML archive, store and read PDFs, and mirror everything into an Obsidian vault. Later phases add the ArXiv browser and the durable deep-research engine, which work alongside the browser and library panes.
Status: implemented (all four phases) — artifact store, page capture
(engine + fallback), PDF pipeline + viewer, saved-page viewer, Obsidian export,
the ArXiv browser, the durable deep-research engine + console, and full
research.* / arxiv.* agent awareness.
The artifact store
The node's first on-disk blob store (backend/modules/artifacts/):
- Blobs live at
$HORRIBLE_DATA_DIR/artifacts/<sha256[:2]>/<sha256>— content-addressed, so saving the same PDF twice stores one blob, and the serving path is derived only from the hash: path traversal is impossible by construction. - Rows live in an
artifactstable inapp.db(id, sha256, kind, mime, filename, size, origin_url, meta). Kinds:pdf,page,report. GET /api/artifacts/:idis the byte route the viewers load from. Captured pages are served withContent-Security-Policy: sandbox; default-src 'self' data:on top of the viewer's iframe sandbox — stored third-party HTML never gets scripts or an origin.POST /api/artifacts/uploadaccepts PDF multipart uploads (100 MB cap);DELETErefcounts the blob (it's unlinked only when the last row referencing its hash goes).
The library gained two artifact-backed source types — page and
pdf — whose catalog rows carry an artifact_id. Ingestion reads the stored blob
(pypdf text extraction for PDFs; trafilatura article extraction for pages), chunks
and embeds it like any other source, and search hits carry the artifact_id so a
result opens straight into a viewer.
Page capture (SingleFile-style, but native)
SingleFile itself is AGPL, and this repo ships no AGPL code (the same rule that picked pypdf over PyMuPDF). Capture is therefore native, with an opt-in escape hatch:
- Engine path — the browser pane's
captureop (full mode): serializes the live post-JS DOM (page.content()), fetches subresources with the session's cookies through the same egress policy as navigation (public-host check per redirect hop), and inlines images and stylesheets asdata:URIs. The artifact is stored server-side; only ids cross/ws. - Fallback path —
POST /api/research/capturewith no engine: the page and every subresource are fetched through the browser module's SSRF guard (safe_fetch_html/safe_fetch_bytes), then inlined the same way. - The inliner core (
backend/modules/research/capture.py) is pure and two-pass —list_resources→ caller fetches →build_page— which is exactly what lets the sync engine thread and the async guard share one implementation. Scripts, inline handlers,javascript:URLs, meta-refresh, and any pre-existing<base>are stripped; a fresh<base href>keeps un-inlined links meaningful. Caps: 5 MB per resource, 25 MB per page; an un-fetched resource just stays an absolute URL (the viewer's CSP blocks it from loading — archives are inert). - Known v1 losses: cross-origin iframe content, JS-lazy assets that never
entered the DOM,
@importchains beyond one level ofurl(...). - single-file-cli (optional): set the
research.singleFileClisetting to a user-installed executable and URL captures shell out to it instead (--dump-content, 60 s timeout). The AGPL boundary is the process boundary; nothing is bundled. Its fetches are its own — only the initial URL is validated against the egress policy.
The browser pane's Save → page (browser.save with target: "page") now
files the capture as a page source pointing at the artifact, and falls back to
the old text-only blog-by-URL ingest if the capture op isn't available.
Viewers
research.pdfViewer(multi-instance, document role) — pdf.js overGET /api/artifacts/:id. Canvas pages with a selectable text layer, zoom, and text-layer find (jumps to the next matching page). Opened with paramsartifactId | sourceId | urland optionalpage. The worker is a Vite-native module worker (new Worker(new URL('pdfjs-dist/build/pdf.worker.min.mjs', …))) — no asset copying, same code in web and desktop.research.pageViewer(multi-instance, document role) — a fully sandboxed iframe (sandbox="",referrerPolicy="no-referrer") over the stored HTML, with the origin URL one click from opening live in the browser pane, and an Export-to-Obsidian button.
Commands: research.capturePage (capture by URL), research.openPdf (upload a
file), research.savePdfUrl (fetch a PDF by URL — SSRF-guarded, 50 MB cap, and
the %PDF- magic decides, not the server's content-type header).
ArXiv browser
backend/modules/arxiv/ wraps the keyless arXiv Atom API
(export.arxiv.org/api/query) with the etiquette it asks for: a process-wide
lock enforces ≥3 s between requests, and a 10-minute TTL cache absorbs repeats
(a panel refining a query and an agent retrying are the same query many times).
Parsing is stdlib ElementTree; arXiv ids are validated against both id
grammars (2107.03374v2, cs/0112017) before any URL is built from them.
Free text is turned into a search_query by build_query. Every term gets its own
all: prefix and the terms are ANDed. Before this, all:large language models
scoped only the first word, so multi-word searches came back loose. A "quoted phrase" stays one phrase. Explicit au: / cat: / ti: terms and AND / OR /
ANDNOT pass through unchanged. search(categories=[…]) ORs several categories; the
Discover pane's default feed uses it for cs.LG, cs.AI and cs.CL. The
arXiv browser's empty state links to that feed.
Spacing requests isn't the whole story — arXiv also rate-limits over longer
windows and answers 429 (or 503 while shedding load), which no local
throttle can predict. Both statuses are retried up to three times with a
backoff that honors Retry-After (falling back to 10 s, doubling per attempt),
and the resulting cooldown is process-wide: a deep-research run fanning out
a dozen queries would otherwise burn a full retry budget per query. A caller
arriving during a cooldown longer than 30 s fails fast instead of parking a
request handler, and a rate-limited lookup falls back to a stale cache entry
when one exists — a ten-minute-old abstract beats an error. Exhausted retries
raise ArxivRateLimited, which the routes pass through as a real 429 with a
Retry-After header rather than flattening it into "arXiv unreachable".
GET /api/arxiv/search?query=&category=&sort=&start=— Atom search (relevance / newest / recently-updated, category filter, pagination).GET /api/arxiv/paper/{id}— full metadata for one paper.POST /api/arxiv/download {arxiv_id, library?}— fetch the PDF through the SSRF guard (one egress policy, no exceptions for known hosts) into the artifact store, and file it as apdfsource with the paper's real title and authors, the abstract page as the citable URL, and anarxivtag.
The arXiv section of the Research window (research.console, sections
Deep research and arXiv; the view is research.arxiv, now embedded) is the
human surface:
search + category/sort filters, a results list, the abstract view, and
Open PDF / Save to library buttons — Open PDF lands directly in the PDF
viewer pane. Agent tools arxiv.search / arxiv.get (read-only) and
arxiv.download (side effect, gated per {arxiv_id}) expose the same flow
server-side.
Deep research: durable multi-agent runs
POST /api/research/runs {query, effort?} starts a run: an
orchestrator-worker research pipeline that survives backend restarts and ends
in a cited markdown report filed into the library (and the Obsidian vault, if
configured). The research.console pane manages runs and streams progress.
The pipeline above is the v1 single pass. It now runs as
plan → [subagents → critique]* → synthesis → verify → citations → export — see
Depth: rounds, verification, steering.
The durable runner
There is no Temporal server — durability is a discipline, not a
dependency. Every stage is a row in research_steps (app.db), and a step's
output + full message transcript are persisted in the same UPDATE that marks
it done, so there is no completed-but-unsaved window. The process-global
ResearchRunner (backend/modules/research/runner.py, started from the app
lifespan):
- Resumes on boot: non-terminal runs re-enqueue; steps stuck
running(a crash caught them in flight) reset topendingkeeping their attempt count. Restarting the backend mid-run costs at most the step in flight. - Retries with backoff:
min(5·2^attempt, 120)seconds, up tomax_attempts(3). A permanently failed subagent does not fail the run — synthesis proceeds with the survivors. A permanently failed plan/synthesis/citations/export step does. - Cancels cooperatively (
POST /runs/{id}/cancel, checked between rounds and steps) and retries manually (POST /runs/{id}/retry— failed steps get a fresh attempt budget; done steps keep their checkpoints). - Enforces a per-run token budget (
research.tokenBudget, default 200k, estimated where providers don't report usage): past it, remaining subagents areskippedand the run synthesizes what it has.
The pipeline
Built directly on the agent module's provider layer (providers.chat /
chat_stream) — not agent.delegate, which is tied to a /ws connection
with a 300 s timeout; runs must outlive the browser tab. Provider/model come
from research.provider / research.model (empty = the node's configured
agent, so the default stays local); cloud keys ride in through the litellm
dialect's encrypted-secrets lookup. research.subagentModel lets subagents run
on a smaller model than the lead.
- plan — the lead model decomposes the query under an explicit
effort-scaling rubric (quick = 1 subagent / 3–10 calls; standard = 2–3 / ≤15;
deep = ≤
research.maxSubagents/ ≤25) into detailed task specs: objective, output format, tool guidance, boundaries, call budget. Strict JSON with one repair round. - subagents — run in parallel under
research.subagentParallelism(default 2 — local providers serialize poorly). Each is a bounded tool loop over the internal toolset:arxiv_search/arxiv_get,fetch_page(SSRF-guarded reader),library_search,save_source(archives evidence aspageartifacts), andweb_search— DuckDuckGo's keyless HTML endpoint behind theresearch.webSearchsetting, degrading to an explanatory tool result when its markup drifts or it rate-limits. Past the call budget the tools are withdrawn and the subagent must report: findings + aSOURCES:list. - synthesis — the lead model writes the report over the merged findings,
citing a globally numbered, URL-deduped source list as
[n]; deltas stream to the console (deltaevents, ~4 Hz). - citations — a separate pass (the synthesis model's reasoning never
grades its own attribution): markers verified/corrected,
## Referencesappended, and a deterministic post-check strips any[n]that doesn't resolve. - export — the report lands as a
reportartifact + a library note (taggedresearch-report) + an Obsidian note when a vault is configured.
Tracing
Every step's complete message transcript is persisted and served
(GET /api/research/runs/{id}/steps?transcript=true) — the console's step
drill-down and the observability pane read the same record. Run/step snapshots
and synthesis deltas stream on the research /ws channel.
A run is one OTel trace (invoke_agent research, source:research), with one child
span per step attempt and the step's model calls inside it; a peer sub-agent's spans
hang off its step when the friend returns them. See OpenTelemetry.
Depth: rounds, verification, steering
Gap-filling rounds
A single pass answers only what it thought to ask at the start. The critique step closes each round: it reads what actually came back, names the gaps, and either declares the findings sufficient or writes new subagent specs targeting only what is missing. The second round asks questions the first round taught it to ask — that is where the depth comes from.
Rounds are capped by effort tier (quick 1, standard 2, deep 3). research.maxRounds is
an override, not a lid — it defaults to 0 meaning "use the tier", so a non-zero
default can't silently stop deep runs reaching their third round.
The current round is always derived from the database (max(round) over subagent
steps), never from a loop counter. That is what makes a restart mid-round resume into
the right round instead of starting over. Steps are keyed by (kind, round)
throughout; keying by kind alone silently collapses several critiques into one.
A failed critique is treated as "sufficient" rather than failing the run — the same principle as a failed subagent. Throwing away a completed round because the reviewer couldn't produce JSON would be the worst available trade.
Verification: what is actually supported
Between synthesis and citations — not after, because the citation pass rewrites the
report and appends ## References, so the audit has to inform that rewrite.
The interesting half needs no model. Claims are extracted by the lead, then their
citations are grouped by registrable domain: three citations to openai.com,
platform.openai.com and openai.com/research are one publisher wearing three hats
and count as one independent source. That is the honest reading of "two independent
sources", it is deterministic, and it is unit-tested.
Verdicts are supported (two or more publishers), single-sourced or unsupported.
A separate single pass over the findings flags where sources genuinely contradict
each other — one call over the whole block, not pairwise comparison of subagent
outputs, which is quadratic and mostly surfaces differences of emphasis.
The citation step then marks single-sourced claims inline and appends a Confidence & caveats section listing contradictions and unsupported claims. It never deletes or softens content — the reader decides what one source is worth.
research.verifyDepth (off | cheap | corroborate) gates it; cheap is the
default and costs two model calls.
Steering a run in flight
Review the plan first. Starting a run with approval_mode: "plan" (the "review
plan" checkbox) parks it at awaiting_plan after planning, before any subagent has
spent a token. POST /runs/{id}/plan approves it, optionally with an edited plan.
The gate returns rather than blocking, and that matters twice over:
research.maxConcurrentRuns defaults to 1, so a blocking wait would halt every other
run on the node, and it wouldn't survive a restart either. awaiting_plan is excluded
from the resume pass — a run waiting on a person must not be re-enqueued on every
boot, or it spins through the gate forever.
An edited plan is written to both the run row and the plan step's output. The pipeline reads it back from the step on resume, so writing only the run row would silently discard the edit the moment the run picked up again.
Ask follow-ups mid-run. POST /runs/{id}/followup queues a question; the next
critique folds unconsumed follow-ups into its "what's still missing" prompt, so they
shape the next round rather than being lost.
Watch the tools live. Each subagent tool call streams on the research channel as
a tool event and persists to research_tool_calls. Without it a subagent shows
nothing for minutes — a step's transcript is only written when the step finishes,
which is exactly the window where you want to know whether it is searching sensibly or
looping. GET /runs/{id}/tool-calls is how a browser that wasn't open catches up.
Schema migrations
init_research_db was CREATE TABLE IF NOT EXISTS only, which is a no-op on an
install that already has the tables — so new columns would have silently never
appeared and every write would have failed on an unknown column. _ensure_column
(PRAGMA + ALTER TABLE) now backfills research_steps.round,
research_runs.approval_mode and research_runs.rounds_used, and there is a test that
builds the old schema first, because a fresh-database test cannot see this class of
bug.
The deliberately separate job system: tasks/queue.py stays the serial
short-job queue (library ingest, Drive sync) — a ten-minute research run on
that worker would starve every ingest behind it.
Distributing a wave to friends
A research wave is embarrassingly parallel, and both halves of distributing it
already existed: a subagent is an isolated unit of work with a spec in and a report
out, and agent.ask_peer is an authenticated agent-to-agent RPC. So this needs
no tunnel, no lease and no new wire type — research.distributeSubagents
(default off) hands at most two steps per wave to trusted, connected peers
advertising the agent capability.
Four things keep it honest:
- A remote turn runs under the peer's permissions. The callee gates on its own
network.allowRemoteAgentandnetwork.remoteAgentMode, which defaults to read-onlyplan. A remote subagent can therefore have fewer tools than a local one, so the brief asks for read-only research and nothing that acts. - A decline is never fatal. A peer that refuses, times out, or answers with
nothing returns
Noneand the step runs locally instead. Distribution can make a wave faster; it cannot make one fail. An empty answer counts as a decline — recording it as done would let synthesis proceed believing that angle was covered. - Provenance travels with the report (
ran_on). The verification pass grades claims by independent publisher, so two peers citing the same domain must count as one source rather than two, and that arithmetic needs to know where each finding came from. The console printsran on <node>beside such a step, and only such a step — its absence already means the work stayed here. - Remote tokens are not charged to this run's budget. They were spent on the peer's, and charging them here would throttle a run for work it did not pay for.
Research uses its own PEER_SUBAGENT_TIMEOUT_S rather than raising
agent_bridge's: a deep subagent turn is minutes of tool calls, while ask_peer's
default is sized for a conversational question, and a chat user waiting on "ask
Rob's agent" should not inherit a ten-minute ceiling.
Assignments are keyed by step id, not index — a follow-up posted mid-wave joins the round (see Depth), and indices would shift under it.
Obsidian export
The node is the source of truth; the vault gets a readable mirror. With
research.obsidianVault set (and optionally research.obsidianFolder, default
Horrible Research), POST /api/research/export writes:
- a
.mdnote — frontmatter (source,source_id,artifact_id,url,saved_at,tags,type) plus the extracted text (pages/PDFs) or the report markdown itself; - the original blob copied under
attachments/and linked with![[...]](reports skip this — the note is the artifact).
Titles are sanitized to Windows-legal, Obsidian-safe names (no #, [[,
reserved characters; 120-char cap), collisions get (2), (3)…, and a
resolve() + is_relative_to assertion backstops the sanitizer so nothing can
write outside the vault.
Agent tools
Backend tools (group research, disclosed progressively; they work with no
browser pane attached):
| Tool | Effect |
|---|---|
research.start | Launch a durable deep-research run; returns the run id (side effect) |
research.status | Status of recent runs or one run (read-only) |
research.report | A finished run's report markdown (read-only) |
research.capture | Save a URL as a self-contained page artifact + library source (side effect) |
research.savePdf | Download a PDF by URL into the store + library (side effect) |
arxiv.search | Search arXiv (read-only) |
arxiv.get | Full metadata for one paper (read-only) |
arxiv.download | Pull a paper's PDF into the store + library (side effect) |
Frontend (UI-touching) tools on the console pane: research.openConsole,
research.openPdf, research.openPage — thin pane-openers, so the agent can
put what it fetched in front of the user. Group blurbs and preload keywords
("deep research", "cite", "paper", "preprint", …) are registered with the
orchestrator's progressive-disclosure tables, and the researcher persona
(roster.py) is scoped to browser/library/github/google/research/arxiv with
research, library, and arxiv preloaded; its prompt tells it when to
answer directly vs. start a durable run.
Security notes
- Every server-side fetch in this module rides the browser module's SSRF guard —
scheme allowlist, public-IP resolution check per redirect hop, content-type
allowlists, and in-transfer size caps. Phase 1 also closed the long-standing
gap where library blog ingest fetched with a bare httpx client: it now goes
through
fetch_readablelike everything else. - Stored pages are inert: scripts stripped at capture,
sandbox=""at render, and CSPsandbox; default-src 'self' data:at the byte route. Three layers, any one of which would have to fail and the other two with it. - The artifact byte route validates ids as 32-hex before touching the database, and blob paths derive from stored hashes only.