Skip to main content

Module: research

Collect, read, and (soon) synthesize research material. The research module is the home of the artifact store viewers and the flows that fill it: capture any URL as a self-contained HTML archive, store and read PDFs, and mirror everything into an Obsidian vault. Later phases add the ArXiv browser and the durable deep-research engine, which work alongside the browser and library panes.

Status: implemented (all four phases) — artifact store, page capture (engine + fallback), PDF pipeline + viewer, saved-page viewer, Obsidian export, the ArXiv browser, the durable deep-research engine + console, and full research.* / arxiv.* agent awareness.

The artifact store​

The node's first on-disk blob store (backend/modules/artifacts/):

  • Blobs live at $HORRIBLE_DATA_DIR/artifacts/<sha256[:2]>/<sha256> — content-addressed, so saving the same PDF twice stores one blob, and the serving path is derived only from the hash: path traversal is impossible by construction.
  • Rows live in an artifacts table in app.db (id, sha256, kind, mime, filename, size, origin_url, meta). Kinds: pdf, page, report.
  • GET /api/artifacts/:id is the byte route the viewers load from. Captured pages are served with Content-Security-Policy: sandbox; default-src 'self' data: on top of the viewer's iframe sandbox — stored third-party HTML never gets scripts or an origin.
  • POST /api/artifacts/upload accepts PDF multipart uploads (100 MB cap); DELETE refcounts the blob (it's unlinked only when the last row referencing its hash goes).

The library gained two artifact-backed source types — page and pdf — whose catalog rows carry an artifact_id. Ingestion reads the stored blob (pypdf text extraction for PDFs; trafilatura article extraction for pages), chunks and embeds it like any other source, and search hits carry the artifact_id so a result opens straight into a viewer.

Page capture (SingleFile-style, but native)​

SingleFile itself is AGPL, and this repo ships no AGPL code (the same rule that picked pypdf over PyMuPDF). Capture is therefore native, with an opt-in escape hatch:

  • Engine path — the browser pane's capture op (full mode): serializes the live post-JS DOM (page.content()), fetches subresources with the session's cookies through the same egress policy as navigation (public-host check per redirect hop), and inlines images and stylesheets as data: URIs. The artifact is stored server-side; only ids cross /ws.
  • Fallback path — POST /api/research/capture with no engine: the page and every subresource are fetched through the browser module's SSRF guard (safe_fetch_html / safe_fetch_bytes), then inlined the same way.
  • The inliner core (backend/modules/research/capture.py) is pure and two-pass — list_resources → caller fetches → build_page — which is exactly what lets the sync engine thread and the async guard share one implementation. Scripts, inline handlers, javascript: URLs, meta-refresh, and any pre-existing <base> are stripped; a fresh <base href> keeps un-inlined links meaningful. Caps: 5 MB per resource, 25 MB per page; an un-fetched resource just stays an absolute URL (the viewer's CSP blocks it from loading — archives are inert).
  • Known v1 losses: cross-origin iframe content, JS-lazy assets that never entered the DOM, @import chains beyond one level of url(...).
  • single-file-cli (optional): set the research.singleFileCli setting to a user-installed executable and URL captures shell out to it instead (--dump-content, 60 s timeout). The AGPL boundary is the process boundary; nothing is bundled. Its fetches are its own — only the initial URL is validated against the egress policy.

The browser pane's Save → page (browser.save with target: "page") now files the capture as a page source pointing at the artifact, and falls back to the old text-only blog-by-URL ingest if the capture op isn't available.

Viewers​

  • research.pdfViewer (multi-instance, document role) — pdf.js over GET /api/artifacts/:id. Canvas pages with a selectable text layer, zoom, and text-layer find (jumps to the next matching page). Opened with params artifactId | sourceId | url and optional page. The worker is a Vite-native module worker (new Worker(new URL('pdfjs-dist/build/pdf.worker.min.mjs', …))) — no asset copying, same code in web and desktop.
  • research.pageViewer (multi-instance, document role) — a fully sandboxed iframe (sandbox="", referrerPolicy="no-referrer") over the stored HTML, with the origin URL one click from opening live in the browser pane, and an Export-to-Obsidian button.

Commands: research.capturePage (capture by URL), research.openPdf (upload a file), research.savePdfUrl (fetch a PDF by URL — SSRF-guarded, 50 MB cap, and the %PDF- magic decides, not the server's content-type header).

ArXiv browser​

backend/modules/arxiv/ wraps the keyless arXiv Atom API (export.arxiv.org/api/query) with the etiquette it asks for: a process-wide lock enforces ≥3 s between requests, and a 10-minute TTL cache absorbs repeats (a panel refining a query and an agent retrying are the same query many times). Parsing is stdlib ElementTree; arXiv ids are validated against both id grammars (2107.03374v2, cs/0112017) before any URL is built from them.

Free text is turned into a search_query by build_query. Every term gets its own all: prefix and the terms are ANDed. Before this, all:large language models scoped only the first word, so multi-word searches came back loose. A "quoted phrase" stays one phrase. Explicit au: / cat: / ti: terms and AND / OR / ANDNOT pass through unchanged. search(categories=[…]) ORs several categories; the Discover pane's default feed uses it for cs.LG, cs.AI and cs.CL. The arXiv browser's empty state links to that feed.

Spacing requests isn't the whole story — arXiv also rate-limits over longer windows and answers 429 (or 503 while shedding load), which no local throttle can predict. Both statuses are retried up to three times with a backoff that honors Retry-After (falling back to 10 s, doubling per attempt), and the resulting cooldown is process-wide: a deep-research run fanning out a dozen queries would otherwise burn a full retry budget per query. A caller arriving during a cooldown longer than 30 s fails fast instead of parking a request handler, and a rate-limited lookup falls back to a stale cache entry when one exists — a ten-minute-old abstract beats an error. Exhausted retries raise ArxivRateLimited, which the routes pass through as a real 429 with a Retry-After header rather than flattening it into "arXiv unreachable".

  • GET /api/arxiv/search?query=&category=&sort=&start= — Atom search (relevance / newest / recently-updated, category filter, pagination).
  • GET /api/arxiv/paper/{id} — full metadata for one paper.
  • POST /api/arxiv/download {arxiv_id, library?} — fetch the PDF through the SSRF guard (one egress policy, no exceptions for known hosts) into the artifact store, and file it as a pdf source with the paper's real title and authors, the abstract page as the citable URL, and an arxiv tag.

The arXiv section of the Research window (research.console, sections Deep research and arXiv; the view is research.arxiv, now embedded) is the human surface: search + category/sort filters, a results list, the abstract view, and Open PDF / Save to library buttons — Open PDF lands directly in the PDF viewer pane. Agent tools arxiv.search / arxiv.get (read-only) and arxiv.download (side effect, gated per {arxiv_id}) expose the same flow server-side.

Deep research: durable multi-agent runs​

POST /api/research/runs {query, effort?} starts a run: an orchestrator-worker research pipeline that survives backend restarts and ends in a cited markdown report filed into the library (and the Obsidian vault, if configured). The research.console pane manages runs and streams progress.

Rounds, verification and steering

The pipeline above is the v1 single pass. It now runs as plan → [subagents → critique]* → synthesis → verify → citations → export — see Depth: rounds, verification, steering.

The durable runner​

There is no Temporal server — durability is a discipline, not a dependency. Every stage is a row in research_steps (app.db), and a step's output + full message transcript are persisted in the same UPDATE that marks it done, so there is no completed-but-unsaved window. The process-global ResearchRunner (backend/modules/research/runner.py, started from the app lifespan):

  • Resumes on boot: non-terminal runs re-enqueue; steps stuck running (a crash caught them in flight) reset to pending keeping their attempt count. Restarting the backend mid-run costs at most the step in flight.
  • Retries with backoff: min(5·2^attempt, 120) seconds, up to max_attempts (3). A permanently failed subagent does not fail the run — synthesis proceeds with the survivors. A permanently failed plan/synthesis/citations/export step does.
  • Cancels cooperatively (POST /runs/{id}/cancel, checked between rounds and steps) and retries manually (POST /runs/{id}/retry — failed steps get a fresh attempt budget; done steps keep their checkpoints).
  • Enforces a per-run token budget (research.tokenBudget, default 200k, estimated where providers don't report usage): past it, remaining subagents are skipped and the run synthesizes what it has.

The pipeline​

Built directly on the agent module's provider layer (providers.chat / chat_stream) — not agent.delegate, which is tied to a /ws connection with a 300 s timeout; runs must outlive the browser tab. Provider/model come from research.provider / research.model (empty = the node's configured agent, so the default stays local); cloud keys ride in through the litellm dialect's encrypted-secrets lookup. research.subagentModel lets subagents run on a smaller model than the lead.

  1. plan — the lead model decomposes the query under an explicit effort-scaling rubric (quick = 1 subagent / 3–10 calls; standard = 2–3 / ≤15; deep = ≤research.maxSubagents / ≤25) into detailed task specs: objective, output format, tool guidance, boundaries, call budget. Strict JSON with one repair round.
  2. subagents — run in parallel under research.subagentParallelism (default 2 — local providers serialize poorly). Each is a bounded tool loop over the internal toolset: arxiv_search/arxiv_get, fetch_page (SSRF-guarded reader), library_search, save_source (archives evidence as page artifacts), and web_search — DuckDuckGo's keyless HTML endpoint behind the research.webSearch setting, degrading to an explanatory tool result when its markup drifts or it rate-limits. Past the call budget the tools are withdrawn and the subagent must report: findings + a SOURCES: list.
  3. synthesis — the lead model writes the report over the merged findings, citing a globally numbered, URL-deduped source list as [n]; deltas stream to the console (delta events, ~4 Hz).
  4. citations — a separate pass (the synthesis model's reasoning never grades its own attribution): markers verified/corrected, ## References appended, and a deterministic post-check strips any [n] that doesn't resolve.
  5. export — the report lands as a report artifact + a library note (tagged research-report) + an Obsidian note when a vault is configured.

Tracing​

Every step's complete message transcript is persisted and served (GET /api/research/runs/{id}/steps?transcript=true) — the console's step drill-down and the observability pane read the same record. Run/step snapshots and synthesis deltas stream on the research /ws channel.

A run is one OTel trace (invoke_agent research, source:research), with one child span per step attempt and the step's model calls inside it; a peer sub-agent's spans hang off its step when the friend returns them. See OpenTelemetry.

Depth: rounds, verification, steering​

Gap-filling rounds​

A single pass answers only what it thought to ask at the start. The critique step closes each round: it reads what actually came back, names the gaps, and either declares the findings sufficient or writes new subagent specs targeting only what is missing. The second round asks questions the first round taught it to ask — that is where the depth comes from.

Rounds are capped by effort tier (quick 1, standard 2, deep 3). research.maxRounds is an override, not a lid — it defaults to 0 meaning "use the tier", so a non-zero default can't silently stop deep runs reaching their third round.

The current round is always derived from the database (max(round) over subagent steps), never from a loop counter. That is what makes a restart mid-round resume into the right round instead of starting over. Steps are keyed by (kind, round) throughout; keying by kind alone silently collapses several critiques into one.

A failed critique is treated as "sufficient" rather than failing the run — the same principle as a failed subagent. Throwing away a completed round because the reviewer couldn't produce JSON would be the worst available trade.

Verification: what is actually supported​

Between synthesis and citations — not after, because the citation pass rewrites the report and appends ## References, so the audit has to inform that rewrite.

The interesting half needs no model. Claims are extracted by the lead, then their citations are grouped by registrable domain: three citations to openai.com, platform.openai.com and openai.com/research are one publisher wearing three hats and count as one independent source. That is the honest reading of "two independent sources", it is deterministic, and it is unit-tested.

Verdicts are supported (two or more publishers), single-sourced or unsupported. A separate single pass over the findings flags where sources genuinely contradict each other — one call over the whole block, not pairwise comparison of subagent outputs, which is quadratic and mostly surfaces differences of emphasis.

The citation step then marks single-sourced claims inline and appends a Confidence & caveats section listing contradictions and unsupported claims. It never deletes or softens content — the reader decides what one source is worth.

research.verifyDepth (off | cheap | corroborate) gates it; cheap is the default and costs two model calls.

Steering a run in flight​

Review the plan first. Starting a run with approval_mode: "plan" (the "review plan" checkbox) parks it at awaiting_plan after planning, before any subagent has spent a token. POST /runs/{id}/plan approves it, optionally with an edited plan.

The gate returns rather than blocking, and that matters twice over: research.maxConcurrentRuns defaults to 1, so a blocking wait would halt every other run on the node, and it wouldn't survive a restart either. awaiting_plan is excluded from the resume pass — a run waiting on a person must not be re-enqueued on every boot, or it spins through the gate forever.

An edited plan is written to both the run row and the plan step's output. The pipeline reads it back from the step on resume, so writing only the run row would silently discard the edit the moment the run picked up again.

Ask follow-ups mid-run. POST /runs/{id}/followup queues a question; the next critique folds unconsumed follow-ups into its "what's still missing" prompt, so they shape the next round rather than being lost.

Watch the tools live. Each subagent tool call streams on the research channel as a tool event and persists to research_tool_calls. Without it a subagent shows nothing for minutes — a step's transcript is only written when the step finishes, which is exactly the window where you want to know whether it is searching sensibly or looping. GET /runs/{id}/tool-calls is how a browser that wasn't open catches up.

Schema migrations​

init_research_db was CREATE TABLE IF NOT EXISTS only, which is a no-op on an install that already has the tables — so new columns would have silently never appeared and every write would have failed on an unknown column. _ensure_column (PRAGMA + ALTER TABLE) now backfills research_steps.round, research_runs.approval_mode and research_runs.rounds_used, and there is a test that builds the old schema first, because a fresh-database test cannot see this class of bug.

The deliberately separate job system: tasks/queue.py stays the serial short-job queue (library ingest, Drive sync) — a ten-minute research run on that worker would starve every ingest behind it.

Distributing a wave to friends​

A research wave is embarrassingly parallel, and both halves of distributing it already existed: a subagent is an isolated unit of work with a spec in and a report out, and agent.ask_peer is an authenticated agent-to-agent RPC. So this needs no tunnel, no lease and no new wire type — research.distributeSubagents (default off) hands at most two steps per wave to trusted, connected peers advertising the agent capability.

Four things keep it honest:

  • A remote turn runs under the peer's permissions. The callee gates on its own network.allowRemoteAgent and network.remoteAgentMode, which defaults to read-only plan. A remote subagent can therefore have fewer tools than a local one, so the brief asks for read-only research and nothing that acts.
  • A decline is never fatal. A peer that refuses, times out, or answers with nothing returns None and the step runs locally instead. Distribution can make a wave faster; it cannot make one fail. An empty answer counts as a decline — recording it as done would let synthesis proceed believing that angle was covered.
  • Provenance travels with the report (ran_on). The verification pass grades claims by independent publisher, so two peers citing the same domain must count as one source rather than two, and that arithmetic needs to know where each finding came from. The console prints ran on <node> beside such a step, and only such a step — its absence already means the work stayed here.
  • Remote tokens are not charged to this run's budget. They were spent on the peer's, and charging them here would throttle a run for work it did not pay for.

Research uses its own PEER_SUBAGENT_TIMEOUT_S rather than raising agent_bridge's: a deep subagent turn is minutes of tool calls, while ask_peer's default is sized for a conversational question, and a chat user waiting on "ask Rob's agent" should not inherit a ten-minute ceiling.

Assignments are keyed by step id, not index — a follow-up posted mid-wave joins the round (see Depth), and indices would shift under it.

Obsidian export​

The node is the source of truth; the vault gets a readable mirror. With research.obsidianVault set (and optionally research.obsidianFolder, default Horrible Research), POST /api/research/export writes:

  • a .md note — frontmatter (source, source_id, artifact_id, url, saved_at, tags, type) plus the extracted text (pages/PDFs) or the report markdown itself;
  • the original blob copied under attachments/ and linked with ![[...]] (reports skip this — the note is the artifact).

Titles are sanitized to Windows-legal, Obsidian-safe names (no #, [[, reserved characters; 120-char cap), collisions get (2), (3)…, and a resolve() + is_relative_to assertion backstops the sanitizer so nothing can write outside the vault.

Agent tools​

Backend tools (group research, disclosed progressively; they work with no browser pane attached):

ToolEffect
research.startLaunch a durable deep-research run; returns the run id (side effect)
research.statusStatus of recent runs or one run (read-only)
research.reportA finished run's report markdown (read-only)
research.captureSave a URL as a self-contained page artifact + library source (side effect)
research.savePdfDownload a PDF by URL into the store + library (side effect)
arxiv.searchSearch arXiv (read-only)
arxiv.getFull metadata for one paper (read-only)
arxiv.downloadPull a paper's PDF into the store + library (side effect)

Frontend (UI-touching) tools on the console pane: research.openConsole, research.openPdf, research.openPage — thin pane-openers, so the agent can put what it fetched in front of the user. Group blurbs and preload keywords ("deep research", "cite", "paper", "preprint", …) are registered with the orchestrator's progressive-disclosure tables, and the researcher persona (roster.py) is scoped to browser/library/github/google/research/arxiv with research, library, and arxiv preloaded; its prompt tells it when to answer directly vs. start a durable run.

Security notes​

  • Every server-side fetch in this module rides the browser module's SSRF guard — scheme allowlist, public-IP resolution check per redirect hop, content-type allowlists, and in-transfer size caps. Phase 1 also closed the long-standing gap where library blog ingest fetched with a bare httpx client: it now goes through fetch_readable like everything else.
  • Stored pages are inert: scripts stripped at capture, sandbox="" at render, and CSP sandbox; default-src 'self' data: at the byte route. Three layers, any one of which would have to fail and the other two with it.
  • The artifact byte route validates ids as 32-hex before touching the database, and blob paths derive from stored hashes only.