Module: symdex
The shared symbol/docs embedding index — one ingestion pipeline whose corpus
feeds two consumers: the roster agents' semantic retrieval (symbols.* tools,
preloaded by the coder and dba agents)
and the editor's completion layer (doc snippets on suggestions,
with no embedding lookup on the keystroke path).
The corpus (five kinds)
| kind | id prefix | source | one document per |
|---|---|---|---|
packages | pkg: | installed curated framework packages (lsp/pyenv.py FRAMEWORK_PACKAGES) in the resolved interpreter's site-packages | public module/class/function/method — ast.unparsed signature + first docstring paragraph |
stdlib | std: | the resolved interpreter's standard library (extract_stdlib.py) | same, one PackageHarvest per top-level module |
sdk | sdk: | the dashboard's own Python APIs (extract_sdk.py): the dash REPL handle and every dash.<facade>, backend.sdk, horrible_train | public function/method. dash rows carry no import path — it is a REPL global, and completion must never offer from dash.lens import grid |
schema | schema: | every saved database connection via driver.introspect | table (qualified name + column list) |
docs | doc: | this repo's docs/**/*.mdx | chunk (library/chunking.chunk_text) |
Package extraction (backend/modules/symdex/extract_packages.py) is static AST
only — packages are never imported (imports run side effects and are slow);
private modules (_*), tests/, and vendored trees are skipped, and per-package
file/symbol caps keep even torch-sized dists bounded. The site-packages probe is
the one subprocess, run on a thread (the Windows --reload loop can't spawn
asyncio subprocesses — see lsp/manager.py). That probe reports site.getsitepackages()
plus the user site-packages, not just the sysconfig prefix scheme — on a
pip install --user layout (the Windows default for a non-venv interpreter) the prefix
directory is nearly empty and every installed package would otherwise be invisible.
The walk is a budget, spent in priority order
The per-package caps are a budget, and the order it is spent in decides what gets
indexed. The walk used to be alphabetical. On torch, that meant 123 of the 200 files it
was allowed were torch.ao (quantization), and torch.nn, torch.optim and
torch.utils.data were never reached at all. walk_rank now orders every file:
- the package's own
__init__; - the
PRIORITY_SUBTREESfor that package, in their listed order (nn/modules,nn/functional,optim,utils/data,autograd, … for torch;trainer,training_args,modeling_utils,generation, … for transformers). A specific entry precedes its parent (utils/databeforeutils), and the first match wins; - everything else, shallow before deep, since the top of a package is its API and the bottom is its implementation;
- the
DEFERRED_SUBTREESlast (ao,distributed,onnx, the quantizednnvariants, andtransformers/models, which is ~400 model directories).
torch and transformers also get larger caps (PACKAGE_FILE_CAPS and
PACKAGE_SYMBOL_CAPS).
Docs that live outside the definition
A parse-only walk reads ast.get_docstring on a def or a class. torch documents most
of its API some other way, and each way is harvested explicitly:
add_docstr(torch.matmul, r"""…"""): every C-implemented function. These are ~970 calls in_torch_docs.pyand_tensor_docs.py(add_docstr_all("view", …)isTensor.view), files the walk skips because they are private.DOCSTR_FILESnames them, and the symbol is taken from the call's target, not from the file. The docstring's first line restates the call, and that line is the signature; the function is C, so there is nothing else to unparse.conv1d = _add_docstr(torch.conv1d, …): the same thing as an assignment, intorch.nn.functional,torch.linalg,torch.fftandtorch.special. This is handled in every walked file.AdamW.__doc__ = (…)after the class body: how everytorch.optimoptimizer is documented.
.format(**common_args) placeholders cannot be evaluated statically. They render as the
argument's bare name ({input} becomes input) rather than as something invented. A
harvested row replaces the doc-less __all__ stub for the same id instead of being
added next to it.
__all__ re-exports
A parse-only harvest can't see two very common cases: symbols implemented in C
(collections.defaultdict, datetime.datetime) and symbols re-exported from a
submodule (pandas.DataFrame, typing.List) — which are exactly the names people
import. Both are listed in the module's __all__, so _emit_reexports takes that list
at its word and emits any name the AST walk didn't already produce. Those rows carry no
signature or docstring, but they carry the right import module, which is what makes
completion insert from collections import defaultdict. (Packages that build __all__
dynamically — numpy, transformers — still slip through; the curated
framework-import source covers the common ones.)
The stdlib's two scopes
The stdlib is harvested into both projections but at different breadth:
- The relational projection (
code_symbols) covers the whole stdlib — it's a cheap SQLite prefix scan and editor completion has to be exhaustive. - The embedding projection covers only
STDLIB_EMBED, a curated set of ~58 modules (the data-structure, concurrency and serialization modules a notebook reaches for among them) worth spending vectors on for agent retrieval. Embedding every stdlib symbol would be tens of thousands of vectors against a local embedder for very little retrieval gain.
Build cost
A full build is ~46k symbols. Getting that from hours down to minutes was mostly about noticing that the embedder was never the bottleneck — it answers in ~65ms, but the requests were going out ~2s apart.
| what | before | after |
|---|---|---|
| LanceDB write | one merge_insert per document (~1.5s) | one per 256-doc chunk (~6ms/doc) |
| model discovery | one GET /api/tags per document | once per build (60s TTL cache) |
| HTTP client | a new AsyncClient per document | one shared client per batch |
| embed request | one per document | one batched /api/embed, or 8-way concurrent |
| documents embedded | all ~40k | only those with a docstring (~17k) |
merge_insert is the one to internalize: it's a whole-table operation, so its cost
tracks table size, not the number of rows you pass it. Handing it one row at a time is
the worst possible use of it — measured at ~1.5s per call on a 5k-row table, which is
16 hours for a 40k-symbol build spent entirely in Arrow.
The docstring filter is a quality decision as much as a speed one: 58% of harvested
symbols have no docstring, and their embeddable text degenerates to the name restated
as a sentence ("class numpy.linalg.LinAlgError"). That's no semantic signal — you
find those by prefix, which the relational index already does instantly — and it
dilutes the vector space with near-duplicate rows.
The same chunked shape is used by code/semantic.py, which had the
identical per-document loop.
Two projections
- LanceDB
symdexcollection (database/vectorstore.py) — the semantic space, searched bysymbols.searchDocs/GET /api/symdex/search. code_symbolsinapp.db(lsp/symbol_store.py, sourcespkg:<dist>andstd:<module>, withdocandimpcolumns) — the same symbols in the relational prefix index, soGET /editor/completereturns signature + doc snippet + the importable module from the SQLite scan it already does. Embedding lookups never sit on the keystroke path.
The relational projection is written even when the embedder is offline. It needs no vectors, and editor completion depends on it — gating it behind a live embedding provider meant intellisense silently stayed empty whenever the local model wasn't up.
The embed-model drift guard
get_embedding picks its model dynamically (whatever embedding model the
provider offers — 384/768/1024 dims) and silently falls back to a 384-dim hash
offline, while a LanceDB collection's vector width is fixed at first write. The
index therefore keeps a meta sidecar ($HORRIBLE_DATA_DIR/symdex.meta.json,
{embed_model, dim, counts}):
- Model changed → the next reindex rebuilds the requested kinds plus any kind that actually had vectors under the old model. A kind that was never built has nothing to invalidate, and rebuilding it here would drag the whole stdlib and every installed package into what should be a targeted repair.
- Only the hash fallback available → reindex refuses (never poisons a
real-model collection); search returns
status: "reindex_needed"with empty results — never aDimensionMismatchcrash.
The sidecar makes a model change survivable, not invisible: between the change and
the rebuild, search answers reindex_needed and returns nothing, which is easy to
read as "the index is empty" rather than "the index is in the wrong space". Set
database.embedModel to stop discovery
changing the model on its own when the provider's catalog changes.
Triggers & status
-
POST /api/symdex/reindex{kinds: ["packages"|"stdlib"|"schema"|"docs"]}— manual, detached. -
Auto-kick: the first
GET /api/editor/python-envresolve with an interpreter starts a background build of whichever ofstdlib/packagesis still empty (_kick_symbol_indexinlsp/routes.py); creating or editing a database connection re-kicks theschemakind.The kick holds a strong reference to its task and logs the outcome.
asyncio.create_taskonly keeps a weak reference, so the build could be garbage-collected mid-flight — and with failures swallowed, the packages corpus could sit empty indefinitely with nothing to show why. -
Progress streams on the
symdex/wschannel ({event: "index", data: {state, kind, done, total}});GET /api/symdex/statusreports{building, total, counts, embed_model}.
Agent tools (group symbols)
symbols.searchDocs(query, kind?, limit?)— semantic retrieval across the corpus (read-only). The dba agent useskind: "schema"before writing SQL against unfamiliar tables; the coder agent uses it before coding against an unfamiliar API.symbols.lookup(name, lang?)— exact prefix lookup via the relational projection (fast, offline).
Backend surface
backend/modules/symdex/: models.py (Pydantic), extract_packages.py /
extract_stdlib.py / extract_sdk.py / extract_schema.py / extract_docs.py
(sync extractors, run on threads), reference.py (whole docstrings and upstream
links for the reference pane), gen_reference_docs.py (writes
docs/reference/python-*.mdx from the sdk harvest; a test fails when they are
stale),
index.py (SymdexIndex singleton + the symdex broadcaster), routes.py
(/api/symdex/*), tools.py (registered from app.py like the training/games
tools). Modeled on code/semantic.py's SemanticIndex.
The reference routes
The Python reference pane browses code_symbols directly:
| Method | Path | Purpose |
|---|---|---|
GET | /api/symdex/reference/sources | Every browsable source (std:json, pkg:torch, sdk:dash) with its installed version and the interpreter's Python version |
GET | /api/symdex/reference/modules?source= | The modules a source defines symbols in, shallow first |
GET | /api/symdex/reference/members?source=&module= | What a module defines, each class carrying its methods (joined by (source, class), so two packages' Linear never pool) |
GET | /api/symdex/reference/search?q= | Prefix search; a dotted query scopes by module |
GET | /api/symdex/reference/doc?source=&module=&name= | The whole docstring by AST (never importing), following a re-export to its definition (definedIn), plus the import line and a version-pinned upstream link |
Browser vs desktop
Identical — everything runs in the backend; the frontend only sees /ws
progress events and enriched completion items.