Skip to main content

Module: symdex

The shared symbol/docs embedding index — one ingestion pipeline whose corpus feeds two consumers: the roster agents' semantic retrieval (symbols.* tools, preloaded by the coder and dba agents) and the editor's completion layer (doc snippets on suggestions, with no embedding lookup on the keystroke path).

The corpus (five kinds)​

kindid prefixsourceone document per
packagespkg:installed curated framework packages (lsp/pyenv.py FRAMEWORK_PACKAGES) in the resolved interpreter's site-packagespublic module/class/function/method — ast.unparsed signature + first docstring paragraph
stdlibstd:the resolved interpreter's standard library (extract_stdlib.py)same, one PackageHarvest per top-level module
sdksdk:the dashboard's own Python APIs (extract_sdk.py): the dash REPL handle and every dash.<facade>, backend.sdk, horrible_trainpublic function/method. dash rows carry no import path — it is a REPL global, and completion must never offer from dash.lens import grid
schemaschema:every saved database connection via driver.introspecttable (qualified name + column list)
docsdoc:this repo's docs/**/*.mdxchunk (library/chunking.chunk_text)

Package extraction (backend/modules/symdex/extract_packages.py) is static AST only — packages are never imported (imports run side effects and are slow); private modules (_*), tests/, and vendored trees are skipped, and per-package file/symbol caps keep even torch-sized dists bounded. The site-packages probe is the one subprocess, run on a thread (the Windows --reload loop can't spawn asyncio subprocesses — see lsp/manager.py). That probe reports site.getsitepackages() plus the user site-packages, not just the sysconfig prefix scheme — on a pip install --user layout (the Windows default for a non-venv interpreter) the prefix directory is nearly empty and every installed package would otherwise be invisible.

The walk is a budget, spent in priority order​

The per-package caps are a budget, and the order it is spent in decides what gets indexed. The walk used to be alphabetical. On torch, that meant 123 of the 200 files it was allowed were torch.ao (quantization), and torch.nn, torch.optim and torch.utils.data were never reached at all. walk_rank now orders every file:

  1. the package's own __init__;
  2. the PRIORITY_SUBTREES for that package, in their listed order (nn/modules, nn/functional, optim, utils/data, autograd, … for torch; trainer, training_args, modeling_utils, generation, … for transformers). A specific entry precedes its parent (utils/data before utils), and the first match wins;
  3. everything else, shallow before deep, since the top of a package is its API and the bottom is its implementation;
  4. the DEFERRED_SUBTREES last (ao, distributed, onnx, the quantized nn variants, and transformers/models, which is ~400 model directories).

torch and transformers also get larger caps (PACKAGE_FILE_CAPS and PACKAGE_SYMBOL_CAPS).

Docs that live outside the definition​

A parse-only walk reads ast.get_docstring on a def or a class. torch documents most of its API some other way, and each way is harvested explicitly:

  • add_docstr(torch.matmul, r"""…"""): every C-implemented function. These are ~970 calls in _torch_docs.py and _tensor_docs.py (add_docstr_all("view", …) is Tensor.view), files the walk skips because they are private. DOCSTR_FILES names them, and the symbol is taken from the call's target, not from the file. The docstring's first line restates the call, and that line is the signature; the function is C, so there is nothing else to unparse.
  • conv1d = _add_docstr(torch.conv1d, …): the same thing as an assignment, in torch.nn.functional, torch.linalg, torch.fft and torch.special. This is handled in every walked file.
  • AdamW.__doc__ = (…) after the class body: how every torch.optim optimizer is documented.

.format(**common_args) placeholders cannot be evaluated statically. They render as the argument's bare name ({input} becomes input) rather than as something invented. A harvested row replaces the doc-less __all__ stub for the same id instead of being added next to it.

__all__ re-exports​

A parse-only harvest can't see two very common cases: symbols implemented in C (collections.defaultdict, datetime.datetime) and symbols re-exported from a submodule (pandas.DataFrame, typing.List) — which are exactly the names people import. Both are listed in the module's __all__, so _emit_reexports takes that list at its word and emits any name the AST walk didn't already produce. Those rows carry no signature or docstring, but they carry the right import module, which is what makes completion insert from collections import defaultdict. (Packages that build __all__ dynamically — numpy, transformers — still slip through; the curated framework-import source covers the common ones.)

The stdlib's two scopes​

The stdlib is harvested into both projections but at different breadth:

  • The relational projection (code_symbols) covers the whole stdlib — it's a cheap SQLite prefix scan and editor completion has to be exhaustive.
  • The embedding projection covers only STDLIB_EMBED, a curated set of ~58 modules (the data-structure, concurrency and serialization modules a notebook reaches for among them) worth spending vectors on for agent retrieval. Embedding every stdlib symbol would be tens of thousands of vectors against a local embedder for very little retrieval gain.

Build cost​

A full build is ~46k symbols. Getting that from hours down to minutes was mostly about noticing that the embedder was never the bottleneck — it answers in ~65ms, but the requests were going out ~2s apart.

whatbeforeafter
LanceDB writeone merge_insert per document (~1.5s)one per 256-doc chunk (~6ms/doc)
model discoveryone GET /api/tags per documentonce per build (60s TTL cache)
HTTP clienta new AsyncClient per documentone shared client per batch
embed requestone per documentone batched /api/embed, or 8-way concurrent
documents embeddedall ~40konly those with a docstring (~17k)

merge_insert is the one to internalize: it's a whole-table operation, so its cost tracks table size, not the number of rows you pass it. Handing it one row at a time is the worst possible use of it — measured at ~1.5s per call on a 5k-row table, which is 16 hours for a 40k-symbol build spent entirely in Arrow.

The docstring filter is a quality decision as much as a speed one: 58% of harvested symbols have no docstring, and their embeddable text degenerates to the name restated as a sentence ("class numpy.linalg.LinAlgError"). That's no semantic signal — you find those by prefix, which the relational index already does instantly — and it dilutes the vector space with near-duplicate rows.

The same chunked shape is used by code/semantic.py, which had the identical per-document loop.

Two projections​

  1. LanceDB symdex collection (database/vectorstore.py) — the semantic space, searched by symbols.searchDocs / GET /api/symdex/search.
  2. code_symbols in app.db (lsp/symbol_store.py, sources pkg:<dist> and std:<module>, with doc and imp columns) — the same symbols in the relational prefix index, so GET /editor/complete returns signature + doc snippet + the importable module from the SQLite scan it already does. Embedding lookups never sit on the keystroke path.

The relational projection is written even when the embedder is offline. It needs no vectors, and editor completion depends on it — gating it behind a live embedding provider meant intellisense silently stayed empty whenever the local model wasn't up.

The embed-model drift guard​

get_embedding picks its model dynamically (whatever embedding model the provider offers — 384/768/1024 dims) and silently falls back to a 384-dim hash offline, while a LanceDB collection's vector width is fixed at first write. The index therefore keeps a meta sidecar ($HORRIBLE_DATA_DIR/symdex.meta.json, {embed_model, dim, counts}):

  • Model changed → the next reindex rebuilds the requested kinds plus any kind that actually had vectors under the old model. A kind that was never built has nothing to invalidate, and rebuilding it here would drag the whole stdlib and every installed package into what should be a targeted repair.
  • Only the hash fallback available → reindex refuses (never poisons a real-model collection); search returns status: "reindex_needed" with empty results — never a DimensionMismatch crash.

The sidecar makes a model change survivable, not invisible: between the change and the rebuild, search answers reindex_needed and returns nothing, which is easy to read as "the index is empty" rather than "the index is in the wrong space". Set database.embedModel to stop discovery changing the model on its own when the provider's catalog changes.

Triggers & status​

  • POST /api/symdex/reindex {kinds: ["packages"|"stdlib"|"schema"|"docs"]} — manual, detached.

  • Auto-kick: the first GET /api/editor/python-env resolve with an interpreter starts a background build of whichever of stdlib / packages is still empty (_kick_symbol_index in lsp/routes.py); creating or editing a database connection re-kicks the schema kind.

    The kick holds a strong reference to its task and logs the outcome. asyncio.create_task only keeps a weak reference, so the build could be garbage-collected mid-flight — and with failures swallowed, the packages corpus could sit empty indefinitely with nothing to show why.

  • Progress streams on the symdex /ws channel ({event: "index", data: {state, kind, done, total}}); GET /api/symdex/status reports {building, total, counts, embed_model}.

Agent tools (group symbols)​

  • symbols.searchDocs(query, kind?, limit?) — semantic retrieval across the corpus (read-only). The dba agent uses kind: "schema" before writing SQL against unfamiliar tables; the coder agent uses it before coding against an unfamiliar API.
  • symbols.lookup(name, lang?) — exact prefix lookup via the relational projection (fast, offline).

Backend surface​

backend/modules/symdex/: models.py (Pydantic), extract_packages.py / extract_stdlib.py / extract_sdk.py / extract_schema.py / extract_docs.py (sync extractors, run on threads), reference.py (whole docstrings and upstream links for the reference pane), gen_reference_docs.py (writes docs/reference/python-*.mdx from the sdk harvest; a test fails when they are stale), index.py (SymdexIndex singleton + the symdex broadcaster), routes.py (/api/symdex/*), tools.py (registered from app.py like the training/games tools). Modeled on code/semantic.py's SemanticIndex.

The reference routes​

The Python reference pane browses code_symbols directly:

MethodPathPurpose
GET/api/symdex/reference/sourcesEvery browsable source (std:json, pkg:torch, sdk:dash) with its installed version and the interpreter's Python version
GET/api/symdex/reference/modules?source=The modules a source defines symbols in, shallow first
GET/api/symdex/reference/members?source=&module=What a module defines, each class carrying its methods (joined by (source, class), so two packages' Linear never pool)
GET/api/symdex/reference/search?q=Prefix search; a dotted query scopes by module
GET/api/symdex/reference/doc?source=&module=&name=The whole docstring by AST (never importing), following a re-export to its definition (definedIn), plus the import line and a version-pinned upstream link

Browser vs desktop​

Identical — everything runs in the backend; the frontend only sees /ws progress events and enriched completion items.