Skip to main content

Module: visualizer

A dynamic workspace visualizer widget that allows the user and agent to generate, edit, run, and control animations, drawings, and simulations.

Status: implemented — frontend in packages/core/src/modules/visualizer/, backend in backend/modules/visualizer/.

Architecture & Rendering Engines​

The Visualizer module supports two execution environments for animations:

1. Client-Side JavaScript Animations​

JS visualizations execute in the main browser thread to achieve high-performance WebGL/2D canvas rendering (60 FPS).

  • Libraries: Native HTML5 Canvas 2D, Three.js (WebGL, pre-loaded), and Babylon.js (WebGL, loaded dynamically via CDN at runtime on first request).
  • Lifecycle Contract: Scripts compiled via new Function(...) may return an object implementing the lifecycle interface:
    return {
    init: (canvas, THREE, BABYLON) => {
    // Initialize scene, camera, mesh, renderer, and lights
    },
    tick: (time, canvas) => {
    // Frame loop called via requestAnimationFrame
    // Use time (ms) to animate objects
    },
    cleanup: () => {
    // Discard geometries, materials, renderers, and event listeners
    }
    }
  • Standalone scripts also work: a script that doesn't return hooks (e.g. an LLM-generated Three.js scene with its own animate() loop) runs as-is. canvas, THREE, and BABYLON are injected into scope; new THREE.WebGLRenderer() is bound to the pane canvas (so the renderer renders in-pane even without a { canvas } arg), requestAnimationFrame is tracked so the loop is cancelled on stop/re-run, and document.body.appendChild(...) is a no-op to keep a detached canvas from escaping the pane. This makes weak models' typical output render without them having to emit the hook shape.
  • Errors show the real cause (run-error.ts). "WebGL is unavailable" is only reported when a throwaway canvas genuinely cannot create a WebGL context; otherwise the script's own error is shown. The earlier handler relabeled any message containing context as a hardware problem, which hid the common real cause: a canvas keeps its first context type, so a script (or the linked editor buffer — Source defaults to the active one) that calls canvas.getContext('2d') makes a later WebGL renderer fail on a machine with a working GPU.

2. Backend-Side Headless Pygame Simulations​

Python simulations run inside an isolated subprocess on the backend, streaming frames in real-time.

  • Headless Mode: Sets os.environ["SDL_VIDEODRIVER"] = "dummy" before importing pygame to allow running on headless servers.
  • Display Interception: Monkeypatches pygame.display.flip() and pygame.display.update() to capture the current frame buffer.
  • Frame Streaming: Encodes the captured frame into JPEG, formats it as a Base64 string, and prints FRAME:<base64_data> to stdout.
  • WebSocket Relay: The backend spawns the subprocess with blocking subprocess.Popen and pumps stdout/stderr on daemon reader threads, relaying each frame onto the event loop with asyncio.run_coroutine_threadsafe and out over the multiplexed WebSocket. Threads instead of the asyncio subprocess API on purpose: on Windows that API raises NotImplementedError under the SelectorEventLoop that uvicorn --reload uses, so the thread-based runner is loop-agnostic and streams under the documented dev command — the same reasoning as the LSP transport (backend/modules/lsp/manager.py) and terminal PTY.

Contributions to the Layout Shell​

  • Widgets: visualizer.home — opens the Visualizer workspace widget, which contains:
    • Canvas Viewer: A responsive layout wrapper displaying the output <canvas>.
    • Control Strip: Controls for Play/Pause, Restart, Engine selector (Canvas 2D, Three.js, Babylon.js, Pygame), and target buffer selection.
    • Editor Link: Dynamically binds to the active editor buffer or any other open file. Detects edits and automatically hot-reloads the animation in real-time (500ms debounce).
    • Export to editor (⤴ Editor button): sends the current script to the editor as a new buffer and links to it (see Editor integration below). Shift-click writes a workspace file instead of a note.
  • Commands: visualizer.open (Visualizer: Open Pane) — opens the visualizer widget.

Editor integration​

The visualizer and editor are bridged through a small editor service registered on the module registry (registry.provideService('editor', …) from packages/core/src/modules/editor/service.ts), so the visualizer no longer deep-imports editor internals. The service exposes openBufferFromContent, getBufferContent (live-or-loadSource fallback), peekBufferContent (sync, live-only), setBufferContent, getActiveBufferSource, and listBuffers.

  • Visualizer → editor (export). openBufferFromContent({ content, language, prefer }) materializes the current script as a new buffer — a backend note by default, or a scratch/viz-*.{js,py} workspace file (prefer:'file', falling back to a note when no workspace root exists). The visualizer then sets that URI as its live Source, so further edits in the editor hot-reload the animation.
  • Editor → visualizer (editor.visualizeBuffer, "Editor: Open in visualizer"). Opens the visualizer pane pointed at the active buffer via the visualizer's setTarget(uri, mode) seam, inferring the engine from the buffer's language.
  • Mode ⇄ language bridge (visualizer/bridge.ts): engines map to a language for export (canvas|three|babylon → javascript, pygame → python); the reverse is ambiguous, so import preserves the current JS engine and otherwise defaults (three for JS, pygame for Python).
  • Unmounted tabs. the frame drops inactive tabs, so the link reads through the service's getBufferContent (which falls back to the persisted bytes) on the initial run; the 500ms hot-reload poll uses the synchronous live-only peekBufferContent.

WebSocket Protocol​

All Pygame control and frame streaming are multiplexed over the shared /ws socket under the "visualizer" channel.

DirectionEventData Shape / Purpose
client → serverstart_pygame{"code": str} — Start a new Pygame subprocess with code.
client → serverstop_pygame{} — Terminate the active Pygame subprocess.
server → clientframe{"frame": "data:image/jpeg;base64,..."} — Streamed JPEG frames.
server → clienterror{"message": str} — Subprocess execution/syntax trace or missing package error.

Agent Tools​

The visualizer registers tools with the agent orchestrator to allow agents to generate and preview animations dynamically:

Tool NameDescriptionParametersSide Effect
visualizer.render_jsCompiles and executes a 2D Canvas, Three.js, or Babylon.js script in the browser.mode (canvas|three|babylon), code (str)Yes
visualizer.render_htmlShows a self-contained HTML document (SVG, charts, D3/Chart.js pages) in a sandboxed allow-scripts iframe: an opaque origin, so it has no access to the app's storage or the agent socket. It unlinks the pane from editor buffers first, so it never overwrites the user's file.html (str)Yes
visualizer.run_pygameRuns a Pygame script on the backend, streaming frames to the canvas.code (str)Yes
visualizer.clearStops any running animation (JS loop or Python subprocess) and clears the canvas.NoneYes
visualizer.get_stateRetrieves the active mode, rendering status, error details, and the full current source code.NoneNo
visualizer.export_to_editorExports the current script to the editor as a new buffer and links the visualizer to it.target (note|file, default note)Yes

To modify what's already on screen ("make the sphere red"), the agent reads the loaded script with visualizer.get_state, applies a minimal edit that preserves the existing structure, then re-renders with visualizer.render_js / run_pygame. The current code is also surfaced in the pane's agent-context snapshot (get_pane_context), so the standard "what's on screen" path sees it too. Before this, get_state and the snapshot exposed only a codeLength, so the agent couldn't read the source and would (correctly, given the tools) ask the user to paste the code.

One JS tool, not one per engine​

render_js covers Canvas 2D, Three.js, and Babylon.js, discriminated by mode — there is deliberately no separate tool per framework. The split that matters is runtime, not framework: the three JS engines share one browser execution contract, so they share one tool; Pygame runs in a backend subprocess, so it gets its own (run_pygame). More near-duplicate tools would only worsen tool selection for small local models without addressing code quality.

The three JS engines share a lifecycle contract (see the TEMPLATES in packages/core/src/modules/visualizer/widgets.tsx): the script must return { init(canvas, THREE, BABYLON), tick(timeMs, canvas), cleanup() }. init builds the scene once against the host-provided canvas, tick draws one frame (the host drives the loop), cleanup disposes. The render_js tool description encodes this contract and the per-mode drawing API so the agent generates correct code (use the given canvas; never window.onload or a self-driven requestAnimationFrame; wrap geometry + material in a Mesh).

Browser vs Desktop​

  • Browser Layout: Direct browser WebGL/Canvas execution. Headless Pygame runs on the backend server host and streams frames.
  • Desktop Layout: Reuses the exact same frontend canvas and WebSockets layer connecting to the local backend process.