Module: visualizer

A dynamic workspace visualizer widget that allows the user and agent to generate, edit, run, and control animations, drawings, and simulations.

Status: implemented — frontend in packages/core/src/modules/visualizer/, backend in backend/modules/visualizer/.

Architecture & Rendering Engines

The Visualizer module supports two execution environments for animations:

1. Client-Side JavaScript Animations

JS visualizations execute in the main browser thread to achieve high-performance WebGL/2D canvas rendering (60 FPS).

• Libraries: Native HTML5 Canvas 2D, Three.js (WebGL, pre-loaded), and Babylon.js (WebGL, loaded dynamically via CDN at runtime on first request).
• Lifecycle Contract: Scripts compiled via new Function(...) may return an object implementing the lifecycle interface:

  return {
    init: (canvas, THREE, BABYLON) => {
      // Initialize scene, camera, mesh, renderer, and lights
    },
    tick: (time, canvas) => {
      // Frame loop called via requestAnimationFrame
      // Use time (ms) to animate objects
    },
    cleanup: () => {
      // Discard geometries, materials, renderers, and event listeners
    }
  }

• Standalone scripts also work: a script that doesn't return hooks (e.g. an LLM-generated Three.js scene with its own animate() loop) runs as-is. canvas, THREE, and BABYLON are injected into scope; new THREE.WebGLRenderer() is bound to the pane canvas (so the renderer renders in-pane even without a { canvas } arg), requestAnimationFrame is tracked so the loop is cancelled on stop/re-run, and document.body.appendChild(...) is a no-op to keep a detached canvas from escaping the pane. This makes weak models' typical output render without them having to emit the hook shape.

2. Backend-Side Headless Pygame Simulations

Python simulations run inside an isolated subprocess on the backend, streaming frames in real-time.

• Headless Mode: Sets os.environ["SDL_VIDEODRIVER"] = "dummy" before importing pygame to allow running on headless servers.
• Display Interception: Monkeypatches pygame.display.flip() and pygame.display.update() to capture the current frame buffer.
• Frame Streaming: Encodes the captured frame into JPEG, formats it as a Base64 string, and prints FRAME:<base64_data> to stdout.
• WebSocket Relay: The backend reads stdout in an async task and pipes each frame back to the frontend over a multiplexed WebSocket.

Contributions to the Layout Shell

• Widgets: visualizer.home — opens the Visualizer workspace widget, which contains:
  • Canvas Viewer: A responsive layout wrapper displaying the output <canvas>.
  • Control Strip: Controls for Play/Pause, Restart, Engine selector (Canvas 2D, Three.js, Babylon.js, Pygame), and target buffer selection.
  • Editor Link: Dynamically binds to the active editor buffer or any other open file. Detects edits and automatically hot-reloads the animation in real-time (500ms debounce).
  • Export to editor (⤴ Editor button): sends the current script to the editor as a new buffer and links to it (see Editor integration below). Shift-click writes a workspace file instead of a note.
• Commands: visualizer.open (Visualizer: Open Pane) — opens the visualizer widget.

Editor integration

The visualizer and editor are bridged through a small editor service registered on the module registry (registry.provideService('editor', …) from packages/core/src/modules/editor/service.ts), so the visualizer no longer deep-imports editor internals. The service exposes openBufferFromContent, getBufferContent (live-or-loadSource fallback), peekBufferContent (sync, live-only), setBufferContent, getActiveBufferSource, and listBuffers.

• Visualizer → editor (export). openBufferFromContent({ content, language, prefer }) materializes the current script as a new buffer — a backend note by default, or a scratch/viz-*.{js,py} workspace file (prefer:'file', falling back to a note when no workspace root exists). The visualizer then sets that URI as its live Source, so further edits in the editor hot-reload the animation.
• Editor → visualizer (editor.visualizeBuffer, "Editor: Open in visualizer"). Opens the visualizer pane pointed at the active buffer via the visualizer's setTarget(uri, mode) seam, inferring the engine from the buffer's language.
• Mode ⇄ language bridge (visualizer/bridge.ts): engines map to a language for export (canvas|three|babylon → javascript, pygame → python); the reverse is ambiguous, so import preserves the current JS engine and otherwise defaults (three for JS, pygame for Python).
• Unmounted tabs. dockview drops inactive panes, so the link reads through the service's getBufferContent (which falls back to the persisted bytes) on the initial run; the 500ms hot-reload poll uses the synchronous live-only peekBufferContent.

WebSocket Protocol

All Pygame control and frame streaming are multiplexed over the shared /ws socket under the "visualizer" channel.

Direction	Event	Data Shape / Purpose
client → server	start_pygame	{"code": str} — Start a new Pygame subprocess with code.
client → server	stop_pygame	{} — Terminate the active Pygame subprocess.
server → client	frame	{"frame": "data:image/jpeg;base64,..."} — Streamed JPEG frames.
server → client	error	{"message": str} — Subprocess execution/syntax trace or missing package error.

Agent Tools

The visualizer registers tools with the agent orchestrator to allow agents to generate and preview animations dynamically:

Tool Name	Description	Parameters	Side Effect
visualizer.render_js	Compiles and executes a 2D Canvas, Three.js, or Babylon.js script in the browser.	code (str)	Yes
visualizer.run_pygame	Runs a Pygame script on the backend, streaming frames to the canvas.	code (str)	Yes
visualizer.clear	Stops any running animation (JS loop or Python subprocess) and clears the canvas.	None	Yes
visualizer.get_state	Retrieves the active mode and script source currently loaded in the widget.	None	No
visualizer.export_to_editor	Exports the current script to the editor as a new buffer and links the visualizer to it.	target (note|file, default note)	Yes

Browser vs Desktop

• Browser Layout: Direct browser WebGL/Canvas execution. Headless Pygame runs on the backend server host and streams frames.
• Desktop (Tauri) Layout: Reuses the exact same frontend canvas and WebSockets layer connecting to the local backend process.