Audio
Status: implemented. Default input, default output, and a routing matrix over both — the dashboard's answer to the question VoiceMeeter answers.
Before it, every module that made a sound opened its own AudioContext and
connected to whatever the OS called "default": the karaoke stage, the game's
synthesized noise, voice comms, the Plaza's level meter, the arcade blips. There
was no way to say "the karaoke video goes to the living-room speakers, my
microphone does not", because nothing in the app modelled which output a sound
goes to. There was also no way to route anything anywhere, because nodes cannot
cross AudioContext boundaries — five private contexts are five islands.
- Backend:
backend/modules/audio/(voicemeeter.py,providers.py,store.py,routes.py,agent_tools.py,events.py) - Frontend:
packages/core/src/modules/audio/— theaudio.mixerpane, a settings section, and the engine every other module now feeds - Routes:
GET /api/audio/status,GET|PUT|DELETE /api/audio/mixer,POST /api/audio/host/launch|send|level,POST|DELETE /api/audio/devices - Channel:
audioon the shared/ws - Agent tools:
audio.status,audio.route,audio.level,audio.start_host_mixer
The shape: strips, buses, and a matrix
A strip is a source of sound. A bus is somewhere sound comes out. Any strip can feed any number of buses at once, and that last clause is the whole feature.
That diagram is the motivating example: a video plays into a virtual cable that another application is recording as its microphone, the singer's own voice goes to the same cable, and the headphones hear the video but not the voice. Three sources, two destinations, five decisions — and it is unexpressible in a model with one "output device" setting, which is why the matrix exists.
The one thing no web app can do
A browser cannot create a virtual audio device. Making audio appear as a microphone to another application requires a kernel or HAL driver. Everything here routes to devices the OS already offers; producing the device itself is the job of VoiceMeeter, VB-CABLE, BlackHole, or PipeWire.
The corollary catches people out: an embedded <iframe>'s audio cannot be
routed either. There is no element to attach to a Web Audio graph. Playing "a
YouTube video" through the mixer means playing it through an element we own —
which is exactly what the karaoke module already does, fetching
with yt-dlp to a local file the backend serves same-origin.
Per platform
The platforms differ in kind, not degree, so ProviderStatus carries two
separate capability flags rather than one.
| Platform | Provider | canCreate | canControl | What that means |
|---|---|---|---|---|
| Linux | PipeWire / PulseAudio | ✅ | ❌ | We make the cable ourselves: one pactl load-module. No install, no driver, no third party — the best-supported platform, which surprises people. |
| Windows | Voicemeeter, VB-CABLE | ❌ | ✅ | The device is the user's install, but Voicemeeter has a remote API, so we can drive the machine's whole mixer — including audio the dashboard does not produce. |
| macOS | BlackHole | ❌ | ❌ | Detect, report, link. No scripting surface exists; hearing audio and sending it needs a Multi-Output Device the user creates in Audio MIDI Setup. |
Nothing is bundled. VB-CABLE is donationware with restricted redistribution and BlackHole is a driver with its own installer, so both get the treatment SearXNG and the AssaultCube content get — supported, detected, linked, never shipped.
Three states, not two
The hardware module's rule, applied to audio:
installed=false, certain=truemeans we looked and it is not there.certain=falsemeans we could not ask.
No pactl on PATH is not evidence that there is no sound server; an unreadable
registry is not evidence that Voicemeeter is absent. Rendering the second state
as the first tells a user with a working Voicemeeter to go install Voicemeeter.
Where each half lives
The graph lives in the browser, because only Web Audio can move samples. The
routing is server state, because the karaoke rule applies here too: the
server holds intent, a pane renders it. A change made by the agent, by a second
window, or by a phone on the fabric arrives on the audio channel and every open
mixer reconciles.
That is also why the matrix is not a setting. SettingValue is a scalar, the
axes are discovered at runtime, and GET /api/settings hands the whole bag to
every plugin. It gets its own table, the way layouts do.
Joining the mixer
A module that makes sound registers a strip and connects to it:
import { mixer } from '../audio/engine';
mixer.declareStrip({ id: 'karaoke', label: 'Karaoke', icon: '🎤' });
const handle = mixer.connectStrip('karaoke');
source.connect(handle.input); // never handle.context.destination
For a media element there is a hook, useMediaStrip(element, decl), which handles
the once-per-element rule below. Pass the element held in state through a callback
ref, not a RefObject: a ref object never changes, so an effect keyed on it does
not re-run when a <video> mounts after the first render, and that element is
never attached.
Until a bus exists, a strip plays straight to the default output. Buses come
from the saved routing, which loads at launch (audio.startOnBoot, on by default)
or when the mixer is first opened. A strip connected before that, or with the
backend down, used to reach a fader wired to nothing. createMediaElementSource
had already taken the element's audio, so a karaoke song played in total silence.
The engine now wires such a fader to ctx.destination and unwires it the moment
apply builds a bus.
Capturing a microphone goes through inputConstraints() so the chosen input
device is honoured — a module that keeps calling getUserMedia({ audio: true })
silently gets the system default whatever the user picked, with nothing on screen
to explain it.
openMicrophone() wraps that: it asks for the chosen input, retries the system
default if that fails (unless the failure was a refusal), and otherwise throws a
message from describeMicFailure that names the error and lists the inputs the
window can see. Chromium's "Requested device not found" is the same text for
"no microphone", "the OS default input is a headset that is switched off" and
"this webview enumerates nothing"; only the device list tells them apart.
Clubhouse voice uses it.
The desktop app runs Chromium's audio service unsandboxed
On Windows the desktop shell starts Chromium with
--disable-features=AudioServiceSandbox (apps/desktop/src/main.ts). Under the
Tauri shell, on a machine where the sandboxed audio service crash-looped inside
WebView2, enumerateDevices() returned no audio devices at all: microphone
permission was granted, the profile was fine, and every capture failed with
"Requested device not found". With the sandbox off, all devices came back. The
switch was carried over to Electron's Chromium rather than re-learned. The
renderer stays sandboxed; this is what the enterprise AudioSandboxEnabled=false
policy does.
The shell also sets --autoplay-policy=no-user-gesture-required, or mixer
playback waits for a click, and grants the microphone (and device labels) to the
app's own page without a prompt — the embedded browser's sites get a per-site
prompt instead.
Failures that are silent
Every one of these produces no error — which is why they are listed rather than left to be rediscovered.
-
A private
AudioContextis an island. Nodes cannot cross contexts, and connecting across them is a no-op, not an exception. A module that callsnew AudioContext()is not in the mixer and nothing says so. -
ctx.close()is not cleanup. The context is shared now. Closing it when a match ends or a microphone is switched off silences the entire app. Release the strip instead — this is the "unmount is not close" rule from the layout shell, with an audible symptom.It is worse than "until the next join", because
getContext()caches the one context for the life of the page and never rebuilds a closed one: every latercreateGain()/createMediaStreamDestination()throwsInvalidStateError, and the resume guard does not catch it either — a closed context isclosed, notsuspended. hassault was fixed and Clubhouse's room teardown was missed, so leaving one Clubhouse room left the microphone dead in every room after it while the button still read Mic Active. The rule is now enforced across the tree bymodules/audio/__tests__/shared-context-is-never-closed.test.ts, which reads the source the waycontrol-padding.test.tsdoes — a closed context is invisible at runtime, so there is nothing else to assert against. -
createMediaElementSourcemay be called once per element, and it moves the audio. After attaching, the element no longer plays to the default output on its own; attach without connecting and the video plays in total silence. -
Cross-origin media yields zeros, not an error. Anything routed must be same-origin or CORS-enabled.
-
A context created without a user gesture starts suspended — wired, silent, and uncomplaining. The engine resumes on the first
pointerdown/keydown. -
enumerateDevices()returns blank labels until a media permission is granted. That is a prompt to show, not an empty device list. -
Device ids are origin-scoped and rotate. Clearing site data renames every device, so a mixer keyed on ids alone silently reverts every bus to the default output at once. Both the id and the label are saved, and
resolveDeviceIdtries the id first (exact; two identical headsets share a label) and falls back to the label (survives rotation). -
setSinkIdis Chromium-only and rejects asynchronously for a device that has been unplugged — while the element happily keeps playing to the default output. The engine catches that and marks the bus as fallen back rather than claiming audio went somewhere it did not.
Voicemeeter specifics
The Remote API is reached with ctypes rather than a PyPI wrapper: the DLL ships
with Voicemeeter, so it exists only where Voicemeeter does, and taking a
dependency for nine entry points would put a Windows-only concern into
pyproject.toml for every platform.
- The API is stateful, single-client and not thread-safe.
VBVMR_Loginbinds the process; a second login without a logout returns-2. One module-global client, one lock, logged out from the app lifespan. - The version counts must be read, never assumed. Voicemeeter is 3 strips ×
2 buses, Banana 5 × 5, Potato 8 × 8. A matrix sized for Banana ignores three of
Potato's strips; sized for Potato it writes
Strip[7].A1on a Banana, which goes nowhere and reads as "the routing didn't take". - The executable names do not match the marketing names.
voicemeeterpro.exeis Banana; Potato isvoicemeeter8.exe. Getting this backwards launches the wrong mixer against the user's saved routing. - Launching waits for the engine.
VBVMR_RunVoicemeeterreturns as soon as the process spawns, and every parameter written in the second or two before the audio engine answers is dropped silently. - Launching is never implicit. It takes over the machine's default audio devices, which every running application hears immediately, so it happens only when someone asks.
Not built
- Level meters. The matrix shows routing, not signal; a meter per strip needs
an
AnalyserNodeper strip and a render loop. - Per-application routing on Linux and macOS. Moving another app's stream is a per-stream operation on PipeWire and impossible on macOS without a helper.
- A media pane. The karaoke stage is currently the only routable media element; a general "play this URL through the mixer" pane is the obvious next source.
- Fabric audio. The phone is already a peer; carrying a bus to it (or a strip from it) would make any paired device a speaker or a microphone.
The Viewers bus
A shared session's guests are an output like any other. The
share module adds a bus whose deviceId is the sentinel
horrible:viewers and whose output is a MediaStreamDestination rather than a
device, so "what the viewers hear" is an ordinary column in the matrix — karaoke
to the viewers but not to your headphones, or your microphone to both, are one
checkbox each.
Two things about it are deliberate:
-
Nothing is routed to it when it appears. The same rule
addBusfollows, for a stronger reason: this output is another person's ears, and defaulting it to carry whatever is playing would make starting a screen share silently broadcast the host's music and notifications. -
It has no
<audio>element. Every other bus plays locally; this one must not, or the host hears every sound twice and their own microphone echoes back. -
The sentinel never goes through
resolveDeviceId.horrible:viewersis not a device id, so looking it up in the OS device list fails and returns''— "system default". A bus resolved that way connects toctx.destinationand builds noMediaStreamDestinationat all, which breaks the two rules above at once: guests get silence, and the host hears the viewer mix out of their own speakers.resolveTargetshort-circuits the sentinel for exactly this reason.
mixer.busStream(id) is how the share module reads it. Going through the mixer
rather than opening its own context is this module's central rule: a private
AudioContext is a module the mixer cannot see, and nothing on screen would
say so.
Note that a bus always carries exactly one audio track once its sink exists,
silent or not, so track-counting cannot answer "is anything being sent". That
question is answered from the routing document by busHasSource.