Skip to main content

The share relay

The public half of the share module: a standalone service that takes one host's screen capture over WHIP and fans it out to anyone holding the link over WHEP. It is what turns a fabric-only session into a URL you can paste to somebody who has never heard of this app.

It is a separate deployable (backend/share_relay/, Fly app horrible-share) rather than part of a node's backend or a corner of the games server. Three reasons, in order of weight: it is the one component strangers talk to, it scales on a different axis from everything else, and a relay falling over must never be able to take a node — or the game server — down with it.

What a public viewer is​

A viewer holds a watch code and gets pixels and chat. Never a grant, never a cursor with authority, never a way to reach the host's dashboard. Interactivity requires an authenticated person on the fabric, which is exactly what keeps this service a dumb pipe rather than something security-critical: the worst thing a compromised relay can do is show video to the wrong people, or stop showing it to the right ones.

Two keys per stream​

Every stream is minted with two values, and they are never interchangeable:

Looks likeAddressesAuthority
Codek7m2x9qp — 8 Crockford base32 characters, ≈40 bits/<code>, /s/<code>, /whep/<code>, /chat/<code>Watch and chat
Token43 URL-safe characters, 256 bits/whip/<token> (and the key-gated /streams, /restream)Publish video

The link a host hands out is the short one, https://horrible-share.fly.dev/k7m2x9qp, served in place (no redirect) so the URL a viewer sees stays the one they were given. The viewer page carries the code and never the token.

Why two. Until this split a stream had one token, used in both the viewer URL (/s/<token>) and the ingest URL (/whip/<token>). WHIP is gated on nothing but that token, and Room.publish closes whoever is publishing and takes over — so anyone holding a public link could POST to /whip/<token> and replace the host's screen with their own video. Registry.get (token) and Registry.get_by_code (code) now refuse each other's value, and test_a_viewer_cannot_publish_with_the_code_from_their_link pins it over HTTP.

Why 40 bits is enough for the code. It would be thin for a permanent secret. It is not thin for a value that dies within hours, can carry a passphrase, and sits behind a miss limiter (ratelimit.py): an address that names 20 well-formed codes that do not exist — or supplies 20 wrong passphrases — in a minute is answered 429 for the rest of that minute, on every code-addressed route, including the real code, since a limit a guesser walks past once they land on a hit is a filter, not a limit. Three rules that are quiet if wrong:

  • Only well-formed codes count. A browser requests /favicon.ico beside every page; charging it as a guess would lock out a viewer who reloaded twenty times.
  • Fly-Client-IP is trusted only on Fly (FLY_APP_NAME is set). Fly's proxy overwrites it; anywhere else it is a header the caller wrote, and trusting it would hand a guesser a fresh "address" per request.
  • A wrong passphrase is a miss too. It is a guess, and each check is a PBKDF2 run — an unlimited stream of them is also a cheap way to spend the relay's CPU.

A code is normalised the way people type it: case-insensitive, hyphens ignored, and Crockford's look-alikes folded (i/l → 1, o → 0).

The token​

The token is publish authority, and the rules below apply to both values:

  • Opaque and random, not signed. A JWT would let the relay verify a link with no state at all, which is tempting for a service meant to scale out — but a signed token cannot be revoked, and immediate revocation is a requirement. Someone who stops a share expects the link to die at that instant, not at expiry. The registry is the authority; the token is a lookup key with no meaning of its own.
  • Unknown, revoked and expired are one answer. Registry.get returns None for all three and every route reports 404. Telling a caller "that existed but expired" confirms the token was real, which is one bit more than a stranger holding a guessed URL should learn.
  • Passphrases are stored hashed (PBKDF2, per-stream salt) and compared with compare_digest, even though the registry is in-memory and short-lived. A process holding other people's live video is not where plaintext secrets should sit waiting for a crash dump.
  • TTL is clamped at both ends — a node asking for a year gets a day.

The registry is in memory and never persisted. Persisting it would make a relay restart resurrect links for sessions that ended while it was down; the correct behaviour for a lost relay is that every link dies with it. The host still has the fabric path, and re-minting is one click.

Who may do what​

ActionCredentialWhy
Mint a streamSHARE_RELAY_KEY headerOtherwise the relay is free video hosting for anyone who can reach it
Publish (WHIP)The tokenThe key must never reach the host's browser; the token already names the stream
Watch (page, WHEP, chat)The code (+ passphrase)This is the public path — meant to be openable by a stranger, and never able to publish
Guess a code—20 misses per address per minute, then 429
RevokeSHARE_RELAY_KEY headerThe same authority that minted it

SHARE_RELAY_KEY is env-only, never a setting: GET /api/settings hands the whole bag to the browser. The relay URL is a setting (share.relayUrl) because it is public by construction — it appears verbatim in every link.

If the key is unset the relay logs a warning at boot and /health reports gated: false, so an operator can see at a glance that they left it open.

The ingest URL is publish authority​

ShareSession.link (the short view URL) is broadcast to every guest. The ingest URL is not, and must never be — anyone holding it can push their own video into the host's stream. It lives on the node's session object and is served only from GET /api/share/link, a route no guest can reach. That protection was hollow for as long as the view URL carried the same token; see two keys per stream.

A node talking to a relay that predates short codes still works: link.mint reads short_url with .get and falls back to view_url, because the node and the relay deploy separately and a KeyError there would break minting outright.

test_share_link.py asserts the string whip never appears in the serialized session model, because that model is exactly what _tell_guests sends over the fabric.

CORS is load-bearing​

Every caller is cross-origin by construction: the host's browser is on its own node's origin — a different one for every user, often a bare LAN IP — and the viewer page is the only thing ever served from the relay itself. There is no allowlist that could be written, so the answer is *, with allow_credentials=False.

That is not a hole, because the origin was never the credential. CORS protects a browser from a page spending its ambient authority (cookies, sessions) against another site; this relay has no ambient authority to borrow.

Getting it wrong fails asymmetrically, which is why it has its own tests: without the headers the host's WHIP POST is refused by the browser and the public link silently carries no video, with nothing but a console message to say so. An ASGI test client does not enforce CORS, so every route test passed while a real browser was blocked — this was found by driving an actual share, not by the suite.

Scaling, and the ceiling​

Run one machine. Like the games server, though for a different reason: the games server is a single authority, while this one simply keeps each stream's registry entry and live tracks in one machine's memory. Fly load-balances across machines, so with more than one a mint lands on machine A and a viewer's WHEP lands on machine B, which has never heard of that token.

That is not hypothetical — it was observed on the first real deploy. Four machines were running, the node reported the stream live, and /health answered streams: 0 because the health check hit a third machine. max_machines_running = 1 in fly.share.toml is what prevents it; fly scale count 1 cleans up machines that already exist.

Scaling out is possible but not implemented. It needs genuine session affinity: the app must return a fly-replay: instance=<machine> response header to bounce a request to the machine that owns the token, which in turn means the token has to record which machine minted it. An earlier version of this file claimed fly-replay-src did that job. It does not — that is the header Fly adds to a request it has already replayed, inbound and informational. It was emitted on every WHIP/WHEP response, looked like affinity, and provided none.

Fan-out uses MediaRelay, so the host uploads one copy of the stream no matter how many people watch. That is the bandwidth win and the reason a public link scales further than the fabric mesh.

It is not a passthrough. aiortc decodes every incoming stream in a decoder_worker thread, and left to itself every outgoing sender holds its own Encoder and re-encodes -- one decode plus N encodes. That was the relay's capacity, and it was small: measured on two cores, fifteen viewers of a 1080p share got 3 fps each out of the host's 15, the CPU pinned, and the starved event loop stalled ICE and DTLS for anyone still connecting. That is what "laggy" and "stuck connecting" were.

So video and audio are each encoded once per room and every viewer is handed the same packets (encoder.py); see Encode once, packetize per viewer.

There are two ceilings and they bite differently. Memory is what crashed the relay; CPU is what made the stream bad. The host captures the desktop at its native resolution on purpose (startCapture sends no width/height; scaled text is an unreadable share). On a 512 MB machine that was an OOM kill 16 seconds after the second viewer connected; the machine is now a 4 GB performance-2x, and the memory mechanism turned out to be queue discipline, not size (next section).

Memory is a queue-discipline bug, not a sizing problem​

The 4 GB machine was OOM-killed too, 2m21s into a single share. That kill is what established the real mechanism, and it is worth stating precisely because every instinct here points at the wrong lever: it was not the viewer count, and raising the memory only moves the deadline.

Every queue on aiortc's receive path is unbounded, and nothing anywhere drops a frame. decoder_worker decodes each inbound frame and put_nowaits it onto RemoteStreamTrack._queue whether or not anyone is reading it, and MediaRelay.subscribe() defaults to buffered=True, giving each consumer its own unbounded queue filled the same way. A decoded frame is freed only when somebody pulls it, and the host is never told to slow down. A 1080p frame is ~3 MB of YUV420; at the capture's 15 fps that is ~45 MB/s of accumulation from any consumer that is slow — or absent.

Absent is the sharper half, because it needs no viewers at all. MediaRelay does not read a source until some proxy of it calls recv(), so a host who mints a link and starts sharing before anyone opens it was not idling. The relay was filling memory at line rate while reporting a perfectly healthy stream.

So fanout.py holds two rules, both pinned by backend/tests/test_share_relay_backpressure.py against real aiortc:

  1. The ingest is pumped from the moment it is published (Room._start_pump), not from the moment somebody watches. The pump discards what it reads; the point is the side effect of keeping MediaRelay's reader running.
  2. Decoded video is never queued. The pump writes each frame into the room's FrameHolder (one slot) and the room's single encoder takes the newest one, so a slow encode skips frames instead of growing a backlog. The audio pump's proxy is buffered (_buffered_for), which is safe only because the pump never stops draining it.

For live video, skipping is the correct semantics rather than a compromise: a frame that arrives late is worthless, and the alternative is latency that climbs until the process dies. Skipping decoded frames before the encoder is always safe. Skipping encoded frames after it is not -- see the keyframe rules below.

The test file asserts the failure mode as well as the fix. If aiortc ever bounds its own queues, the control case goes red, and that is a signal to revisit the approach rather than a test to repair.

CPU is the quality ceiling. The decode and the encode are software. A shared vCPU is throttled to a fraction of a core under sustained load, and a continuous encode is exactly that — the burst credits that make shared CPUs fine for HTTP buy nothing here. shared-cpu-1x was visibly laggy with a single viewer. The relay runs on performance-2x.

An OOM is worse than it sounds, because the registry is process memory: the kill takes every token with it. The link goes dead for everyone holding it, and the host's browser notices nothing at all — WebRTC to a departed peer does not raise. That is why the node polls GET /streams/{token} while a share is running.

Encode once, packetize per viewer​

RoomEncoder takes the room's newest frame, encodes it once with its own libvpx context on its own thread, and pushes the av.Packet to every viewer's PacketTrack. aiortc's sender, handed a packet instead of a frame, takes its passthrough path (encoder.pack()), which only packetizes. RoomAudio does the same for Opus. The cost is one decode plus one video and one audio encode per room, plus a cheap packetize per viewer.

Audio had to be shared too, and that was found by measuring rather than guessed: with video shared, 25 viewers ran at full frame rate on two cores -- until the publisher carried audio. Each viewer's sender then Opus-encoded the same samples in the loop's default executor (about 1,250 jobs a second), the one video encode queued behind them (median 15 ms to 83 ms), and every viewer fell to 2 fps. The video encode now also has its own thread, so nothing else can queue in front of it.

Sharing encoded packets imposes rules that sharing decoded frames did not:

  • A viewer starts on a keyframe. A PacketTrack queues nothing until its sender first pulls (so a viewer mid-handshake never starts on a stale backlog), then asks for a keyframe and discards packets until one arrives.
  • A viewer never skips a single encoded frame. Dropping a delta frame breaks every frame after it. A viewer that falls behind (VIEWER_QUEUE_MAX) drops its whole backlog and resumes at the next keyframe, which it requests.
  • PLI goes to the room. aiortc ignores its own force-keyframe flag on the passthrough path, so Room.subscribe points each sender's _send_keyframe at the room encoder. Requests coalesce to one keyframe per MIN_KEYFRAME_GAP_S, and one is forced every KEYFRAME_INTERVAL_S regardless, because RTCP is lossy too.
  • Codecs are pinned. Each viewer is negotiated onto VP8 and Opus only. A viewer that preferred H.264 would otherwise get VP8 bytes split as NAL units: connected, and garbage.
  • One bitrate for everyone (SHARE_RELAY_VIDEO_KBPS, default 3000, CBR). A viewer's REMB retunes nothing. aiortc's own Vp8Encoder clamped to 1.5 Mbps -- unreadable for a desktop -- and rebuilt its codec, keyframe included, on every 10% swing of each viewer's estimate. Frames taller than SHARE_RELAY_MAX_HEIGHT (1080) are scaled down once before the encode.

The host's upload needed the same care. aiortc estimates the ingest bandwidth by REMB alone, tracking about 1.5x whatever is currently arriving, so a still desktop pinned the host's Chrome at 53 kbps (qualityLimitationReason: bandwidth, resolution halved) and the next scroll arrived as a smear for every viewer. The WHIP answer now carries x-google-min-bitrate and x-google-start-bitrate on its video codecs (ingest_bitrate_hints; floor SHARE_RELAY_INGEST_MIN_KBPS, default 1000). Measured after: 1280x720 at about 2 Mbps, limitation none.

Measured capacity​

scripts/share_relay_loadtest.py publishes a synthetic 1080p desktop (scrolling text plus a moving bar, with audio) and opens N real WHEP viewers in separate processes, then reports each viewer's frame rate and the relay's CPU and memory. --pin-cores 2 restricts a local relay to the two cores the Fly machine has and keeps the harness off them.

Relay on 2 cores, 1080p at 15 fpsViewer fpsRelay CPU (of 200%)RSS
Per-viewer encoders, 15 viewers3.0177%1.5 GB
Shared encode, 15 viewers15.064%329 MB
Shared video, per-viewer audio, 25 viewers1.980%542 MB
Shared video and audio, 25 viewers15.076%284 MB

The ceiling is SHARE_RELAY_MAX_VIEWERS, set in fly.share.toml next to the machine it was sized for, now 20: inside the 25 measured above, with room for the real network. It is per stream, not per machine. Run the load test against the deployed relay after any change to this path (mint with --key), and watch /health -> load: an encode_ms_p50 creeping toward the frame interval (66 ms at 15 fps) is the early warning, and viewer frame rate falls after it.

A seat is given back when its viewer leaves​

The seat count used to be written only when a viewer joined, and the capacity check read that number -- so viewers who left never gave a seat back, and a stream that once filled turned everyone away after it, reconnects included. The count is now read from the room at every check. The WHEP answer also carries a resource URL (Location: /whep/<code>/<viewer_id>), and the viewer page DELETEs it on pagehide and before every redial (keepalive, so it outlives the page), so a reload does not hold two seats until ICE consent expires. A viewer stuck disconnected for VIEWER_DISCONNECT_GRACE_S (10 s) is reclaimed too.

Viewers need TURN too, and that costs something​

STUN alone gets a viewer a server-reflexive candidate, which is enough on most home networks and not enough behind a symmetric or carrier-grade NAT. The failure is asymmetric and therefore confusing: the host, on a friendly network, watches their own link perfectly while some of their audience sits on a black rectangle. That is how this was found.

So SHARE_RELAY_TURN_FOR_VIEWERS = '1' is set in fly.share.toml. The trade is real and deliberate rather than an oversight: the viewer page is public, so a credential on it is public, and the operator's TURN bandwidth becomes usable by anyone holding a link. It lives in [env] and not in fly secrets because it is a flag, not the credential — SHARE_RELAY_TURN_URL/USER/PASS stay secrets. Rotate them if the exposure ever stops being acceptable.

Cloudflare TURN removes that trade, and is the preferred source. With CLOUDFLARE_TURN_KEY_ID and CLOUDFLARE_TURN_API_TOKEN set (as fly secrets), ice.refresh() mints a credential with a 24-hour lifetime, caches it for the whole process and re-mints when less than 12 hours remain. A leaked one expires, so it goes to viewers without the opt-in. Viewers fetch a fresh set from GET /whep/<code>/ice before every attempt (passphrase-gated and charged like WHEP), and the host fetches one from GET /whip/<token>/ice before publishing, since the host's own node settings rarely carry TURN and the host is half the path. Port-53 URLs are dropped for browsers, which block that port. aiortc uses only the first TURN URL it understands, so the relay chooses plain UDP for itself. A failed mint falls back to the static server, then to STUN only, and is retried after a minute rather than once per connection.

Before this, the deployed relay had no TURN at all (/health said "turn": false). Fly routes no inbound UDP to aiortc's ephemeral ports, so both legs relied on hole-punching and worked or not depending on each end's NAT.

A viewer that still cannot form a path now says so rather than showing black — see below.

The viewer page is one viewport, never two​

body is pinned with height: 100vh and height: 100dvh (the second for mobile browsers, so it measures the visible viewport rather than the one hiding behind the URL bar) with overflow: hidden. Everything that can grow — the chat log, the diagnostics panel — scrolls inside its own column.

It used to be min-height: 100vh, which is the same thing right up until a child gets tall, and then the page grows and the stage slides off the bottom. A shared screen you have to scroll to is a shared screen you cannot watch. The matching half is min-height: 0 on main, .stage and aside: a flex item defaults to min-height: auto and refuses to shrink below its content, which is the actual mechanism by which a long conversation breaks a fixed layout.

Progress, not a spinner​

A viewer waiting on a connection is shown which of six steps is in progress — offer, gather, relay, negotiate, path, live — as a bar and a label. A spinner says "alive"; this says which thing is being waited on, which is also the first question worth asking about a viewer that never starts.

Waiting on the host is deliberately not drawn as progress. The host may never start sharing, so a bar creeping toward 100% would be a lie; that state sweeps instead, and when it is polling it counts the retry down out loud (retrying in 3s), which is what keeps the page from looking abandoned.

The relay pushes stream state​

The chat socket is the one channel a viewer page holds that the relay can push on, so it also carries {"kind": "stream", "state": ...}:

statesent whenthe page
livea WHIP publish succeedsconnects now, superseding any wait or poll
pauseda room closes and the link is still validdrops its connection, shows Paused, waits
endedthe link is revoked or expiredstops every retry; the chat closes with 4404

Before this a viewer learnt the host had come back only by polling WHEP, and that they had left only when ICE consent ran out about thirty seconds later. With the socket up, waiting for the host polls every 20s as a backstop rather than every 4s; without it (socket down, reconnecting with backoff) it polls as before.

paused comes from the Rooms close hook, not from the WHIP DELETE route, because a room closes for more reasons than anyone asks it to — a host whose network dropped closes it from inside aiortc's state handler. The same hook forgets the room: one that closed itself used to stay in the table with _closing latched, so the host's next publish landed in a room whose close() was a no-op forever.

The page recovers by itself​

Somebody watching a screen share is not looking at the controls, so nothing a network can fix waits for a click:

  • A dropped connection redials with backoff (2s, 4s, 8s, then every 15s), reset by the next decoded frame. disconnected gets a 4s grace first, since ICE often repairs it alone. An unreachable relay — a blip, or a machine waking from scale-to-zero — is treated the same way.
  • A full stream checks for a seat every 15s instead of stopping.
  • Blocked autoplay plays muted with a labelled Unmute button on the video, instead of holding the picture behind a click-to-play overlay. Muting used to be rejected here because it was silent; a visible Unmute is not. Only if muted autoplay is refused too does the page fall back to asking.

A fullscreen button (and double-click) sits beside it. The page's <title> and og:title are the stream's own title, so the link's preview in a chat app says what it is rather than showing a bare hostname.

The page carries its own diagnostics​

A Diagnostics panel logs every phase, ICE state change, candidate type, track arrival, autoplay block and connection transition, timestamped, with a Copy button — plus a getStats sample every 10s once live: resolution, frame rate, frames dropped, freezes, packets lost, jitter, jitter-buffer delay, round trip, and which candidate pair won.

This exists because diagnosing this service meant reading the relay's Fly logs, and the relay can only see requests. It cannot see ICE failing, a track arriving, or autoplay being blocked — which is where every real failure has actually been. A viewer who says "it is black" can now copy the whole story.

Two constraints on it: the buffer is bounded (people watch for hours), and candidates are logged as type/protocol only — a raw candidate line carries the viewer's own IP addresses, and this panel is meant to be handed to someone else.

Closing a tab releases the slot​

The page closes its peer connection on pagehide. Without that the relay holds the viewer until ICE consent expires about 30 seconds later, and a viewer slot is a full-resolution encoder on a machine whose first ceiling is memory.

One connect attempt at a time​

connect() awaits five times — offer, ICE gathering, the WHEP fetch, the body, setRemoteDescription — and every one is a window in which the page can call it again: the Retry button, an unlock, the 409 timer.

It used to read and write the module-level pc throughout. So a second attempt replaced pc while the first was still in its fetch, the first then applied its answer to the second attempt's connection, and the second's own setRemoteDescription found the thing already stable:

InvalidStateError: Failed to set remote answer sdp: Called in wrong state: stable

The attempt died there, as an unhandled promise rejection, and the viewer got a black rectangle.

Each attempt now takes a generation and its own self reference. The shared pc exists only so the next attempt can close the previous one; every other line uses self, every await is followed by a staleness check, and the event handlers return early when superseded — a dead connection must not overwrite the live one's status. setRemoteDescription is wrapped, because a rejection there is invisible except as black video.

backend/tests/test_share_relay_viewer.py pins these structurally: this page is JavaScript inside a Python string, with no compiler, linter or type checker between writing it and a stranger watching through it, so node --check is the only build step it will ever get.

A viewer says 'live' only when a frame has decoded​

The page used to call itself live the moment the relay's SDP answer arrived. An answer means the relay agreed to send; it says nothing about whether a path between the two ends exists. A viewer whose ICE never completed therefore got a hidden overlay, a green live chip and a black rectangle — indistinguishable from a host who really was sharing a black screen, and the single most confusing state this page can produce.

Moving live onto connectionstatechange fixed half of that and left the other half, because connected is not a picture either. It means ICE and DTLS completed — that the two ends can talk. The relay answers a WHEP offer for a room whose track has stopped yielding frames exactly as happily as for a live one, so a host who stopped sharing left viewers on the same black rectangle under the same green chip, with the overlay that would have explained it hidden by an event that had proved nothing about the video.

live therefore comes from the first decoded frame and nothing else: requestVideoFrameCallback where it exists, loadeddata everywhere else. Both, because neither fires for a track that carries no media — which is the whole point. Until one does, the chip reads connected and the phase says "waiting for video".

The 12s deadline is guarded on that same frame rather than on the connection state, and reports which of the two failures happened: "could not reach the stream" when no path formed, "connected, but no video" when one did and nothing came down it. Guarding it on connectionState === 'connected' is what let the silent case run out the clock unreported — the one watchdog that could have named it disarmed itself on the evidence of the thing that was not wrong. The deadline is needed at all because ICE can sit in checking for a long time and may settle on disconnected rather than failed, in which case the failure handler never runs.

The viewer page retries, but only once at a time​

A viewer arriving before the host starts gets 409, not 404 — the link is real, the picture is not there yet — and the page retries every 4s.

That retry holds its timer in retryTimer and every entry into connect() cancels it. Without that, each path into connect() (the initial call, the Retry button, an unlock) forked its own loop that nothing ever stopped, and the loops compounded: observed at roughly one WHEP per second against a relay that was being asked to wait.

The noise is the smaller half. Each loop that eventually succeeds becomes a real viewer session, and a viewer session is a full-resolution encoder — so an uncapped retry storm is a memory leak on the relay wearing a network-noise costume, on a machine whose first ceiling is memory. A single page was seen taking two WHEP sessions this way minutes before the relay was OOM-killed.

Restreaming to RTMP​

A live room can also be pushed to Twitch, YouTube Live, or any RTMP server. The frames are already decoded (see above), so the relay pays nothing extra to feed them out — the expensive half is the H.264 encode, and that happens in an ffmpeg subprocess, not in the relay process.

That split is the whole design decision. PyAV is already a dependency, so encoding in-process would need nothing new — and would be wrong twice over: encoding is the expensive thing and this process is latency-critical, so a 1080p encode on the event loop makes every viewer stutter to serve a broadcast nobody in the room can see; and a transcode is the component most likely to wedge or crash, which in a subprocess is an exit code and in-process is a dead relay and 25 dropped viewers.

Three things about the ffmpeg invocation are quiet if wrong, which is why build_args is pure and tested:

  • Raw video on a pipe has no header. -s, -pix_fmt and -r must all be correct, and a wrong one does not fail — it produces a skewed, miscoloured or wrong-speed picture that reads as a broken encoder rather than a bad argument.
  • -tune zerolatency. Without it x264 buffers frames looking ahead and the broadcast lags the room by seconds — invisible in a test, obvious on a stream.
  • Forced keyframes every 2s. Every platform requires them, and without them a viewer joining mid-stream waits for the next natural keyframe, which on a static screen share can be a very long time.

A resolution change mid-stream (a host resizing a shared window) scales to the size the stream started at rather than restarting the encoder: RTMP cannot renegotiate, and a broadcast that drops for four seconds every time somebody drags a window edge is worse than one that is very slightly soft.

Stopping closes ffmpeg's stdin and waits, rather than killing it — EOF lets it finish the current GOP and close the RTMP session properly, where a kill leaves a corrupt tail on whatever recording the platform kept.

The stream key​

An RTMP stream key is a password that does not expire, so:

  • It lives in the streaming connector (Fernet-encrypted in secrets.db), never in a setting — every setting is served to the browser.
  • It is resolved on the node and travels node → relay. The browser never sees it; the pane gets a label ("Twitch"), never the target URL.
  • POST /restream/{token} is gated on SHARE_RELAY_KEY, not on the share token — unlike WHIP. The token is the credential for publishing to this relay; starting an outbound broadcast on someone's behalf is an operator action, and the body carries a key.
  • Every log line goes through streaming.redact, which keeps the host and app (diagnostic) and drops the last segment (the key). A key in a log is a key in a bug report, a screenshot, and a pasted stack trace.

Revoking a link or stopping a stream stops the restream too. Otherwise the link dies and the broadcast carries on to Twitch, which is the worst possible reading of "stop".

What is not built​

The HLS branch is not implemented. It is the honest answer to the SFU ceiling for large audiences, and ffmpeg would produce the segments happily — but serving them needs a player, and Chrome does not play HLS natively. Keeping the viewer page's no-CDN promise would mean vendoring hls.js into this repo, which is a real decision rather than an afternoon's work. The RTMP path covers "broadcast to an audience" today by handing that problem to a platform that has already solved it.

TikTok is out of reach, not merely unimplemented: it has no generally available live-ingest API — RTMP endpoints are issued per-account through a partner programme — so no amount of effort here would produce one.

ICE, and the failure with no symptom​

The relay's own peer connections need STUN. Without it aiortc gathers host candidates only, which on a hosting platform are private addresses — so a browser on the public internet has nothing dialable and the connection never completes. Nothing reports this: WHIP returns a valid SDP answer, /streams says the stream is live, the viewer page sits on "connecting", and no log line anywhere mentions ICE. It works perfectly on one machine, where a host candidate is exactly the right answer, which is precisely what makes it easy to ship.

Measured, before and after configuring it:

Candidate types the relay offers
No ICE servershost
SHARE_RELAY_STUN set (the default)host, srflx

Configuration is env-only, like everything else here:

VariableDefaultMeaning
SHARE_RELAY_STUNstun.l.google.com:19302host:port; the scheme is added. Empty disables STUN.
SHARE_RELAY_TURN_URL—Full URL (turn:host:3478)
SHARE_RELAY_TURN_USER / _PASS—Both required; an incomplete TURN entry is dropped rather than offered, because a malformed entry can take the whole configuration down with it
SHARE_RELAY_TURN_FOR_VIEWERSoff (on in fly.share.toml)Whether the viewer page also gets the static TURN credential
CLOUDFLARE_TURN_KEY_ID / _API_TOKEN—Mint expiring TURN credentials from Cloudflare; preferred over the static server and always given to viewers

The semantics match buildIceConfig on the frontend and _ice_servers in network/transport/webrtc.py — STUN is a bare host:port, TURN is a full URL. Three readers of one idea that disagreed about whether the scheme is included would fail as "ICE just does not connect".

TURN is off for viewers by default and that is a real trade, not an oversight: a TURN credential in a page anyone can open makes the operator's relay bandwidth free for the whole internet, while a viewer behind a symmetric NAT cannot connect without one. The decision belongs to whoever runs the relay.

/health reports whether STUN and TURN are configured, and turn_source (cloudflare, static or null). It reports presence only, never the credential, so "is ICE set up" can be answered without reproducing the failure.

A still screen must still reach the viewer​

Chrome sends a desktop that isn't changing at close to zero frames per second. Three things in the media path assumed a steady flow of frames, and each one turned a still screen into a black viewer:

  • aiortc's jitter buffer releases video frame N only when frame N+1 starts arriving, and ignores the RTP marker bit. The newest frame of a still screen stayed undecoded in the relay. jitter.py completes a frame on its marked packet; that is installed by fanout._aiortc() before any receiver exists.
  • Per-viewer MediaRelay proxies feed an encoder only on a new frame. A late joiner got nothing, and a lost keyframe was never resent, because aiortc's PLI response only flags the next encode. frames.py holds each room's newest frame. A new viewer starts on it, and repeat_while_idle re-publishes a copy about once a second while someone is watching. Timestamps come from the room's own clock, so each viewer's sequence only ever increases.
  • "Live" was claimed on the WHIP answer. A room is now live only once its first frame has decoded. /whep answers 409 "the host is connecting" until then, live is pushed to viewers at that moment, and /streams/<token> adds publishing, receiving and last_frame_age_s. The host's node maps "publishing but not receiving" to a connecting state, and its WHIP watchdog (relay-status.ts judgePublish, whip.ts onLost) republishes with backoff when the leg stays dead.

test_share_relay_frames.py and test_share_relay_jitter.py each keep a control case asserting aiortc's own behaviour, so a library change that makes either workaround unnecessary shows up as a failure rather than going unnoticed.

There is no UDP service block​

An earlier version of fly.share.toml declared protocol = 'udp' on port 9100. That was wrong twice: 9100 is the HTTP port, and aiortc allocates ephemeral high UDP ports for RTP, so no fixed port declaration corresponds to where media actually lands. What makes media work here is ICE, not a port mapping.

Peer connections are closed under a timeout​

RTCPeerConnection.close() waits on the DTLS transport, and a peer that never finished its handshake — a viewer who closed the tab mid-negotiation, a host whose network dropped during the offer — never completes that wait. Unbounded, tearing such a room down hangs the request that asked for it, so revoking a link would hang forever exactly when somebody urgently wanted it revoked.

CLOSE_TIMEOUT_S bounds every close, viewers are closed concurrently rather than in series, and Room.close guards against re-entry — closing a peer connection fires its own connectionstatechange, whose handler closes the room, so a plain teardown otherwise re-enters itself and pays every timeout twice.

Deploying​

fly deploy -c fly.share.toml
fly secrets set SHARE_RELAY_KEY=... -a horrible-share

fly.share.toml lives at the repo root for the same reason fly.toml does: flyctl resolves the build context relative to the config file, and the image needs pyproject.toml, uv.lock and backend/.

Two things in that config are easy to get wrong:

  • UDP must be exposed. WebRTC carries RTP over UDP; a relay reachable only over TCP/443 degrades to TURN-over-TCP at best and simply fails at worst — the kind of thing that works perfectly in a local test and never in production.
  • One worker per machine. The registry and the rooms are process memory, so a second worker would answer WHEP for streams it has never seen. Scale by adding machines, not workers.

Scale-to-zero is safe and wanted: a relay with no live stream is doing nothing, and the cold start costs one second on the first mint. The media path itself never cold-starts — WHIP only happens after a successful mint.

Running it locally​

uv sync --extra webrtc
uv run uvicorn backend.share_relay.app:app --port 9100

Then point a node at it by setting share.relayUrl to http://localhost:9100. aiortc is an optional extra on a node (nothing but the relay needs it) and required in the relay's own image.