Shipping cross-browser voice AI chat

Voice makes AI products feel magical—until Safari disagrees with your assumptions. On Vivi we shipped cross-browser text and voice AI chat using the Web Audio API and WebSockets. The hard parts weren’t the model calls; they were the browser edges.

Treat audio as a product surface

Mic permissions, interrupted tracks, background tabs, and Bluetooth headsets all show up as “the AI is broken” to users. We designed the chat UI to make state obvious:

Connecting / listening / processing / speaking
Explicit mute and cancel
Clear recovery when the socket drops mid-turn

WebSockets are the easy half

A durable socket for tokens and control messages is straightforward compared to audio. We kept message types boring and versioned so the client could ignore unknown events instead of crashing the session.

Web Audio gotchas we hit

Autoplay and permission policies differ enough that “works on my Chrome” is not a test plan
You need a deliberate strategy for buffering and barge-in (user interrupts the assistant)
Always have a text fallback path—voice is enhancement, not the only door in

UX principles that survived QA

Never leave the user unsure whether the assistant heard them
Prefer short, interruptible speech turns over monologues
Log enough client telemetry to debug “it failed once on iOS” without a repro video

If you’re adding voice to an existing chat product, ship the state machine and fallbacks first. The waveform animation can wait.