Ever since OpenAI launched their improved voice mode, I've been looking for a capable LLM with builtin voice-in and -out. Llama 4 was supposed to be it but turned out to be a dud. I haven't followed the topic closely lately, did I miss anything? Are there capable (!) open weights omni models that allow low latency voice chat? If so, what software do you use them with? Can you use a PWA on your phone? WebRTC? WebTransport?

Actually, we measured exactly this recently. The strongest open-weights speech-to-speech model we have run is NVIDIA's NemotronLabs VoiceChat 11B - no provider serves it, so we hosted it ourselves and ran the same scripted call every model on our board gets. Remarkably stable, 40+ sessions with zero errors - but by turn thirteen it was answering nine tries in ten without saying a word. Stable engine, but degrades on long calls.

Wider picture: we rebuilt the speech-to-speech test around a hard 20-turn concierge call, and today only the closed models finish it cleanly. Board: https://benchmarks.speko.ai/s2s

shared some write-up: https://benchmarks.speko.ai/blog/can-s2s-replace-the-cascade

And jakswa's Gemma datapoint matches what we measure, the fastest rows on our LLM board are the small models like Gemma 4 is performing incredibly well for voice agents

FWIW, I have seen a couple phone apps using react native and sending directly to backends using a socket. Seems to avoid some overhead but I don’t have first hand experience with the client side.

Gemma 4 (both E4B + 12B) performed really well as ears+brains. I mostly comment because I too am always scouting for a nice local all-in-one model.