Funnily enough - I built this (delegation) over the weekend with Fable for a local voice chat running 100% on local LLMs, Parakeet and Kokoro. I say "...ask the thinking model..." and that redirects it to Qwen 3.6 27B on vLLM.

Can't claim originality though - it was inspired by Sesame - where their models will invoke a search, or check the weather etc, and make a vocalisation to keep you engaged.

Turn taking is one of the hardest things to get right for the exact reasons mentioned - but does seem to be the way that Claude.ai's voice works - in a very obvious way.

Anthropic + OpenAI both rug-pulled voices I liked and got used to and OpenAI really dumbed down their voices at the same time - Arbor went from Estuary English and almost "jack the lad" to some generic English accent. Claude had a Birmingham accent and said things like "shit", ending sentences like "So you're telling me that they asked for a 90% discount yeah?" - then it changed overnight to a mock Derbyshire accent with a dull tone.

ChatGPT's voice also gaslights me for conventional opinions - "my Eastern European neighbour helped me lift a wardrobe upstairs - something you just can't ask your typical neighbour neighbour"... then you get a full on left-leaning lecture from the safety layers rather than a head nod or "what luck!"

Claude + Sesame are nowhere near as overbearing.

In both cases - from edgy and engaging to something that just didn't gel.

The point of making my own assistant is that I can talk for as long as I want, episodic memory is personal and private, there's no "trust me bro, we're a big corporation" vibes.

This was not my first attempt - when I had a bunch of Opus credit around Jan/Feb - I tried really hard and created something that was not good enough. What I have now, is working, and each session is training Claude/Codex on what to tune, and to fix.

"Just had a convo - can you look into what happened?" And if it's one I don't mind sharing with the model - I'll say, "and what did you think of the questions I asked?" Sometimes it'll give a lovely commentary on how the model did.

af_heart is probably the smoothest voice - but yes it's more like another commented - more "StarTrek" than "telesales assistant that pauses and laughs at your jokes".

If you're on a similar path and want something full duplex - the go to solution is PersonaPlex from Nvidia based upon Moshi.