This latency discussion is incredibly relevant to real-time voice AI applications. When you're building a voice agent that needs to respond conversationally (not just generate text), the inference speed directly determines whether the interaction feels natural or robotic.

In practice, humans perceive conversational pauses >800ms as awkward. So for a voice pipeline (STT → LLM inference → TTS), you have maybe 400-500ms budget for the LLM portion. At typical Sonnet speeds (~80 tok/s), you get ~35 tokens in that window — barely enough for a sentence. At Cerebras/Groq speeds (1000+ tok/s), you get 400+ tokens, which changes what's architecturally possible.

This is why the small-model vs. big-model tradeoff matters so much for real-time applications. We've found that a well-tuned smaller model with domain-specific context can outperform a larger model for constrained tasks (like navigating a user through a website or answering product questions), while staying within the latency budget. The "council" approach — multiple specialized small agents instead of one large general agent — lets you get both speed and quality.

The speculative decoding point is underrated here. For voice AI specifically, you can predict likely response patterns (greetings, confirmations, common Q&A) and pre-generate TTS for those, then only hit the full inference pipeline for novel queries. Gets you sub-200ms for ~60% of interactions.