You can also avoid the problem all together. I made a "speculative regeneration" at least thats what i call it pipeline i been using for many months now and it works great. basically as long as you have access to the inference server you are good to go. idea is as such, every n milliseconds of silence (usually 300ms) for me, you send the asr transcript for the llm to process immediately. you let the model start answering , but if the user starts talking again before the end of turn token is reached you cancel the generation and again wait for the 300ms. rinse repeat. there is a safety window you have to set up as well that way the final llm answer gets finalized regardless of its inference speeds. works like a charm. basically the user gets 300ms latency at the highest for that "silence" window. so the only other thing you have to worry about as faras latency is llm inferance speeds and asr and tts latencies. in my pipeline the moment the user stops talking, the model ,is already responding within about 900ms - 1.2 sec.. its my gemma 26b moe model , 4 bit quant thinking on... with mtp getting about 300 tokens per second... asr and tts are both under 350ms for total... anyways, speeds could be improved if turning off thinking but i like my voice agent to be smart as he has tool calling and common sence capability is reduced when thinking is off.... fun times!