This is the 3rd or so time I've posted this -

Run a tiny shitty llm, one that is super fast. Or better yet try out a decision model now days.

Feed it the previous conversation turn and all the STT (ASR output) you have so far and just ask it for a probability that the user is done speaking.

Use that probability to shorten your VAD time.

It isn't as good as a true speech to speech model but this will cut a few hundred ms compared to pure VAD. Oh as a bonus it also lets you wait longer when the user isn't done talking yet.

▲

julius a day ago | parent | next [-]

Thanks I had not heard of that idea yet. Especially like it for the waiting longer part when I am not done talking.

▲

nmstoker a day ago | parent [-]

It's quite cool. Look into Semantic End-of-Turn Detection, also called Semantic Turn Detection or Semantic VAD.

	▲	com2kid 18 hours ago \| parent [-]
		Oh it has a name, cool! I first did this around 2023 because I needed to make things fast and I was out of other options!

▲

antman a day ago | parent | prev [-]

I had tried language models but seemed slower, which model responded faster in your app?

	▲	com2kid 13 hours ago \| parent [-]
		Whatever is cheap and has less than 20b parameters. It really doesn't matter, asking "is this a complete reply to the previous statement" is hardly an earth shatteringly difficult question for any LLM made in the last 3 years.