Or know when to shut up.

A tangent, but can anyone ELI5 how models "know" when to stop generating tokens? Or what the method to stop them at the right point is?

▲

bulder 4 hours ago | parent | next [-]

The model doesn't "know" how to generate tokens any more than it knows how to stop generating tokens. The sampler simply stops pulling values when it outputs a "stop token", which is a token the same way every other token is.

That is to say, it stops when it's statistically the most likely to.

	▲	phatfish 3 hours ago \| parent [-]
		OK thanks, so the neural net (that no one can explain fully) generates a "stop" signal at a certain point.

▲

bigfudge 4 hours ago | parent | prev [-]

I might be out of date but my understanding was that STOP was just another token that gets predicted.