Seems like they are closer to scratch than reasoning... Generating some scratch to draw from helps make it easier to compute the real answer.

▲

forgotTheLast 3 hours ago | parent | next [-]

That's my personal theory too. The model is stuffing its own context with vaguely related tokens, which helps the attention heads retrieve the right tokens.

	▲	clhodapp an hour ago \| parent [-]
		Yup. You basically just need something for probability to push off of

▲

cyanydeez a day ago | parent | prev [-]

I assume theyre searching the local gradient to see if theres a better descent before proceeding.

	▲	eigenspace 10 hours ago \| parent \| next [-]
		LLMs dont do gradient descent to generate tokens. They are trained by gradient descent, but inference doesnt involve it.
	▲	c0_0p_ 14 hours ago \| parent \| prev [-]
		I don't think there's anything like that going on. They just word vomit into a secondary area, and then there is an internal prompt that says "clean this up and summarize for the user".