I assume theyre searching the local gradient to see if theres a better descent before proceeding.

LLMs dont do gradient descent to generate tokens.

They are trained by gradient descent, but inference doesnt involve it.

I don't think there's anything like that going on. They just word vomit into a secondary area, and then there is an internal prompt that says "clean this up and summarize for the user".