> At this point I think human brains probably do something analogous to next token prediction when we think

That's reasonable, but it doesn't mean that LLMs are close to being brains.

For a start, when humans think/talk, we often think ABOUT something - whatever is swirling about in our mind, or what we are currently seeing/feeling/etc. An LLM generating tokens/words is doing so only based on it's weights and the word sequence it is currently generating ... the human parallel would be more like a rapper spitting out words based on prior words, essentially on auto-pilot, or when we get triggered into spitting out stock phrases like "have a nice day".

If you want to compare an LLM to a human brain, it's basically equivalent to our language cortex if you ripped out all the external connections and ripped out all the feedback paths that make it capable of learning.

Of course there is a lot more to our brain than just our language cortex, but that alone should make you realize there is no real comparison beyond the fact that our language generation is also going to be based on prediction, and partly auto-regressive.