I see them collapse when faced unusual problems. They’ll go down a weird rabbit hole and then just keep going, even if their idea is completely irrational and not working.

They’re incapable of changing their mind, because they’re a Markov chain. They’re cool and they’re good at doing things, but they’re not sentient in the slightest.

Not at all the same idea. People also rathole into doomed approaches. What we're talking about here is literally LLMs producing line noise, or English words that don't fit into sentences.

(I don't think LLMs are "sentient".)

I've seen this reflex quite a few times now. To diminish the impressive capabilities of the latest LLMs, one resorts to the assertion that they're not sentient, or not truely intelligent, or various other semantic diversions. No one claimed that. GP was all about usefulness.