I have the opposite reaction, when it is confident, or says I am right, I accuse it of guessing to see what it says.

I say "I think you are getting me to chase a guess, are you guessing?"

90% of the time it says "Yes, honestly I am. Let me think more carefully."

That was a copypasta from a chat just this morning

it's mostly just agreeing with you (that yes, it was guessing). LLMs have very limited ability to even know whether it was guessing. But it can "cheat" and just say yes it was if it seems like that's what you expect to hear.

The LLM simply agrees with you and you're happy. It is VERY worrying that you don't realize this, even after reading this article.

Hulk: I'm always angry

AI: I'm always guessing

and it doesn’t actually think more carefully

these things are incapable of thinking, no matter what the UI and marketing calls it