These models are incapable of saying they don't know, because they have no concept of knowing. They simply predict the next word which is most likely.

Maybe it's because they are trained on Internet comments, and the most rare thing to find on the Internet is someone admitting they don't know something.

The saddest part is when people take their experience with Google's idiotic AI implementation and assume that's how all LLMs work. Frontier-class models will, in fact, generally admit when they don't know something. That includes the one I run at home on my own graphics cards, but it seems that Google just doesn't GAF.

Your point about "predicting the next word" mostly means that your post was very easy to predict.