What's more alarming isn't that AI is limited to existing domain data, it's that when people push it to deviate outside those known data points it confidently hallucinates nonsense.

Models are not trained to self-evaluate. Or only as an afterthought during tuning. So they are poor at it.

It isn’t mysterious.

Humans are trained incrementally, educated informally and formally, and along the way tested by context and classroom. Evaluated by people in their circle and strangers. The training to evaluate ourselves is near constant.

Even then, many people habitually believe they understand things they clearly don’t.

And can even be hostile to feedback.

Many forms of hallucination are canonized, or socially encouraged at varying demographic scales. While others are idiosyncratic.

Once self-examination is a first-class part of models training process, I expect they will respond like they have to being trained on vast troves of information, which is far excel human beings.

But, until then, their poor self-assessment isn’t mysterious, it is a mundane result of being trained not to do that. Or only as a tuning after thought.

Not as different from humans as we might like to think.

And models are noticeably improving on this measure. Humans in my experiencing may be regressing.

And many of today's publicly-accessible platforms are designed to steer results away from nonsense back to... too much sense. "Hypernormal" is a good word for it. I've spent a lot of time prompt-yelling, "Please make something as weird as I'm imagining, stop veering back to what normal people want to see/read."

So much nonsense.

If I had a nickel for every AI-poisoned "researcher" I'd seen with a preprint full of nonsense buzzwords like "quantum fractal holographic resonance matrix"... well, I wouldn't be rich, but I'd probably at least have enough to buy a coffee.