https://openai.com/de-DE/index/why-language-models-hallucina...

tl;dr: The models are optimized against a test that evaluates incorrect answers just as well as no answer at all. Therefore, they guess when in doubt.

I really don't mind guessing but the model should be encouraged to say when it's guessing.