Wouldn't it be better for users if, rather than having to puzzle this out (how to provide examples, etc) the LLM was somehow aware of its lossy areas, and replied "I don't know"?

Or maybe, to be more useful: "I don't know, but if you give me an example maybe we can figure it out"?

The problem is not only that it resembles a "lossy encyclopedia", but also that it's an extremely confident encyclopedia that doubles down on the confidence even when it doesn't have the data.