I like this version much better because most people don't write books and AI is much better at writing than the average person, probably even a few standard deviations above the average.

It still produces wrong answers regardless though, not because of the training data but because of just... intrinsics. The question is what an acceptable error rate is, how severe those errors are, and whether a human would make comparable errors.

But this debate has parallels with self-driving cars; even if the numbers say that self-driving cars are not perfect but safer than human drivers, anything but perfection will be considered broken or outright illegal.

I wonder how much knowledge can be decoupled from experience, if at all.

If I read thousands of books that explain the details of another civilization in another galaxy, very thoroughly and consistently, but it it just happens to be all made up - did I gain knowledge? More importantly, does what I have in my brain now flip from being fiction to being knowledge if that civilization flipped from not existing to existing? How so, if nothing in my brain, or how I live out the rest of my life, changes in the least, if not a single atom in this galaxy changes (let's ignore that gravity has infinite reach and all that, for the sake of argument)?

If yes, how? What in your definition of knowledge makes that possible?

It also does not have access to any knowledge that isn't public or written down or even not in their training data.

With AI, everyone will be average in no time!

Internet started it, hopefully LLMs will finish it.