As with a lot of AI stuff, Borges already wrote about it.

https://www.sccs.swarthmore.edu/users/08/bblonder/phys120/do...

That’s more elegantly put than I ever can.

Btw I am not disagreeing with the utility of LLMs, my point is it can never be 100% accurate with current architecture (unless you blow up the size).