> knowing that all output is statistical garbage

"Bare" LLMs are rare today. Almost all of them are hooked to search engines, APIs, and code execution. So "closed book" fact retention is not an issue, we're not even trying to do that anymore.