Pretend learning is absolutely the key point, for me. There is danger in shifting our reasoning from knowing "stuff", to knowing a symbolic summary of "stuff" (helpfully generated by an LLM at varying levels of accuracy).

Previously, we saw a shift with search engines where we no longer needed to learn data because we could use a search engine as a mental signpost to the data, freeing up capacity for other thought.

LLMs are shifting knowledge creation to this mental pointer model. We don't need to know real "stuff" because we know how to look it up later (never?).

Each of these summaries is a secondary source, delivered through an agent biased by whatever is in its current context window. Like a game of telephone the summaries are inherently lossy, and each one may be 95% correct and we crucially don't understand which 5% may be incorrect.

When our basis for decision making is a collection of 100s or 1000s of LLM generated "Schrodinger's facts", we risk cumulative cascading errors. We will be wrong in unpredictable, chaotic ways.

We are voluntarily capping ourselves as this childish level of thought, because it feels like we are exercising our critical judgement the same as ever. However, the integrity of the inputs has been compromised. Bad inputs always lead to bad outputs.