The Python example is fascinating, and a good rejoinder to anyone still dismissing LLM’s as stochastic parrots.

Indeed, I found this part extremely interesting. The more general vision of "testing a vintage model on something invented after its training data ended" seems like quite a strong test of "true cognition" (or training data contamination, if you haven't stopped up all the leakage...)

It's shocking!