Arguing that LLMs spontaneously evolve new capabilities outside of their training set (being able to naturally interpret weights and manipulate them to change their function) is a completely different proposition from being able to pattern-match Pac Man really fast from a pseudo-graphical representation (which is what they actually do here).

It's like ChatGPT coming up with a brand new brownie recipe that's surprisingly good vs. coming up with a working design for a cold fusion reactor.

Yeah the brownie is cool, but it's not that cool.