>It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill?

I mean, this question indicates that your understanding of how these models work is, indeed, lacking.

Despite what some people having being hollering about, treating LLMs as just "stochastic parrots" isn't doing their capabilities any justice. They are constantly exhibiting emergent behavior which wouldn't be found directly in their training data. They are capable of combining different ideas that they are explicitly taught to perform novel tasks.

I'm not saying you're guilty of this, but the people claiming LLMs just regurgitate existing data don't understand how LLMs, and likely their own cognitive experience, actually works.

> emergent behavior

This hand-wavy phrase has applied to neural networks in general since it was possible to train them. That was more than half a century ago. It means both nothing and everything simultaneously.

Every learned ability of any neural network is an emergent property.