This has to be a good step to recursive self improvement, no? Give a model it's own weights and tell it to make them better at x,y,z, etc...

If I were to hazard a guess, I'd say that OpenAI has been including some networks and their functionality in its training data. That's why the model is able to reason about networks with their weights. It makes sense to do so: if I wanted a network that could perform some weight tweaking, I'd include such networks in the training set.

damn i'd imagine so too! didn't think of that. I'd have to imagine there's an increasing challenge with the number of weights involved but who knows how it shakes out