If I told you, "I can write a Python program to repeatedly run a machine learning model and calculate statistics over the weights and activations to find and fix a specific problem", would you be surprised by that? Because this is very much a part of what ML engineers do.

And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.

It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.

▲

antoineMoPa 3 hours ago | parent [-]

> Because this is very much a part of what ML engineers do.

Do you realize most people aren't ML engineers?

I even built tiny neural nets myself and never played with the weights directly post-training. The original post was eye-opening for that reason. I knew it was possible to visualize weights, but as far as I understood, it was not really something I could edit with any success. Now, it seems possible without a phd just by asking another LLM. This is fun!

▲

akkad33 3 hours ago | parent [-]

I think you misunderstand. Yet the model is doing something impressive still by writing code to fix model weights on problems. But that's a whole different ballgame than just reading weights and knowing what to change. That would mean that they have an internal representation of how the model weights map to information or a specific task, which if true, throws all the interpretability research out of the window

	▲	antoineMoPa 3 hours ago \| parent [-]
		To me, the original article is more about what LLMs can do in weight space, I don't really see the part where it would tie to an internal representation.