We keep moving the goalposts, I love it. A few years ago LLMs writing simple functions was downplayed and now this. I lovehate the future this will bring.

▲

aesthesia 4 hours ago | parent [-]

This isn't goalpost moving, it's pushing back against a misinterpretation of the paper's results (one that seems to have been encouraged by the authors).

▲

kenjackson 2 hours ago | parent | next [-]

Can we get a single human on this planet to go against Astra-6 on experiment 4 & 5? And for experiment 4 the human also has to do it "In a single try with no tools".

I don't think this is something I've seen a human do or even seen hinted that a human can do. So I'd love to see a human do it. My guess is virtually no standard ML engineer could do this. Maybe some savant could, but I suspect that is what it would take.

▲

lnenad 3 hours ago | parent | prev [-]

How is it pushing back, if anything it's mostly clarifying and the conclusion aligns with the original article:

> It’s cool that Astra was able to make direct weight updates. But we shouldn’t think of this like a radically new capability that will change how AI research is done. We should think of this as analogous to the ability of LLMs to emit machine code and tweak compiled binaries: it’s an impressive trick, but the best way to work with binaries is still to write source code and compile it.

It's like people are immediately forbidden to be impressed that AI was able to do this. The original article itself mentions the mechanisms by which Astra achieved this. They just of course wanted to gain more attention so they used a sensational title, like 90% of scientists that want their stuff read.

How is it not moving goalposts, it knew how to detect which weights to update and a way to do it? Even this article says so in the part that I quoted. The original piece never stated humans are unable to do this, I don't even think it implied it. So being impressed with this accomplishment is actually *ok*.

▲

aesthesia 2 hours ago | parent [-]

Here are some quotes from the original HN thread:

> I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.

> No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.

> How is it even possible though I can’t think of any reason why they would have the intuition.

> This is the "recursive self-improvement" they've been warning about.

> I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.

> I’m usually quite pro-AI, yet when I read this my first thought is that this is the first concrete sign that the world is about to end.

I think the paper was deliberately written and framed to elicit these kinds of reactions, and it is completely legitimate to push back on that.

	▲	lnenad an hour ago \| parent [-]
		Ok, if I quote other ones written in the oposite spirit will your point be moot? Have you seen some of the things the model did in the paper? It edited weights directly with no tools.