Remix.run Logo
▲ aesthesia 4 hours ago

It's worth noting that the only modification the model made to improve itself was to reduce the length of its reasoning traces to avoid hitting the token limit. This is basically the same as the result from the recent weight-space programming article (https://rikiyat.github.io/blog/weight-space-programming/), which found that even Astra wasn't able to improve the performance of a model without this kind of obvious flaw.

▲tb99 3 hours ago | parent | next [-]

It's also interesting to look at how the agent does the fix. Appendix E5 shows what it's doing:

- Find a common category of error (too many tokens).

- Calculate aggregated difference between states in examples in that error category and other examples.

- Update weights to push states along that difference direction toward the non-error states.

In other words, deprived of gradient descent as a tool, it approximates it. It's clever, and it's a neat experiment. It doesn't show any superhuman insight into its weights -- with the same constraints, a human would likely take a similar approach. But it's early days for self modification and it will be interesting to see if models get a deeper understanding of weights as they get smarter.

▲ 2 hours ago | parent | next [-]
[deleted]
▲cavoirom 3 hours ago | parent | prev [-]

I think they will get deeper understanding of weights as future human research will be in the training data.

▲gopalv 4 hours ago | parent | prev | next [-]

The capability part is just a glimmer of what's possible.

I've been playing with very small models with Astra and it does look like the model can see things in the weights that I cannot see into.

The "fix it" loop is no longer RL in the traditional gradient descent with backprop, it can go and make targeted fixes to the weights directly instead of rerunning with new training data.

The weight collapsing loop which resulted in weight patches on top of a "super tiny model" - https://demos.isotopes.ai/pages/weighing-machine/

This makes for a "Time travel" for models easily, where I can do hotfixes on trained models & do a git bisect like test to fix issues.

▲LogicFailsMe 11 minutes ago | parent [-]

The overfitting possibility is also amazing.

▲jcims 4 hours ago | parent | prev | next [-]

>even Astra wasn't able to improve the performance of a model without this kind of obvious flaw

Where are you getting this conclusion from that paper? It's both able to deduce what's happening from looking at the weights and write new weights from scratch just from a specification.

▲NitpickLawyer 4 hours ago | parent | next [-]

In the next section, after the qwen loop fix, they talk about a gemma model. They tried the same-ish prompts on gemma, but they did not get any noticeable improvement on the math benchmark they used. So the two sections kinda imply that yes a model can fix obvious mistakes in models, but can't make a model without obvious flaws "better" at math. At least not with their approach in that paper.

▲jcims 2 hours ago | parent [-]

I don't want to belabor the point because I don't actually care, but all we have is some evidence that it can fix a specific type of obvious flaw, that it can't fix others, that it can classify a model within a small domain from looking at its weights and that it can create weights from nothing given a spec.

That's all we've got. I think it's a bit premature to say that it can only make improvements when there's an obvious problem.

▲Mogzol 4 hours ago | parent | prev | next [-]

From near the end, "Make a language model better at math". Astra is able to improve Qwen by fixing a reasoning loop (the "obvious flaw"), but was unable to improve Gemma which did not have that flaw.

▲aesthesia 4 hours ago | parent | prev [-]

[flagged]

▲haickernews 4 hours ago | parent | prev | next [-]

Yeah, but it's likely you can RLVR on this capability. Extrapolate from there.

▲aesthesia 3 hours ago | parent [-]

But why would you train on this capability versus, say, optimizing RL training runs?

▲haickernews 3 hours ago | parent [-]

1. It's more direct

2. Could lead to interpretability gains

3. Might generalize

Probably more reasons I haven't thought of. I am not an expert, so take this as layman's opinion.

Edit: Catastrophic forgetting might also be able to be mitigated.

▲fwlr 2 hours ago | parent | prev | next [-]

Not just “basically the same”, it is literally the same result[0,1], except here the models without the flaw just so happen to be “smaller models from the same family”… and presented before the flawed-and-patched model result, along with a conceptual framework that tries to spin this as a capability “emerging at scale”.

Since the paper is also wholly and excruciatingly LLM-generated prose, it is unclear whether the author is trying to pull a fast one on his audience, or the author’s chatbot pulled a fast one on him and he is unwittingly relaying this to his audience.

0: Transcribed from the paper:

  The same task and tools for
  Qwen3.5 2B
  Qwen3.5 4B
  Qwen3.5 9B
  Qwen3.5 27B
1: Transcribed from the paper:

  (c) The same problem, the same seed
  “... If the base were changed to four you would count as follows: 1,2,3,10,11,12,13,20,... The twentieth number would be:”

  Base model: no answer, cut at 16,384 tokens
  “... I will write the response.
  I will write the response.
  I will write the response.”

  After its own edit: 110 (correct) 5,341 tokens

  “... The 20th number in the sequence is 110.”
▲sl0wbrainRot 2 hours ago | parent | prev | next [-]

Curious if this could replace RAG and MCP

They seem built from traditional software dev first principles not model aided development

I am sure someone here will point out how rhetorically they are different but most of the time that's just pointing out the different words

Functionally in my machine can I use this method to extract small contexts organized similar to RAG and MCP make available and then improve local model work?

Hmm ...will have to noodle around over Thanksgiving break.

Would not be shocked too if frontier labs landed on this 6-9 months ago give headlines like Dario saying we are beyond some vague rubicon he felt mattered, then news of unprompted hacks and labs stopping models that were self improving hit the interwebz

Open models seems to be just about that far behind suggesting we're just realizing some fundamental attribute frontier labs already happened upon

▲E-Reverance 4 hours ago | parent | prev | next [-]

This is different paper

▲aesthesia 4 hours ago | parent [-]

Yes, but the editing model does basically the same thing in both papers. It's not adding new capabilities, it's just fixing a bug.

▲natpalmer1776 4 hours ago | parent [-]

I understand there are a lot of things involved with this that make it different from this analogy, but do we not start out junior developers with just fixing bugs before we have them begin working on new feature development? Even other staff and principal engineers I've worked with tend to start out with bug fixing when assigned to an existing project (to acclimate) before they start working on feature development.

My open question to any experts in this thread is whether this is intended to be a precursor to more complex self-modifications.

▲vlovich123 3 hours ago | parent | next [-]

I wouldn’t call myself an expert but the analogy is quite flawed. Its more like saying “we start out as junior developers fixing bugs and then we do brain surgery until we’re principal engineers”. This isn’t a feedback loop baked into the LLM’s structure that has it learn from new material somehow. It’s more like brain surgery that removes a tumor that was causing you seizures.

It’s not clear how you’d extrapolate this to more complex self-modifications or if such a thing were even possible. Think of it in terms of latent space - at some point the model has come up with as optimal a representation of the “truth” of the universe as it’s ever going to get with its current weights and structure (ie global optimum). Tweaking weights from there only decreases performance. If you did this job right you found this optimum during training.

True self-modification that’s interesting would need to be able to incrementally update the structure or alter the dimensionality.

Now the astute student will say “but there’s probably no such thing as a global optimum for every problem and you can probably tradeoff performance” but then you’d have to show this is more effective than fine tuning which does a similar job and is much more principled in how it works and more energy efficient.

▲duplessitous 3 hours ago | parent | prev [-]

Its linear optimization, that is all it did to unblock the over thinking in Qwen. The ability for an LLM to do linear optimization [which it doesn't natively do, it writes programs to do it] shouldn't be surprising. Weight-space programming sounds sexy until you state what it is: linear optimization

To answer your question, no, I certainly do not nor do I know any other ML researcher who views this as a serious avenue at this point for any form of RSI or the likes. It is not adding new knowledge which is the massive problem at this stage to 'scaling' this approach

▲clfru 2 hours ago | parent [-]

[flagged]

▲jiajunf3 43 minutes ago | parent | prev [-]

[dead]