Remix.run Logo
▲ jcims 4 hours ago

>even Astra wasn't able to improve the performance of a model without this kind of obvious flaw

Where are you getting this conclusion from that paper? It's both able to deduce what's happening from looking at the weights and write new weights from scratch just from a specification.

▲NitpickLawyer 4 hours ago | parent | next [-]

In the next section, after the qwen loop fix, they talk about a gemma model. They tried the same-ish prompts on gemma, but they did not get any noticeable improvement on the math benchmark they used. So the two sections kinda imply that yes a model can fix obvious mistakes in models, but can't make a model without obvious flaws "better" at math. At least not with their approach in that paper.

▲jcims 2 hours ago | parent [-]

I don't want to belabor the point because I don't actually care, but all we have is some evidence that it can fix a specific type of obvious flaw, that it can't fix others, that it can classify a model within a small domain from looking at its weights and that it can create weights from nothing given a spec.

That's all we've got. I think it's a bit premature to say that it can only make improvements when there's an obvious problem.

▲Mogzol 4 hours ago | parent | prev | next [-]

From near the end, "Make a language model better at math". Astra is able to improve Qwen by fixing a reasoning loop (the "obvious flaw"), but was unable to improve Gemma which did not have that flaw.

▲aesthesia 4 hours ago | parent | prev [-]

[flagged]