I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.

▲

augment_me an hour ago | parent | next [-]

The math example on Qwen was based on 50 batched iterations, and the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing.

So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.

As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into

▲

ex-aws-dude an hour ago | parent [-]

I don't think its that crazy that you could look at activations and tweak weights directly to get a certain outcome, but isn't the question whether you can do that without side effects or degrading/breaking other parts of the model?

If I want to do that in a generalizable way that preserves performance across all domains then am I not just back to training again?

	▲	augment_me 13 minutes ago \| parent [-]
		Yep youre right, I don't see an eval on this. I think you could still essentially have a really cheap Lora-Post training on a frozen base model by doing this. And instead of creating 200 checkpoints for different post-trained directions just adjust a single one on the fly.

▲

jstanley an hour ago | parent | prev | next [-]

At some point we have to accept that there is enough training data on "being good at thinking" and that being good at thinking generalises really well.

There doesn't need to be specific narrow training data on every conceivable task.

	▲	ERcel 4 minutes ago \| parent [-]
		I agree, but also, I kind of feel like real judgement and understanding are a different thing from this "being good at thinking", and that in fact, this is the reason AIs hallucinate in the first place (A generated text can fit the pattern of a good response because its accurate, or because it's bs but has similar words and structure to other texts on the topic. Without real understanding, LLMs can't tell that the truth is important) It's not directly related to this but I'd like to add that, in my opinion (admittedly uninformed but it makes sense to me), when they manage to, not just reduce the frequency of the LLM stating made up dates, laws etc, or made up facts, but manage to make an AI understand what the truth is and deliberately make responses accurate, we'll have real AGI that will be able to do engineering, accounting, medicine... unsupervised.

▲

Wheen an hour ago | parent | prev | next [-]

One consideration is that Qwen3.5 has bigger/later versions that are already better at math. If it was indeed training on ML research, it's possible that what it is actually doing is applying lessons learned from the Qwen3.8 27B weights (which independently solved the problem) to Qwen3.5 0.8B.

I doubt it's that simple, but I could totally see an LLM being able to make that association and apply a fix from an improved LLM to a smaller one with the same architecture.

▲

btian an hour ago | parent | prev | next [-]

You don't think an LLM can perform tasks not in its training data?

▲

37 minutes ago | parent | next [-]

[deleted]

▲

krona 38 minutes ago | parent | prev | next [-]

Not one which can easily be represented as an interpolation of other tasks it was trained on.

▲

worldsavior an hour ago | parent | prev [-]

Well, it can't, by definition.

▲

adamsb6 32 minutes ago | parent | next [-]

Almost nothing I prompt into existence already exists, otherwise I'd just use the existing thing.

▲

triceratops 36 minutes ago | parent | prev [-]

What definition is that?

	▲	dearing 31 minutes ago \| parent [-]
		model

▲

nater5000 an hour ago | parent | prev [-]

>It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill?

I mean, this question indicates that your understanding of how these models work is, indeed, lacking.

Despite what some people having being hollering about, treating LLMs as just "stochastic parrots" isn't doing their capabilities any justice. They are constantly exhibiting emergent behavior which wouldn't be found directly in their training data. They are capable of combining different ideas that they are explicitly taught to perform novel tasks.

I'm not saying you're guilty of this, but the people claiming LLMs just regurgitate existing data don't understand how LLMs, and likely their own cognitive experience, actually works.

	▲	krona 27 minutes ago \| parent [-]
		> emergent behavior This hand-wavy phrase has applied to neural networks in general since it was possible to train them. That was more than half a century ago. It means both nothing and everything simultaneously. Every learned ability of any neural network is an emergent property.