> There are certain tasks, like improving a given program for speed, for instance, where in theory the model can continue to make progress with a very clear reward signal for a very long time.

This makes me think: I wonder if Goodhart's law[1] may apply here. I wonder if, for instance, optimizing for speed may produce code that is faster but harder to understand and extend. Should we care or would it be ok for AI to produce code that passes all tests and is faster? Would the AI become good at creating explanations for humans as a side effect?

And if Goodhard's law doesn't apply, why is it? Is it because we're only doing RLVR fine-tuning on the last layers of the network so all the generality of the pre-training is not lost? And if this is the case, could this be a limitation in not being able to be creative enough to come up with move 37?

[1] https://wikipedia.org/wiki/Goodhart's_law

▲

lemming 2 days ago | parent | next [-]

I wonder if, for instance, optimizing for speed may produce code that is faster but harder to understand and extend.

This is generally true for code optimised by humans, at least for the sort of mechanical low level optimisations that LLMs are likely to be good at, as opposed to more conceptual optimisations like using better algorithms. So I suspect the same will be true for LLM-optimised code too.

▲

username223 2 days ago | parent | prev | next [-]

> I wonder if, for instance, optimizing for speed may produce code that is faster but harder to understand and extend.

Superoptimizers have been around since 1987: https://en.wikipedia.org/wiki/Superoptimization

They generate fast code that is not meant to be understood or extended.

▲

progval 2 days ago | parent [-]

But there output is (usually) executable code, and is not committed in a VCS. So the source code is still readable.

When people use LLMs to improve their code, they commit their output to Git to be used as source code.

▲

Wowfunhappy 2 days ago | parent [-]

...hmm, at some point we'll need to find a new place to draw the boundaries, won't we?

Until ~2022 there was a clear line between human-generated code and computer-generated code. The former was generally optimized for readability and the latter was optimized for speed at all cost.

Now we have computer-generated code in the human layer and it's not obvious what it should be optimized for.

	▲	erichocean 2 days ago \| parent [-]
		> it's not obvious what it should be optimized for It should be optimized for readability by AI. If a human wants to know what a given bit of code does, they can just ask.

▲

franktankbank 2 days ago | parent | prev [-]

Ehh I think if it ends up being a half good architecture you wind up with a difficult to understand kernel that never needs touching.