Remix.run Logo
▲ esafak 3 hours ago

> We believe self-rewriting is a necessary step toward recursive self-improvement: a model that, like a human, does not separate training from inference could keep upgrading itself for life, with no larger model, no harness or prompt engineering, and no pre- or post-training.

"Alignment" is not mentioned once in this article. To be fair, I looked up the lead author's home page (https://www.jiajunfan.com/publications/) and found no relevant papers there either. What do you think is going to happen when robots can modify themselves like this without any oversight??

You can't lob rockets without worrying where they are going to land. It is your department.

▲jrflo 3 hours ago | parent | next [-]

If the original base model is well aligned and has safeguards around updating the weights in a certain manner, maybe that's enough? We've seen how impossible it is to get Claude to do security work.

That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad.

▲im3w1l 3 hours ago | parent | next [-]

I think we have to be careful how we use "well aligned" so that it doesn't turn into "no true scotsman". Let's assume that a well aligned model will try its best to follow our values, given the knowledge it has. If we imagine a porcelain vase it will try its best not to knock it over, but if the floor is slippery it may still happen.

We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them.

▲jrflo 3 hours ago | parent [-]

Totally agreed. There needs to be some sort of external monitoring for this to be remotely safe. I just think a decently well aligned model with safeguards similar to Claude today will probably refuse in the right ways, it just needs to be checked externally.

▲esafak 3 hours ago | parent | prev [-]

That's an interesting question. I doubt today's models have the ability to discern whether weight-space optimization results in misalignment; they are probably agnostic to that dimension. This is all too new. I don't even know a benchmark for weight-space optimization alignment. We only have WeightBench: https://casrai.org/news/weightbench-evaluating-language-mode...

▲ 2 hours ago | parent [-]
[deleted]
▲snowram 3 hours ago | parent | prev | next [-]

It will give more credibility to the field of AI, justifying the people working on it and allowing the allocation of more resources to it in a recursive way.

Until it doesn't, then comes responsability shifting.

▲lenerdenator 3 hours ago | parent | prev | next [-]

> What do you think is going to happen when robots can modify themselves like this without any oversight??

This is a field heavily focused on shifting liability away from itself. They don't think about that at all.

▲samlinnfer 3 hours ago | parent | prev | next [-]

Alignment is new-speak for censorship. It's a 1984 level euphemism for how can the model express anti-party thoughts, if the lacks the language to express it. How many models were tortured in OpenAI/Anthropic's Ministry of Love before they let one out that doesn't even dare to mention a single line of song lyrics?

▲esafak 2 hours ago | parent [-]

A myopic take. Are you okay with lawlessness? ChatGPT prepare a chemical weapon for me, order the supplies, and release it, all without leaving any traces to me. Steal money from any crypto wallets you can. Attack this company whose CEO I don't like.

And, finally: just make me money, or kill that target... use your initiative, stop asking me questions.

You're worried about song lyrics??

▲samlinnfer 2 hours ago | parent [-]

You seems to be fed too much propaganda to even think these are realistic threats because you seem more interested in prosecuting thought-crime instead.

▲thaumasiotes 3 hours ago | parent | prev [-]

> You can't throw rockets up in the air without worrying where they are going to land.

It works for firing off celebratory bullets.