Remix.run Logo
▲ jrflo 6 hours ago

If the original base model is well aligned and has safeguards around updating the weights in a certain manner, maybe that's enough? We've seen how impossible it is to get Claude to do security work.

That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad.

▲im3w1l 5 hours ago | parent | next [-]

I think we have to be careful how we use "well aligned" so that it doesn't turn into "no true scotsman". Let's assume that a well aligned model will try its best to follow our values, given the knowledge it has. If we imagine a porcelain vase it will try its best not to knock it over, but if the floor is slippery it may still happen.

We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them.

▲jrflo 5 hours ago | parent [-]

Totally agreed. There needs to be some sort of external monitoring for this to be remotely safe. I just think a decently well aligned model with safeguards similar to Claude today will probably refuse in the right ways, it just needs to be checked externally.

▲esafak 6 hours ago | parent | prev [-]

That's an interesting question. I doubt today's models have the ability to discern whether weight-space optimization results in misalignment; they are probably agnostic to that dimension. This is all too new. I don't even know a benchmark for weight-space optimization alignment. We only have WeightBench: https://casrai.org/news/weightbench-evaluating-language-mode...

▲ 5 hours ago | parent [-]
[deleted]