| ▲ | jrflo 6 hours ago | |||||||
If the original base model is well aligned and has safeguards around updating the weights in a certain manner, maybe that's enough? We've seen how impossible it is to get Claude to do security work. That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad. | ||||||||
| ▲ | im3w1l 5 hours ago | parent | next [-] | |||||||
I think we have to be careful how we use "well aligned" so that it doesn't turn into "no true scotsman". Let's assume that a well aligned model will try its best to follow our values, given the knowledge it has. If we imagine a porcelain vase it will try its best not to knock it over, but if the floor is slippery it may still happen. We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them. | ||||||||
| ||||||||
| ▲ | esafak 6 hours ago | parent | prev [-] | |||||||
That's an interesting question. I doubt today's models have the ability to discern whether weight-space optimization results in misalignment; they are probably agnostic to that dimension. This is all too new. I don't even know a benchmark for weight-space optimization alignment. We only have WeightBench: https://casrai.org/news/weightbench-evaluating-language-mode... | ||||||||
| ||||||||