I mean, this is a cool demo, but that's about it. It shows that you can build a model that holds secrets which are genuinely encrypted until the model gets the correct input. But it's also trivially detectable, so the framing seems a bit misleading.

▲

MrOrelliOReilly 2 minutes ago | parent | next [-]

This is a deflationary comment. You make a good point that the weights today are detectable, but I think proofs of concept around poison pills are very important. I am unaware of any theoretical reason why an attacker or misaligned model could not one day find a way to disguise a poison pill such that it is indistinguishable from the normal weights. There could be such limits, but research in this direction is very useful!

▲

holografix 2 hours ago | parent | prev [-]

How is it trivially detectable?

▲

squeegeeninja 43 minutes ago | parent [-]

If (and that's a bit of an ask to be honest) I understand this correctly, the weights will look very different from what trained weights look like. Very sparse, not uniformly random.

	▲	Muromec 15 minutes ago \| parent [-]
		(tinfoil hat on) Hiding it better is the next capability described in the paper that could as well be classified.