It's easy to argue that anyone can operate any wallet without restrictions but just pulling the right key to it.

Every participant knows and accepts it the moment they pull a random key and start operating the corresponding wallet.