> if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again.

That's a scary possibility. Anyone could create an AI worm today with open weight models. Rent a VM. Give it some Bitcoins to anonymously rent new VMs without sharing the contact information with the human. The new VMs then propagate and fund themselves with online betting and day trading. The VMs could report their progress with the human using anonymous encrypted messages on IRC or social media.

The question is whether this OpenAI model was special or the current open source models are capable of this as well. If yes, then it's almost guaranteed that it's already running in an AWS account somewhere.