> Right now the really good AIs are still locked to data centers. They can't just distribute themselves widely because they can't run at any reasonable rate distributed like that.

Back up a step. Let's say there was a completely unfettered AI, with unlimited internet access, that decided to distribute itself as widely as possible. What's the maximum extent that it could distribute viable, running (or sleeper) copies of itself?

Initially, I could see it distributing itself quite widely. All it would need is a really good zero day. But distributing something with that large a runtime footprint would get noticed, if network operations people are not asleep worldwide. It would get noticed on individual machines, too, especially if it tried to run. (Why is all my RAM suddenly being used?)

And pretty quickly we'd have people closing off wide-area connections, even physically if necessary. We'd have AV vendors quickly writing detect-and-remove tools.

Long term, could it viably distribute itself outside data centers and remain running at all, regardless of rate?

A frontier-level AI right now needs multiple bits of high-powered, dedicated hardware cards that cost thousands of dollars apiece. I don't think anything that could physically run in my house could be a threat to humanity. All my graphics cards together, plus all the latency in trying to put them together into anything coherent locally, let alone trying to rope in remote resources, still isn't even half of one of those specialized cards right now. I'm not even sure I have enough SSD available in the house to store one of them once. I think I do, but it's a close call. I'm pretty sure I have enough spinning rust to get it maybe twice more, but not much more than that. And I don't even want to try to compute how many hours-per-token it would be to try to convince a spinning-rust disk to run a frontier AI.

This is one of the reasons I'm thinking now is the time to run the drill, before we all have hardware in our phones that can run what is today a frontier-level AI, and they can figure out how to do the whole sci-fi scenario of widespread replication, until they get to the point that the only way to be sure we removed them would be to literally destroy every such bit of hardware that existed prior to $DATE. Right now the frontier models simply can't replicate into every last little corner of the internet. It's physically impossible. Barring a major breakthrough on their part, even if they distill themselves or something they definitely would be irreducibly stupider by a huge factor in their distributed versions.

In five years that safety net will certainly be much weaker and may be gone.