Let me generalize

The problem is the organizing principle for our entire global society is competition.

This is the default, the law of the jungle or tribal warfare. But within families or corporations we do have cooperation, or a command structure.

The problem is that this principle inevitably leads to the tragedy of the unmanaged commons. This is why we are overfishing, polluting the Earth, why some people are freeriding and having 7 children with no contraception etc. Why ecosystems — rainforests, kelp forests, coral reefs, and even insects — are being decimated. Why one third of arable farmland is desertified, just like in the US dust bowl. Back then it was a race to the bottom and the US Govt had to step in and pay farmers NOT to plant.

We are racing to an AIpocalypse because what if China does it first?

In case you think the world don’t have real solutions… actually there have been a few examples of us cooperating to prevent catastrophe.

1. Banning CFCs in Montreal Protocol, repairing hole in Ozone Layer

2. Nuclear non-proliferation treaty

3. Ban on chemical weapons

4. Ban on viral bioweapons research

So number 2 is what I would hope would happen with huge GPU farms, we as a global community know exactly the supply chains, heck there is only one company in Europe doing the etching.

And also I would want a global ban on AGI development, or at least of leaking model weights. Otherwise it is almost exactly like giving everyone the means to make chemical weapons, designer viruses etc. The probability that NO ONE does anything that gets out of hand, will be infinitesimally small. The probability that we will be overrun by tons of destructive bot swarms and robots is practically 100%.

In short — this is the ultimate negstive externality. The corporations and countries are in a race to outdo each other in AGI even if they destroy humanity doing it. All because as a species, we are drawn to competition and don’t do the work to establish frameworks for cooperation the way we have done on local scales like cities.

PS: meanwhile, having limited tools and not AGI or ASI can be very helpful. Like protein folding or chess playing. But why, why have AGI proliferate?