Physical hardware isn't a good argument against risk.

The hardware we ourselves are dependent on is increasingly automated; and even if it wasn't, there's plenty of failure modes where an agent destroys its own host and then itself dies off. Happens with predator-prey dynamics, happens with cancer, happens with ebola. People are worried it will happen with human-caused environmental degradation.

> Rather than covid picture armed Boston Dynamics dogs except there are multiple different factions and some of them are at least loosely in favor of preventing the wanton murder of humans.

Or imagine an LLM, with no hardware access at all, that's just used for advice. Much better than current ones, so that while it still has some weird edge cases where it makes mistakes (because the humans it learned from would also make mistakes), it's good enough that everyone ends up just trusting it the way we trust Google Maps for directions. (Trust is the key: people already trust current LLMs more than they ought to, so the quality of this hypothetical future LLM doesn't need to be high enough to logically deserve the trust, just good enough for real people to actually use it like this).

And then we metaphorically drive over the edge of a destroyed bridge.

Putting too much trust in a system — to the extent everyone just blindly accepts the answer — is both normal, and dangerous. At every level, from "computer says no" in customer support roles, to Bay of Pigs, to the 2007 financial crisis, to the Irish potato famine and the famine in the Great Leap Forward, etc.

It won't be a mistake along the lines of "there's a fire in Las Vegas and not enough water to put them out, so let's empty the dams in north California that aren't hydrologically connected" followed by six months later "Why are the farms in Central Valley not producing any food?"

But here's an open question with current tech: are algorithmic news feeds and similar websites, which optimise for engagement, making themselves so engaging that they preclude real social connection, and thereby reduce fertility?

Or another: are dating apps and websites motivated to keep people paying subscriptions, and therefore inherently prefer to pair people together only if they're going to split quickly afterwards and be back in the dating market, again leading to lower fertility rates?

There's many ways to die — More than we can count, which is why we're vulnerable. If it was just about killer robots, we could say "don't build killer robots"*. The problem is that you could fill a library with non-fiction books about people who have already lived, getting exactly what they wished for and regretting it.

* for now, at least — Red Queen race dynamics mean that even that isn't stable in the long run.