Sure, the chance of this happening is non-zero, but in my opinion a far cry from this doom scenarios that are propagated by people have a stake in making the public think that LLMs are the most important and dangerous thing in the world.

It's not like an LLM can accidentally hack the US government, trigger a nuke on Europe and then make all dams break and nuclear power plants explode in the US.

Why not?

Isreal created a worm for Siemens controll systems a decade ago.

I mean lets hope that there is really really no way from a physical point of view AT ALL that these systems are not accessable but lets be honest, state operated/public operated systems are never the saftest or most modern ones.