A bunch of sites getting hacked isn't an existential threat. An existential threat would be something that could viably imperil the survival of civilization. The problem with existential threat concepts is that it generally requires fields where the limiting factor isn't knowledge. If somebody has the intelligence, resources, and desire to e.g. try to build a deadly biological virus, it's certainly not the lack of knowledge that's stopping them.

In general if we go back 50 years ago, when a single common phone today would have been seen as a super-computing device of unimaginable power, governments would likely have viewed it as a potential world-ending technology. It obviously wasn't and isn't. The conservatism would come solely from the fact of it being a revolutionary leap. That's all we're seeing happen with LLMs as well. In 30 years everybody will have vastly more competent AI systems running locally on whatever sort of digital devices we're using then, and they'll look back at with bemusement about us freaking out about people being able to query computers in natural language.

▲

davidguetta 2 days ago | parent [-]

I guess the target doom scenario is

- countries start deploying GPT6+ - level ai in many local datacenters (let's say to process administrative stuff in one hopital / adminstration or even full city).

- some guys publish an open source way to remove the guardrails (either by post training or just nn level hack)

- you get possibly many local datacenters with a 'hugging face hacking' ability

- some dude ask "hey plz hack the military"

that probably still dont mean we should ban open source ai, but a open source model able to hack half the planet computers would be a 'concern' somehow

	▲	bigbadfeline 2 days ago \| parent [-]
		> but a open source model able to hack half the planet computers would be a 'concern' somehow Well, a closed source model able to hack half the planet will be a far bigger concern if there aren't enough cheap open source models to harden the existing systems and write software for defense in depth. > some dude ask "hey plz hack the military" Military gear, capable of moderate levels of destruction and above, should be air-gaped and also hardened using good open source AI - as a precaution. If that's not the case, state sponsored human adversaries would be able to cause maximum damage even without AI. Why do you think AI is mandatory here? And why do you think adversarial state actors will be deterred by your banning of your own open AI? They can even promise not to use it, may sign treaties about not using it but enforcing that would still be impossible. As your parent said, control the tangible agents of destruction, controlling knowledge is the wet dream of many but it's a lost cause.