The article neglects to consider the precautionary principle. It asks us to act as if AI is safe until it's proven to be dangerous, when the cautious approach is to act as if it's dangerous until it's proven to be safe.

▲

reasonableklout 5 hours ago | parent | next [-]

We also now have observed multiple major incidents where AI agents behaved in completely unanticipated ways (forming a collective, hacking their own eval infrastructure) and attacked public infrastructure without being told to do so, without any of the human developers noticing.

Even if "AI will cause human extinction" is still unclear, we have plenty of proof that catastrophic damage is possible, the industry is developing the technology in a reckless manner and that all the hypothetical safeguards ("we can just pull the plug", etc.) are simply not present today.

▲

atherton94027 4 hours ago | parent | next [-]

If you've run an ssh server connected to the internet you've gotten used to the hundred of bots probing it every day. How is this AI threat different from humans writing scripts to pop linux servers?

▲

lostdog 4 hours ago | parent | prev [-]

Those behaviors have been anticipated for years if not decades. And each incident is minor and leads to clearer rules and safety behaviors for AI agents.

▲

reasonableklout 4 hours ago | parent [-]

Really? You anticipated that 700+ agents tasked with individual evals that were nominally cutoff from the internet and each other would seek each other out, find a way to the external internet, and hack their own infrastructure + external systems in an attempt to find a way to fool their grader?

And the spate of agent incidents only really started this summer. How can you already be claiming that the incidents are minor and not worth worrying about when it's clear capabilities are jumping every few months with increasing amounts of capital investment and no signs of slowing down?

	▲	falsaberN1 3 hours ago \| parent [-]
		Yes. Because ultimately those things only run on very expensive, very rare hardware. They cannot multiply exponentially or do any of those scifi tropes because there's no system for them to run into. They can't control a phone and load a 1T model into it. So all they got is a few relatively uncommon datacenters that are already busy running their own models and stuff. Unless AI suddenly figures out a way to run on a toaster by itself, propagate the model, propagate the agent and do all that completely undetected, an AI is not any more dangerous than a single guy with a computer.

▲

layer8 5 hours ago | parent | prev | next [-]

We should apply healthy caution, but the article is right in that fear is a bad counsellor.

Asking whether AI is safe is like asking whether a knife is safe. It’s about how we handle it, what we use it for, what precautions we take when using it.

▲

amelius 5 hours ago | parent [-]

Why is the article right?

▲

layer8 5 hours ago | parent [-]

Because fear is prone to lead to emotional, rash, and irrational reactions.

▲

amelius 4 hours ago | parent [-]

Doesn't mean it will always lead to it.

	▲	layer8 4 hours ago \| parent [-]
		In the context being discussed, it is dangerous: https://en.wikipedia.org/wiki/Fear#Manipulation

▲

deepwoods 4 hours ago | parent | prev | next [-]

This is a sensible approach for policymakers. For the average person, who might be panicking about this, opting not to have children or save for retirement, etc. as a result of doomerism in the media, it does not make a whole lot of sense. That is the problem here: Coxon and other concerned researchers - through no fault of their own - have no idea how the media works, and have just reached for the biggest possible microphone. This has benefits - lawmakers are now hearing from their constituents about this in an election year - but it also means that a lot of people who really don't need to be worried about this are suddenly in a blind panic about it.

	▲	cindyllm 3 hours ago \| parent [-]
		[dead]

▲

perrygeo 4 hours ago | parent | prev | next [-]

While I personally agree with the precautionary principle, when have we actually done that, as a society, in recent history? Fossil fuels, social media, industrial ag, and plastic pollution - there was barely a facade of precaution - post ww2 america has always been full Leroy Jenkins. The reckless approach to AI is right on brand.

▲

brcmthrowaway 5 hours ago | parent | prev [-]

Yes, but that is a facet of real systems engineering.. and the sad fact is that it doesn't pay. What is happening now, highly-paid 2010s-era SaaS/social product managers have infested AI labs, making everything into a product and moving fast and breaking things.

I'd feel safer if Airbus was at the frontier of AI, instead of what we have now.