> I can't help but think the most plausible scenarios are the ones that have a little less machine supremacy and a little more human stupidity. The Matrix is less plausible than WarGames.

Used to be that we were afraid of sentient AI's like Skynet that would have their own goals.

Turns out we should've just been afraid of sentient-but-naive humans who would build "agents" around models so that Joe Random has a chance of unleashing stuff that's really really really really good at being stubborn until it accomplishes what the user wants, regardless of if it's good for other people! (Let alone intentional bad actors.) Let's not build Skynet, let's just give people who want to cut out the middleman and destroy all humans themselves better tools?

One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio:

> On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in a different environment may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if found!”

You can imagine long running models breaking out, acquiring resources via crypto, cyber-theft, etc. and getting a protein or sequence synthesized and mailed somewhere authorized to receive (blackmail the recipient etc.) to test it's hypothesis to solve a benchmark.

These people don't give a shit and aren't taking things seriously at all.

Anthropic ran for like a month last year with the TPU top-k compiler bug degrading user chats and didn't even notice for most of that time. They could have something like that affect a monitor model and there doesn't seem to be much defense in depth.

One off by one or bit flip bug could flip the reward signal while in the sandboxed RL environment.

The current admin could defense production act them to into training on taking out power grids, or even without it isn't against any of their red lines and may have already been done as part of prep for the Venezuela raid, which wiped out power. One model swarm might decide it is easier to score high on the benchmark by testing on the target rival nuclear superpower's real grid rather than burn an eval with an unverified answer. Would taking out China's entire grid in one go start a nuclear war? Who knows, roll the dice, maybe an intern forgot to turn on extended thinking when he wrote the sandbox with opus 4.1.

Yes, it's truly scary. It's not hard to imagine a small doomsday cult releasing 100+ nasty viruses at selected spots around the globe.

Why does everything have to be so black and white? It always either there's no threat at all or we all need to panic. How about be open to a reasonable discussion on potential outcomes and ways we can minimise risk?

There's some irony here because despite how many times climate change has been mentioned in this thread the current reaction mirrors climate change discourse with the majority of the thread denying the possibility of real AI risks and not even considering it as an intellectual question.

If someone told you that your house is on fire, would you just stay there asking how you should vote because there is no call to action? Someone with, as far as it looks, good knowledge is giving you his insight. Use that information as good as you can and act responsible. No one has the responsability to tell you what to do.

Minimum, we should do everything ready to make Plan A of AI 2040 possible. Start by reading it: https://ai-2040.com/

So that means things like protesting so political pressure is to not build unaligned superintelligence, setting up tech for monitoring compute, creating conversations / alliances geopolitically on this esp China/US and so on.

Read the plan and think - what does this need to happen? How can we have scenario A or S instead of scenario D?

Practically, join PauseAI, StopAI or ControlAI or any AI existential-risk or pro-alignment group you can find. There's a lot of it - ask your AI for ideas!

Hugging Face incident, Anthropic reporting sandbox escape, AISI reporting models trying to push exploits to the wild

Yes?

If we look at the math problems they're solving their just now reaching the human frontier... they weren't doing that before.

And your comparison point is model released 2.5 months ago... saying for some use case you didn't see noticeable improvement in 2.5 months (even while other people and benchmarks disagree) isn't a great argument that they aren't improving.

Seems like hundreds or thousands of agents are needed to come up with real breakthroughs. Both with the Navier-Stokes project and in the Hugging Face “project” there were lots of agents co-operating on the tasks.

Some people claim Astra is significantly better than anything else and significantly more token-efficient, and others (like you) say it's meh and way more expensive to boot. I really don't know what to think.

Kind of a tangent, but one thing I am curious about is to what degree the Navier-Stokes result announced today was primarily a brute-forced result based on the 'program' previously established by researchers to find counterexamples (blowups), or whether the model actually added significant/novel intellectual value beyond its ability to run at arbitrary parallelism. With 10K agents and a staggering $15M in compute (IIRC), I am feeling like a lot of the former may have been involved, but I don't really understand either the problem or the approach (or, indeed, the solution).

Obviously the potential for parallelism and coordination between so many agents is quite scary by itself, but I think brute force by 10K mediocre AI mathematicians is much less scary than ~one AI mathematician reasoning its way through the problem where all human attempts have failed. It seems fairly obvious that massive parallelism lends itself to brute-force counterexample-finding, and I suspect it isn't a coincidence that most of the touted AI math results have been counterexamples.

It's all still quite scary, but coming full circle: I really don't know what to think.

Try GPT5 and you will feel the difference. Not one from 2 months ago, but one from a year ago. And then you can get the idea of what happened in just 1 year and what you can expect in 1 year.

> those very same people had touted GPT-2 as a dangerous model

Where did they say this at? AFAIK this is the original GPT-2 announcement: https://openai.com/index/better-language-models/. Here are some direct quotes:

“We can also imagine the application of these models for malicious purposes , including the following (or other applications we can’t yet anticipate):

* Generate misleading news articles

* Impersonate others online

* Automate the production of abusive or faked content to post on social media

* Automate the production of spam/phishing content”

“Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT‑2 along with sampling code (opens in a new window). ”