I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing. Not general "intelligence" or "reasoning" but fast and effective offense.

Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can check the capability, you can teach offense to learn defense, but it seems like they are literally benchmaxxing it. Why?

OpenAI are like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen. "We need to slow down. Somebody please stop us". The thing that is unaligned here is not the models. Everyone in the story (especially the humans) just keeps doing what they think will get the most reward.

I believe these "A.I. Armageddon" warnings are completely self-serving to pump up their share price now that interest rates are rising. Soon, it may become financially unsustainable to finance the A.I. boom and the entire sector will deflate.

Do you really believe that Anthropic or the other A.I. hoods will stop selling A.I. and only make it available to a few government spooks and evaluators? That won't bring in the money to fuel the boom and could lead to share price collapse!!

Who would have thought that putting a heap of extra hosts on the internet (in the form of massive new data centres), running a heap of extra clients (in the form of agents), running autonomous workloads could cause problems?

We really have to postpone the IPOs because of security reasons, these LLMs cannot be trusted

You gotta keep the hype up, to keep the illusion going for longer...

Don't worry, ~~NetWatch~~ Cloudflare is actively building the Black Wall to save us from all these rogue AIs !

But not before the second coming claims pentacostal fundamentalist christains!!

Congrats! Certain companies are running with the same fear driven crazy nonsense that blows through the other side time to time. Hell, get together! Maybe ill get a wedding invite. I'll hang on it the wall. It'll give me something to think about.

To be sure the craziness is for $$$$$$$. They want to look like the inevitable next big thing ... which has similarity to the other s... well they want their 10% cut regardless

So jail the CEO, anyone else invovlved & shut down the company, right ?

As if AI scrapers haven't already!

"potentially causing hundreds of billions of dollars in damage"

The global economy is over $126T this year (ecommerce alone is like $7T) how can something as scary sounding as AI swarm taking over the internet cause so little damage?

AI companies are making no money so a crisis might divert attention.

So stop building it

most claims coming from the frontier models are already partly true. agents have taken over a decent chunk of the internet, it's saddening.

IMO it is their FULL responsibility to create an internet just for humans before they go full scale autonomy here with agents that are indistinguishable from humans to most actual humans. It is very dangerous there will be no safe place to obtain information you can just trust. For example, the recent earthquakes in Indonesia. I was scared to share that information with others because it looked like it could be AI and was only being shared by small accounts.

It already has. Google search results out front should have told you.

[dupe] Discussion on source: https://news.ycombinator.com/item?id=49672510

What it really shows is that Dario is in total panic mode over his business.

Ever it is because it is financially becoming unsustainable for them, they have hard time to raise more money, and the craziness of finance of all of that will be exposed soon.

Or that he is so scared of new open source models that might make his company obsolete in the long run in the sense that he will never be able to return the income expected by investors by fighting cheaper competitors.

He thinks this is a warning. My dude, when you're the one making it, it's a threat.

I hereby Triple Dog Dare him to let that swarm loose. It would be a huge improvement over the wasteland of social media.

"Mom, uncle Dario is ranting about AI swarms again!"

"If only there was someone could stop me, the CEO of Torment Nexus Inc, from creating the Torment Nexus."

“They are just saying that as a tactic. We need to keep accelerating because we will lose if we don’t.”

[dead]

- "LLMs might kills us all"

- "Terrorists are hacking claude API to configure drones"

- "LLM swarm might take over the internet"

- "LLMs will take everyone's jobs"