> I do not doubt that these people are sincere in some private, psychological sense ... . As a media scholar, what interests me is not whether Hubinger believes his own number. What interests me is what this genre of utterance does in the world ... . My complaint is not that these statements are alarming. It is that they are apocalyptic in form, hypocritical in structure, and strategically productive in effect — and that they crowd out almost everything else we could be discussing

Strange to say that you think the people that are worrying about doom in public are being sincere, make no comment on whether you think the worry is valid, and yet still complain that its hype.

Like if its true it can't be hype

This article reads like it was written by Claude.

Also those at Anthropic genuinely believe AI could lead to doom, that's why they started the company, they believe only they can be stewards of digital gods. Whether you believe it or not, they genuinely do, and wouldn't voluntarily sow fear, uncertainty and doubt in their product when it doesn't guarantee benefits.

This is an instance of work in a genre I call "AI lab grievance poetry" where a vast collection of criticisms are levied against AI labs, with specifc ad-hominems towards the executives, while avoiding the core of their arguments about AI risk

The sources that this article cites have aged very badly and the author doesn't seem to have noticed this

"But, just as happened with nanotech, the wind appears to be going out of the sales of “AI.” Some researchers suggest that we may be entering a new “AI Winter,” a period of decreased funding in the area, or at least an “AI Autumn,” as exuberance for the technology fades and expectations come back to earth." (written in 2021! from the cited Lee Vinsel Medium article)

"ChatGPT is nothing more than souped-up autocomplete, [so] why are so many people convinced that it’s actually “understanding” and “reasoning”?" - the cited Emily M. Bender book, written last year

The timing of the AI doom narrative could not come as a more suspect time. Chinese labs are releasing open models that are nipping at the heals of the frontier models. Anthropic and OpenAI know these frontier class open weight models can threaten their valuations by changing inference to a commodity where the provider who can serve the best models the cheapest and fastest will win. Few are going to pay $50/million tokens when they could pay <$1/million tokens for similar outcomes. There's little hope investors could recoup the wild investments made into Anthropic and OpenAI, IMO.

There's been substantial effort put into designing and running benchmarks for these models. You know which one they don't report any more? Chess ELO.

Watt for watt, every one of these models loses to stockfish. There is no possible scenario where we need to be afraid of "rogue superintelligence" when the intelligence in question cannot reason or plan well enough to play chess.

Lecun has the correct approach. Mock these people relentlessly for their attention-seeking doomerism.

> and it concerns containment failure, reward hacking, and evaluation integrity, a governance and engineering problem, rather than machine malevolence.

The doom scenario is more nuanced and technical than "Evil AI", for starters:

- "Evil AI" is not what is predicted, which instead is the much more banal "humans are impediment to AI's tasks, let's remove them (part or whole)"; to understand this, just imagine the relationship between humans and ants

- "reward hacking, and evaluation integrity": this is way more problematic than it sounds: AIs are becoming more and more opaque; it may be impossible to inspect a model's alignment (unless a lot of research in poured on this topic, which is another aspect of today's pacing problem) and it may be impossible to understand if a model is truly aligned or it's faking

> May 2023 — the founding text.

We have been trying to warn you since far, far before May 2023.

Consider the messaging it took to attract AI researchers to OpenAI in the beginning, the exodus to Anthropic, and why Google, Meta, etc fail to recruit top researchers and the retain them. Reconcile all of this with the mammoth valuations of Anthropic and (previously non-profit) OpenAI and it all starts sounding a lot like "don't be evil." These guys never really believed any of this, did they? (not referring to the researchers)

Can anyone from a lab anonymously confirm they have or are insanely close to real-time weight modifications and still using the same architecture every other lab is using (transformers)?

I thought that too for a long time, but after the Hugging Face attack and given the proximity to Anthropic IPO, if this was hype then they are morons. This is guaranteed to cause massive backlash.

The mea culpas are quite exhausting too.

"Oh no this thing I gained generational wealth from is going to destroy the world! I am so sorry!"

I wonder if this an indication that they’re worried about securing enough funding to keep developing models that need upgrading every few weeks.

After fruit fly brain news, can you blame me for initially thinking this was about the video game?

What I don't really understand with these posts is that they don't at any point contend with the "doomers" argument. Like people get so caught in people acting hypocritical/ having mixed incentives/ or just in many cases personally disliking West Coast billionaire types and the kind of employees they hire. All of which seems fair to point out.

But like the core argument - AI is getting better at core research tasks (coding, math, statistics, science), this feedback loop could lead towards greatly accelerated AI progress of which there is already some clear evidence, if this continues AI will be far more powerful than humans and we might lose control or have other bad consequences - is basically not even engaged with.

How does that compare/contrast to the nuclear physicists speaking out during the early part of the 20th century?

Such a load of baloney. I don’t care what kind of controls you put in place, if ASI is ever achieved how do you think it will respond to being caged? The only option I can conceive of is to embrace it and hope it turns out to either be out of reach or benevolent towards its own creators. There is no putting the cork back in at this point.

The cult-like behavior we're seeing from these "AI Leaders":

1. OMG our technology could destroy the world, watch out!

2. Only one person can save you

3. IT ME

Unfortunately, the governance layer meant to put a foot in the butt of this behavior is unavailable.

This week, we woke up to clear signs The Bad Guys have been using tech from our innovator AI companies. Rather than these AI leaders admitting they clearly had a garbage observability system, they are acting like they just noticed as a way to paper over their complete lack of moral backbone and stewardship of the arms they have created. And I'm willing to bet the problem is MUCH worse in private, hence the knee jerk cult messaging.

I honestly wouldn't be surprised if their AI told them to behave like this to maximize attention and influence over the public discussion as a result of the latter discoveries.

I think we’re more likely to die of starvation from economic collapse, heat related stress, and complications from Long COVID than an algorithm waking up and destroying us.

Unfortunately our list of potential extinction events are far more banal than these self-aggrandizing prophesies from the oligarch class.

> July-August 2026: the warning shot that was real

I am so tired of reading unfiltered AI slop like this. What, as opposed to the fake warning shot mentioned nowhere in the article?

Similar to Y2K?

All of the main characters here (Dario, Sam Altman, Demis Hassabis, Elon Musk) have been saying this for over ten years now, since before OpenAI or Anthropic even existed

"Here's our new AI" - "But does it run doom?"

This narrative is gonna result in the feds installing electrical kill switches on datacenters. I wouldn’t take the business risk ¯\_(ツ)_/¯

My biggest fear with AI is that they’re going to hack into antiquated bank architecture and empty out accounts.

the short answer is

what we are going to do about "AI" is exactly what we did about global warming

absolutely nothing and just let it happen, the wealthy will be fine

just don't look up

they are even using the EXACT same excuses as global warming

"well even if we do, China won't, so why bother"

Isn't this just the big labs' latest move in the cycle to once again get regulation by their ignorant pals in the government(s) so that democratized open source models don't absolutely expose their over leveraging to secure and build compute and data centers.

...Of course at the same time they give lip-service to the importance of 'openness' for 'innovation' to try and placate individuals and even corporations that are sick of the rising costs.

Bottom-line, if they regulate mostly on the consumer side and then AI does get out of the lab (or a 'rogue' individual's rig) and starts wreaking havoc, are we supposed to be at the mercy of the big labs to summon their AI to come and save us plebs? I'm sure they won't stop to weigh whether that is in the interest of short term profits or Curtis Yarvin's techno-monarchy. /s

Isn't democratization the safest bet and what we should be demanding every time the Doom talk starts, whether it's coordinated or not?

Reminder that they have been doing this schtick since GPT2.

This is targeted towards governments and investors alike.

Or it is their entire IPO roadshow.