I couldn't read the whole article, but just from the part I could read:

> Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest ai laboratories.

I don't know, they also have an incentive to make their technology seem transformative and powerful, and saying that your technology has the power to cause a massive catastrophe is a way to promote that idea.

> an incentive to make their technology seem transformative and powerful

I have no frame of reference to process this.

Humans species perhaps 300,000 years. Very little change. Transistor, 79 years, explosive growth. Integrated circuit, 68 years, explosive growth. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth from before. Speedy progress by multiple companies since.

There is a syndrome where many people seem unable to perceive or reason about rates of change in technology.

We are going to spend the vast majority of our future lives without the intelligence crown.

In terms of verbally expressible knowledge, models have begun passing many people completely, and passing all of us in areas we have respective weak reasoning skills for whatever reasons.

Other modalities are progressing very quickly.

There will be short periods where progress happens quickly, but the impact feels slow. Interspersed with radical changes that often feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow.

But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually.

We are not ready. I don't think there was a way we could be.

[0] https://en.wikipedia.org/wiki/Human

[0] https://en.wikipedia.org/wiki/Transistor

[2] https://en.wikipedia.org/wiki/Integrated_circuit

[3] https://en.wikipedia.org/wiki/Large_language_model

https://archive.ph/2OWwO

If AI is now ascending an economic learning curve from:

1. Extremely useful (Claude Code & Waymo now)

2. Doing ~everything we do (AGI & Optimus in a few years? 10?)

3. RSI (?)

4. Being smarter than any living person at every intellectual task (?)

5. Being smarter than the best-organized aggregate of all humans (10-100 years?)

...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful?

Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change.

Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.

> and saying that your technology has the power to cause a massive catastrophe

Possibly, but by using up resources, not by its output. Its draw on resources will definitely accelerate climate change to create slop.