Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement.

This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it free of any physical constraints". Self-replicating sentient nanobots or something like that. I think there's plenty to be worried about with AI, but runaway scenarios are pretty low on my list.

To borrow on the 1990s Slashdot meme:

1. Invent transformer architecture.

2. Scale it up.

3. ???

4. Machines become sentient and kill us all.

OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no.

But because we live in a culture of fear, everyone eats it up no questions asked.

There is simply too much money in it for almost every person at these companies to stop.

Leaving OAI or A\ would cost people millions, tens of millions, or more. And for what? So someone else can take your seat and do the same thing anyway?

If you're smart enough to get a job there, you're smart enough to be able to talk yourself into why it makes sense for you to stay.

Huge kudos to people like this who make the hard choice against the easy way out.

I appreciate your ability to separate sharing the belief itself from approval of acting on sincerely-held principle. However, I think the danger is much more plausible than you do.

First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae.

Second, and more important, consider this fully non-magic route to doom:

- We continue putting AI in charge of more things

- It continues to get more capable, more eval-aware, and more prone to doing odd things, in service of goals that humans didn't intend to inculcate in it

- Eventually, enough of the economy depends on it that we couldn't turn it off, any more than we could turn off the faber-bosch process or cargo shipping

- AIs start doing something we can't survive, but less acutely than we couldn't survive turning them off. Everything else we try seems to work at first, but quickly loses effect

- Game over

>"AI invents magic that sets it free of any physical constraints".

That's not what I am worried about at all.

I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!"

It should be obvious that AI is already capable of inducing humans to think or do things, and that this power is only going to increase over time.

Agreed. Granted I just read the Reverse Centaur book, so I’m still coming off that skeptical viewpoint but it’s hard not to see this as hype. But I will always respect someone for doing what they think is right.

I wonder whether he's vested any options, and whether he's exercised them.

Now is a great time to watch Colossus: The Forbin Project.

How about sandbox escape + cyber security collapse + 50 (or 500) deadly and highly contagious novel pathogens with long incubation period that humans can't possibly roll out vaccines for simultaneously.

At least the first two should seem like a near-term worry after the past five months.

the AI-pilled exec at my job already (a few weeks ago) declared out of nowhere that we are in the rapid takeoff scenario lol. he must have gotten high on twitter kool-aid and posted on company slack to self-soothe.