I don't disagree with your point, but I did read a post by Sean Goedecke[1] (whose opinions on the world of LLM-stuff I've generally come to respect) that I found relevant. It's not that the second-order effects don't matter to them, it's more that both the cat's probably out of the bag on those negative externalities regardless of the progress of frontier models, and that they are truly, sincerely, in-their-bones worried about the first thing and therefore focused on it since that's something that they might still have some agency over.

[1] https://www.seangoedecke.com/they-really-do-think-ai-might-k...

Doesn't it feel like the exact opposite? If you care a lot about the AI being a vengeful god, you do not do what OpenAI and Anthropic are doing. These companies only pay lip service to the idea of that aspect of AI safety in the hope that they can use it to regulate open source AI out of existence.

Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.

I am worried about those things too.

But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".

And the real threats are mostly economic and environmental - the concentration of the means of production in the hands of few who use it to exploit us all, and a surge in energy usage accelerating climate change.

Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.

So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.

I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.

It's the problem of the banality of evil. Stopping someone from dramatically pressing a big red button doesn't solve humanity's largest problems because that's not what caused them. We need to stop millions "boring" actions done by systems blindly following instructions without regard for the consequences.

There’s also an element of it which is total misdirection.

We should be paying at least as much attention to the people who want to use AI to consolidate their wealth and power, and how they’re trying to do that. They’re a clear and present immediate danger to our societies, not something we can only speculate about. And if we deal with them, better control of AI will be a side effect.

Yeah, there are very real effects happening now regarding labor as well but to dismiss it all and worry about science fiction that is on par with evangelical beliefs is just extremely weird.

One of the times I regret that I have only one upvote.

Yes I trust this government with that power. /extreme sarcasm

> and to have democratic control of the development of the technology

Why? I don't care about democratically participating in a closed model's development. It doesn't belong to me.

China will develop whatever they want, a federal stake in OpenAI or Anthropic punishes Americans and shields US labs from legitimate competition.

Yikes. Hopefully you never gain any power or influence. The last thing we need is more economic central planning. I sincerely hope that you fail.

to have democratic control of the development of the technology

Not super interested in what the people who elected Donald Trump POTUS twice think about AI.

(Of course, with Musk and Brockman in the C-suites at two of three major labs, that's what we'll get either way.)