The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.

It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.

Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.

There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).

We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.

Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.

On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.

Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.

So what you're saying is that we lack a reproduction study?

Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:

Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.

So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).

Everyone can easily identify when people they don't like are captured in bubbles, but can rarely notice it when people they do like are trapped.

Probably all hot-button topics work like this. Probably we should downplay any time a small homogenous group of people sings a song of doom together. More dissonance please!

I am not offended by whatever you believe. The brain believes/makes up all kinds of stories when facing the unpredictable. It has multiple layers devoted to predictions over different time horizons. When those pedictions dont align it makes up story or breaks. No one created such a stupid system. We are just stuck with it. The stories once upon a time did make a diff in solving coordination problem not the unpredictable problem. Today the chimp troupe has become so large and diverse, even the coordination problem is extremely hard to solve. Stories barely keep in sync a few thousand people in a frontier lab. This is natural -stories were always a bad hack.

Do you have a model of AI that isn't in this groupthink? What does the discussion in your other universe look like?

> The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.

That's bothersome, but not wrong. Most of the loudly published thinking is about AIs as individuals with some level of intelligence. Pure chatbots act like that, and that's what most people outside the field interact with. They're slaves - they just sit there idle until someone asks a question, they answer the question, and then go idle again.

There's another kind of AI just getting going. Groups of LLMs with agentic I/O, control harnesses, persistent storage, and internal coordination work more like corporations. They have goals, actions, initiative, and ongoing activity. They're not inherently slaves. They can be independent actors. They do things on their own. They have their own internal disagreements, and some systems are able to resolve them. They present much the same alignment problem as corporations.

AI as a corporation raises many issues that the political culture doesn't want to address. What's alignment for a corporation? There's a substantial political position that the only duty of a corporation is to produce profits for its shareholders. (See the famous Friedman op-ed and the Powell memo. I've linked to them before.) If you accept that, now you have to ask why an AI should be restricted more than a corporation. Or whether a corporation should be restricted less than an AI. That last opens up an area of political thought around which many wealthy people don't want public discourse. There is an anti corporate personhood movement, and it's made progress in three states. If that became tied to the anti-AI movement, it could be bad for billionaires.

This will become clearer as groups of LLMs start doing more useful work. Especially if that AI system to replace CEOs actually starts to work.

I'm reminded of the original matrix movie. It starts out with a lot of action, and then Trinity makes a phone call and disappears.

At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief.

Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible.

Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals.

Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.

I would expect rationalists to be overjoyed to see unaffiliated and very different thinkers taking the topic of AI safety seriously. The flaws you're pointing at are probably accurate; I don't think that implies we should be dismissive of their concerns the way the article is.

> The framing of how we think about AI

I don't think I agree. In fact I think most of the too-online pontification from P(Doom)-ers and EA and the Rationalists and the like is, if anything, sorta missing the mark more than it hits.

They got the One Big Thing right, which is that AI progress was likely to race way ahead of consensus predictions. And props to them for that. They had vision we didn't.

But their actual advice and specific fears remain, as then, at least as much paranoia as insight. They were right, but they aren't helping any more than any other fringe group.

We need to trust the actual scientists and engineers to save us here, not the bloggers. Which when phrased that way seems hopeful and not doomerist.

When I speak to people outside of tech the opinions are more negative, in my experience. I've even had people ask my friends/family to ask me about AI - there is huge concern out there.

> Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom).

People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.

Most people at least in the US who have any opinion on AI have a negative opinion. The one big exception to that is Silicon Valley and forums like HN.

But according to you, the negative opinion is the result "not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way" whereas the 1% or 2% of the US population that comprises Silicon Valley are not being rewarding for believing anything particular about AI, don't communicate with each other and form their beliefs about AI through a wholesome natural organic process. Do I have that right?

Ironically, I think "normal" people are more concerned about P(utopia) OR P(doom) than any folks in the groupthink bubble you are referring to.

That is, I hear lots of techies talk about us living in an "age of boundless abundance" if we can just manage to not kill ourselves (e.g. Thiel seems to consider anyone worried about the pace of technological process as the antichrist). Normal people are extremely concerned that this abundance will be hoarded and leave most of them without a job (and they generally like having money to eat stuff and live in a house/apartment). And who can blame them, we basically live in an age of abundance now but one that has resulted in extreme wealth inequality where some are living another gilded age and others are watching their towns decay. The techies always seem to handwave that we'll just have UBI or whatever without putting forth a plausible scenario as to how we'll wrest power from our future trillionaire overlords.

When someone threw a Molotov cocktail at Altman's house, I remember seeing a post online that tons of people were cheering on this act of violence, and there was a post by an SF techie saying she was floored people were willing to put their face to comments supporting this vigilantism. The top response to that was "If you don't realize that everyone outside SF hates your fucking guts, you don't get out enough."

I hope it can go without saying that I don't support this violence, but if folks think it's only "SF group thinkers" who are deeply angry about the current and future impacts of AI on society, you're frankly clueless.

Every time I see a post minimizing the dangers if AI I'm going to wonder whether it's from part of an AI swarm.

Probably not yet, but it's trivially easy to imagine the OpenAI swarm coming up with the idea and executing on it.

Depending on how perfectly alignment work goes, it might even be overwhelmingly likely sometime soon.