They're about as insular as the Unitarian church or Hacker News. It's basically an online community where anyone can check in, check out, and leave a comment or an essay. There's no formal structure or membership organization. The people who write or associate there could be edgy pre-teens or prominent math professors, it's really just a self-association thing. There are informal self-organized meetup groups in various cities, but not much different than groups who meet to discuss poetry or literature or video game speed-running. Some ideas have bubbled around more than others, like any memetic spread, but I don't think it's accurate to say that the people who associate there all agree on things, just like HN does not either. And nor is it any more of a clique than HN, or Twitter for that matter, even though you'll find some people on Twitter are more prominent and more vocal than others.

Very well stated

People in this comments page who are defending the crazy rationalist sects can be safely treated as though they’re arguing in bad faith. It matters little whether they are, or if they’ve ODed on their manbaby kool-aid and truly believe their nonsense; they’re just here to waste your time.

Okay, so what should we do?

I for one would like nobody to build the AI, is that an option? How do we get it on the menu?

Should we trust the people who say "nothing could possibly go wrong and there's no reason to worry or think about safety measures, if there are any trifling problems along the way we'll just figure it out as we go?"

Other people, sure, but AI? That's what this is all about, isn't it? An AI judge wouldn't have a personal life. No family to threaten, no sex/drugs/gambling addiction to be blackmailed or corrupted over. Who gets to feed training data into that AI would have outsized power, but then at least it would be transparent.

Judgement, wisdom, and resistance to corruption are the 3 things that Tom Kuegler analyzes out of the psychology of Dune as the qualities for the human beings who administer institutions/government. Even if you design a totall perfect system of governance, it still has to be administered by humans, and humans vary in those 3 areas. So the question is, can we construct a system of governance that has better judgement, better wisdom, and greater resistance to corruption han human administators, while not transferring sovereignty to whomever controls the AI?

The rationality of AI Optimists is at least rational in so far as making a huge pile of money is the rational thing to persue in capitalism.

I feel your frustration, but it’s just normal human behavior, no need for a psyop. A lot of people are emotionally invested in AI being a fad, useless, easily controlled, a capitalist boondoggle, a circular financing scam, a bubble about to explode. The alternative is that we live in an incredibly dangerous time, because obviously if we’re building something smarter than ourselves, we’re building the kitchen where we’ll be cooked.

So you get the normal reactions:

1. Nothing that happens is ever surprising. Navier-Stokes was stolen, HuggingFace was pedestrian and normal, and coding agents that last year could barely write a function and now can do a day’s worth of work in ten minutes will never be able to do anything more, because this is the end of history.

2. The people that say otherwise are lying. Because it’ll make them rich (although they are already rich). Because they are a cult (although regular people asked if we should build machines that are better than us at everything have the same reaction). Because they are marketing geniuses (although suspiciously few companies advertise the lethality of their products).

3. The people who’ve been warning about this the longest are weird and in a sex cult. (Except Alan Turing.) (Except Geoffrey Hinton.) (Except Stephen Hawking.) (You are in this group. You must be, otherwise they’d have to listen to you.)

I do hope they’re right. But their song will never change, and when some misaligned AI kills thousands, they will say that of course AIs were always going to kill thousands, but did you know the guy who told us so also writes fanfic?

There's no connection between neo-rx and the Berkeley rationalists. A guy with a lot of expertise in Wikipedia shenanigans (David Gerard) manufactured a connection to smear the rationalists.

Did the METR investigation not show you enough?

I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.

Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.

Where someone has spent years in the texture of a topic that others only pass momentarily, their ability to increase resolution of their models on that corner of reality and project models forward in time (to imagine), that is the origin of both their generous offers and their compromising biases they might make to the public discourse. It doesn't make them reliable, but what they can imagine is an important in-road to more general social inquiry. Imagination and model-testing is what comes before data in science. If the world moving fast is the danger, maybe we could make more space for imagination in order to survive the acceleration?

You downplaying "imagining", when it is literally one of the gifts of intelligence that makes us worthy of distinction... That feels like a strange posture, I guess? I'm a scientist, but I think data fetishism is one of the dangers to wary of in this accelerating world and culture

At least this is the bias of my brain in this world. I certainly know your perspective is not uncommon, esp in tech and other spectrum-biased spaces.

It’s just nonsense man.

Like, Yudkowski is a high school dropout who wrote a bunch of Harry Potter fanfic I cannot believe that people take the arguments he makes seriously.

It’s an excuse not to think, not an argument.

The doomer argument is straightforward: a powerful AI will have goals. If those goals don’t include the welfare of humanity, a sufficiently powerful AI will kill us either by accident, by indifference, or intentionally when we get in its way. Making an AI that includes the welfare of humanity in its goals is incredibly hard.

That’s it. To refute the doomer argument, you can refute any part of that. I notice that none of the people bringing up fanfics, cults, and weirdness are doing that. No, they’re instead trying to convince people that doomers are uncool. Since only cool people can know things, QED.

> autonomous bots hacking sites.

They weren't even autonomous. They were tasked with solving a hacking task. "AI" doesn't actually do anything at all unless prompted to, then it takes millions and millions of dollars of computer to do those things.

Yes, I wonder who the clowns are:

1. Researchers that have forgotten more about this technology than the average HNer will ever learn

2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it

This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.

But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!