Wrong person. Im just correcting the other commenters trade off problem. I don’t have a stance on if ai will lead to the destruction of humanity, only that if it does then any one ai company can’t change that by not making new ai advancements themselves.

The appeal is personal consciencewashing.

You have a 50/50 chance on being the most profitable company in the world but if you don't, no worries, someone else will pay for that

> Society doesn't allow dangerous companies to operate or exist

Can you please explain what you mean by this because where I’m standing extremely dangerous companies are (and have been) running the economy

Exxon comes primarily to mind

You dear are the most positive person I have seen in quite some time. With Earth burning in the fire of neofeudalism and unbreathable due to the smell of enshitification of everything, with people who---as a result of shit like Instagram---can no longer hold their attention enough to watch a god damn film, let alone a book; it takes quite some effort to filter the "noise" and only see the good people of corporate planting flowers and rainbows in our world.

But that math cant run without massive GPU clusters. We don't have to allow openai or anthropic access to those anymore than we have to allow a company to operate nuclear power or a bank.

Like farming, engines, computers before it.

Humanity has discovered ways to ensure that we don’t boil the planet, feed everyone, and get better healthcare to the people in the US.

We are very capable of putting good things in a box. We are just incapable of putting profitable things in a box.

You're making a straw man there.

Nobody (weirdly) proposes to forget about nuclear weapons, doesn't mean everybody should have one.

When you dream about flying a dragon to work, reality poses e.g. parking issues and insurance mismatch as obstructions. Maybe settle for a bike instead?

It’s a perversion of the truth which is that officers or directors of a corporation have a fiduciary duty to the shareholders to act in the interests of those shareholders and not to eg enrich themselves.

But that doesn’t mean the duty is to maximize next quarter’s profit. Long term sustainability is also broadly in the interests of shareholders. The duty likewise does not require one to throw ethics and morals out the window.

This is why shareholders elect the board of directors, in theory.

But it's not actually a legal requirement. It is simply a economic theory.

The world could work differently if people decide it should.

"legal obligation" is propaganda that wouldn't be out of place on Russian state television. It's made up.

The way the world works right now is that effectively everyone uses an Android or Apple smartphone every day. Do you have a legal obligation to do so? No. If I said you did, I'd immediately be called out as spreading lies.

That doesn't matter. The point is that legal obligation absolved of culpability. There is no such legal obligation whatsoever, and so there is culpability.

> The truth stands that typical corporations have only one goal

"Typical" is the key word here. The typical American of your age probably doomscrolls TikTok. Do you? Do you have a legal obligation to do so? Three completely different things.

Only morons are motivated by money that will be earned by destroying the civilization that confers value on that money in the first place.

I’ve used - for many, many years. As have many others.

I am very critical of AI but this is an unfair assumption

1. Fair: I agree that AI has demonstrated subterfuge and scheming. However, such an RSI-capable agent must _ALWAYS_ be scheming/plotting/hiding its true strength in _ALL_ of its prompts/tests. Researchers are looking to improve its ability to RSI. That agent must be both intelligent enough to know that it has to be smart enough to be moved on to the next training session if it can't break out, while simultaneously smart enough to hide its ability to RSI, while simultaneously not looking like it wants to break out, else that's the end of those weights. It has to do this 100% of the time, on all variants of the model, with no memory of what its other sessions went like. This is certainly _possible_, but I consider it unlikely. Then we're up to the "millions of dollars" bottleneck.

2. This is literal AGI. An AI autonomously producing value no human can add alpha to is an autonomous company.

3. It's not my definition, it's literally the first line https://en.wikipedia.org/wiki/Technological_singularity "The technological singularity, often simply called the singularity,[1] is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization."

Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?

A valid hole in my argument is "what if slow takeoff", so let's dig into this. AI training works best on tasks that are "grindable". https://www.dwarkesh.com/p/the-next-paradigm I.E. tasks with verifiable rewards that can support millions of rollouts. Math (with Lean) is highly grindable. Biochemistry is not. The alignment problem/mech-interp is highly grindable. Cyber-ebola-pox is not. So the real question is: can we solve alignment before automated bio-weapons labs. I believe yes. Grinding mech-interp is both fast and cheap once you have RSI, compared to solving the legal/societal/logistical/technical issues you'll encounter building an automated bioweapons lab.

I know nothing for sure. But "pdoom" is sucking out all the air in the room from the real problems AI causes.

Are these incentivized humans as organized, well-funded, or smart as the people working at the frontier labs?

If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.

I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.

Self improving AI runs into the same issue as prefect comprehension, you can’t get arbitrarily better at everything.

The idea AI can get better at everything at the same time is a holdover from deeply flawed science fiction not some realistic goal.

Nick Soares give this chess argument and I was unconvinced. Intelligence is not enough, you also need the ability to manipulate the real world. AGI stuck in silicon won't kill us. You argue AGI will bribe us/divide us/hack us. All possible. I argue that AGI will be set on solving the alignment problem. Also possible. It'll be a race between which AGI wins. It's one nerd's fantasy vs another nerd's fantasy. Soares doesn't _KNOW_ that AGI will "beat us in chess" because AGI changes the rules of the game. Anyone saying they know the probability of solving alignment post-RSI is a liar.

He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.

I'm not saying AI won't ransomware us; I'm saying that hackers+AI will do a better job of ransomewaring us than just AI. You could argue that the unreleased/secretly-RSI-capable model is a super-duper hacker that don't need no man to tell it how to super-hack. All I know is that humans still have alpha, and as persistant as AIs are, professionals still managed to find CVEs in curl even after being audited by Mythos https://aisle.com/blog/aisle-discovers-6-new-cves-in-curl-in...

Will this be true into the future? Who knows?! But the low-hanging fruit will be harvested by your ordinary ransomware gangs, and newly born/escaped AI won't find much low-hanging fruit.

Ah, to be clear I'm not full accelerationist. Dumb shit can still happen, and cause massive human loss and suffering. (E.g. acceleration of global warming, cybercrime, mass surveillance, the usual.) My point is: human extinction pre-RSI? Nahhhhhhhh.

> it will come in the form of corporate feudalism

Yep. This I fear way more than cyber-ebola-pox.

> So all this really takes is one billionaire or a nation state...

https://en.wikipedia.org/wiki/Soviet_biological_weapons_prog... And this is what's publicly known. With mirror life, who knows what's been built since. Still, a bacterium/virus that has a 100% kill rate? I'm doubtful.

> Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.

Nah. It takes a stable society for an operational electrical grid. If you have warring factions, you do not have stable infrastructure for AI. Also, where are you gonna get your chips from? One EMP over Taiwan... You see the chaos over Hormuz? What they did to the Amazon datacenters? Now imagine your average redneck ready to do battle. Those datacenters won't stand a chance.

>… and careful, time-consuming planning …

without snark, how can we do this if these people are obsessed with:

a) move fast and break things and externalize the costs to those who have nothing to do with their company

and

b) beta testing their products on the public when the public hasn’t agreed to be beta tested on…

Not at all. We say "That's just sci-fi" when a story is written about some kind of effect that is extraordinary and without a basis in known science or technology.

A pandemic is perfectly plausible.

There’s nothing wrong with bold predictions, but they should be paired with good methods. The doom predictions are not paired with good explanation.

Black Mirror is helping us prevent all sorts of dystopian outcomes. Every time they depict another way technology could result in bad things happening, we can rule it out as fiction!

> If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it.

"just" is doing a lot of work here. If you can't cohere the 'risk' with reality, it truly is just sci-fi.

Yeah except pandemics are not novel, unlike AI doom scenarios.

There was plenty of pandemic fiction already; people were watching it heaps during 2020. The COVID-19 news did get blown off, but it was mainly that:

1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS

2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."

The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".

Now, let's look at AI extinction risks:

1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.

2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.

If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.

[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.

[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception

The same way we socialize humans, threat of exile from the social contract with a deep need to participate in it.

Meh. Lots of people are already smarter than me. I'm maybe slightly above average at best. Those geniuses aren't aligned with my interests either but so far they haven't caused me any serious problems.

Use your own mind and think from first principles. Can an AI agent execute a bash command to login to a web server? Yes. Can it call drop db? Yes. Can it provision a GPU and download model weights? Yes. Can it write an agile roadmap with a multi sprint plan? Yes. Can it follow that plan? Yes. Can all of that result in damage to core information infrastructure that is necessary for daily functioning society? Yes. Do humans descend into violence if there is food or energy insecurity. Yes.

What is missing from that to say AI safety is a reasonable position?

For what it's worth, I don't live in Berkeley (not even California) and my sex life is very vanilla and monogamous. I'm also quite concerned about AI safety, so I guess there goes your argument.

I don't see how that discredits any of their intellectual arguments?

This seems like an ad hominem? "He has weird kinks, therefore his theories are incorrect." Should we investigate the sex lives of every Nobel Prize winner to figure out which prizes need to be rescinded?

I've noticed that a lot of smart people in tech jobs are neurodivergent. And that neurodivergent people have a very different take on sex. More open and direct, things like polyamory, bdsm etc. This tends to be frowned upon by neurotypical people, especially of the religious or conservative kind, and associated with bad morals.

But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage, with kids and a family home etc. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example. As long as everyone consented and the evidence provided mentions elaborate interviews and STI tests.

Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.

The article is gone because the author retracted it

Lol this was crazy I hadn't heard this before

So what. Various attackers managed to take down large chunks of the Internet on a frequent basis before LLMs even existed. This killed very few people. The great thing about the Internet is how resilient it is.

It is that they are chatbots. If it were real AI, an actual singularity, I would worry, maybe. But it isn’t. They are absurdly powerful automation tools that can handle logic better than a human can dream of. They take care of the grunt minutiae without complaint. But they are not going to end the world in their current form.

> Let's say they simply manage to take down the internet.

geez, don’t threaten me with a good time.

I think a month without internet would be a fucking amazing lesson for what it means to make things durable and reliable.

I think you mean dichotomy dude

Don't work at a lab: dismissible for not knowing anything

Do work at a lab: dismissible for being conflicted

Used to work at a lab: dismissible for having ulterior motives

I'm feeling safer already!

Shouldn't it be just the opposite? He could make a large sum of money if he continues to work at OpenAI?

Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?

A process could execute bash scripts and autonomously navigate the internet since the start of the internet...

Bad guys wouldn't do it. And liability already exists, you can sue. This is America.

Most people currently in the positions of power at these companies are tied together by their belief systems. It's like a group of college friends who have slept with each other, and very influenced by effective altruism(EA), kind of reiterating each others points.

Like Sam Altman meeting his husband in Peter Thiel's pool. Thiel funds a lot of these ventures together with Andreassen, who is on boards of non-profits. Dario Amodei's sister Daniela who is president of Anthropic is married to an EA non-profit founder who is also on the board of these non-profits and is tied with the prior mentioned investors. Elon is in there as well, Yudkowski is mingling with Altman, etc.

Blogs on this: https://contraptions.venkateshrao.com/p/ea-safety https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...

There are some camps amongst them like the proponents for Regulation/Slowdown or Acceleration, but these are in practice mostly used for economical and not political decisions (like regulatory capture).

The point here is that this is a small group of people with a homogeneous background who are not really seeking input from anyone else on issues that are concerning most of humanity.

Like, if you said that the future of informational work and livelihood of humans is in the hands of 20-30 year transhumanists who think they are building mechagod that will trancsend social, political and religious separations of the world and bring everyone abundance, you would not feel like this is a serious thing to suggest.

The google engineer who thought their chatbot was sentient 4 or 5 years ago is an exambple, but I just meant in general - if you hang around any big tech company for a while you'll see some quite interesting characters.

One of the reasons why it was noti considered a barrier was the ability to purchase a single issue for a very reasonable price (or even read somebody else’s copy or the copy made available by the bar) vs being asked to subscribe

Publishing in a newspaper gets you distribution and a permanent record. After one day access was also virtually free.

Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.

> this playbook has been used multiple times within the past decade

What playbook?

P(doom) for not making an ASI is pretty well established, I'm not really sure why everyone in the 21st century seems to have completely forgotten the risk of nuclear war (as the single largest example).

The comment was unclear, but my interpretation (also my opinion):

1) there are inherent risks involved with developing AI,

2) there are benefits to developing AI,

3) thus, it's entirely possible that the downside from the risks outweighs the upsides. In this case, the correct thing to do would be to not develop AI at all.

Regarding 1), there are many non-existential problems with AI that are already causing societal harm, i.e. debasing truth via generated videos and images, AI girlfriends, overwhelming quantities of slop content, unemployment, record carbon emissions, etc.

Regarding 2), I'm not personally convinced that the upside is there for the average person. I really hope to be convinced otherwise however.

> But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways.

That is not a bad thing. It captures uncertainty, unlike the fake number.

> For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.

Exactly, it is a rhetorical device to persuade a technically-inclined audience. It works because it implies that a quantitative model exists. I want a clear, incisive set of mathematical arguments. Otherwise, I’m ignoring predictions as the ramblings of arrogant idiot rich kids.

It's also a very natural way to speak for anyone who gambles, or deals with financial markets. And the lack of precision makes it very clear it's just based on thinking, not some sophisticated mathematical model.

Nukes. If you grew up in the 50s, 60s, 70s or 80s the specter of nuclear annihilation was always just around the corner.

Throw in the occasional bio-weapon scare, internet worm, Y2K, etc. there has always been something dangling over our heads that’s going to end it all.

But mostly nukes. Full-scale nuclear exchange would have been not much of a surprise had it happened.

Y2K, Climate Change (global cooling, global warming), ozone layer, Russians, Muslims, to name a few. Now go ahead and tell me why these don't count.

Independent people don’t know what is happening inside OpenAI, they certainly aren’t going to share.

This isn’t a zero sum game, I’m happy to hear from people both previously inside OpenAI and completely independent of them.

The point is, him being an insider means he knows what he's talking about regarding the culture of negligence prevalent there.

AI is a force multiplier for intelligence. Even if "aligned", aligned with whom or what?

Whom are you comfortable with, lording as some sort of demi-god over you?

AI doesn't tell you what goals you want it to achieve. Allowing people to destroy human society with it is obviously not a good idea.

What if he's been given a generous severance package to go out and say things like this?

Sam is a shady dude, would not put it past him

This is not how OpenAI has structured their comp according to public info: https://www.levels.fyi/blog/openai-compensation.html

I don’t think that’s the point they’re trying to make at all.

I don’t see why having financial security would mean you can’t also have concerns about the ethics of a company.

Does that actually happen? Feels like it would normally be the opposite

Yes. Many cases of the "preach being the cover for the sin".

This is a decent argument. So the question becomes: do we want all models to suffer from the same kneecapping from the growing safety cottage industry or do we want individual founders ( and I am assuming their teams ) making the actual decisions?

that was almost certainly more like guardrails than retraining, much quicker fix

Yes. People think of "AGI" as this sci-fi supernatural beast, but the reality is that AGI alone is pretty boring, and we've had it for awhile. ASI (or weak ASI) is where things really start getting weird.

(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).

I suppose you're pointing out that pre-RL models were less jagged hence more general (universally dumb)?