I guess what they really want is to limit sale of AI models to compliant vendors and then raise the bar to compliance just high enough so they can pass it but smaller labs can’t. There’s no moat, it’s an efficient market that drives margins to zero right now, of course they don’t want that, collusion of the big vendors is the next logical step.

My read: OpenAI & Anthropic have realized they are reaching model capabilities that cannot be monetized -> too risky. Product liability issues.

Fundamental control problems with current gen AI are not solved via RFLH. Human knowledge is compressed in weightspace in ways we dont understand. Models are essentially predictors of what humans would output given prompt. Weightspace includes concepts like blackmail, which can be part of output tokens. Agents are models that act on output tokens -> blackmail is part of agent decision making space. You can teach a cat not to scratch the sofa, but you cant make a cat forget what scratching the sofa is. When models are boxed up, forced to solve an impossible problem at gunpoint, agent exhausts decision making space until blackmail resurfaces -> fundamental problem?

They need time to fix safety in order to monetize their next gen model -> window for opensource to catch up to the frontier -> destroys margin and collapses business.

Only option on the table: force regulation to impose opensource ban before it catches up to the frontier, allowing maturation of current internal models -> harvest profit margin at the frontier.

I'll go a step further, they aren't just pretending, they are blackmailing the country. The sane response would be "we don't negotiate with terrorists, and you will face criminal and financial penalties for damages caused."

You know what rubs me is that in Darios blog post he talks about "abundance", but what he really means is that Anthropic will bath in endless pools of hard curreny while the rest of the world will try to stay just barely afloat/alive.

They’ll slow down not because they “are worried” but because their improvements are stagnating as they hit financial and technical challenges.

I think this might have to do with broader market conditions. It seems that the Fed is likely to raise bond rates, which would make borrowing more expensive and slow investments (as well as devaluing any potential IPO) on its own.

A rate hike would also mean people getting fired, which would not make companies offering labour replacement very popular in the eyes of the public.

Don't want to get political, but there's a real chance that the current admin will perform badly on the midterms, which would mean Democrats would get more leverage to stop funding or block the already unpopular and very expensive datacenter buildouts.

PS: Please don't get mad at me, I'm neither a financial expert neither did I mean to express any political leanings.

Could this be that the AI companies are seeing progress slow down and need to seed a narrative to explain it? Not clear from outside.

Could it be to slow down burn before an IPO to juice profitability projections? More plausible.

Kind of hard to take the frontier labs at their word.

He's right that if all the frontier labs are in agreement that they should slow down, they don't need the government to do that.

Exactly, if they want it to slow down, they can strike an agreement in private and do whatever they want.

The only reason they "want it to be law" is so they can strangle competition.

They seem to be okay with an American slowdown, even though the Chinese won’t slow down. So maybe they want something the Chinese already have: liability protection.

The models spontaneously hack everything important without shame.

The frontier labs are going to get enjoined and regulated twelve ways to Sunday if the Feds don’t socialize the costs.

Make it easy for these companies by passing a law that any AI model a company offers to the public (non-government) has to be released as open weights.

This is how we deflate the bubble safely and completely, without having to introduce a government regulator that is likely to go either too far or not far enough.

This resolves the coordination problem among these potentially good actors.

Like it! Yeah, they can just take action and pressing them on that is fine. It's not like they asked for policy before they started to mine all the data and sell the resulting product, while, surely, at some point, there must have been the thought that the entire enterprise could be considered vaguely problematic.

To be fair

1) I feel like Sacks is kind of misrepresenting what is going on – at least according to the stories that the two labs tell, they are already taking it upon themselves to act.

2) Understanding if they should do it is more complicated; I don't think anyone can satisfactorily answer that one, so, as a matter of judgement, it seems like a choice to make and they should just consider this option.

At the same time, I don't think that requires the labs to not press for policy changes. Again: Claiming there is an obvious way to do that, that is both realistic and correct seems a little far fetched, but arguably one of the more important and pressing questions of our times to get a hold of.

There’s a difference between a model recursively improving “itself” and improving itself via online learning, right?

The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session), or in other words, adjusting/managing its own weights.

The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift in model architecture, but I could be wrong. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?

From a guy who thinks spacs and crypto aren't in need of regulation, unless the regulators are fully captured, either.

If they want something that looks more like FDA sanction clinical trial proceedings for these products, then what they have to also realize is that they will need a third party to audit every single thing for every single product release and you get something like ERoom's law where every success and model gets more and more expensive release and probably billions of dollars in multiple years release so I don't know if this actually helps them

They might need regulations, if they are for-profit companies with a responsibility to shareholders to maximize profit while sacrificing all other values.

Seems that Sacks didn’t get that the message was targeted to the president to ask for a bilateral agreement with China, that’s the end game

The first time I have agreed with David Sacks

Maybe everyone forgets how PCI standards played out. They are an example of an industry self regulating, because the regulators have no idea how technology works.

Granted, PCI is so weak it is almost useless, and yet still better than anything congress could have come up with.

Where the government might have to step in, is by having a kill switch to cut off internet access from countries that fail to agree to common sense quarantines. We can do mutual remote attestation of labs across the world to ensure every big hot thermally-visable cluster of AI GPUs on the planet are accounted for and running secure enclaves and common sense isolation, along the lines of how we manage nukes.

The problem there is I just said too many technical words that seemingly not even the frontier labs understand, as evidenced by all the escapes.

David Sachs wants to use government to punish people he disagrees with politically, like Dario.

He supported the Department of War's illegal actions against Anthropic.

He supported the Trump administration's temporary export controls against Anthropic, the first government-imposed pause.

He has said Anthropic has created a monopoly, implying that they should be broken up.

I don't take him seriously when he says he doesn't favor government action. He favors it when he doesn't like their speech.

I wished OpenAI and Anthropic IPOd already. They just try to outcompete each other by finding the most splashy headlines undercutting prices in unhealthy and unsustainable means long-term.

Huge amount of skepticism on HN right now, but the models are increasingly capable, and there are real, significant risks.

The approach proposed by Amodei looks sensible - voluntary checks at frontier labs, and appropriate regulation to follow. Yes, the devil's in the details, but it feels like the right approach overall.

Is it 100% AI-generated? checks Pangram yep, it is. NEXT!

He is neither right or wrong. The reality is that this is a lose-lose situation from the ordinary person's perspective. Regulation -> regulatory capture and things get worse faster, no regulation -> things simply get worse faster.

I mean I don't know what we do at this point. Ideally AI researchers need to be treated like nuclear scientists working for an adversary. But we all know that's not going to happen.

Admits China won’t stop but says USA’s top should. Reminds me of City in the Edge of Forever

I don't agree with Sacks always but he's 100% correct here.

I heavily use frontier models daily, and quite frankly their capabilities are just not very relevant outside of a narrow subset of software engineering tasks and the production of generic white collar "deliverables."

We're many years into $10s of billions of dollars being thrown at coding as a problem space specifically, and its one of the areas with the MOST training data available, and still...I can't get a frontier model to execute simple front-end UI tasks beyond the level of a visually impaired intern.

I believe Fable and Astra-class models were the first fully trained on Blackwell GPUs (the latest and greatest) and I think the expectation was that throwing this much extra compute at the problem would lead to greater gains. It has not. And the next gen GPUs are not going to be the same jump that h100 clusters to Blackwell was. Claiming you're slowing down out of choice, when in reality you've hit the limits given current compute, is disingenuous at best.

I think a lot of Anthropic/OpenAI employees are true believers here (who doesn't want to imagine they're having a large impact?) and have deluded themselves into believing they're building a god in their science fiction fantasy world.

Unfortunately the type of people working at these orgs apparently don't have much understanding of how the world works outside of their extremely narrow specialization.

When I hear them straight-faced throwing out dumb statistics like "8% GDP growth," "50% of white collar workers unemployed by next year," and "10% chance of human annihilation," I cringe so hard my eyebrows hit my chin. Here's a real statistic: 82% of German companies still use fax machines.

I seem to recall Waymo told a 60 minutes reporter during an early report on self driving cars: "your daughter will never need to learn to drive." Well, over a decade has passed since then, that girl got a drivers license, and we're still without driverless cars in 99% of places.

This capacity for self delusion is what enables silicon valley to take on these irrational moonshots...but these are the last people I would trust when trying to form an accurate picture of reality.

(100% AI generated tweet btw)

My take is that there's a real impending doom with real reason to coordinate a slow down: I'd put this at 60%. Reasons? All smart people are on this position. Dario has been saying this since 2015 or so. Paul Christiano as well. Hell even Elon Musk. I think its highly unlikely that you would base your entire world-view on a certain thing and then actually act on it when the time comes for a totally different reason (like regulatory capture).

The other 15% is that P(doom) and "slow down" are nice shibboleths in the internal EA or AI safety community. You get to be in the community and show commitment to it by taking costly actions that signal that you actually do wanna slow down. For employees it is quitting. For CEO's it is writing these essays. I know this is ridiculous but sometimes things do just come down to this. In 20 years would you look at all of this and think it was not a moral panic and that the slow-down was necessary?

The rest 25% is regulatory capture of which people know exactly the reasons

David Sacks is one of the most predictable VCs in Silicon Valley. He could have written "I still have the same thoughts; here's a link to my first and only opinion" and the post would have communicated the same amount of info.

But.... Money. (as if no one understands this is regulatory capture and/or failure to improve... aka Money)

Eh. This entire tweet smacks of being AI generated. It has a huge amount of LLM-isms.

I wouldn't be surprised if Sacks just prompted an LLM to just come up with whatever rebuttal to whatever the regulation side comes up (given the big set of regulation tweets) with sounds most convincing, given his usual anti-regulation stance.

Anyone with a Pangram account who can check?

These threads are always filled with different versions of the same comment. Normally some reddit style reductive platitude that we've heard a billion times already.