If anything, his entire argument leads to the eventual premise that AI labs need to nationalize.

He is already arguing that commercial competition creates dangerous incentives, and that labs cannot slow down because competitors may overtake them. He also asks for what would boil down to significant government intervention to preserve a Western lead.

If this is true, why preserve the commercial incentive? The U.S. government would already have the necessary goal of remaining ahead of China and other competitors. Why not remove entirely domestic race for market share, valuation, and investor returns rather than keeping private labs in competition and then asking regulators to counterbalance the incentives that competition creates.

That doesn't mean nationalization would be better, but given the severity of the risks he describes to national security, it seems like an obvious conclusion that nationalization is the eventual end result of the argument.

The potential of AI may give us the opportunity to graduate to a post-capitalism, or to move to a sort of neo-capitalism, which is governed by different rules.

Capitalism thrives in the realm where there is a scarcity of resources, either physical or informational. Imagine breakthroughs in energy science such that the cost of energy drops to zero, which means the cost of physical resources declines precipitously. Ok so where is the capital now? Must we retain a model based on the premise of scarce physical capital?

They want to spend over $200B/year on $100B of revenue so they still need investment.

That level of revenue is apparently not enough for them to feel confident in the stability of their competitive position.

You also IPO if your investors want to cash out.

Don't underestimate the ability of the courts and the state to pull the rug out from under the legal system. Speaking as an outsider from Canada, it looks very much like all bets are off in terms of respecting checks and balances in the United States and it's very realistically possible that unless there is a big upset and turn around the next couple of years any traces of democracy in the US will be a farcical nod to what the founders built as a way to paper over the abuses.

I really hope I am wrong as my perspective is not anti-American, it's specifically anti-corruption and pro-democracy.

As long you are burning more cash than you bring in, you are beholden to investors - whether it is retail, VCs, banks or a government giving you a bailout it is still someone signing you a check.

Golden shares, vetos, PBC, charter are all paper tigers , they only matter if/when the firm is self-sustaining business with no outside capital needed, the alternative to not listening to investors till then is crash and burn.

After that point, you will have to listen to the paying customers (sometimes but not always they are also users ) as they are ones now funding your organization.

Bottom line you are always listening to someone.

google dropped "dont be evil" despite being the definition of the company

these are just words at a time and place.

sama showed that you can futz with it, and as long as you spend enough in court on judges, aint nobody gonna stop you

From Jakub Pachocki, chief scientist at OpenAI last week: https://openai.com/index/an-alien-mind/

> Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.

> are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction

This was already the case even before the "frontier AI" age. The big question is, will these glaring AI arms-race threats be obvious to enough people to rethink the underlying systemic flaw driving it all?

Not holding my breath on that one.