Suppose the upside is as Dario claims, very positive. Why should frontier companies be the ones controlling access to the model weights and gatekeeping research? Shouldn't you be giving it to everybody as quickly as possible?

It's my understanding that Dario has two answers to this question: 1) bio/cyber/misalignment risk. 2) a democracy should gatekeep frontier capabilities from authoritarian adversaries, due to the aforementioned risks in 1). So it's really just one reason.

There's a big problem with that, and I'm not sure if he genuinely does not understand: the bio/cyber/misalignment risk is precisely the highest when you concentrate ownership of compute and general intelligence in the hands of a few.

Consider bio/cyber risk, or more generally, any sort of novel adversarial action that arises from increased intelligence. Those offensive/defensive strategies are always symmetrical in the long run, otherwise we would not observe more intelligent species being more prosperous as a general evolutionary trend, or perhaps intelligence would not have evolved at all on this planet. It looks as if asymmetry in existing bio/cyber strategies will continue into the future, but that's because we're currently too stupid to realize the countermeasures and new cooperative dynamics.

Regarding misalignment risk, consider what "aligned" means. Is it not alignment to humanity, as the company name "Anthropic" suggests? Or are you modifying humanity's objective function with your own aesthetic preferences? If it's not your intention to do the latter, then releasing the model weights and research results will create a much more aligned scenario than access control and research behind closed doors. I explained why that would be the case in an older comment, if you'd like to read it: https://news.ycombinator.com/item?id=49313739

There's also an additional risk that I believe Dario overlooks: a democracy can only continue to exist if individual citizens are capable of keeping eachother in check, especially keeping people in government and large corporations in check. When some people own AGI but others do not, there will be no more democracy. And by "own", I mean ownership that cannot be taken away: possessing the hardware, research papers, inference stack, and weights.

As soon as anyone starts talking about alignment, my question is why we don’t seem to be paying nearly as much attention to alignment when it comes to CEOs, billionaires, and national politicians.

The answer is of course that those people are exactly the people using “alignment” as yet another red herring to divert attention away from themselves and their actions.