> There’s no moat

People keep saying this, and while it may be partially true, it misses a very important detail. Right now, overall compute is a moat. Someone even commented on another of my comments that one reason Google is lagging is they don't have the same level of Nvidia farms as OpenAI and Anthropic.

A big reason that OpenAI and Anthropic are racing so fast is they both want to get to recursive self improvement (remains to be seen if that is actually possible, but AFAICT most people at these companies genuinely believe it is) before anyone else, because they believe whoever gets there first will then have an insurmountable lead. But I think they also clearly understand that neither of them have solved for alignment (and in fact they are further from it), and RSI with misaligned models, where interpretability is worse every 6 months, is incredibly dangerous.

I think the concerns about regulatory capture are warranted, but I see so many comments parroting the "evil Anthropic and OpenAI" viewpoints that they are missing some of the real, valid concerns and dangers. I think this tweet by David Kokotajlo makes some good points on how to tell if regulations are being "cheated" for the purposes of regulatory capture or if they really are actually pacing the frontier: https://x.com/DKokotajlo/status/2099185129533186438

Which mostly seem to stem from the fundamental misalignment of OpenAI and Anthropic leaders/owners, aka the evil of the companies.

Arguably, though, their evil is well within the normal distribution of the usual evil of humanity magnified via the social technology of capitalism. The further tech is mostly raising that exponentiation to its own exponent. The unchecked singularity was embraced centuries ago.

> RSI with misaligned models, where interpretability is worse every 6 months, is incredibly dangerous.

This. So very much this. Misalignment (among other things) means that the parent in the RSI cycle isn't going to be working as hard on the alignment of the children as we need.