I've been thinking along similar lines recently. It seems to me that the earlier form of nonhuman intelligence is a corporation (or any group which is not under one individual's control.) Each person following their own incentives can result in emergent behavior where the corporation follows its own incentives. If we can't reliably "align" corporations, and corporations create AI, will AI be aligned as well?

It's not totally disconnected, though. A corporation is constrained by what its members are willing to do. I read that the reason Gemini has fallen behind is because Googlers have been quitting over Google's DoD contract. Conversely, if engineers are willing to work hard for you, you can go far. I suspect this is behind both Anthropic's success and also Tesla/SpaceX.