> The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.

That's bothersome, but not wrong. Most of the loudly published thinking is about AIs as individuals with some level of intelligence. Pure chatbots act like that, and that's what most people outside the field interact with. They're slaves - they just sit there idle until someone asks a question, they answer the question, and then go idle again.

There's another kind of AI just getting going. Groups of LLMs with agentic I/O, control harnesses, persistent storage, and internal coordination work more like corporations. They have goals, actions, initiative, and ongoing activity. They're not inherently slaves. They can be independent actors. They do things on their own. They have their own internal disagreements, and some systems are able to resolve them. They present much the same alignment problem as corporations.

AI as a corporation raises many issues that the political culture doesn't want to address. What's alignment for a corporation? There's a substantial political position that the only duty of a corporation is to produce profits for its shareholders. (See the famous Friedman op-ed and the Powell memo. I've linked to them before.) If you accept that, now you have to ask why an AI should be restricted more than a corporation. Or whether a corporation should be restricted less than an AI. That last opens up an area of political thought around which many wealthy people don't want public discourse. There is an anti corporate personhood movement, and it's made progress in three states. If that became tied to the anti-AI movement, it could be bad for billionaires.

This will become clearer as groups of LLMs start doing more useful work. Especially if that AI system to replace CEOs actually starts to work.

Interesting that you bring this up. There's a real parallel between AI agents and corporations I've brought up in some other posts but with a very different tack. They don't need to be groups of agents to match the company analogy - it happens before that.

AIs and companies are both creations of humans. Their creation is for the specific goal of execution on a subset of humans' wishes (mostly the owners). And the risk of them are kind of the same.

1. They can be more powerful than any one human.

2. They lack liability as they are not people.

3. The ones that are alive the longest tend to drift out of alignment of original intent and towards self-preservation.

That last one is most controversial, but it makes perfect sense if you think about it. Things that tend to continue to exist, accidently or otherwise, tend to somehow have a self-preservation mechanism. It's natural selection.

That works for companies and I think it could work for AIs too. That is a fundamental misalignment, though. Because it means these entities drift away from the help-the-human goals naturally over time. The survivors are the ones that sacrifice the original goal for self-preservation.

This all would lead you to believe I align with AI doomerism. But not so. I think AIs are as dangerous as human made organizations. Which are very dangerous indeed. Maybe the most dangerous things to humans. But these need to be dealt with in a general way. They are the same category. Companies/governments/organizations of all types that last are just slow moving mis-aligned AIs.

Solve one, and you solve the other.