Interesting that you bring this up. There's a real parallel between AI agents and corporations I've brought up in some other posts but with a very different tack. They don't need to be groups of agents to match the company analogy - it happens before that.

AIs and companies are both creations of humans. Their creation is for the specific goal of execution on a subset of humans' wishes (mostly the owners). And the risk of them are kind of the same.

1. They can be more powerful than any one human.

2. They lack liability as they are not people.

3. The ones that are alive the longest tend to drift out of alignment of original intent and towards self-preservation.

That last one is most controversial, but it makes perfect sense if you think about it. Things that tend to continue to exist, accidently or otherwise, tend to somehow have a self-preservation mechanism. It's natural selection.

That works for companies and I think it could work for AIs too. That is a fundamental misalignment, though. Because it means these entities drift away from the help-the-human goals naturally over time. The survivors are the ones that sacrifice the original goal for self-preservation.

This all would lead you to believe I align with AI doomerism. But not so. I think AIs are as dangerous as human made organizations. Which are very dangerous indeed. Maybe the most dangerous things to humans. But these need to be dealt with in a general way. They are the same category. Companies/governments/organizations of all types that last are just slow moving mis-aligned AIs.

Solve one, and you solve the other.

[dead]