> the rational thing for these rational competitors to do would be to agree

This is the bit that is tripping you up. The market is composed of more than just the these potentially agreeing parties, so there is no case where all parties can even come together to agree. It's rational for any party to break the agreement or be stealthy and attempt to take the upper hand while the visible players are all patting themselves on the back for their self-inflicted restraint. It's rational for one party to encourage everyone to come together to agree, to distract from their own actions to undercut everyone else.

On top of that, such agreement would potentially qualify as collusion, allowing an upstart to cry foul.

It's all game theory.

then it isn't possible in that situation to limit ai deployment, so limiting ai deployment can't be optimal because it's impossible.