> If the behavior of the system is specified well enough

Then it becomes code: a precise symbolic representation of a process that can be unambiguously interpreted by a computer. If there is ambiguity, then that will be unsuitable for many systems.

The word “many” is carrying a lot of weight here. Given the probabilistic nature of AI I suspect that systems that are 98% correct will be just fine for all but the “this plane will crash” or “this person will get cancer” use cases. If the recreation of the system failed in that 2% by slightly annoying some end user, who gives a shit? If the stakes are low, and indeed they are for a large majority of software use cases, probabilistic approximation of everyone’s open source will do just fine.

If you’re worried about them achieving the 98%, worry no more, due to the probabilistic nature it will eventually converge on 9’s. Just keep sending the system through the probabilistic machine until it reaches your desired level of nines