The problem is the "interface" of software engineering being oblique to outsiders. It's alchemy in essence.

The logical extrapolation of AI capabilities stress-testing human software is all bugs being discoverable. Hence, you have to aim for actually bug-free software.

The only way to do that is to make your code provable. That again is only realistically doable using AI.

In the process, people will have to turn the current mere pretense of software engineering into reality, making the engineering part explicit.

That in turn only works when humans actually understand what they're doing and can communicate it to outsiders.