Would you say you’re able to draw a diagram of the application architecture out of your head or do you treat it as a black box? Do you need an AI to debug issues or not? In my experience with spec driven development, even if reviewing every single PR, it is hard to develop a mental model of the codebase structure unless you invest on it. It might be fine to treat it as a black box, not arguing the opposite but will all software be a black box in the future?

For a completely new project it is a high risk. While the AI is fantastic at brainstorming and writing detailed architecture, it is difficult to get the "big picture" and even more difficult to verify that it is being done correctly or which things can be improved/reused, because on this situation you don't look into the code.

I don't believe people will spend time looking at the code beyond the small blurbs they can read from the command line while talking with the AI, so I agree with you that it ends being treated as a blackbox.

Did an experiment for a server implementation in Java (my strong language), gave the usual instructions and built up the server. When I went to look into the code, it was a far smaller and more concise code base than what I would write myself. AI is treating programming language on the level of a compiler for javascript, it will make the instructions super efficient and uses techniques that on my 30 years experience I'm not able to pair-review because we tend to have our own patterns of programming while these tools use everything, no matter how exotic they will use it to their advantage.

After that experience I don't look at generated source code any longer. For me it is becoming the same as trying to look at compiled binary data.