You deploy an application that ostensibly has the desired capabilities but it fails at scale, or the architecture is not extensible to add new capabilities or fix bugs. Those are expensive mistakes and in safety-critical domains could be deadly.

The thing is I've worked on plenty of code like this written by humans. AI just accelerates the process and puts this type of code within reach of more people. It could be seen as a step backwards in good software engineering practices we've painfully learned over the last two decades. I see it as a cycle repeating itself in a new iteration, with the hope that we'll end up with still better tools and processes in another few years. But I agree it can and will result in real harm in the meantime.