We always talk about damage, what's the damage here?

You deploy an application that ostensibly has the desired capabilities but it fails at scale, or the architecture is not extensible to add new capabilities or fix bugs. Those are expensive mistakes and in safety-critical domains could be deadly.

The thing is I've worked on plenty of code like this written by humans. AI just accelerates the process and puts this type of code within reach of more people. It could be seen as a step backwards in good software engineering practices we've painfully learned over the last two decades. I see it as a cycle repeating itself in a new iteration, with the hope that we'll end up with still better tools and processes in another few years. But I agree it can and will result in real harm in the meantime.

Code that accidentally ships every 100th order out for free... exploits releasing customer PID and causing regulatory nightmares... self-driving cars that reboot at 70mph... The only limit is your imagination!

A giant pile of unmaintainable, brittle code nobody understands. At some point, even LLMs struggle to make changes without introducing bugs.

For example, doing a marketing push assuming that the app works, and then having it fail in trivial ways at release.

Waste of time and therefore money is a big one.

The next decade of software consulting will be defined by repairing the Big Ball of Mud https://www.laputan.org/mud/