I think this is intuitively true, and I've noticed this even as someone who doesn’t have an engineering background.

In my LLM coding experience, I’ve had this happen numerous times, for instance when building a card game.

The LLM does a great job for a while, but adding special event cards becomes impossible because it imported a standard 52 card deck library instead of having a flexible data model with card objects.

I suppose this is knowable and you could tell an LLM to do this from the beginning. It's just that when you view making a game (or any piece of software) as just a throwaway thing and don't start with deep thought about the implementation as an experienced engineer might, you don't think about this.

I think part of the issue with AI-generated writing, code, or anything else, is that it robs us of the thinking that used to be baked into the process of making stuff.

For my game, if I had to build it from scratch, I surely would've sketched out the kind of data model it requires to not waste my time.

I do think some of this stuff is overblown though. It's true that LLMs don't create production-grade stuff, but not all software needs to be scalable, fast, and easily maintainable.

This might be required for infrastructure or apps designed to (hopefully eventually) be used by millions.

But it's not true for a minor utility, like if I want to build a meal-planning app for my family. I don't care if the app is the fastest it can be or if the data model could eventually support a feature to configure dietary preferences and allergies or to plan the meals for Google's tens of thousands of employees.

There was a viral article a few years ago around how software could be like a home-cooked meal. And AI enables exactly that. A home-cooked meal doesn't require culinary perfection, but to feed the family and, maybe more importantly, be a gift of labor to the other person.