I think about this a lot. I started my teenage-to-young-adult life in the literary world as a poet who loved programming, and made a living as a ghostwriter. At some point, I fell in love with mathematics and wound up in ML in research/engineering for the last 8 years or so. So, I've thought a lot about writing and ML and their intersection.

I think one of the underappreciated things about writing "substantive" work is that the work you see at the end isn't the first attempt. And I don't mean the first draft of the piece. I mean that almost always, writers iterate on the same topic many times, either with complete published pieces or abandoned drafts or even just conversations and sessions of unproductive daydreaming. It's a cliche that your best work typically also comes out fastest, but it's not because of divine inspiration, it's because you've whittled the big idea you actually care about down so much in your mind that you instinctively know exactly how to write it.

My experience has been that for people who don't work this way or don't write a lot, LLMs can give them this incredible feeling of leaping straight from inkling to "substantive" writing. And because they haven't built up those muscles or "taste", they don't immediately recognize that it's imprecise and hard to follow.

That's not to say they're not brilliant in their own right, just that they haven't spent a lot of time on this particular thing. Sort of like a very gifted programmer who doesn't have a ton of experience yet (speaking as someone who is gifted at nothing and frequently has to do things they're inexperienced at).

So I don't think the problem is that LLMs just write bad. It's that LLMs are so wonderfully powerful that they allow you to confidently leap forward to create something that is a little beyond your experience.

And that's why I have different reactions to heavily AI generated writing. When it feels like marketing at scale, it grosses me out. But when I feel like it's just someone who is excited to write an idea and maybe doesn't have a lot of experience doing it, I'm not judgemental. My hope is that it makes them more excited about writing, and that trying to make their next piece better will lead them inevitably to start thinking about where the last piece fell short. And if there's some slop along the way, eh, I'm not compelled to read it.