It's definitely that publishers don't want it.

This is actually the trajectory of both Apple News and iAd before it, which is what started out providing the ad service for Apple News. Apple would like to do a high quality solution, and then keeps relaxing their standards when there's not enough buy-in from the content providers. They were forced to allow the non-curated news formats to have sufficient content.

I wonder why they don't just prioritize the ~500 most popular of those content providers that are feeding them sludge articles, and write (AI-generate, even) logic to manually parse and transform said sludge into their format?

It'd be a big one-time lift; and of course there'd be annoying constant breakage to fix as sites update; but News.app could always fall back to rendering the original article URL if the News backend service's currently-deployed parser-transformer for a given site failed on the given article. It's make things no worse and often better than they are today.

I can't imagine it's a great deal for publishers. It's probably why NYT, Economist and other prestige publications aren't on it. (Save for Atlantic, New Yorker). I. Assuming they use the Spotify model ( paying commissions on articles per reader)?