Let me give you an example I just ran into now which illustrates why #1 is so pernicious.

I was just linked, by an intelligent, educated, skeptical person who regularly uses AI generative tools, to an interesting report on the darknet markets (which I used to be an expert on): https://tornews.org/legendary-darknet-vendor-crimsonlotus-ta... I was not familiar with this site, but I figured it was probably one of many successors to DeepDotWeb, and I've been out of the scene for a long time and wouldn't know about it; but I eventually realized, after several minutes of reading and then checking, that this was an AI slop website. This is because 'Obsidian Bay' doesn't appear to be real, the photo is obviously AI-generated, and the closing commentary by 'Ella Vargas, darknet researcher at the Cyber Crimes Institute' appears to be a fake person at a non-existent organization. Is anything in the article real? Probably not.

Ahhhhh, but you say you don't care about it being real, you say you only care about the prompt - in which case you're fine: some things in it are old events relabeled. (eg. there really was a large darknet market which was famous and had eluded LE and was taken down in a global operation in part due to packet timing attacks! You did learn something true from that accurate prompt/input yielding that slop output! It was just called 'Silk Road 2' and that happened over a decade ago.) So, I guess you have no issue with tornews.org. Maybe you should subscribe. All these articles sound quite exciting and there's a lot of them...

Personally, I think it's a bad site. I could have fallen for it - I simply got lucky that this one was so easy to note the style, factcheck, and debunk. For example, I could have instead read https://tornews.org/massive-dark-web-drug-operation-busted-i... : if you read this, this sounds very plausible and exciting, and it doesn't immediately come off as AI slop because it is so detailed and doesn't sound too much like 4o.

This, it turns out, is because it's based on a real bust: https://www.suffolkcountyda.org/suffolk-county-district-atto... https://www.newsday.com/long-island/fentanyl-bust-suffolk-k7... Heavily rewritten to obscure the sources being plagiarized. How much of it is real? Well, the assertions I spotchecked seemed to be real (ie. copied without too much distortion from the police press conference)... but I have no idea about the rest of it, and this article lends credibility to the other articles.

(Why does this week-old site exist at all and even has a Twitter account? https://x.com/tornews_org It's almost certainly either for the affiliate marketing revenue, in the best-case scenario, or is a phishing scam, in the worst-case scenario. Since it's so young and so reliant on fake content, likely the latter. I didn't check any of the onion links they so helpfully provide, but even if they are all legit right now, that simply means they are not yet scamming. To pay for the AI and domain name and labor here, you only need to phish one DNM user who will deposit a few hundred dollars of cryptocurrency. And this site has already started to pollute Google with its 'facts'.)

> What I can't figure out is why you seem so confident that the OP didn't verify the LLM output and/or would have published anything written by the model, whether it was faulty or not (which again, in this case it wasn't).

Because I routinely catch people, often very intelligent and educated people, confidently posting LLM materials that they have not factchecked and which are wrong (such as, say, about history, where they might make an argument about human evolution based on cave-dwelling where 4o was merely off by a few hundred thousand years), and I still find confabulations in my own use of even frontier models like o1-pro or Gemini-2.5-pro (leaving aside o3's infamous level of crimes, which I think is probably unrepresentative of reasoning models and idiosyncratic to it).

And in this case, the prompt looks like it was very low quality. The author had plenty of chances to put in details from his own real experience as an actual Iranian - any intelligent, observant college undergrad routinely buying coffee ought to have plenty to say - and instead, it's super-vague waffle that... well... an LLM could have written about just about any country, swapping out a few clauses. ("Why drinking coffee in America has become so complicated")

> You're clearly allergic to basic LLM-style

This is not a 'basic' LLM style. It is a very specific, recent chatbot style. (Note that visarga was able to instantly tell it was the recent 4o, because that style is so distinct compared to the previous 4o - never mind Claude-3, Gemini, Llama, Grok etc.)

Further, there should be no single 'LLM-style'; it makes me sad how much LLM writing capability has been collapsed and degraded by RLHF tuning. Even my char-RNN outputs from 2015, never mind GPT-3-base in 2020, showed more occasional sparks of flair than a 2023 ChatGPT did.

> or at least the masquerading of LLM text as human, so I'm curious what you'd consider worse: 1. LLM-generated text reflecting an accurate prompt/input, or 2. genuine human BS wanting to be taken seriously? (e.g. The Areas of My Expertise by John Hodgman if it wasn't in jest)

#1 is worse (if unedited/factchecked/improved etc and just dumped out raw), because there will be much more of it and the intermingling of fact and fiction makes it harder to factcheck, harder to screen out of future training corpuses, and overall more insidious. Human BS serves as costly proof-of-work and because it is costly, once you recognize you are reading BS from someone like Elon Musk or Sam Altman, you can switch modes and ignore the factual content and ask, 'why is he writing this? what purpose does this BS serve? who is the audience here and how are they using it?' and get something quite useful out of it. I have learned a lot from statements by humans where little or none of it was factually true. Whereas a LLM output may mean nothing more than 'some unattended code spent $0.0001 to spam social media with outputs from a canned prompt', if even that.