Interesting methodology. The results for Pangram are surprisingly good.

These tools are definitely not 100% perfect, which is the primary complaint used to dismiss them. However the error rate is also getting impressively low.

In most cases I see socially and online, there is a high suspicion that the content is AI generated before someone thinks to submit it to Pangram. It’s used on-demand as a tool to confirm suspicions. I have seen several cases where Pangram had some false negatives where the content was judged to be likely human written but the author later admitted it was written by an LLM.

Pangram is very interesting in the context of Substack because the platform was a target for lazy AI newsletters. People realized they could start 10 (or maybe many more) newsletters and spend only a few minutes getting ChatGPT to write posts for them. Starting an AI generated substack and trying to get paid subscribers for it was becoming one of the popular ways to use AI to try to get a little cash. Having a tool that makes it a little bit harder, at least until the LLMs get good enough to evade it, was important for the platform.

▲

vidarh 5 hours ago | parent | next [-]

I think outside of a few niche uses, e.g. schools/universities, services like Pangram will struggle.

Currently people care because so much of AI output is still poor quality. But given I personally know of someone with published AI-generated works where the reviews are gushing and positive across the board, I doubt people will care whether works are AI generated for very long vs whether the text is good.

A way of certifying a work as above a certain literary quality would likely have far more longevity, as I doubt the market of people who care whether a given work is AI or not irrespective of quality will be particularly large.

▲

SamoyedFurFluff 4 hours ago | parent | next [-]

I actually disagree, I think B people care a lot if a created work is an intended artistic thing vs produced by a machine. It’s like the difference between getting someone a generic hallmark cards vs actually writing something inside. Theres a place for the hallmarks but a plain one with no human touch is still considered tacky

	▲	vidarh 4 hours ago \| parent [-]
		Survey people about who created a given work, and you'll find a whole lot of people who like something without having the faintest clue who made it, or anything about why, before even considering the people who can even name the people involved. It's hard to believe that suggest people are so quick to care about the artistic merit. As a concrete example relating to books, the success of the Stratemeyer Syndicate in packaging up book series ghostwritten by "nameless" authors under single pseudonyms is an example. Stratemeyer series like Nancy Drew has sold more than 80 million copies (for Nancy Drew alone). At one point 98% of American children listed a Stratemeyer title as their most popular books. How many of those do you think saw those works - most sold as a work for hire for next to nothing and churned out by the hundreds - as "an intended artistic thing"? People who care a lot about literature tends to seriously overestimate how readers engage with writing. As it stands it's hard to get people to care enough to read, much less to get them to care about artistic intent. I think people who bet on people caring about that as a means to stem the tide of AI content will be very disappointed.

▲

bakugo 2 hours ago | parent | prev [-]

> Currently people care because so much of AI output is still poor quality. But given I personally know of someone with published AI-generated works where the reviews are gushing and positive across the board, I doubt people will care whether works are AI generated for very long vs whether the text is good.

I think this has less to do with the quality of the text and more to do with whether people have standards or not.

For every person who reads a paragraph of AI text and is instantly disgusted, there are probably 10 or more people who will gladly read pages upon pages of the sloppiest slop imaginable copy-pasted straight from chatgpt.com without batting an eye. That's just how things are, most people have very low standards.

Getting any of the current codemaxxed models to output natural text without obvious AI-isms is not an easy feat, and I doubt this person you mentioned has somehow cracked the case.

	▲	vidarh 2 hours ago \| parent [-]
		I've seen the text, and it was genre typical. You get 80% of the way there just by giving them a writing sample and asking them to copy the style paragraph by paragraph. You get a lot further with some basic statistics to look for seriously out of distribution word use. The hard problem isn't prose, but consistency over a long text. I don't know how he's achieved that, but I've seen the reviews and none complained about consistency issues, which readers tend to be sticklers about.

▲

NoahZuniga 5 hours ago | parent | prev [-]

> I have seen several cases where Pangram had some false negatives

Yep, Pangram says they think false positives are way worse than false negatives, so if their AI isn't sure enough they prefer a "not ai" output to an "is ai" output.