Remix.run Logo
breezybottom 8 hours ago

That's about as useful as saying you asked the magical sky fairy.

meowface 7 hours ago | parent | next [-]

Pangram has an extremely low false positive rate. Even on adversarial examples.

One trade-off is even some obviously LLM text won't get detected by them, but they work really hard to ensure false positives are rare since a false accusation is much worse for society than someone getting away with LLM meatpuppetry.

jchw 8 hours ago | parent | prev [-]

I think you can't trust Pangram in a high stakes situation, but it is absolutely better than random noise at detecting AI-generated text. Which isn't surprising. If the distribution of probabilities can yield blatant Claudisms, it's not surprising it would also have more subtle deviations.

(Addendum: As I recall, LLM-generated outputs roughly follow Zipf's law, but the distribution still tends to have some subtle distinctions vs human text; pretty interesting, but I don't know where I heard this, so nothing to cite. Sorry.)