I’m a teacher that requires students to write, so I’m always on the lookout for reliable AI detection. Haven’t anything close to reliable yet, but Pangram definitely seemed to be better than anything else in my (very limited) tests.

That said, it did incorrectly flag something that I personally wrote as AI, which baffled me. I spent a while playing around with the text and feeding it back into Pangram, trying to figure out what the issue was. Turns out, I had a section in the text with three bullet points. I removed the bullets (literally just the bullets themselves, no actual text) and it passed as human. While it seems to perform better than anything else, I’m a bit skeptical of their claimed success rate.

I wrote a novella with epigraphs at the beginning of the chapters. Every epigraph was flagged as AI-generated likely because they were written as curt, clipped statements of fact as if taken from a historical record entry.

It's interesting that it cares about formatting choices like bullet points.

On the one hand, it makes the system more brittle and arguably overfit, since it's not making a decision on the content of the writing itself. Most people would say that AI writing is still AI writing even if bullet points are removed, and vice versa.

On the other hand, we know that AI writing does has a very specific formatting signature (e.g. em-dashes, bullet points) so it seems unwise to completely ignore it.

The tools will never be perfectly reliable. It’s good that people like you are aware that it’s a statistical tool, not a perfect oracle.

For student work I would think it’s reasonable to look for patterns across each student’s work. A single Pangram positive across 10 assignments shouldn’t be enough to bring suspicion upon a student, but when 8 out of 10 of someone’s assignments are coming up as AI it’s time to talk.

The trend I’m hearing from my teacher friends is that students are generating AI papers and then putting them into tools like ZeroGPT and Pangram before submission. They do what you did: Remove sections, rewrite things, and make adjustments until the AI score goes down enough.

I heard that one classroom had developed a trick of adding spelling mistakes to ChatGPT output to try to make it look human written. So the teacher was getting pure AI slop but every paragraph had a couple typos inserted.

As for the 3 bullet points: That was a huge tell for one of the recent popular models. Almost all writing output would get sections of plus or minus 3 bullet points in a certain style. The newer frontier models do it less but that was a big LLM pattern for a while. Unfortunate for anyone who liked to write with 3 bullet points commonly. They’re feeling the pain of people who were emdash enjoyers before 2023