Remix.run Logo
WhitneyLand 19 hours ago

I’m not sure what you’re saying I’m wrong about.

The comment I was responding to, about changing LLM output, referred to prompting, not temp/sampling tricks. I’m not aware of Pangram being beat by clever prompting. There’s some interesting work on creative writing using contrastive prompt techniques, but I haven’t seen it tried as evasion.

Even if you control temp and sampling, they’re not magic. If you raise the temperature too much writing can go to hell, so you may beat the detector but end up with junk. There are some ways to mitigate such a quality drop like raising temperature in conjunction with min-p, but still, I haven’t read any research that shows it getting good results at anything close to 10.

Now you want to get more clever and manipulate logprobs…well ok, you could come up with elaborate strategies designed to evade specific detection methods. But I don’t see that getting done as a weekend project while maintaining writing quality. And if it does happen there’s no guarantee the detector can’t train on its characteristics and start an arms race.

Der_Einzige 17 hours ago | parent [-]

As min-p approaches 1, the temperature you can get away with approaches infinity. Also more modern samplers like top-n-sigma are explicitly designed to get away with temperature of infinity.

WhitneyLand 10 hours ago | parent [-]

Not convinced just using top n sigma is going to beat a SOTA detector. The fingerprints Pangram uses should in principle be able to detect style above the token selection level.

Other papers have tried to beat it with temperature and it didn’t work, although I haven’t seen anyone try insane levels.

Give it a shot and let me know if you have any success.