This was good, the tldr point is LLMs suck at natural writing, particularly long form. Or more abstractly they don't have complex original ideas, so can't do anything that requires this.

It's not surprising as it's very hard to train for or benchmark.

Also should add I don't think anyone serious thinks that long form writing or ideation is what they're for - assuming an LLM would be good at this is a side effect of anthropomorphism / confusion. It doesn't mean an LLM isn't good at summarizing something or changing unstructured data into structured or all of the other "cognitive tasks" that we expect from AI.

I suspect that gpt 6 will write great diverse essays when prompted with single words, ace specifically this benchmark, and piss people off when they upgrade siri to got 6, say “time?” to their smartwatch, and get a 3600 word eloquent response.