Author here. I think the term "widespread appeal" may be misleading.

There are comedians who can attract a very large audience, large enough to make them major celebrities, but that doesn't mean that aggregating preferences doesn't produce mediocre jokes.

Instead, comedians build an audience of like-minded people, and get to know that audience very well. It's a little bit like the process of finding product-market fit for startups. You can achieve great success by catering to the needs of a very large market, even if you can't cater to everyone's needs.

> It would be interesting to compare how well LLMs can estimate how funny a joke is vs how good they are at generating jokes.

Academic psychologists have not found a quantitative measure of "how funny a joke is." If there were such a measure, LLMs could try to optimize for it.

But there isn't such a measure, and, if my argument is right, there couldn't possibly be a measure like that, because jokes have to be surprising but inevitable in hindsight, and different jokes will be surprising/inevitable to different people.

I mainly agree with your argument, but I think "inevitable in hindsight" needs reconsideration. Following the incongruity model, humor comes when the punchline forces a change in perspective on the setup. So it is not that the new perspective is inevitable, but that it was such a less likely interpretation of the setup that it didn't occur to the audience.

Considered this way, the question is whether an LLM could be trained to create joke setups that are ambiguous in the right way, with one obvious interpretation and a hidden but possible interpretation that will be forced by the punchline.