Interesting essay. But it attributes something magical (ability to solve undefined problems) to humans and says that AI doesn't have it.

It seems pretty meaningless and not engaging with the real problem to say that AI doesn't "actually" write movie scripts or paint pictures. Like this doesn't line up with my definitions for doing those things which AI clearly fulfills.

And human intelligence arises from a well defined problem: maximizing f(environment, self) -> babies.

Also: if it were possible to measure, which it isn't, I strongly suspect that ability to solve well-defined problems and ability to solve poorly-defined problems are highly correlated, not totally uncorrelated. Happiness is a poorly defined problem, but it's just one of many, and has its own pile of things to consider that can isolate it from the general ability to solve poorly-defined problems.

I do like the framing. seems to be describing something similar to Goodhart's Law.