Doesn't this just suggest that the next frontier for powerful AI models is to ask challenging questions, not simply solve them?

Terry even says this: "In fact, it is now the identification of a promising problem which is the scarce and precious resource."

The creativity and insight needed to ask a question that Terry gets excited about is the next step. Perhaps OpenAI should create a set of challenging questions and offer a prize to solve them.

The incentives are massively skewed towards the AI labs investing their massive amounts of compute into being the first to solve an outstanding problem.

It's a marketing game for them, any societal benefits are secondary. Winning a prize is going to get headlines and feed into the "AGI soon, machine replaces another career" narrative they crave unlike coming up with some (possibly) interesting problems.

I think the problem with AI asking questions is that it will ask questions that are interesting to it but not necessarily us. AI, as a model, will never be a perfect copy of a human. It will always be a simulation, and thus to some extent, will ask questions that humans find irrelevant and solve problems that humans find irrelevant.

For anyone facing an existential crisis on AI, your ace in the hole is your humanity. Only you have it, and only you will be the best judge of what is good and interesting (to a human at least).

If AI can generate questions and then answer them, what are the people for?