This might be key, I think Grant Sanderson of 3blue1brown fame gave the example of the Turing machine being this small lemma that was introduced in a paper about computability, and if no one actually spent any time thinking about what it was and what it meant, we have solved the paper and thought there wasn't any interesting or useful application of this math.

Thankfully the machine was built, I do wonder if how we're going to solve the understanding part of this, there's a lot coming out, drinking from the firehose is not something I can reconcile with my experience that a lot of good ideas need serious contemplation time to bear fruit.

Importance is a function of culture and biology. We already gave AI access to culture, so it just needs to capture some wild mathematicians for vivisection. Once it has a good model of mathematician biology they can be efficiently recycled.

'Figuring out what is important/what success looks like' in math and other fields seems much easier for humans than 'achieving important success', which is why we celebrate success.

Why do you think AI will be better at achieving success than humans, but worse than humans at identifying it?

Unless you mean super high level stuff like prompting Galactica 8.5 with "reorganize human society for maximum flourishing, make no mistakes"

Eh, do you? And more importantly, will you?

"What's important" can be generic stuff like "fusion", and current gen LLMs are already more than able to go from there and iteratively explore what they might contribute with. It's not exactly a stretch to take the extra step of asking an LLM about what the top 10 unsolved scientific problems are.

I say this because I see this mantra repeated often, but largely it makes very little sense. Do you have any concrete examples of unsolved important problems that an LLM is inherently unable to locate?