Agreed, I keep thinking to myself that there's a huge mathematical question right in front of us today that is in exactly this vein - all the various nuances of why LLM's work so well is a mathematical question. As far as I know, it's not really understood beyond "we do this basic thing (that makes sense) to predict that a noun is followed by a verb, and then we scale it up a bazillion fold and it can contribute to mathematics research".

As a comparison, classical computing has been scaled up a bazillion fold too, and can do things which are absolutely miraculous, but every layer of abstraction is discretely understandable.