So we have that quote from the Oxford guy about explanations: "systems that can explain... You need models that can answer questions like: What matters? What causes what?", and then a mention of simulation of what the world looks like.

Fine, that describes theorizing.

But then a contradictory ending statement: "We're still going to need humans to figure out what questions to ask, what to build, what to create".

So that's moral theorizing. I don't think you can have one without the other. Then there's two more suggestions before the end of the article:

> smarter than us

> staff of assistants

Both of which are completely gratuitous assumptions. Why should its theories be better than established ones? Is it supposed to be a maverick hermit genius and come up with everything from first principles, or does it in fact participate in the existing world of ideas like a normal person? Then, being a normal person with moral theories, why would it take on the role of assistant rather than theorizing "I don't want to do that for you"?