Yup, I think that's fair.

I'm not sure how many humans know how to be genuinely innovative; nor if it's learnable; and also, assuming that it is learnable, whether or not known ML is sample-efficient enough learn that skill from however many examples currently exist.

As you say, we don't understand what we're trying to build. It's remarkable how far we got without understanding what we build: for all that "cargo cult" is seen as a negative in the 20th century onwards, we didn't understand chemistry for thousands of years but still managed cement, getting metals from ores, explosives, etc.

Then we did figure out chemistry and one of the Nobel prizes in it led to both chemical weapons and cheap fertiliser.

We're all over the place.