Why wouldn't it be desirable? Is knowledge beyond that a child can understand undesirable because the child can't understand it? I think not, same with anything AI figures out that we can't easily understand ourselves. If GPT-10 Quasar grinds tokens out for 6 months and out pops a warp drive, and even it's executive summary is difficult for anyone to understand, do we get out the pitchforks and burn the data centers or do we go "Sweet, we've got warp drives!"

Why is it always a false dichotomy between two ridiculous extremes? Maybe inventing some dangerous technology benefits from human understanding for a bunch of obvious reasons, like human beings being responsible it goes well?

Maybe fix energy and heat on earth first. Fusion would be nifty. How's the AI progress there?