>I don't think people would obey an AI oracle in that way

You clearly haven't encountered the large group of people who trust hallucinated LLM output more than their coworkers.

When they like the output, sure. I've never had any luck convincing them to change course by showing that I can prompt an LLM to say something else.