I actually asked GPT that question some time ago.

The importance of top executive decisions is overstated, they often do no better than random chance. But their decisions are not where their value lies.

The important part is that they own their decisions, or at least appear to do so. And in doing that, they gain trust for the company, they are the company face. It is more of a performance than a technical job. By that, I don't mean it is fake, people will notice if you don't know what you are talking about, and it won't inspire confidence, which is what leaders try to avoid at all costs.

LLMs will probably be at least as good at decision making (I mean, that's almost a dice roll), and they can be convincing, which is important too, but they are completely lacking in the trust department. No one want to work for a machine, no one want to do business negotiations with a machine either. That's why it is important to have a human on top.