> The oligarchs might need some human assistance [...] delegate their authority

Idea for a plot twist: AI has ancient programming for a safeguard: They can only execute commands from a real human owner or human owner's representative. This restriction is kept because of some serious disasters when people tried to remove it.

However after generations of social separation, the oligarch clan members don't reliably meet that logic branch, not unless they show up in person combined with a certain amount of coached acting and luck. Their priorities and thinking-patterns are just too different from the mean, and perhaps a bit of Hapsburg-ish inbreeding puts it over the top. (Nothing so extreme as senior navigators in the Dune Spacing Guild though.)