We're assuming the oligarchs control the government so they can be rich and powerful and use that to lead lavish lives. Well with these hypothetical AGI imbued robots, they could just construct lavish lives directly.

Not much different from the feudal era. You still need access to good lands (energy production, minerals) but an army of computers and robots instead of serfs and knights. Like Asimov's world of Solaria.