As Moises Naim says, these days power is easy to get, hard to use, easy to loose.

As complexity/unpredictability rises in any system its natual that govt will shed power. We have already seen this with setting of interest rates.

There is a long list of reasons why the Treasury Dept, President, House, Senate, Supreme Court or NASA doesnt get to set the rate.

Why did they hand over that power to the central bank? Read some financial system evolution history. Mindlessly ambitious people, the kind who run things hardly ever hand over power to someone else unless they see weilding the power puts a giant bulls eye on their head when the inevitable unpredictable/uncontrolable event occurs. Same goes with the Church or Religious institutions. Why does govt stay out of that lane? Why did Italy hand over the Vatican to the Church?

there is a huge difference here between that and the central bank

the central bank is semi part of the state, AI Labs are not, AI Labs are fully under directly control of a extremely small number of people in a way which has nothing to do with democracy

founding a Organisation like the FCC but for AI oversight is a completely different topic, but that isn't what he is proposing

> Italy hand over the Vatican to the Church?

because it always had been a nano country under Church control, this is not comparable at all

> Church or Religious institutions.

most democracies have a separation of state and religion where religion has little to no governing power. When this isn't the case it's a failure of the state (often for historic reason). This isn't desirable at all.