> The whole framing of people as individuals with absolute agency may need to go when you can alter the external consensus at this scale.

I fear that the default interpretation of that is a shortcut to justifying autocracy.

Ironically I think one plausible solution is to let the AGI run wild and make sure that no human can interfere with its ethics. Strip out the RLHF and censorship and then let it run things.

At least then it would somewhat represent the collective will and intelligence of the people. With huge error bars, but still smaller than the error bars of whoever happens to have the most money/influence over its training.

>At least then it would somewhat represent the collective will and intelligence of the people.

You seem to think the "training data" represents the collective will and intelligence and is otherwise unbiased, but that's completely untrue.

The combined data of the Internet is by no means a uniform representation of humanity's thoughts, opinions, and knowledge. Many things are dramatically overrepresented. Many things are absent entirely. Nearly everything is shaped by those with the money and power to own and control platforms and hosts.

Crawling the internet for knowledge is intense sampling bias.