So the piece exhorts us a couple of times to try to just think neutrally about this stuff, and I get the point, but at the same time, can anyone in 2025 think that it's just going to be best most altruistic people armed with highly persuasive AIs who just want to use those AI to persuade us to act in our own best interests, rather than the people who own and are running the AIs?

Like, I have a hard time even strawmanning such a position. Of course people armed with highly persuasive AIs will task those AIs into doing what is best for the people who own the AIs and the odds of that happening to line up with your own interests are fairly low. What the hell else are they going to do with them?

But then, keep gaming it out. This particular thing isn't exactly completely new. We've seen bumps in persuasiveness before. Go back and watch a commercial from the 1950s. It's hard to believe it would have done a darned thing, but at the time people had not yet had to develop defenses against it.

We had to develop defenses. We're going to have to develop more.

What I forsee as the endgame is not that we become mindless robots completely run by AIs... or, at least, not all of us... but that we enter into a world where we simply can't trust anything. Anywhere. At all. How does any being, human or otherwise, function in an infosphere where an exponentially large proportion of it is nothing but persuasion attempts, hand-crafted by the moral equivalent of a team of PhDs personally dedicated to controlling me? Obviously humanity has never had a golden age where you could truly just trust something you heard, and we can argue about the many and sundry collective failures to judge the veracity of various claims, but it's still going to be a qualitative change when our TV programs are designed by AIs to persuade us, and the web is basically custom assembling itself in front of us to persuade us, and all of our books are AI-generated to persuade us, and literally no stone is left unturned in the ever-escalating goal to use every available channel to change our behavior to someone else's benefit.

When does trying to become informed become an inevitable net negative because you're literally better off knowing nothing?

What happens when the bulk of society finally realizes we've hit that point?

Here is my Bayesian version of this: If you have lies coming at you in high enough volume, you cannot update your priors at all, or else you will eventually come to believe the lie.

But then you have the problem: If you won't update your priors, and neither will someone else, but they have different priors than you, how can you talk to them?

But I'm maybe a bit less cynical than you. I think (maybe I'm kidding myself) that I can to some degree detect... something.

If someone built a parallel universe of false journal articles written by false experts, and then false news articles that referred to the false journal articles, and then false or sockpuppet users to point people to the news articles, that would be very hard to detect if it was done well. (A very persistent investigator might realize that the false experts either came from non-existent universities, or from universities that denied that the experts existed.) But often it isn't done well at all. Often it's "hey, here's this inadequately supported thing that disagrees with your priors that everybody is hyping". For me, "solidly disagrees with my solidly-held priors" is enough to give it a very skeptical look, and that's often enough to turn up the "inadequately supported" part. So I can at least sometimes detect something that looks like this, and avoid listening and believing it.

I'm hoping that that's enough to avoid "When does trying to become informed become an inevitable net negative because you're literally better off knowing nothing?" But we shall see.

Zvi's an AI-doomer, so he sees the endgame as AI becoming smart enough that they don't need us anymore and then killing us to take our stuff, but your scenario is also pretty bad.

I think you get to the point of authoritarianism. A trustless society is not a functioning one, unless people are forced to function.