I too fear what governments will actually do in this area. But I think you may be underestimating the threat to personal agency.

Imagine you are trapped in a groundhog day like time loop - but you are not the person who remembers previous loops. "Z" is. He tries to convince you to do something, over and over and over, thousands or millions of times, refining his approach based on your reactions while you remember nothing. Are you really confident that your free will protects you from being taken advantage of in this situation?

Now imagine that instead of a time loop, Z has a million clones of you. He tries his persuasion on one of them at a time, refining it until it works reliably before using it on you. You are just as vulnerable.

Now suppose he has a billion people, not identical to you but drawn from the same distribution. He has a harder computational problem, mapping the high dimensional manifold of their responses to create a model of you sufficiently accurate to manipulate you. But with enough data he can approximate the results of the previous case without more than a tiny fraction of his experimentation being visible to you.

Any relationship where one party gets to surveil and monitor not only the other party, but millions or billions of like parties, has the potential to be a deeply abusive one. We should not tolerate such situations whether the surveilling party is a government or not.