I'm vaguely sympathetic to this argument.

But only vaguely. Google uses its monopoly position in advertising to basically ensure that you allow them to scrape your site (or if not you personally, the majority of revenue driving sites). They have the benefit of being allowed by default.

They also then scrape again at the user level for users operating chrome.

They also conveniently ignore global blocks for their adsbots (you have to specifically name them to block them).

If you're not Google, you likely don't have this luxury.

My preference would be that governments force search indexes to be public. The exact mechanisms for this can be debated.