I will add, the news websites didn't start paywalling just because of LLM scrapers. They started doing it after certain parts of the GDPR passed, because they could no longer sustain themselves as much from targeted advertising and data sales. News has supplemented itself with advertising for a long time, but targeted advertising has sadly become perceived as mandatory by advertising companies, even before Google's dominance.

So publishers are in a no-win situation: if they lock down their content completely (server-side paywall), they disappear from Google Search and lose traffic. If they keep the "leaky" paywall, their content gets hoovered up for free by Common Crawl to train models that will then compete directly against them. They're trapped