Remix.run Logo
rvz 2 hours ago

I am going to assume that this also destroys millions of AI agents and bot scrapers this time which is why some “AI Engineers” were complaining about this recently.

Well, this is how Google will kill all the scrapers on its search data.

jeroenhd 2 hours ago | parent | next [-]

Not entirely, Google's own page says:

> Fraud Defense leverages a sophisticated and adaptable risk analysis engine to shield against automated software. It is specifically designed to orchestrate trust for the agentic web, neutralizing malicious scrapers while welcoming legitimate AI agents.

I'm sure it'll block a whole bunch of awful scrapers but if Google doesn't hate a bot, it'll be able to pass.

dns_snek an hour ago | parent [-]

Sounds like an anti-competitive play to build an AI moat. They'll introduce a sham "verification program" and only allow bots operated by entities large enough to sue them for anti-competitive conduct.

hsbauauvhabzb an hour ago | parent | prev [-]

Does mass scraping need google for content discovery? Surely most sites contain a site map or index that would effectively self enumerate once you know the domain, which is more often than not publicly disclosed?