Is an agent-ready website so obvious poison? If I'm running a plumber shop in East London, then I'd want agents to know that just as much as I want Google (Search) to know that. The same will be true for most real-world businesses. Only sites that make money by selling their users' data and eyeballs obviously stand to suffer.

Or the website of someone who makes things for people to see, or art for people to consume, and would prefer to avoid being automatically plagiarized as much as possible. It's not always about business.