> The bots have a different model -- they assume that the website is adversarially "hiding" its content.

this should give us pause. if a bot considers this adversarial and is refusing to respect the site owners wishes, thats a big part of the problem.

a bot should not consider that “adversarial”