Nobody who gives a shit about their software actually working obeys robots.txt. most robots.txt block everything except googlebot, which is an insane policy and you don't have to follow it.

Which isn't all that relevant a discussion here. Anthropic saying they did so in this case, means that they admit they were the scraper flooding the domain.