Bots and spiders hang onto old URLs longer than I hold onto a grudge.

We had some SSR that followed up with JavaScript listeners to do interactive updates instead of loading a whole new page. But the team responsible didn’t add nofollow and some bots ignore it anyway. Some significant fraction of our cluster size was answering those requests. But one deployment someone broke the fuck out of those links and it showed up immediately in our 4xx error statistics and we rolled the change back within a week.

Three years later I was still seeing those busted URLs being submitted.

Then the fix for the nofollow got implemented and declared done but the flag never got made permanent. We were pretty peeved when someone learned that in some offhand comment in a conversation between teams.

We were already identifying “good” and “bad” bots at the border and telling them to fuck off, so all of this was coming from our supposed friends. And that’s when I realized that a company owning a search engine and a cloud service provider is a protection racket. Google cloud is selling you servers in order to be talked to by Google bots.