| ▲ | shakna 2 days ago | ||||||||||||||||||||||
I have a single <a> element in my website's head, to a route banned by robots and the page is also marked by noindex meta tags and http headers. When something grabs it, which AI crawlers regularly do, it feeds them the text of 1984, about a sentence per minute. Most crawlers stay on the line for about four hours. | |||||||||||||||||||||||
| ▲ | dbalatero 2 days ago | parent [-] | ||||||||||||||||||||||
That's hilarious, can I steal the source for my own site? | |||||||||||||||||||||||
| |||||||||||||||||||||||