Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.

▲

NooneAtAll3 5 hours ago | parent [-]

it's hard to separate spambot that only accesses 3-5 links per IP and a legit user. Changing content for legit user can be devastating

	▲	inigyou an hour ago \| parent [-]
		Put a cookie wall in front. The bot will either load the cookie and have a persistent identifier, or not load the cookie and not get in