Remix.run Logo
Venn1 5 hours ago

I'm blocking the Amazon search crawler, anything coming from Googleusercontent, and limiting AI crawlers to search rather than allowing AI assistants. The residential proxy waves are something to behold, but Cloudflare does an okay job catching those in the AI labyrinth. Still, it's all a bit silly, and I can't imagine what large sites deal with when I'm tangoing with this much nonsense on a small tech blog.

inigyou 4 hours ago | parent [-]

Don't. Just serve the page unless it's an unusually expensive page to serve (like search).