Remix.run Logo
spockz 4 hours ago

Is there some existing mechanism already that counts how often an ip only scrapes the page and not the css and then block those origin IPs if it occurs “too often”?

Unfortunately, the best practice is to make css cacheable so you need to keep long histories.

Bender 4 hours ago | parent [-]

I thought about that but to your point CSS is cachable. In fact I made mine immutable. No I just visually spot patterns and use that to study other facets of the agent, other client headers or lack thereof, supported protocol, accepted encoding and so on.

spockz 4 hours ago | parent [-]

Maybe it is enough to include some css/js which is served without cache and is loaded after all user visible css/js is loaded. Make it small enough to not cause too much bandwidth for the server and legitimate clients. Then anyone who doesn’t hit that CSS file gets banned.

Bender 3 hours ago | parent [-]

Possibly. I've noticed however that if people default to reader view they don't always pull the CSS. Seems to vary by client software. I'm weird enough to just stick with visual inspection and add characteristics when botters get too clever. Playing with the botters is almost as fun as Megabonk.