I wonder if it’s proof of work bullshit. For those to be effective, they have to slow down whatever machine is running in an LLM farm.

That generally means they have to peg a desktop GPU for long enough to make a dent larger than LLM inference time.

Browsing with a 2 core box is like using a musket to blow a hole in a wall that’s optimized for armor piercing rounds.

Is this dumb? Of course. Your best recourse is probably to work around the anti-LLM blocks by having an agent read and summarize the page for you.

Are there any decent cloud based web renderers yet? Something like vnc or rdp backed by a shared 4090 could solve the problem. For static content, it’d only have to run proof of work once, then serve the result from cache.

That makes no sense. You require PoW before providing the information, not when the page is already sent. For a large code repository like GitHub for LLM training, it's also trivial to just git fetch to scrape. The simpler explanation is that modern web devs are frequently incompetent and can't get static text to appear on an idle page without pegging cores, can't get subsecond server page generation for simple CRUD, etc. This was the trend before LLMs were ever a thing.