| ▲ | jeroenhd 2 hours ago |
| > Most users aren't spoofing their user agent headers to be a different operating system. The people behind the LLM behind this blog post are. They're trying to pretend their robots are people to sell other websites' data to their customer. It's easier to pass bot detection gates if you pretend to be a physical machine running Windows or macOS than if you honestly admit you're using Linux on a VM. |
|
| ▲ | nradov 20 minutes ago | parent | next [-] |
| The HTTP User-Agent header was a mistake from the beginning. There is no legitimate need for the server to know what software the client is (or claims to be) running. |
| |
| ▲ | esrauch 2 minutes ago | parent [-] | | I feel like this is with 2026 view where browsers are so mutually compatible. In the bad old days there were so many differences between html, css and js behaviors that if you wanted your site to be nice you had to change it for the browser. The way css padding worked wasn't even the same. Feature detection was rarely viable for any of this. No user agent would probably have only entrenched IE6 dominance even more by blocking you from making a site that works at all on other browsers and even made it so IE7+ couldn't make the standards conformance progress that they did. |
|
|
| ▲ | acters 2 hours ago | parent | prev | next [-] |
| It's sometimes easier to lie than to tell the truth, and being on Linux telling the truth gets me more scrutiny than those pretending to be legit. |
|
| ▲ | baby_souffle 25 minutes ago | parent | prev | next [-] |
| Some scrape activity happens because I can't obtain the data any other way. I would be thrilled if certain retailers had price and availability data as an API so I could not bother with the bulk of scrape and process. |
|
| ▲ | reactordev 2 hours ago | parent | prev [-] |
| The Internet is a cesspool of scams now |
| |
| ▲ | pocksuppet 2 hours ago | parent | next [-] | | scraping, however, is not intrinsically a scam. | | |
| ▲ | mplewis 2 hours ago | parent | next [-] | | It is when you're doing it like the LLM companies are: at scale, to the degree that you're taking down my site, without my consent by masking your user-agent, for the purpose of stealing data I didn't authorize you to have. | | |
| ▲ | Bender 39 minutes ago | parent | next [-] | | I documented some crude methods that can stop most of that without a CDN. [1] There will be some false positives so I guess it depends on ones priorities which methods if any to implement or test on a throw-away test site. Not perfect, nothing is. I am watching hundreds of bots sending SYN's and the daemons are oblivious to them. The only method I have not played around with yet is #7 ssl fingerprinting. There are additional methods I chose not to document such as limiting access to logged in accounts that require double-opting-in to acceptable use policies and terms of use, not that most scrapers would give a toss. That it too much whack-a-mole for me personally. That method requires progressively adding friction to account creation and that comes with some pros and cons. [1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som... | |
| ▲ | nradov 16 minutes ago | parent | prev | next [-] | | What data is being stolen? Are you referring to copyright violation or something else? If you don't want LLM companies to scrape a site then just restrict access to authorized users. Simple. | |
| ▲ | lsaferite an hour ago | parent | prev | next [-] | | Unfortunately the response to that right now is to nuke everything that isn't within a strict set of constraints. That helps with the bad bulk-scrapers, but hits everyone else as well as collateral damage. | |
| ▲ | VorpalWay an hour ago | parent | prev | next [-] | | Something like Anubis in front of the server to protect it might be an option. It sucks that we have to resort to that yes, but it seems the least bad option currently (better than the entire internet going through Cloudflare at least). | | | |
| ▲ | userbinator an hour ago | parent | prev [-] | | The idiotic "stealing" argument again? At least use "piracy" if you want to be correct... The moment you openly publish information on the Internet, you have already given consent. There are other solutions to bandwidth usage. User-agent discrimination should be illegal. All it does is further the control that Big Tech has, and help authoritarian governments with their control too. | | |
| ▲ | pdpi an hour ago | parent [-] | | As with anything IP-related, the data is licensed under whatever terms you choose to impose. That’s the extent of the consent you’ve provided. | | |
|
| |
| ▲ | duped 34 minutes ago | parent | prev [-] | | No it's just intrinsically antisocial behavior | | |
| ▲ | GaryBluto 23 minutes ago | parent [-] | | Scraping has never "intrinsically" been anti-social behaviour, you just don't like it. |
|
| |
| ▲ | Mistletoe 17 minutes ago | parent | prev | next [-] | | Has the internet made the average human’s life better? As someone that lived in both eras I have to say no. | |
| ▲ | d1ss0nanz 2 hours ago | parent | prev [-] | | Always was. |
|
