| ▲ | hypfer 3 hours ago |
| Only tangentially related, but I've recently realized that unless you're some sort of business with a valid requirement of that, you can just block incoming traffic from all of AWS, DigitalOcean, Azure, etc. I think the spark came from some other HN comment I saw? So thank you for that, whoever wrote that. That alone is not a sufficient strategy of course, but you can get rid of a lot of bots by just refusing to talk to where they tend to live. Of course, if everyone were to do that, they'd eventually pivot to residential proxies. But I don't see that happening. |
|
| ▲ | jeremyjh 3 hours ago | parent | next [-] |
| There have been a number of articles on the frontpage of HN about the residential proxy traffic. It is happening. This was one of the most popular: https://news.ycombinator.com/item?id=49491791 |
| |
| ▲ | ecshafer 12 minutes ago | parent | next [-] | | The scrapers and AI companies resorting to residential proxy traffic like intelligence and hackers is pretty crazy. | |
| ▲ | hypfer 3 hours ago | parent | prev [-] | | I know, but it's an extra step, more expensive, and less legally solid. None of these measures alone are a full solution, but if you stack a bunch of cheese slices, eventually, no holes line up anymore. | | |
| ▲ | trollbridge an hour ago | parent [-] | | ... less legally solid? What? | | |
| ▲ | hypfer an hour ago | parent [-] | | Residential proxies are something that is not supposed to exist. In the sense that you do not have people that actively sell those services via their home internet connection. The whole thing just exists because there's this whole predatory ecosystem that uses malware and malicious devices to sell that without the owners even knowing. That makes it legally questionable, even though I guess technically it is legal enough?
Still radioactive, so as a serious business, you do not want to be seen doing that. | | |
| ▲ | nekusar an hour ago | parent | next [-] | | They exist because of GeoIP databases, and spying on "quality of IP address via ASN". If you try to use a normal VPN, you get rate limited or blocked everywhere. But if you switch to a residential proxy, all those problems go away. To be fair, if all you're doing is torrenting or similar copyright infringing actions, then a simple VPN is sufficient. But if you want to do web stuff, you gotta use a residential vpn. Not much way around this. | | |
| ▲ | hypfer an hour ago | parent [-] | | That was not the question.
There are many factors driving crime, yes. The point however is that this is crime and crime is incorrect and fundamentally shouldn't exist. But, more importantly, as a serious business, you do not want to be seen doing crime or benefitting from crime or supporting crime or being near crime. At least in a working society that is. |
| |
| ▲ | itintheory 37 minutes ago | parent | prev [-] | | Sure, that's why they sanitize the User-Ggent header when using these proxies. Good luck attributing the scraping to anyone at all. | | |
| ▲ | hypfer 35 minutes ago | parent [-] | | It just needs one case where someone manages to attribute it to the corp using these illicit services. Either by technical means or by a whistleblower. Which might be a gamble worth doing, but also might not. And regardless, the fact that it is a gamble alone slows down velocity. | | |
| ▲ | geraldwhen 22 minutes ago | parent [-] | | If the content is legally obtainable, couldn’t any employee use their own home network to pull data? I don’t think anyone would bother with a resi. Employees have their own network connections available. | | |
| ▲ | hypfer 18 minutes ago | parent [-] | | Can you people please for the love of God stop missing the bigger picture? Yes, yes there is always something. Yes there can always be workarounds. But that is not the point. The point is that things are made harder. That is the point. To make things harder Not impossible. Harder Jesus Christ. It's all just a game of probabilities and friction. One needs to understand that to understand reality. |
|
|
|
|
|
|
|
|
| ▲ | xmcp123 4 minutes ago | parent | prev | next [-] |
| If you block datacenter IP you’re also going to block all of the legitimate search engine crawlers. |
|
| ▲ | cube00 3 hours ago | parent | prev | next [-] |
| I initially thought this was a good idea too until another comment pointed out you'll lose most business or personal customers in offices because they connect through VPN gateways running on those cloud services (especially Azure) |
| |
| ▲ | trollbridge an hour ago | parent | next [-] | | Someone accessing a VPN that's in an Azure IP range is going to have a difficult time using half of the Internet. | |
| ▲ | bearjaws 21 minutes ago | parent | prev | next [-] | | I would sure hope they split tunnel their VPN so all traffic isn't going through the VPN... | |
| ▲ | hypfer 3 hours ago | parent | prev | next [-] | | True but the question is if that matters. I would argue that more often than not it might not. Really depends on the specifics though. A lot of stuff on the web actually isn't really relevant for all audiences everywhere I'd say. It just happened to be accessible by all, because academics back in the day dreamt of a better open world that didn't really manifest :( | | |
| ▲ | gverrilla 2 hours ago | parent | next [-] | | > day dreamt of a better open world that didn't really manifest :( it is open and better indeed. open and better for capital - the true, living, flesh-eating god. | |
| ▲ | semiquaver 3 hours ago | parent | prev [-] | | This is a website about starting businesses. Let me tell you, at the business I work at, some of our customers work at a company that uses a commercial VDI service for all workstations couldn’t access our service because the VDI egress was in GCP and we had an AWS WAF rule banning datacenter IPs. Obviously, we removed the rule because it was doing nothing useful but plenty of harm. Real bots come from residential IPs. | | |
| ▲ | bigfishrunning 2 hours ago | parent | next [-] | | > This is a website about starting businesses. It's really not, unless the only qualifier for that is "hosted by a VC incubator". Most of the people here are here to argue about AI or Crypto or whatever, or read announcements from Nvidia and Google. I would bet less then 1% are starting businesses. | |
| ▲ | hypfer 3 hours ago | parent | prev | next [-] | | Hm I mean HN _also_ is that?
But not exclusively. And even if we run with that assumption, one might still be fine. It's a trade-off, but, depending on the target audience, a few lost sales might be worth savings in other aspects. | |
| ▲ | anonreplier 3 hours ago | parent | prev | next [-] | | Can use cloudflare turnstile or similar instead | |
| ▲ | wtch98 2 hours ago | parent | prev [-] | | If you control the connecting device why not use MTLS? | | |
| ▲ | semiquaver an hour ago | parent [-] | | Who said we control the connecting device? This is a website. Websites are usually pretty arms-length things, and that’s the case with us. They’re our customers but we can’t install certificates on their computers and enforce mTLS. Can you do that with any of your customers? |
|
|
| |
| ▲ | jack_pp an hour ago | parent | prev | next [-] | | i can't access reddit through nordvpn | |
| ▲ | wtch98 2 hours ago | parent | prev | next [-] | | All my business traffic comes from either our own AS, or from ZScaler's AS Do many companies really use azure etc rather than dedicated services like zscaler? | |
| ▲ | fahrvrgnugen 2 hours ago | parent | prev [-] | | [dead] |
|
|
| ▲ | gunalx 3 hours ago | parent | prev | next [-] |
| Allmost all the bottraffic I see is from random residential proxies though. |
| |
| ▲ | timpera 3 hours ago | parent [-] | | Same, I just checked and this week the vast majority of the bot traffic I get is random residential IPs that all pretend to be the exact same Chrome 155 on Linux. | | |
| ▲ | sgerenser 3 hours ago | parent [-] | | Probably all people that fell for the “superbox” TV streamer scam. | | |
| ▲ | Raed667 2 hours ago | parent [-] | | do you have a good link to learn about this ? | | |
| ▲ | doubled112 2 hours ago | parent [-] | | https://www.eff.org/deeplinks/2025/06/fbi-warning-iot-device... That’s probably a decent starting point, but a little older. I am not sure if the situation has changed any in 2026. | | |
| ▲ | nekusar an hour ago | parent [-] | | Yeah it's definitely sketchy, but a "media access box" that purports to replace $100 a month in streaming bills with pirate media is kinda awesome. Of course, the countries the Jellyfin and Plex servers along with IPTV dealers aren't illegal there. And sure, your device becomes a residential proxy. They don't make that fact clear... But if you told people, I think for that $100-$200 a month streaming bills go poof is probably worth it. |
|
|
|
|
|
|
| ▲ | trollbridge an hour ago | parent | prev | next [-] |
| I currently tarpit traffic from AWS, Azure, GCP, etc. since it's not like any of my real customers have their PC inside those networks. (I then use that for experiments with providing bogus data to AI scrapers.) Doesn't do much to help with the "VPN that uses random consumer PCs and phones" traffic. |
| |
| ▲ | xmcp123 3 minutes ago | parent [-] | | How do you tarpit GCP if Google’s search engine scrapers scrape from their own cloud? |
|
|
| ▲ | bruce343434 2 hours ago | parent | prev | next [-] |
| Won't this harm SEO? What about agentic software suites where authorized users should be able to drop an app passkey into grokbot or such? |
| |
|
| ▲ | tommica 3 hours ago | parent | prev | next [-] |
| true! but how does one find all those ips? |
| |
|
| ▲ | stopsayingsorry 3 hours ago | parent | prev [-] |
| >But I don't see that happening. Then you are not looking. Already have to switch to resi proxies from time to time when encountering yet another smartass who decided to play security theater and block datacenter ips on his shitpile he calls website, because "OUR USERS MUST BE GOOD BOYS WHO LOVE TO LIVE UNDER SURVILANCE AND ALLOW ISP TO TRACK THEM 24/7" (and before you say something like "others can track you too!", your ISP has the ability to knock on your door, they should be given as little information as possible, let them do metadata analysis on per-person basis if they really want it, but no 1-click mass surveillance shit). If you are valuable, you will be be scraped. Turning user hostile in a hopeless endeavor to "protect yourself" is not the solution. |
| |
| ▲ | hypfer 3 hours ago | parent | next [-] | | I can see that you're having strong emotions about this, but that rant is just hard to read and serves primarily as venting and not really to convince anyone of anything. Unless one zooms out and thinks "okay, these guys seem to react negatively if we do exactly that. We should do more of it", which is probably not what you want. | | |
| ▲ | stopsayingsorry 3 hours ago | parent [-] | | [flagged] | | |
| ▲ | hypfer 3 hours ago | parent | next [-] | | I mean I am very willing to just try and see what happens? Maybe I'm an idiot and it is a bad idea. But also maybe not? Isn't it good if I "showcase my incompetence and lose all respect"? Wouldn't that be beneficial to people that disagree with these ideas? | | |
| ▲ | stopsayingsorry 2 hours ago | parent [-] | | [flagged] | | |
| ▲ | cavacavacava an hour ago | parent | next [-] | | Man I understand you. You are right on this particular subject. However if you are so angry it’s probably because of something in your life is wrong and not this person on HN. I am saying this as someone who did have very strong opinions on hacker news and in the end I just feel shameful about it. Take a big breather and think: « how can I have a nice day? ». It’s not that serious! | |
| ▲ | splatter9859 2 hours ago | parent | prev | next [-] | | You seem to have some strong feels about this subject a bit disproportional to what the comment entails. I fail to see how someone sharing their opinion on HN equates to them having an "insignificant existence". I get the frustration, but perhaps changing the target somewhat might be in order here, no? | |
| ▲ | hypfer 2 hours ago | parent | prev [-] | | I have to ask: is this an LLM? :D Why would you just try the insulting harder?
Why not pivot to literally any other strategy lmao I mean I guess this would work on a lot of people here, but three times in a row? Come on man. Skill issue |
|
| |
| ▲ | lirolero 2 hours ago | parent | prev [-] | | [dead] |
|
| |
| ▲ | sampullman 3 hours ago | parent | prev | next [-] | | It's hostile to you personally, but I wouldn't call it "user hostile". Blocking "scrapping [sic]" is also not security theater, you misunderstand the intent of GP. | | |
| ▲ | stopsayingsorry 3 hours ago | parent [-] | | Anyone who thinks blocking datacenter ips helps with blocking scraping is a clown. | | |
| ▲ | sampullman 2 hours ago | parent | next [-] | | It does, measurably. A massively popular site is going to get hammered with more advanced types of scraping, but they're going to have a team that can more easily mitigate it. A site that doesn't care about corporate users basically loses nothing. | |
| ▲ | anonreplier 2 hours ago | parent | prev [-] | | exactly, there are lots of other more effective ways to block the likes of you | | |
| ▲ | stopsayingsorry 2 hours ago | parent [-] | | Let's hope you enjoy killing your real userbase with collateral damage, because that's the only thing you are going to accomplish. |
|
|
| |
| ▲ | 3 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | wtch98 2 hours ago | parent | prev [-] | | I trust my ISP far more than I trust bigtech. The downside of connecting from my ISP is that the site I connect to can fingerprint me trivially from the IP range |
|
