Remix clone Hacker News

There's a fork of this that has some great improvements over to the top of the original and it is also actively maintained: https://github.com/lexiforest/curl-impersonate

There's also Python bindings for the fork for anyone who uses Python: https://github.com/lexiforest/curl_cffi

▲

nyanpasu64 18 hours ago | parent | next [-]

I suppose it does make sense that a "make curl look like a browser" program would get sponsored by "bypass bot detection" services...

▲

ImHereToVote 11 hours ago | parent [-]

Easy. Just make a small fragment shader to produce a token in your client. No bot is going to waste GPU resources to compile your shader.

▲

kelsey978126 10 hours ago | parent | next [-]

Why do people even think this? Bots almost always just use headful instrumented browsers now. if a human sitting at a keyboard can load the content, so can a bot.

▲

simpaticoder 10 hours ago | parent | next [-]

Security measures never prevent all abuse. They raise the cost of abuse above an acceptable threshold. Many things work like this. Cleaning doesn't eliminate dirt, it dilutes the dirt below an acceptable threshold. Same for "repairing" and "defects", and some other pairs of things that escape me atm.

▲

abofh 6 hours ago | parent [-]

That's the same argument as CAPTCHA's - as far as I know there are no bots protesting them making their lives harder, but as a human - my life is much harder than it needs to be because things need me to prove I'm a human.

Clean for data ingestion usually means complicated for data creation - optimizing for the advertisers has material cash value downstream, but customers are upstream, and making it harder is material too.

▲

ImHereToVote 6 hours ago | parent [-]

What is so hard about running a fragment shader after the site has loaded?

	▲	abofh 5 hours ago \| parent [-]
		I have to assume /s, but lacking that -- Why can't you just allow `curl`? You need a human for advertising dollars or a poor mechanism of rate limiting. I want to use your service. If you're buying me a fragment shader, I guess that's fine, but I'm feeding it to the dogs, not plugging in your rando hardware in to my web-browser.

▲

ImHereToVote 10 hours ago | parent | prev [-]

We are talking about Curl bots here. How is what you are saying relevant?

	▲	cAtte_ 9 hours ago \| parent [-]
		no, nyanpasu64's comment extended the discussion to general bot detection

▲

gruez 7 hours ago | parent | prev | next [-]

Can't they use a software renderer like swiftshader? You don't need to pass in an actual gpu through virtio or whatever.

▲

ImHereToVote 5 hours ago | parent [-]

Maybe you can call a WebGL extension that isn't supported. Or better yet have a couple of overdraws of quads. Their bot will handle it, but it will throttle their CPU like gangbusters.

	▲	gruez 4 hours ago \| parent [-]
		Sounds like a PoW system with extra steps?

▲

zffr 9 hours ago | parent | prev [-]

Can't a bot just collect a few real tokens and then send those instead of trying to run the shader?

	▲	ImHereToVote 9 hours ago \| parent [-]
		How do you automate that? Just generate a new token for each day.

▲

illegally 13 hours ago | parent | prev | next [-]

There's also a module for fully integrating this with the Python requests library: https://github.com/el1s7/curl-adapter

▲

RKFADU_UOFCCLEL 9 hours ago | parent | prev [-]

All these "advanced" technologies that change faster than I can turn my neck, to make a simple request that looks like it was one of the "certified" big 3 web browsers, which will ironically tax the server less than a certified browser. Is this the nightmare dystopia I was warned about in the 90's? I wonder if anyone here can name the one company that is responsible for this despite positioning themselves as a good guy open source / hacker community contributor.