Remix.run Logo
fibers 3 days ago

you have to feed it multiple arguments with rate limiting and long wait times. i am not sure if there have been recent updates other than the js interpreter but ive had to spin up a docker instance of a browser to feed it session cookies as well.

m_ke 3 days ago | parent [-]

Yeah we had to roll through a bunch of proxy servers on top of all the other tricks you mentioned to reliably download at a decent pace

trenchpilgrim 3 days ago | parent [-]

What are your thoughts on the load scrapers are putting on website operators?

immibis 2 days ago | parent [-]

What are your thoughts on the load website operators are putting on themselves to block scrapers?