Remix.run Logo
grahamj a day ago

Note I said outgoing, as in the crawlers should be throttling themselves

bflesch a day ago | parent [-]

Sorry for misunderstanding your point.

I agree it should be throttled. Maybe they don't need to throttle because they don't care about cost.

Funny thing is that servers from AWS were trying to connect to my system when I played around with this - I assume OpenAI has not moved away from AWS yet.

Also many different security scanners hitting my IP after every burst of incoming requests from the ChatGPT crawler Azure IP ranges. Quite interesting to see that there are some proper network admins out there.

jillyboel a day ago | parent | next [-]

They need to throttle because otherwise they're simply a DDoS service. It's clear they don't give a fuck though, like any bigtech company. They'll spend millions on prosecuting anyone who dares to do what they perceive as a DoS attack against them, but they'll spit in your face and laugh at you if you even dare to claim they are DDoSing you.

grahamj a day ago | parent | prev [-]

yeah it’s fun out on the wild internet! Thankfully I don’t manage something thing crawlable anymore but even so the endpoint traffic is pretty entertaining sometimes.

What would keep me up at night if I was still more on the ops side is “computer use” AI that’s virtually indistinguishable from a human with a browser. How do you keep the junk away then?