| ▲ | not2b 2 hours ago | |||||||
I understand why OpenAI is trying to reduce its costs, but it simply isn't true that AI crawlers aren't creating very significant load, especially those crawlers that ignore robots.txt and hide their identities. This is direct financial damage and it's particularly hard on nonprofit sites that have been around a long time. | ||||||||
| ▲ | stingraycharles 15 minutes ago | parent [-] | |||||||
These are ChatGPT and Claude Desktop crawlers we’re talking about? Or what is it exactly? Are these really creating significant load while not honoring robots.txt? Genuinely interested. | ||||||||
| ||||||||