Respecting robots.txt is not a legal obligation. If you don’t want the traffic, ban the offending IP block or ASN, or put the origin behind Cloudflare for their crawler blocker.

You could go down the rabbit hole and try to find someone at Amazon to tell their crawler to chill. Don’t expect success but would make a fun blog post.

https://www.cloudflare.com/learning/ai/how-to-block-ai-crawl...

▲

pera 21 hours ago | parent [-]

My concern is more in relation to trying to use an endpoint from non-public source code, it seems negligent to randomly try endpoints like this

	▲	toomuchtodo 21 hours ago \| parent [-]
		Whether it’s negligent isn’t terribly relevant. Is it illegal? It is not, they aren’t bypassing access controls. No different than using Shodan, scanning public IPs, crawling open directories, etc. It would be different if they were attempting to brute force credentials to access an endpoint, but they aren’t.