Robots.txt anyone?

Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions.

Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we should make that clearer in the README.

	▲	bilekas 10 minutes ago \| parent \| next [-]
		> comparing publicly listed product prices across e-commerce sites Those prices and information is for the public viewers, the reason why some people have ROBOTS.txt for example is to reduce the traffic load that slop crawlers generate. The bandwidth is not free so why would you assume to ignore their ROBOTS.txt when you're not footing the bill ?
	▲	reyqn 3 hours ago \| parent \| prev \| next [-]
		https://news.ycombinator.com/item?id=47340079
	▲	zendist 2 hours ago \| parent \| prev [-]
		Regardless. You should still respect robots.txt..