robots.txt is a way of leaving the door unlocked but kindly asking bots to stay outside.

account42 2 hours ago | parent | next [-]

Which in a law-abiding society should be enough. It's also how we do things in the real world in many cases - i.e. here you can just write on your mailbox "no ads" and companies have to respect that.

Even when we do actually put physical locks on things they are mostly there to show that someone breaking in did so intentionally and not at all designed to prevent motivated attackers.

	▲	dpark an hour ago \| parent [-]
		> here you can just write on your mailbox "no ads" and companies have to respect that Where do you live? In the US it’s actually illegal for anyone except the USPS to deliver to a mailbox.

▲

dpark an hour ago | parent | prev [-]

You might be interested to know that entering an unlocked door into a space you do not have permission to be in is still illegal.

▲

throw1234567891 an hour ago | parent [-]

You might be interested to know that the “illegality” depends on the intent. If I rest on your unlocked door handle, it opens, I enter, it’s an accident.

▲

dpark 22 minutes ago | parent [-]

Sorry, what? In this scenario are you claiming that you accidentally fell inside the restricted area because you were leaning on the door? Or are you claiming that you accidentally opened the door and then walked through intentionally? In the former case, you are guilty of breaking and entering in most US jurisdictions if you don’t promptly get out. Any sane court would likely agree an accidental trespass is probably not a criminal act, but it’s not an accident if you stay. In the latter case, you’re clearly trespassing illegally.

Also this has gotten pretty far away from the web scraping scenario. There’s no door accidentally opening here.

	▲	dminik 8 minutes ago \| parent [-]
		Oops, I just accidentally fell into every website. Don't know how that happened ...