> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.

I think you've hit the nail on the head here. I think that's a big reason why people want to ban bots.

▲

aomix 4 hours ago | parent | next [-]

The social contract of accepting scraping for visibility was always tenuous and is now fully dead. But it’s shocking the number of people who are essentially victim blaming here. You should spend YOUR time to optimize your free website so MY use case is not impacted. How is that not incredibly selfish on its face?

▲

iwontberude 3 hours ago | parent | next [-]

[dead]

▲

autoexec 3 hours ago | parent | prev [-]

The social contract for putting a website on the public internet involves following standards and making your website accessible. You SHOULD make sure that your website is readable to people no matter what user agent they are using as long as it's standards compliant.

For example, making a website that only works in Edge and refuses to load for anything else would be a shitty practice. You are free to do it. There's no one stopping you, but you should expect people to bitch about it and they might think you're kind of a jerk or maybe just bad at making websites. It's the same thing when a site actively rejects traffic from curl, wget, or any other common utility that many people find useful.

▲

bakugo 3 hours ago | parent [-]

The social contract for putting a website on the public internet was built on the assumption that browsers would be used to access those websites, and that most browsers would generally behave in a similar way (click on page, display page, pretty simple).

LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all. They do not display the page for a real person to read, they merely ingest its contents and then spit out something completely different.

▲

autoexec 2 hours ago | parent | next [-]

> The social contract for putting a website on the public internet was built on the assumption that browsers would be used to access those websites, and that most browsers would generally behave in a similar way (click on page, display page, pretty simple).

No, not browsers, user agents (some of which would be browsers) and there was never an assumption of how a website would behave on the user's end, that's the job the user agent. All that HTML and CSS are only suggestions, but the power was always intended to be left to the user to decide if/how they wanted that data presented to them and it was always intended that the user be able to choose whatever tools they wanted to collect, process, and display content pulled down from the internet. That same principle is how we have ad-blockers. You are free to infest your website with ads, but as the person requesting the website I'm under zero obligation to display any part of that site I don't want.

> LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all.

LLMs are just another tool used by people to collect and process the information available on websites. Maybe there is a distinction to be made between people using LLMs to get web content and corporations scraping websites to take training data, but even scraping has always been a common and expected practice. It's the current scale that is making things different.

▲

coreyp_1 2 hours ago | parent | prev [-]

Don't tell this person about translation plugins. They change everything! The end user never reads what the website actually served!

	▲	autoexec 2 hours ago \| parent [-]
		It's probably a pretty common misconception that websites should have total control over how their content is viewed, but it's strange to see that on a website like this. Part of the reason apps are so popular with companies is that it gives them control over how content is presented which is something webpages were never intended to give them.

▲

paul7986 4 hours ago | parent | prev [-]

Exactly and Cloudflare (there should be others out there too - open source options) has proposed that AI bots need to pay for access to our websites. If they and or others could pull off blocking AI access until it pays creators then AI is forced to pay as it should and always should've!

Overall systems and different business models need to be created that forces AI to pay it's fair share! Elon says there will be an abundance thanks to AI and we wont have to work -- ok how's that gonna work without different systems in place paying us?

I feel strongly about this topic and proposed some systems in this Substack post https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans...