Remix.run Logo
dotancohen 2 days ago

Yes it would matter. The AI might be I in your eyes, but it is still A.

varenc 2 days ago | parent | next [-]

I'll poke the bear:

As a user, the browser is my agent. If I'm directing an LLM to do something on a page in my browser, it's not that much different than me clicking a button manually, or someone using a screen reader to read the text on a page. The browser is my user agent and the specific tools I choose to use in my browser shouldn't be forbidden by a webpage. (that's why to this day all browsers still claim to be Mozilla...)

(This is very different than mass scraping web pages for training purposes. Those should absolutely respect robots.txt. There's a big difference between a user operated agentic-browser interacting with a web page and mass link crawling.)

debazel 2 days ago | parent | prev | next [-]

My understanding of this product is that this isn't an automated AI scraper, it's simply helping the user navigate pages they've already navigated to themselves.

If any type of AI based assistance is supposed to adhere to the robot.txt, then would you also say that AI based accessibility tools should refuse to work on pages blocked by robot.txt?

lolinder 2 days ago | parent | prev [-]

So is Chrome. Very artificial. It's still not a robot for the purposes of robots.txt.

What coherent definition of robot excludes Chrome but includes this?

dotancohen 16 hours ago | parent [-]

The meatsack at the end of the technology chain.

No meatsack in the loop making decisions and pushing the button? Robots.txt applies.