Remix.run Logo
Dotnaught 3 days ago

https://localghost.dev/robots.txt

User-Agent: * Allow: /

charles_f 2 days ago | parent | next [-]

I contacted the author, she said because no-one respects it, she hasn't even tried.

thrance 3 days ago | parent | prev [-]

Not like anyone respects that anyways.

a3w 3 days ago | parent [-]

Also, I wanted tldrbot to summarize this page. /s

criddell 3 days ago | parent [-]

That's a good point. It's not a black and white issue.

I personally see a bot working on behalf of an end user differently than OpenAI hoovering up every bit of text they can find to build something they can sell. I'd guess the owner of localghost.dev doesn't have a problem with somebody using a screen reader because although it's a machine pulling the content, it's for a specific person and is being pulled because they requested it.

If the people making LLM's were more ethical, they would respect a Creative Commons-type license that could specify these nuances.