Does any of the LLM providers actually use llms.txt?

If I remember correctly this "standard" was setup by someone but without involvement of any of the major AI players.

▲ solumos 3 minutes ago | parent | next [-]

No, requesting "Accept: text/markdown" in the headers and returning markdown is the more agreed upon standard at this point.[0]

[0] - https://acceptmarkdown.com/

▲ HermanMartinus 3 hours ago | parent | prev | next [-]

I can definitively say llms.txt is not used by any AI players. I run a blogging platform with around 80k blogs and /llms.txt is not requested by anything (other than humans checking to see if there's an llms.txt path).

All regular pages are aggressively scraped to the extent it's a problem I have to consistently manage, but not llms.txt.

▲ nickserv 3 hours ago | parent | next [-]

I'm seeing quite a bit of request for these on my work's GitBook documentation site.

But perhaps these are developers specifically targeting these pages to feed whatever LLM they are using.

▲ isaachinman 3 hours ago | parent | prev | next [-]

How is a static blog being scraped a problem? Do you not use a CDN?

▲

nickserv 3 hours ago | parent | next [-]

> a blogging platform with around 80k blogs

But nah, I'm sure OP doesn't know about CDNs.

▲

the_real_cher 3 hours ago | parent | prev [-]

Are all blogs static though?

	▲	johannes1234321 2 hours ago \| parent [-]
		Very few blogs require frequent updates. Even with user comments.

▲ sunshine-o 2 hours ago | parent | prev | next [-]

Amazing, I didn't know.

So it get even stranger, I am the only one reading those /llms.txt ...

▲ 0123456789ABCDE 3 hours ago | parent | prev [-]

> I can definitively say llms.txt is not used by any AI players.

  https://developers.openai.com/llms.txt
  https://docs.anthropic.com/llms.txt
  https://geminicli.com/llms.txt
  https://github.com/llms.txt
  https://docs.aws.amazon.com/llms.txt
  https://openrouter.ai/docs/llms.txt

▲

m4tthumphrey 2 hours ago | parent [-]

OP clearly meant that the AI players are not reading and/or honouring llms.txt of other websites when scraping.

	▲	0123456789ABCDE 2 hours ago \| parent [-]
		i stand corrected, but what was clear to you, obviously was not clear to me.

▲ 0123456789ABCDE 2 hours ago | parent | prev [-]

yes, they do.

anyone who's, even slightly, clued into how agents access documentation, has been making changes to their pages. ex: https://searchtxt-web.fly.dev/search?q=aws