Thank you! I think that's a great idea, and will definitely look into implementing this.

I would just like to add some cautionary anec-data: there are widespread cases in certain jurisdictions where rightsholders are known to seed the same torrents themselves, just to turn around and send love letters to leechers that connect to them. A good example is Germany with movies and TV shows.

Now, I don't know if, say, Wolters Kluver would/does the same thing, and what the realistic risk of an individual receiving such a letter is, but I think it makes it worthwhile to go over the actual law in your jurisdiction before diving head first on things like this.

I'm not saying it's wrong to seed these things, I'm just saying it might be a good idea to weigh the risks if you don't have a cool 500€ in cash to part ways with.

DMCA letter sounds like small potatoes when we talk about letting random people write stuff to your disk space and using your bandwidth.

Happens every day in the US. Mostly video and music (MPA/RIAA). There's also been some effort put into extorting ISPs for the activities of their customers, but the effectiveness of that is still being determined as cases work their way through the court system. We should have a better idea this summer after the supreme court decides on the $1 billion in damages one ISP was ordered to pay to a bunch of RIAA labels.

It will be a lot more profitable to sue ISPs than it is to try to sue poor parents and grandparents for what children do online.

In Germany you can expect to get a letter from some law firm, confirmed by some judge that orders you to pay 100s or 1000s of euros if you don't use a vpn

They will attempt to download DMCA files from you as often as possible and then calculate the amount of times times price of the product to come up with a fictional damages amount

I'm in Italy. Most people I know have been pirating movies, series and games [1] for 20+ years, via torrents and eMule (yes, eMule is still big in Italy), and nobody ever received any letters.

But there's a big exception: as soon as you start pirating soccer, they're going to come after you.

[1] I've personally stopped pirating games a long time ago, because it's just easier and safer to buy them on Steam or GOG. Gaben was 100% right when he said "Piracy is almost always a service problem".

In France, for movies/music you get 2 warning letters, then a scary one that says you can now get to court possibly.

Didn't really hear about people getting fines for this, but the law exists.

US colocated seedbox with ~10k film and tv torrents seeding at any given time, the last letter I got was ~2014 IIRC, before that it was several a year. I never responded to any of them.

I don't think I'm especially good at covering my tracks, so either they've abandoned individual enforcement in favor of going after distributors or they no longer bother with non-residential IPs.

I've heard Finland sends out letters, same with Japan. Are there actual consequences, or can they just be ignored?

Norway I haven't heard of anyone getting anything in the past decade. The ISPs supposedly get letters from lawyers but just toss them, since the intersection of the burden of proof and our privacy laws make it such that nothing can really be done.

I think there was some ISP that gave out names and IP addresses to one of the firms years ago, but nothing happened and the police said "we have better things to do".

I haven't heard about someone ever getting a letter for seeding books, but maybe I'm lucky. In any case, I'll add a notice to the README, thank you for the suggestion.

You configure Levin to "always leave 2GB available". Levin checks the available diskspace using a simple statvfs call, deducts 2GB, and sees that as its budget. It then checks your diskspace every minute (more or less, depending on the device) to see if anything changes. If more free space is suddenly available, it will download more content. If there's less than 2GB available, it will immediately start deleting its own files until 2GB are free.

You can check the content of the torrents, just like any torrent. The client isn't a "one shot" LLM produce, I've been spending quite some time on it. What actual concerns do you have?

Just like you can read source code written by humans (and should if you take this stance) you can also read source code generated by LLMs. Then, when you find something unsavory and feel that your sentiment is warranted, make a contribution.

Who do you think Anna is

What do you have in mind?

It's related in the sense that it works in the background, using the spare resources you have. Whether you see the thing it does as a good thing or theft is really up to you. I guess some people had their own reasons for not supporting the SETI@home objectives either. In any case, I'm perfectly happy with an analogy like "it's like going to the library, making a copy of all the books and making the copies available for everyone for free".

Yes. It's a basic scraper that fetches the document, parses it for URLs using regex, then fetches all those, repeat forever.

I've done honeypot tests with links in html comments, links in javascript comments, routes that only appear in robots.txt, etc. All of them get hit.

Absolutely.

I assume that there are data brokers, or AI companies themselves, that are constantly scraping the entire internet through non-AI crawlers and then processing data in some way to use it in the learning process. But even through this process, there are no significant requests for LLMs.txt to consider that someone actually uses it.

I think it depends. LLMs now can look up things on the fly to bypass the whole "this model was last updated in December 2025" issue of having dated information. I've literally told Claude before to look up something after it accused me of making up fake news.

I also noticed this LLMs.txt at bun.sh, so for me it looks like some sort of advertising.

Thanks for the clarification.

>for use in LLMs such as Claude (1)

From your website, it seems to me that LLMs.txt is addressed to all LLMs such as Claude, not just 'individual client agents' . Claude never touched LLMs.txt on my servers, hence the confusion.

1. https://llmstxt.org

And to try to get them execute bb(5) ;)

Actually, I noticed an interesting behaviour in LLMs.

We had made a docs website generator (1) that works with HTML (2) FRAMESET and tried to parse it with Claude.

Result: Claude doesn't see the content that comes from FRAMESET pages, as it doesn't parse FRAMEs. So I assume what they're using is more or less a parser based on whole-page rendering and not on source reading (including comments).

Perhaps, this is an option to avoid LLM crawlers: use FRAMEs!

1. https://github.com/tirrenotechnologies/hellodocs

2. https://www.tirreno.com/hellodocs/

Good question, at least OAI-SearchBot is hitting robots.txt.

I assume the real issue is that what overloads the servers like security bots, SEO crawlers, and data companies — are the ones that don't respect robots.txt in full, but they wouldn't respect LLMs.txt either.

In tirreno (our product), we catch every resource request on the server side, including LLMs.txt and agents.md, to get the IP that requested it and the UA.

What I've seen from ASNs is that visits are coming from GOOGLE-CLOUD-PLATFORM (not from Google itself), and OVH. Based on UA, users are: WebPageTest, BuiltWith, and zero LLMs based on both ASN and UA.

1. https://github.com/tirrenotechnologies/tirreno

This is only done at the DNS level, so using a different DNS (such as Quad9) solves that issue. For background info, I can recommend [1, 2].

[1]: https://www.youtube.com/watch?v=Uxmu25mUZgg [2]: https://cuiiliste.de/

Yay, MITM in the wild :)

I got it on my phone, but not with my local ISP.

I can access the site just fine from Germany. Tried Vodafone and Congstar but I don't use their DNS servers.

In other news, Project Gutenberg not completely censored in Germany. Well done, Germany. https://cand.pglaf.org/germany/index.html

And the works that previously had lead to Project Gutenberg being unavailable from Germany IP addresses will go into public domain in 2027.

I'm on Vodafone in Spain and I see

> Error code: PR_CONNECT_RESET_ERROR

If I try the http version, I get redirected to https://bloqueadaseccionsegunda.cultura.gob.es/ (which also fails with PR_CONNECT_RESET_ERROR).

If it wasn't enough that half the internet gets unusable whenever there is football on TV (which is fucking stupid), now we're also getting rid of free (text!) information it seems.

For Virgin Media, redirects to https://assets.virginmedia.com/site-blocked.html

> Virgin Media has received an order from the High Court requiring us to prevent access to this site.

Appears that UK EE has it blocked too. Tried this morning waiting for the train in to work.

https://en.wikipedia.org/wiki/Anna%27s_Archive#United_Kingdo...

>In December 2024, the UK Publishers Association won an order from the High Court of Justice requiring major ISPs to block Anna's Archive and other copyright-infringing sites, extending a list of sites blocked since 2015 under section 97A of the Copyright, Designs and Patents Act

Others have already posted, but the biggest domestic British ISPs block a variety of things, like SciHub, Libgen, Pirate Bay, or Anna's Archive. Coverage varies a lot though, so I assume ISPs have some discretion and enforcement is patchy.

Also in the UK and can also see it fine.

I wonder if it's blocked simply by DNS manipulation and therefore only people using the ISP DNS have issues.

In the UK I'm currently getting:

Hmmm… can't reach this page

Check if there is a typo in annas-archive.li.

DNS_PROBE_FINISHED_NXDOMAIN

I am in the UK and I can't see it unless I use a VPN. I get

This site can’t provide a secure connection annas-archive.li sent an invalid response. ERR_SSL_PROTOCOL_ERROR

Neither was the data LLMs were trained on.

At least this isn't saddled with a profit motive and the destruction of the consumer computing market.

Data doesn't belong to anyone, data is free :) zero-copy cost, delivery at speed of light.

It is. They gathered it. They stored it. They served it. That's how data should work and eventually will.

I’m not completely sure there was an API from the start. I’ve thought the only way is to get a DB dump (which sounds pretty reasonable to me).

They did already? OK. I somehow missed that.

Thing is, when they're pirating books, they're flagrantly violating ip laws in ways which big tech companies do themselves. When they're pirating music, they're flagrantly violating IP laws on a type of IP the big tech companies are directly selling. They're making a lot of new enemies.

they have individual IPFS links but they don't work 100% of the times

Thanks. I also enabled DNS-over-HTTPS for good measure.

https://wiki.archlinux.org/title/XDG_Base_Directory

https://specifications.freedesktop.org/basedir/latest

originally published as a standard in 2003, apparently.

HTTP equivalent:

https://www.rfc-editor.org/rfc/rfc8615

https://en.wikipedia.org/wiki/Well-known_URI

Big corps are bad, human culture is great. Thats the red thread here.

They only derived payment because other humans find value in the robots output. In the end it’s still benefiting humans.

Well yes, it would pretty clearly be classed as "prompt injection" given that it's trying to get the LLM to give them money or "persuade" a human to give them money. Of course the fault lies mainly with whoever deployed the LLM in the first place, but I still think it's misguided to try to convince LLM "agents" to make financial transactions in order to benefit yourself. It'd be much more ethical to just block them.

Well, I can go along with your analogy, and say that yeah, I'd be annoyed at the owner of the lemonade stand. Those marathon runners are trampling all my vegetables, and you're just trying to make a quick buck selling lemonade? People (me included) are annoyed at LLM creators scraping the web and gobbling up all copyrighted material, but it's mis-calibrated to get annoyed at Anna's Archive performing some sort of digital selling of stolen goods?