How are the download rates of these things so high?

Are there 1000 people running 100 CI pipelines/day where downloads aren’t cached?

swiftcoder 10 hours ago | parent | next [-]

Yep, every single transitive downstream consumer pulls a fresh copy on every CI run. It's kind of ridiculous, and it's like this in every modern ecosystem.

For example, this is a completely empty rust crate, which I started years ago and never released, and it's still downloaded multiple times per day... https://crates.io/crates/ruble

▲

dwoxctbvgq 8 hours ago | parent | next [-]

Could it be caused by Crater runs?

https://github.com/rust-lang/crater

▲

ModernMech 3 hours ago | parent | prev | next [-]

Oh so you're the reason we got btleplug!:P

https://github.com/deviceplug/btleplug

fwiw the downloads are probably automated crates.io indexers.

▲

fainpul 10 hours ago | parent | prev [-]

I'm confused. Why did you publish this? Who "uses" it?

▲

swiftcoder 9 hours ago | parent [-]

I published it purely to placeholder the name (and then life got in the way, and I never released the actual library). Nobody uses it at all, but some CI system is still polling at regular intervals...

▲

BobbyTables2 4 hours ago | parent | next [-]

Wonder if someone has an internal crate named the same, but yours is getting pulled down for some tertiary use (license check, etc).

Stuff like Docker that defaults to downloading things from Internet repos is kinda scary. There isn’t always a clear dividing line between the thing as a “tool” and the thing as a “repo”.

At least with “git”, I know they I’m not going to end up cloning an artifact from an Internet repo just because I made a small typo…

▲

fainpul 8 hours ago | parent | prev | next [-]

But if no other crate depends on your crate and nobody directly uses it, why would any CI system download it?

	▲	swiftcoder 6 hours ago \| parent [-]
		I think some of the CI systems just monitor all published crates. For example, docs.io potentially pulls it to see if there is any new documentation

▲

qludes 8 hours ago | parent | prev [-]

So someone could reserve all the pronounceable crate names?

	▲	swiftcoder 6 hours ago \| parent [-]
		Yeah, it’s a known problem in the (non-namespaced) crates.io ecosystem. If someone else wants this name for their Bluetooth LE crate, I’ll happily hand over the keys, but obviously not every crate-squatter is so principled

▲

kingstnap 14 hours ago | parent | prev | next [-]

Yep. True webscale is astronomical levels of inefficient and wasteful usage of bandwidth and CPU.

▲

nerdponx 13 hours ago | parent | prev | next [-]

It doesn't help that container images typically require you to "opt in" to sharing a cache with the host system. Docker even has a special file mount type for it. I find that people on average don't have a lot of interest in making such things work, they see that it's not available right in front of them and shrug and say "oh well" because ultimately it doesn't hurt them much.

	▲	phito 12 hours ago \| parent [-]
		Yeah I'd say most people view it as such, until they hit (docker hub) rate limits.

▲

zahlman 12 hours ago | parent | prev | next [-]

It seems to happen in the Python world, too. Check out the stats for the most commonly downloaded packages from PyPI (https://pypistats.org/top), and also consider which packages are on the list.

(Pip used to be right up there next to setuptools. It's still pretty popular — https://pypistats.org/packages/pip — but uv has clearly changed the equation.)

▲

GenerocUsername 15 hours ago | parent | prev [-]

Yes