Remix.run Logo
nananana9 11 hours ago

Package managers are a failed experiment.

We have libraries like SQLite, which is a single .c file that you drag into your project and it immediately does a ton of incredibly useful, non-trivial work for you, while barely increasing your executable's size.

The issue is not dependencies themselves, it's transitive ones. Nobody installs left-pad or is-even-number directly, and "libraries" like these are the vast majority of the attack surface. If you get rid of transitive dependencies, you get rid of the need of a package manager, as installing a package becomes unzipping a few files into a vendor/ folder.

There's so many C libraries like this. Off the top of my head, SQLite, FreeType, OpenSSL, libcurl, libpng/jpeg, stb everything, zlib, lua, SDL, GLFW... I do game development so I'm most familiar with the ones commonly used in game engines, but I'm sure other fields have similarly high quality C libraries.

They also bindings for every language under the sun. Rust libraries are very rarely used outside of Rust, and C#/Java/JS/Python libraries are never used outside their respective language (aside form Java ones in other JVM langs).

staticassertion 4 hours ago | parent | next [-]

They're not a failed experiment. No one has ever "experimented" by making a safe package manager for their new language. And it is not that insane to do so. Very basic things will get you very far:

1. Packages should carry a manifest that declares what they do at build time, just like Chrome extensions do. This manifest would then be used to configure its build environment.

2. Publishers to official registries should be forced to use 2FA. I proposed this a decade ago for crates.io and people lost their minds, like I was suggesting we drag developers to a shed to be shot.

3. Every package registry should produce a detailed audit log that contains a "who, what, when". Every build/ command should be producing audit logs that can be collected by endpoint agents too.

4. Every package registry should support TUF.

5. Typosquatting defenses should be standard.

etc etc etc. Some of this is hard, some of this is not hard. All of this is possible. No one has done it, so it's way too early to say "package managers can't be made safe" when no one has tried.

philipwhiuk 2 hours ago | parent | next [-]

> Publishers to official registries should be forced to use 2FA. I proposed this a decade ago for crates.io and people lost their minds, like I was suggesting we drag developers to a shed to be shot.

How is this enforced when it's pushed via a pipeline?

staticassertion an hour ago | parent [-]

Your account is separate from your publishing. That is, in order to go to my account to change configuration values, 2FA must be required.

Publishing should be handled via something like Trusted Publishing, which would leverage short lived tokens and can integrate with cryptographic logs for publish information (ie: "Published from the main branch of this repo at this time").

bbkane 4 hours ago | parent | prev [-]

Love these ideas!

pjc50 10 hours ago | parent | prev | next [-]

Package managers are now basically a requirement for language adoption. Doing it manually is not a solution, in an automated world.

What is a problem is library quality. Which is downstream of nobody getting paid for it, combined with an optimistic but unrealistic "all packages are equal" philosophy.

> High quality C libraries

> OpenSSL

OpenSSL is one of the ones where there's a ground up rewrite happening because the code quality is so terrible while being security critical.

On the other end, javascript is uniquely bad because of the deployment model and difficulty of adding things to the standard library, so everything is littered with polyfills.

hresvelgr 8 hours ago | parent [-]

> Package managers are now basically a requirement for language adoption. Doing it manually is not a solution, in an automated world.

Absolute nonsense. What does automated world even mean? Even if one could infer reasonably, it's no justification. Appealing to "the real world" in lieu of any further consideration is exactly the kind of mindlessness that has led to the present state of affairs.

Automation of dependency versions was never something we needed it was always a convenience, and even that's a stretch given that dependency hell is abundant in all of these systems, and now we have supply chain attacks. While everyone is welcome to do as they please, I'm going to stick to vendoring my dependencies, statically compiling, and not blindly trusting code I haven't seen before.

nailer 7 hours ago | parent | next [-]

> Automation of dependency versions was never something we needed

How do you handle updating dependencies then?

PedroBatista 7 hours ago | parent | prev | next [-]

Relax, while mentioning the real world without any criticism for the soundness of the solution is absolute nonsense, some would say idiotic, thinking only in the absolute best solution given your narrow world view is not any better.

hresvelgr 5 hours ago | parent [-]

While I agree that my view is narrow, the "best solution" in question is what we used to do, and it was fine. There are still many places that manually manage dependencies. Fundamentally automatic software versioning is an under-developed area in need of attention, and technologies like semantic versioning which are ubiquitous are closer to suggestions, and not true indicators of breaking changes. My personal view is that fully automatic dependency version management is an ongoing experiment and should be treated as such.

pjc50 6 hours ago | parent | prev [-]

> What does automated world even mean?

People are trying to automate the act of programming itself, with AI, let alone all the bits and pieces of build processes and maintenance.

squeaky-clean 8 minutes ago | parent | prev | next [-]

SQLite had 6 CVE's last year. FreeType had an RCE bug published last year. libcurl's last CVE reported was 2 weeks ago. Libpng had 4 vulnerabilities published this year....

doginasuit 6 hours ago | parent | prev | next [-]

> We have libraries like SQLite, which is a single .c file that you drag into your project and it immediately does a ton of incredibly useful, non-trivial work for you, while barely increasing your executable's size.

I'm not sure why you believe this is more secure than a package manager. At least with a package manager there is an opportunity for vetting. It's also trivial that it did not increase your executable's size. If your executable depends on it, it increases its effective size.

hvb2 10 hours ago | parent | prev | next [-]

If you're developing for the web your attack surface is quite a bit bigger. Your proposed solution of copying a few files might work but how do you keep track of updates? You might be vulnerable to a published exploit fixed a few months ago. A package manager might tell you a new version is available. I don't know how that would work in your scenario.

10 hours ago | parent [-]
[deleted]
layer8 10 hours ago | parent | prev | next [-]

For some reason, NPM is the only ecosystem with substantial issues with supply-chain attacks.

SoKamil 9 hours ago | parent | next [-]

Popularity

fsflover 2 hours ago | parent [-]

The number of issues is disproportionately larger than the one for Debian.

techterrier 10 hours ago | parent | prev | next [-]

apart from that python one the other day

indy 9 hours ago | parent | prev | next [-]

The culture within the npm/js community has mainly been one of using the package manager rather than "re-inventing the wheel", as such the blast radius of a compromised package is much greater

progmetaldev 5 hours ago | parent | next [-]

It's more to do with the standard library being so barren of common application needs, and looking for a solution that the community has gotten behind. Axios has been a common dependency in many codebases, because it is a solid solution that many have already used. Every developer could try building all the libraries that they would reach for themselves, but then each company has now taken on the task of ensuring their own (much larger) codebase is free from security issues, on top of taking care of their own issues and bugs.

christophilus 7 hours ago | parent | prev [-]

It’s not just NPM, though. Every Rails project and every Rust project I’ve seen ended up with massive numbers of dependencies vs what an equivalent project in Go or C# would have needed.

5 hours ago | parent | next [-]
[deleted]
anthk 4 hours ago | parent | prev [-]

CPAN too, just try Hailo under Perl to test an old-fashioned chatbot based on Markov chains where very small LLM's and Hailo converge if used with the advanced training options for it. Yes, it will pull tons of dependencies, (less with cpanminus if run with 'cpanm -n Hailo'), but contrary to NPM, Pip and the like CPAN's repos are highly curated and before PHP and ubiquitoous Python Perl was used everywhere, from a sysadmin language (better than Bash/Sh for sure) to CGI, IRC bots and whatnot. How many issues did we have? Zero or near zero.

rvz 5 hours ago | parent | prev [-]

It is because it has the lowest barrier to entry with no quality control. Ever.

This is what happens when there is no barrier to entry and it includes everyone who has no idea what they are doing in charge of the NPM community.

When you see a single package having +25 dependencies, that is a bad practice and increases the risk of supply chain attacks.

Most of them don't even pin their dependencies and I called this out just yesterday on OneCLI. [0]

It just happens that NPM is the worst out of all of the rest of the ecosystems due to the above.

[0] https://news.ycombinator.com/item?id=47577183

allreduce 9 hours ago | parent | prev | next [-]

I don't think this community of professionals is going to come around to a solution which requires marginally more effort.

If no one checks their dependencies, the solution is to centralize this responsibility at the package repository. Something like left-pad should simply not be admitted to npm. Enforce a set of stricter rules which only allow non-trivial packages maintained by someone who is clearly accountable.

Another change one could make is develop bigger standard libraries with all the utilities which are useful. For example in Rust there are a few de facto standard packages one needs very often, which then also force you to pull in a bunch of transitive dependencies. Those could also be part of the standard library.

This all amounts to increasing the minimal scope of useful functionality a package has to have to be admitted and increasing accountability of the people maintaining them. This obviously comes with more effort on the maintainers part, but hey maybe we could even pay them for their labor.

vincnetas 10 hours ago | parent | prev | next [-]

no no, please we don't want to get back to dragging files to your project to make them work.

Tarraq 7 hours ago | parent [-]

And manual FTP uploads, while we're at it.

voidfunc 10 hours ago | parent | prev | next [-]

I'd really like to see package managers organized around rings where a very small core of incredibly important stuff is kept in ring 0, ring 1 gets a slightly wider amount of stuff and can only depend on ring 0 dependencies and then ring 2+ is the crapware libraries that infect most ecosystems.

But maybe that's not the right fit either. The world where package managers are just open to whatever needs to die. It's no longer a safe model.

regularfry 8 hours ago | parent | next [-]

The OS distro model is actually the right one here. Upstream authors hate it, but having a layer that's responsible for picking versions out of the ecosystem and compiling an internally consistent grouping of known mutually-compatible versions that you can subscribe to means that a lot of the random churn just falls away. Once you've got that layer, you only need to be aware of security problems in the specific versions you care about, you can specifically patch only them, and you've got a distribution channel for the fixes where it's far more feasible to say "just auto-apply anything that comes via this route".

That model effectively becomes your ring 1. Ring 0 is the stdlib and the package manager itself, and - because you would always need to be able to step outside the distribution for either freshness or "that's not been picked up by the distro yet" reasons - the ecosystem package repositories are the wild west ring 2.

In the language ecosystems I'm only aware of Quicklisp/Ultralisp and Haskell's Stackage that work like this. Everything else is effectively a rolling distro that hasn't realised that's what it is yet.

swiftcoder 10 hours ago | parent | prev | next [-]

In practice, "ring 0" is whatever gets merged into your language's standard library. Node and python both have pretty expansive standard libraries at this point, stepping outside of those is a choice

anakaine 10 hours ago | parent | prev [-]

Malicious actor KPI: affect a Ring 0 package.

pie_flavor 9 hours ago | parent | prev | next [-]

Rust libraries are infrequently used outside of Rust because if you have the option, you'd just use Rust, not the ancient featureless language intrinsically responsible for 70% of all security issues. C libraries are infrequently used in Rust outside of system libc, for the same reason; I go and toggle the reqwest switch to use rustls every time, because OpenSSL is horrendous. This is also why you say 'rarely' instead of 'never', when a few years ago it was 'never'; a few years from now you'll say 'uncommonly', and so on. The reason C libraries are used is because you don't feel like reimplementing it yourself, and they are there; but that doesn't apply more to C libraries than Rust libraries, and the vast majority of crates.io wouldn't be usefully represented in C anyway, or would take longer to bind to than to rewrite. (No, nobody uses libcurl.) Finally, this only happens in NPM, and the Rust libraries you pull in are all high-quality. So this sounds like a bunch of handwaving about nonsense.

physicsguy 8 hours ago | parent [-]

Rust is terrible for pulling in hundreds of dependencies though. Add tokio as a dependency and you'll get well over 100 packages added to your project.

estebank an hour ago | parent | next [-]

Even side stepping that tokio no longer pulls multiple packages, it used to be split into multiple packages, in the same way that KDE in Rust would be hundreds of packages.

Rust projects tend to take their project and split it into many smaller packages, for ease of development, faster compiles through parallelization, ensuring proper splitting of concerns, and allowing code reuse by others. But the packages are equivalent to a single big package. The people that write it are the same. They get developed in tandem and published at the same time. You can take a look at the del tree for ripgrep, and the split of different parts of that app allows me to reuse the regex engine without dealing with APIs that only make sense in the context of a CLI app or pulling in code I won't ever use (which might be hiding an exploit too).

Counting 100 100 line long crates all by the same authors as inherently more dangerous than 1 10000 line long crate makes no sense to me.

pie_flavor 8 hours ago | parent | prev [-]

pin-project-lite is the only base dependency, which itself has no dependencies. If you enable the "full" feature, ie all optional doodads turned on (which you likely don't need), it's 17: bytes, cfg-if, errno, libc, mio, parking_lot+parking_lot_core+lock_api, pin-project-lite, proc_macro2+quote+syn+unicode-ident, scopeguard, signal-hook-registry, smallvec, and socket2. You let me know which ones you think are bloat that it should reimplement or bind to a C library about, and without the blatant fabrication this time.

jonkoops 8 hours ago | parent | prev | next [-]

> We have libraries like SQLite, which is a single .c file that you drag into your project

You are just swapping a package manager with security by obscurity by copy pasting code into your project. It is arguably a much worse way of handling supply chain security, as now there is no way to audit your dependencies.

> If you get rid of transitive dependencies, you get rid of the need of a package manager

This argument makes no sense. Obviously reducing the amount of transitive dependencies is almost always a good thing, but it doesn't change the fundamental benefits of a package manager.

> There's so many C libraries like this

The language with the most fundamental and dangerous ways of handling memory, the language that is constantly in the news for numerous security problems even in massively popular libraries such as OpenSSL? Yes, definitely copy-paste that code in, surely nothing can go wrong.

> They also bindings for every language under the sun. Rust libraries are very rarely used outside of Rust

This is a WILD assumption, doing C-style bindings is actually quite common. YOu will of course then also be exposing a memory unsafe interface, as that is what you get with C.

What exactly is your argument here? It feels like what you are trying to say is that we should just stop doing JS and instead all make C programs that copy paste massive libraries because that is somhow 'high quality'.

This seems like a massively uninformed, one-sided and frankly ridiculous take.

nananana9 7 hours ago | parent [-]

> You are just swapping a package manager with security by obscurity by copy pasting code into your project

You should try writing code, and not relying on libraries for everything, it may change how you look at programming and actually ground your opinions in reality. I'm staring at company's vendor/ folder. It has ~15 libraries, all but one of which operate on trusted input (game assets).

> fundamental benefits of a package manager.

I literally told you why they don't matter if you write code in a sane way.

> doing C-style bindings is actually quite common

I know bindings for Rust libraries exist. Read the literal words you quoted. "Rust libraries are very rarely used outside of Rust". Got some counterexamples?

vablings 44 minutes ago | parent [-]

https://github.com/memflow/memflow

https://github.com/PyO3/pyo3

https://slint.dev/

https://github.com/dora-rs/dora

It is VERY common in existing codebases that are migrating from C++/C to make heave use of FFI/ existing C

TacticalCoder 2 hours ago | parent | prev | next [-]

Then you've got ecosystems like Clojure where many projects are just considered done and used by many. You can pin these (and be warned if a new version still comes out, say for an actual security fix). There are Clojure projects so stable, without any know exploit (we're certainly not talking about daily npm exploits here), that haven't been updated in years because they are... Done. Simply done. Perfection.

Something to reflect upon too.

anthk 4 hours ago | parent | prev | next [-]

Package managers are older than some users here. From CPAN/CTAN to ports under BSD's.

Some pm's are badly maintained (Pip/NPM), while others are curated enough.

Again, if you have GNU/Linux installed, install Guix, read the Info manual on 'guix import' and just create a shell/container with 'guix shell --container' (and a manifest package created from guix import) and use any crap you need for NPM in a reproducible and isolated way. You $HOME will be safe, for sure.

victorbjorklund 7 hours ago | parent | prev [-]

I think you can do copy paste in most languages. But it will be a pain to update when there are improvements / security fixes.

You got a project with 1-2 depencies? Sure. But if you need to bring in 100 different libs (because you bring in 10 libs which in turn brings in 10 libs) good luck.

skydhash 5 hours ago | parent [-]

> But if you need to bring in 100 different libs (because you bring in 10 libs which in turn brings in 10 libs

So don’t?

With manual deps management, everyone soon gravitates to a core set of deps. And libraries developer tends to reduce their deps needs, That’s why you see most C libraries deals with file formats, protocols, and broad concerns. Smaller algorithms can be shared with gists and blog articles.