Just to add some more context. Zig cc is a wrapper around clang. It can handle cross compiling to specific glibc versions. See https://andrewkelley.me/post/zig-cc-powerful-drop-in-replace... I imagine it would help with the glibc problems they are taking about. Glibc tries to provide a backwards compatible abi.

> I think the problem in a nutshell is that it's not trivial(?) to build an executable on a modern Linux distro that links against an old glibc version number (and if it is trivial then it needs to be better communicated).

I wouldn't say it's trivial, but it's not rocket science either. Basically there are two main approaches. One is to just build inside a chroot or container with a sufficiently old distro inside. This is generally the path of least resistance because your build system doesn't really have to have any awareness of what's going on. You just build normally inside the chroot/container. The main downsides with this approach are that it's kind of wasteful (you have a whole distro's filesystem) and if you want to use a newer compiler than what the old distro in question shipped with you generally have to build it yourself inside said chroot/container.

The other main approach is to use a sysroot. gcc and clang both take an optional --sysroot parameter which is an alternative root for header and library lookups. This lets you use a compiler on the normal host, but old headers and libs. You can also bake this parameter in when compiling gcc (and also I assume clang, but less sure there) if you want a dedicated cross-toolchain.

The PEP-600 [0] Rationale section touches on this a bit. The basic problem is that there are things beyond glibc that would be nice to use from the environment for a number of reasons (security updates, avoiding clashes between multiple wheels that depend on the same lib, etc.), but since most libs outside of glibc and libstdc++ don't really have an ABI policy and the distros don't necessarily have a policy on what libraries are guaranteed to be present you sort of have to guess and hope for the best. While the initial list in PEP-513 [1] was a pretty good guess, one of the libraries chosen (libcrypt.so.1) got dropped in Fedora 30 and replaced with an ABI incompatible version. Crypto libraries are an example of something that's actually important to keep up to date so I find this rather unfortunate.

[0] https://peps.python.org/pep-0600/

[1] https://peps.python.org/pep-0513/

> It's the exception which is why it's talked about the most.

It's definitely not the only exception. libgl is another obvious example since different GPUs need different userland code. I would be surprised if there there had never been compat regressions in those.

If you have 3 hours or less of development experience on linux.

Way off topic but you just reminded me of all the time I spent playing Warlords 3 (not Warlords Battlecry 3, the original Warlords games were turn-based). One cool feature it had that I'm surprised I haven't really seen other turn-based games do is a "play by email" option similar to correspondence chess, except you're just emailing save files back and forth and the game makes importing/exporting the save files via email a bit more streamlined.

> Witness Windows, which has little to no problem with multiple libcs in a process

Only so long as you don't pass data structures from one to the other. The same caveats wrt malloc/free or fopen/fclose across libc boundaries still applies.

Well, not anymore, but only because libc is a system DLL on Windows now with a stable ABI, so for new apps they all share the same copy.

> The same applies to PAM word for word.

That's one of the reasons that OpenBSD is rather compelling. BSDAuth doesn't open arbitrary libraries to execute code, it forks and execs binaries so it doesn't pollute your program's namespace in unpredictable ways.

> It's true that there's something of a problem where two copies of a libc can't coexist in a process...

That's the meat of this article. It goes beyond complaining about a relatable issue and talks about the work and research they've done to see how it can be mitigated. I think it's a neat exercise to wonder how you could restructure a libc to allow multi-libc compatibility, but question why anyone would even want to statically link to libc in a program that dlopen's other libraries. If you're worried about a stable ABI with your libc, but acknowledge that other libraries you use link to a potentially different and incompatible libc thus making the problem even more complicated, you should probably go the BSDAuth route instead of introducing both additional complexity and incompatibility with existing systems. I think almost everything should be suitable for static linking and that Drepper's clarification is much more interesting than the rant. Polluting the global lib directory with a bunch of your private dependencies should be frowned upon and hides the real scale of applications. Installing an application shouldn't make the rest of your system harder to understand, especially when it doesn't do any special integration. When you have to dynamically link anyway:

> As for why you’d actually want to dlopen from a static executable, there’s one killer app: exokernels, loading (parts of) system-provided drivers into your process for speed.

If you're dealing with system resources like GPU drivers, those should be opaque implementations loaded by intermediaries like libglvnd. [1] This comes to mind as even more reason why dynamic dependencies of even static binaries are terrible. The resolution works, but it would be better if no zlib symbols would leak from mesa at all (using --exclude-libs and linking statically) so a compiled dependency cannot break the program that depends on it. So yes, I agree that dynamic dependencies of static libraries should be static themselves (though enforcing that is questionable), but I don't agree that the libc should be considered part of that problem and statically linked as well. That leads us to:

> ... when done right they don't even require both sides to share a malloc implementation

Better API design for libraries can eliminate a lot of these issues, but enforcing that is much harder problem in the current landscape where both sides are casually expected to share a malloc implementation -- hence the complication described in the article. "How can we force everything that exists into a better paradigm" is a lot less practical of a question than "what are the fewest changes we'd need to ensure this would work with just a recompile". I agree with the idea of a "narrow waist of an interface", but it's not useful in practice until people agree where the boundary should be and you can force everyone to abide by it.

[1] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28...

As long as the library is available.

Neither static nor dynamic linking is looking to solve the 20 year old binaries issue, so both will have different issues.

But I think it's easier for me to find a 20 year old ISO of a Red Hat/Slackware where I can simply run the statically linked binary. Dependency hell for older distros become really difficult when the older packages are not archived anywhere anymore.

Software running for 20 years is not always a reasonable requirement.

How do you troubleshoot and figure that out?

At least for some apps, perhaps it’s Wine and the Win32 API which is the answer.

Are you saying that a statically linked binary cannot make an HTTP request to `google.com` because it would be unable to resolve the domain name?

There are entire distros, like alpine, built on musl. I find this very hard to believe.

This blows my mind, that in 2025 we still struggle with a simple task such as "read in a string, parse it, and query a cascade of resolvers to discover it's IP". I just can't fathom how that is a difficult problem, or why DNS still is notorious for causing so much pain and suffering. Compared to the advancements in hardware and graphics and so many other areas.

I was able to run Age of Empires 2 on Windows 11. It didn't like running on an ultrawide, but it worked well enough at smaller resolutions.

The 2006-engine version of Half-Life 2: Episode 1 runs on Windows 10/11 with no configuration [1], outside of getting Steam to download it. I recall installing The Elder Scrolls IV: Oblivion on a Windows 11 machine, which just needed Directx 9c to run.

[1] https://steamcommunity.com/sharedfiles/filedetails/?id=28643...

It kinda is, and kinda isn't.

It is, in the sense that the package is bigger, and the package ships "everything it needs".

It isn't in the sense that those parts can be updated independently as long as the DLL interface is backward compatible.

For example, I ship OpenSSL dlls with my app. Which means swapping in a later (compatible) OpenSSL can be done (by the user if necessary.)

If I'm making a small utility I static link it - and I still use utilities daily I compiled 25 years ago. Obviously those dynamically link to KERNEL etc, but Microsoft has insane levels of compatibility there.

And perhaps that's the fundamental root of the issue. Windows has one provider, very committed to the longevity of software. Linux, well, does not.

That's OK. The world has room for different philosophies. And each one will have strengths and weaknesses.

isn't this tu quoque?

>I don't mean disrespect towards people running Alpine (hi), Arch, or Gentoo, but you wouldn't be running these distros if you aren't ready to handle their quirks.

You'd think so, but you'd be surprised how many gamers switching from Windows to Linux get recommended Arch and Manjaro

You’re my hero for this.

One of my side projects is building a toolchain to enable C++ cross-compile using the Zig header/source libs.

I didn’t love Zig as a Clang++ replacement because it has a bit too much magic. And it might go ahead? But the underlying library code is a God send

The problem is the opposite: they are trying to run executables built using a newer glibc in a system that has an older glibc. glibc keeps all the old function definitions since practically forever.

Frankly, I do not understand who would think glibc symbols themselves would be the challenge in this case. Even if you statically link glibc there's zero guarantee the syscalls will be present in the older Linux (cue .ABI-tag failures). Or even damn ELF format changes (e.g. gnu-style hashes). The simple solution is to build in the older Linux (&glibc).

In my long experience with ancient binaries, glibc has almost never been the problem, and its ability to _run_ ancient binaries is all but excellent; even Linux is more of a problem than glibc is (for starters paths to everywhere in /proc, /sys change every other half-decade).

The legacy symbols are still around. It's just that the linker won't use them by default.

> a lot of user-space in Linux-land does not make the same effort

I believe that, what the article misses is that glibc is maintained and extended with an entirely different community and development model. Windows remains compatible over decades because Microsoft (a) is the sole distributor, and (b) puts an immense effort towards backwards compat. In Linux userspace, it's simply a non-goal across distributions. If you want to ship a binary for a particular distro, you need to build the binary on / for that distro; even within a distro, a major release bump (or especially a major release downgrade) may break a binary.

Ultimately, it's a consequence of Conway’s Law. Microsoft is the sole distributor of Windows, so they can enforce compatibilty with an iron fist, and there are people working for Microsoft whose pay depends on said compatibility. With "Linux" in general, there is no common authority to appeal to, and (again) most vendors don't even promise a seamless userspace upgrade path from one major release to another.

This is unfixable; it will never change -- as long as independent parties are permitted to distribute different operating systems yet call them all "Linux".

Ship multiple binaries, or distribute the source code (and let users build it).

EDIT: you'll notice that "ship multiple binaries" is what distros (especially commercial distros) do. They maintain separate branches, backport fixes to old branches, and employ software maintenance engineers to focus on this kind of work. If you want to target multiple major releases, this is what you have to do, too.

If you (as a commercial ISV) target a commercial distro with long-term support, and can convince your users to use / license the same distro, you'll have a good, stable development experience. You only need to port like once every decade, when you jump major releases.

The Linux user base / the Linux market is fragmented; that's the whole goal. The technical proliferation / inconsistency is just a consequence. Unless you take away the freedom of users to run their own flavors of "Linux", there won't be a uniform Linux target.

In a way, it's funny to even expect otherwise. Why do you expect to ship the same binaries when the foundations are diverse, with no standardization authority that all Linux distributors recognize as such? And even POSIX is an API spec, not an ABI spec.

And, any authority that controls binary aspects will immediately accrue political capital. This is exactly what shouldn't happen in Linux. The fact that anyone can fork (or start) a distro, and contribute to the chaos, is good for freedom.

I thought that Go invoked syscalls directly instead of going through libc.

Rust toolchain has already a bug since 2015: it cannot link statically libgcc, namely it does not have "-static-libgcc" option since 2015.

I got the bug with "TinyGlade" video game (extremely good BTW), which is written in rust, and with the dev we hit that bug. Namely... better have a libgcc with the right ABI... and I can tell you, this has a been a HUGE issue since valve started to distribute games more than a decade ago.

No, end users need not get involved, it could/should be handled by the operating system.

The idea is that software would only be shipped if it actually works, and part of this would be ensuring it compiles. I'm not suggesting we use Make files or some other foul approach like that. Code should ideally be distributed as a single zip file with an executable header that triggers it to be unpacked in some cache directory by a program analogous to the linker, and compiled by the simple process of compiling all source files. Additionally this could be sped up drastically by having the compiler be a single executable with uniform flags across files. Header files could be parsed once and shared between build threads in parallel, which alone would safe much work on the part of the compiler. If this still proves insufficient, a language with superior semantics to C could be chosen which can be compiled faster. Most code these days shipped is JS which is compiled on the user's computer. This would be an order of magnitude improvement on that from a performance perspective.

This is somewhat exaggerated.

The compilation of firefox could take a few hours on some laptop dual-core Skylake CPU from 10 years ago.

Nowadays, on any decent dektop CPU with many cores the compilation of Firefox should take significantly less than an hour, though it remains one of the handful of open-source applications with a really long and non-negligible compilation time.

The Linux kernel is normally compiled much faster than Firefox, except when one would enable the compilation of all existing kernel modules, for all the hardware that could be supported by Linux, even if almost all of that is not present and it would never be present on the target computer system.

Just because you are used to slow compilers doesn't mean fast ones are impossible. As I said in the original post, code optimisation can be done before compilation on the developers machine, so all that need be done on the target is a simple debug build. An example is Jonathon Blow's JAI compiler which compiles around 250,000 lines of code per second. Even on very slow hardware this is reduced to perhaps 80,000LoC/s, two minutes for 10 million lines of code. Most users would be tolerant of an "install" box with a progress bar that takes at most two minutes for only the largest application on the computer. Or perhaps the browser people would opt not to distribute their software in this way. Advanced software may have advanced needs, beyond that of the average application. Most code is already written in Javascript these days, meaning it begins life being interpreted and is JIT compiled on the user's computer. Few seem to notice the compilation but many complain about the general sluggishness.

If you offered users a deal: spend 1 minute installing the software and it is snappy and responsive when you use it, I suspect all would accept.

Slow and usually pointless for minor updates.