Can somebody explain:

1. The exact problem with the Linux ABI

2. What causes it (the issues that makes it such a challenge)

3. How it changed over the years, and its current state

4. Any serious attempts to resolve it

I've been on Linux for may be 2 decades at this point. I haven't noticed any issues with ABI so far, perhaps because I use everything from the distro repo or build and install them using the package manager. If I don't understand it, there are surely others who want to know it too. (Not trying to brag here. I'm referring to the time I've spent on it.)

I know that this is a big ask. The best course for me is of course to research it myself. But those who know the whole history tend to have a well organized perspective of it, as well as some invaluable insights that are not recorded anywhere else. So if this describes you, please consider writing it down for others. Blog is probably the best format for this.

The kernel is stable, but all the system libraries needed to make a grapical application are not. Over the last 20 years, we've gone from GTK 2 to 4, X11 to Wayland, Qt 4 to 6, with compatibility breakages with each change. Building an unmodified 20 year old application from source is very likely to not work, running a 20 year old binary even less so.

Linux API/ABI doesn't cover the entire spectrum that Windows API covers. There is everything from lowest level kernel stuff to the desktop environment and beyond. In Linux deployments, that's achieved by a mix of different libraries from different developers and these change over time.

There is no ABI problem. The problem is a lack of standardization for important APIs and infrastructure. There once was a serious effort to solve this: the Linux Standard Base: https://en.wikipedia.org/wiki/Linux_Standard_Base Standardization would of course be the only way to fix this, instead of inventing even more packaging formats which fragment the ecosystem even more. LSB died due to lack of interest. I assume also because various industrial stakeholders are more interest in gaining a little bit of control over the ecosystem than in the overall success of Linux on the desktop. The other major problem is that it is no fun to maintain software, which leads to what was described as CADT: https://www.jwz.org/doc/cadt.html As you see with Wayland and Rust rewrites CADT still continues today always justified with some bullshit arguments why the rewrites are really necessary.

Together this means that basically nobody implements applications anymore. For commercial applications that market is too fragmented and it is too much effort. Open-source applications need time to grow and if all the underpinnings get changed all the time, this is too frustrating. Only a few projects survive this, and even those struggle. For example GIMP took a decade to be ported from GTK 2 to 3.

You never ran into a GLIBC version problem?

The model of patching+recompiling the world for every OS release is a terrible hack that devs hate and that users hate. 99% of all people hate it because it's a crap model. Devs hate middlemen who silently fuck up their software and leave upstream with the mess, users hate being restricted to whatever software was cool and current two years ago. If they use a rolling distro, they hate the constant brokenness that comes with it. Of the 1% of people who don't hate this situation 99% of those merely tolerate it, and the rest are Debian developers who are blinded by ideology and sunk costs.

Good operating systems should:

1. Allow users to obtain software from anywhere.

2. Execute all programs that were written for previous versions reliably.

3. Not insert themselves as middlemen into user/developer transactions.

Judged from this perspective, Windows is a good OS. It doesn't nail all three all the time, but it gets the closest. Linux is a bad OS.

The answers to your questions are:

(1) It isn't backwards compatible for sophisticated GUI apps. Core APIs like the widget toolkits change their API all the time (GTK 1->2->3->4, Qt also does this). It's also not forwards compatible. Compiling the same program on a new release may yield binaries that don't run on an old release. Linux library authors don't consider this a problem, Microsoft/Apple/everyone else does. This is the origin of the glibc symbol versioning errors everyone experiences sometimes.

(2) Maintaining a stable API/ABI is not fun and requires a capitalist who says "keep app X working or else I'll fire you". The capitalist Fights For The User. Linux is a socialist/collectivist project with nobody playing this role. Distros like Red Hat clone the software ecosystem into a private space that's semi-capitalist again, and do offer stable ABIs, but their releases are just ecosystem forks and the wider issue remains.

(3) It hasn't change and it's still bad.

(4) Docker: "solves" the problem on servers by shipping the entire userspace with every app, and being itself developed by a for-profit company. Only works because servers don't need any shared services from the computer beyond opening sockets and reading/writing files, so the kernel is good enough and the kernel does maintain a stable ABI. Docker obviously doesn't help the moment you move outside the server space and coordination requirements are larger.