> Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong.

I've run systems with live code updates for userland, and would have considered live kernel updates if it was reasonable on our systems.

The thing is you typically build your system to tolerate reboot or unscheduled stop of a single node. Scheduled stop is nicer, but systems sometimes lock up even when you're not doing risky behaviors, so you know.

But just because the system can tolerate a reboot or restart doesn't mean it's not disruptive. A lock up / etc during hot load is also disruptive, of course. But when you can push code without having to stop anything, with limited impact on users, it makes it easier and faster to do updates. You can use whatever rollout pattern you like to contain risk too; same as you would for an upgrade with restarts.

For us, we have servers with hundreds of thousands or millions of tcp connections from mobile clients. Restarting a server would make all those clients have to reconnect and connecting is expensive. Restarting all the servers would result in many clients reconnecting several times. It was better to avoid it when possible.

We've used ksplice for good amount of years (till bought by Oracle and they stopped publishing patches for other Linux distros) and all in all it was very stable technology 10 years ago

> I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong.

Because you haven't worked at that level in organization. Doing restart in some case might involve paperwork with your client and maintenance window outside of working hours even if service is redundant. And some customers are fine with a little bit of downtime and don't want active/active level of redundancy but still insist of maintenance windows for any work like that.

Live patching makes that a whole lot easier

Public goods are not generally funded by large corporations.

Linux distros are not npm. It doesn't mean they are infallible to malicious actors, but I believe it is possible to make them infallible for some small set of packages at least.

Attacks are still possible, but if we look at xz backdoor attack[1] it was insanely complicated attack and it still failed. Its fail doesn't look promising, attack could succeed just the attacker was unlucky. Still it shows that the success is not guaranteed.

Theoretically npm can be improved in this way, if there were a separate "distro" for packaged, with dedicated maintainers for packages, who don't write code, just pull it from a mainstream and review it. It is not being done because of tragedy of commons, not because it is impossible.

[1] https://en.wikipedia.org/wiki/XZ_Utils_backdoor

Easier: Do not start with a "allow all" configuration in the first place.

Maybe all of those userspace-work-done-in-kernel-because-muh-performance features should be restricted to (the "real") CAP_NET_ADMIN, unless positively enumerated as free-for-all-containers. And then subtract from that free-for-all list every time you learn that some kernel module in its currently available version cannot be trusted to do its own memory shuffling.

> we're now into the "users installing our distro are getting hacked left and right" territory

Are we? Are users actually getting hacked, or have they theoretically been exposed to problems that could allow local privileged escalation if exploited but that nobody's seen used in the wild?

(Edit: To be clear, I'm skeptical but this isn't a completely rhetorical question. If there are actual reports of these vulns causing problems, that would strongly incentivize a stronger response.)

It used to be relatively standard even on the "big" distros to compile your own kernel if you needed something outside of the bog-standard. Modularization and all the related auto-detect auto-mod tools have resulted in most distros shipping a "works for almost everyone" kernel that has everything available as a module.

Perhaps we should tend toward the first.