Remix.run Logo
stackghost 3 hours ago

Hi Chris,

One of the most grating pain points of the early versions of systemd was a general lack of humility, some would say rank arrogance, displayed by the project lead and his orbiters. Today systemd is in a state of "not great, not terrible" but it was (and in some circles still is) notorious for breaking peoples' linux installs, their workflows, and generally just causing a lot of headaches. The systemd project leads responded mostly with Apple-style "you're holding it wrong" sneers.

It's not immediately clear to me what exactly Amutable will be implementing, but it smells a lot like some sort of DRM, and my immediate reaction is that this is something that Big Tech wants but that users don't.

My question is this: Has Lennart's attitude changed, or can linux users expect more of the same paternalism as some new technology is pushed on us whether we like it or not?

sandebert 3 hours ago | parent | next [-]

Thank you for this question, it perfectly captures something that I believe many would like answered.

2 hours ago | parent [-]
[deleted]
chaps 3 hours ago | parent | prev [-]

As someone who's lost many hours troubleshooting systemd failures, I would like an answer to this question, too.

microtonal 3 hours ago | parent [-]

You won't believe how many hours we have lost troubleshooting SysV init and Upstart issues. systemd is so much better in every way, reliable parallel init with dependencies, proper handling of double forking, much easier to secure services (systemd-analyze security), proper timer handling (yay, no more cron), proper temporary file/directory handling, centralized logs, etc.

It improves on about every level compared to what came before. And no, nothing is perfect and you sometimes have to troubleshoot it.

chaps 2 hours ago | parent | next [-]

"In every way"

About ten years ago I took a three day cross-country Amtrak trip where I wanted to work on some data analysis that used mysql for its backend. It was a great venue for that sort of work because the lack of train-internet was wonderful to keep me focused. The data I was working with was about 20GB of parking ticket data. The data took a while to process over SQL which gave me the chance to check out the world unfolding outside of the train.

At some point, mysql (well, mariadb) got into a weird state after an unclean shutdown that put itself into recovery mode where upon startup it had to do some disk-intensive cleanup. Thing is -- systemd has a default setting (that's not readily documented, nor sufficiently described in its logs when the behavior happens) that halts the service startup after 30 seconds to try again. On loop.

My troubleshooting attempts were unsuccessful. And since I deleted the original csv files to save disk space, I wasn't able to even poke at the CSV files through python or whatnot.

So instead of doing the analysis I wanted to do on the train, I had to wait until I got to the end of the line to fix it. Sure enough, it was some default 30s timeout that's not explicitly mentioned nor commented out like many services do.

So, saying that things are "much better in every way" really falls on deaf ears and is reminiscent of the systemd devs' dismissive/arrogant behavior that many folk are frustrated about.

notabee 44 minutes ago | parent [-]

I had a situation like that with an undocumented behavior and systemd-tmpfiles. I wanted it to clean up a directory in /var/tmp/ occasionally. The automation using that directory kept breaking, however, because instead of either finding a whole intact git repo to update or a deleted repo, it instead found only a scattering of files that were root-owned with read-only permissions. There was yet another undocumented feature in systemd-tmpfiles where it would ignore root-owned, read-only files regardless of explicit configuration telling it to clean up the contents of those directories. Eventually this feature was quietly removed:

https://bugzilla.redhat.com/show_bug.cgi?id=1780979

https://github.com/systemd/systemd/commit/a083b4875e8dec5ce5...

That was far from the only time that the systemd developers decided to just break norms or do weird things because they felt like it, and then poorly communicate that change. Change itself is fine, it's how we progress. But part of that arrogance that you mentioned was always framing people who didn't like capricious or poorly communicated changes as being against progress, and that's always been the most annoying part of the whole thing.

toast0 2 hours ago | parent | prev | next [-]

> systemd is so much better in every way,

How can I cancel a systemd startup task that blocks the login prompt? / how is forcing me to wait for dhcp on a network interface that isn't even plugged in a better experience?

Nextgrid 2 hours ago | parent [-]

Your distribution has configured your GDM or Getty to have some dependency on something that ultimately waits on dhcpcd/network-online.target.

It’s not really the fault of systemd; it just enables new possibilities that were previously difficult/impossible and now the usage of said possibilities is surfacing problems.

toast0 2 hours ago | parent [-]

It is the fault of systemd that there's no interactive control.

On other inits, I can hit ctrl-C to break out of a poorly configured setup. Yes, it's more difficult when there's potentially parallelism. But systemd is not uniformly better than everything else when it lacks interactivity.

And it might not be better than everything else if common distributions set it up wrong because it's difficult to set it up right. If we're willing to discount problems related to one init system because the distribution is holding it wrong, then why don't we blame problems with other init systems on distributions or applications, too? There's no need to restart crashing applications if applications don't crash, etc.

shrubble 2 hours ago | parent | prev | next [-]

There’s a reason why Devuan (a non systemd Debian) exists. Don’t want to get into a massive argument, but there are legitimate reasons for some to go in a different direction.

smartmic 2 hours ago | parent [-]

And Void Linux. And Gentoo. And Alpine Linux. And Slackware. And others.

forty 2 hours ago | parent | next [-]

Systemd has recently added experimental support for musl libc, which should eventually allow Alpine to upgrade though

eth0up an hour ago | parent | prev [-]

After over a decade of Debian, when I upgraded my PC, I tried every big systemd-based distro, including opensuse, which I wholly loathed. I finally decided on Void and feel at home as I did 20+ years ago when I began.

There are serious problems with the systemd paradigm, most of which I couldn't argue for or against. But at least in Void, I can remove network-manger altogether, use cron as I always have, and generally remain free to do as I please until eventually every package there is has systemd dependencies which seems frightfully plausible at this pace.

Void is as good as I could have wanted. If that ever goes, I guess it's either BSD or a cave somewhere.

I'm glad to see the terse questions here. They're well warranted.

jamespo an hour ago | parent [-]

How is systemd stopping you use cron?

eth0up an hour ago | parent [-]

Not stopping. Just clashing with that and a hundred other things that I never wanted managed by one guy. Systemd.timer, systemd.service, yes, trivial, but I don't catalog every thing that bothers me about systemd - I just stay away from it. There are plenty of better examples. So where ever I wrote 'stop', it should read hinder.

foresto 2 hours ago | parent | prev | next [-]

Here are a few examples of problems systemd has caused me:

System shutdown/reboot is now unreliable. Sometimes it will be just as quick as it was before systemd arrived, but other times, systemd will decide that something isn't to its liking, and block shutdown for somewhere between 30 seconds and 10 minutes, waiting for something that will never happen. The thing in question might be different from one session to the next, and from one systemd version to the next; I can spend hours or days tracking down the process/mount/service in question and finding a workaround, only to have systemd hang on something else the next day. It offers no manual skip option, so unless I happen to be working on a host with systemd's timeouts reconfigured to reduce this problem, I'm stuck with either forcing a power-off or having my time wasted.

Something about systemd's meddling with cgroups broke the lxc control commands a few years back. To work around the problem, I have to replace every such command I use with something like `systemd-run --quiet --user --scope --property=Delegate=yes <command>`. That's a PITA that I'm unlikely to ever remember (or want to type) so I effectively cannot manage containers interactively without helper scripts any more. It's also a new systemd dependency, so those helper scripts now also need checks for cgroup version and systemd presence, and a different code path depending on the result. Making matters worse, that systemd-run command occasionally fails even when I do everything "right". What was once simple and easy is now complex and unreliable.

At some point, Lennart unilaterally decided that all machines accessed over a network must have a domain name. Subsequently, every machine running a distro that had migrated to systemd-resolved was suddenly unable to resolve its hostname-only peers on the LAN, despite the DNS server handling them just fine. Finding the problem, figuring out the cause, and reconfiguring around it wasn't the end of the world, but it did waste more of my time. Repeating that experience once or twice more when systemd behavior changed again and again eventually drove me to a policy of ripping out systemd-resolved entirely on any new installation. (Which, of course, takes more time.) I think this behavior may have been rolled back by now, but sadly, I'll never get my time back.

There are more examples, but I'm tired of re-living them and don't really want to write a book. I hope these few are enough to convey my point:

Systemd has been a net negative in my experience. It has made my life markedly worse, without bringing anything I needed. Based on conversations, comments, and bug reports I've seen over the years, I get the impression that many others have had a similar experience, but don't bother speaking up about it any more, because they're tired of being dismissed, ignored, or shouted down, just as I am.

I would welcome a reliable, minimal, non-invasive, dependency-based init. Systemd is not it.

plagiarist 2 hours ago | parent | prev | next [-]

The problem is not systemd vs SysV et al, the problem is systemd spreading like a cancer throughout the entire operating system.

Also trying to use systemd with podman is frustrating as hell. You just cannot run a system service using podman as a non-root user and have it work correctly.

storystarling 2 hours ago | parent | next [-]

Quadlet actually solves this. It's the newer way to define containers for systemd and handles the rootless user case properly. I migrated my services to it recently and it's much more robust than the old generate scripts.

plagiarist 30 minutes ago | parent | next [-]

Could you give an example system-level quadlet that accepts connections on a low port, like 80, but runs the actual container as a non-root user (and plays nice with systemd, no force kill after timeout to stop, no reporting as failed for a successful stop)?

My understanding is quadlet does not solve this, and my options are calling "systemctl --user" or "--userns auto". I would love to be wrong here.

storystarling 9 minutes ago | parent [-]

I solved the port 80 issue by adding AmbientCapabilities=CAP_NET_BIND_SERVICE to the Service section of the unit file. That lets you bind privileged ports while still defining a User= line to run non-root. The lifecycle management seems solid in my experience, no force kills required.

forty 2 hours ago | parent | prev [-]

Quadlet are great but running podman via systemd as a non root user worked perfectly well before quadlets and I have no idea what your parent is talking about (I'm currently in the process of converting my home services from rootless podman over systemd to quadlet)

storystarling 14 minutes ago | parent [-]

Fair, it worked, but podman generate systemd is deprecated now. I found the generated unit files pretty brittle to maintain compared to just having a declarative config that handles the lifecycle.

cyberax an hour ago | parent | prev [-]

> You just cannot run a system service using podman as a non-root user and have it work correctly.

Err... You just need to run `podman-compose systemd`?

I have my entire self-hosted stack running with systemd-controlled Podman, in regular user accounts.

jamespo an hour ago | parent | prev [-]

I'd be interested in what other init alternatives offer the security options systemd does