"In every way"

About ten years ago I took a three day cross-country Amtrak trip where I wanted to work on some data analysis that used mysql for its backend. It was a great venue for that sort of work because the lack of train-internet was wonderful to keep me focused. The data I was working with was about 20GB of parking ticket data. The data took a while to process over SQL which gave me the chance to check out the world unfolding outside of the train.

At some point, mysql (well, mariadb) got into a weird state after an unclean shutdown that put itself into recovery mode where upon startup it had to do some disk-intensive cleanup. Thing is -- systemd has a default setting (that's not readily documented, nor sufficiently described in its logs when the behavior happens) that halts the service startup after 30 seconds to try again. On loop.

My troubleshooting attempts were unsuccessful. And since I deleted the original csv files to save disk space, I wasn't able to even poke at the CSV files through python or whatnot.

So instead of doing the analysis I wanted to do on the train, I had to wait until I got to the end of the line to fix it. Sure enough, it was some default 30s timeout that's not explicitly mentioned nor commented out like many services do.

So, saying that things are "much better in every way" really falls on deaf ears and is reminiscent of the systemd devs' dismissive/arrogant behavior that many folk are frustrated about.

I had a situation like that with an undocumented behavior and systemd-tmpfiles. I wanted it to clean up a directory in /var/tmp/ occasionally. The automation using that directory kept breaking, however, because instead of either finding a whole intact git repo to update or a deleted repo, it instead found only a scattering of files that were root-owned with read-only permissions. There was yet another undocumented feature in systemd-tmpfiles where it would ignore root-owned, read-only files regardless of explicit configuration telling it to clean up the contents of those directories. Eventually this feature was quietly removed:

https://bugzilla.redhat.com/show_bug.cgi?id=1780979

https://github.com/systemd/systemd/commit/a083b4875e8dec5ce5...

That was far from the only time that the systemd developers decided to just break norms or do weird things because they felt like it, and then poorly communicate that change. Change itself is fine, it's how we progress. But part of that arrogance that you mentioned was always framing people who didn't like capricious or poorly communicated changes as being against progress, and that's always been the most annoying part of the whole thing.