I was always curious why they created their own format rather than leveraging SQLite, OpenLDAP's LMDB, etc.

simoncion 3 hours ago | parent | next [-]

Given that the ethos of the Systemd Project is to almost-always reinvent so that they retain complete control [0] over the code, I'm never surprised when they choose to reimplement and fold that implementation into the project rather than to cooperate and improve the state of the world for all projects.

[0] As demonstrated by many vertically-integrated successful businesses -SpaceX being a recent example- there are substantial benefits to doing everything in-house. However, if you choose to pull an assload of things in-house to do them yourself, you must be capable of doing all of that work yourself. Given the state of SystemD, [1] its historical and current reaction to reports of both subtle but severe bugs and of totally reasonable system configurations that SystemD makes impossible, I don't believe they have the capability required to do a good job at what they've set out to do. Choosing to not cooperate with the existing ecosystem was a short-term win, but -IMO- a huge long-term mistake.

[1] ...this is spelt "SystemD" not as a slur, but to distinguish systemd(1) from The Systemd Project it is a part of. It's damn annoying that they share the same name...

▲

rcxdude 2 hours ago | parent | prev [-]

sqlite is not drastically better in this regard: it's designed as a rewritable database, not a log store, and so it's also going to have quite a big write amplification if you do lots of small writes. (Probably the best mitigation is to buffer up the log lines and write them out periodically, but for an idle linux system this might need to be pretty long to make much of a different, and then you would inevitably get complaints about logs being lost during a power failure or kernel panic)

▲

otterley 2 hours ago | parent [-]

Is that true even with PRAGMA journal_mode = WAL?

https://www.sqlite.org/wal.html

▲

amluto an hour ago | parent [-]

That design doc explicitly talks about what is, essentially, compression of duplicate values in the same column. Many column-oriented databases do this.

With SQLite you’re looking at third party extensions that compress the data, still in row-oriented format, and might rather inefficiently recover some benefit. But WAL probably does help with the write amplification above and beyond this.

journald-style logs really want a column store IMO. It would be highly entertaining to try something like ducklake with SQLite as the catalog — the whole stack is pretty lightweight and there’s support for inlining inserts in the catalog to avoid creating silly numbers of Parquet files.

	▲	otterley an hour ago \| parent [-]
		That would make for a fun experiment for a syslog service. I encourage anyone who wants to give it a shot!