Systemd should just use DuckDB. It's perfect for this job.

"But isn't it an OLAP database? Shouldn't you use SQLite for something that's vaguely real-time?"

Eh, in this instance, I think I'd prefer the columnar design and automatic compression DuckDB affords. Log entries have lots of little fields, many of which are unchanging from row-to-row, and DuckDB excels at storing this kind of data.

BTW: no, you don't need O(N*log(N) writes for DuckDB. No, you're not doing a whole block-group write for every message. No, Parquet is not a magical solution. I mean, maybe it's fine, but DuckDB is already columnar, and arguably better at it.

Seems like there are a lot of mistaken impressions about DB storage engines out there.

▲

marginalia_nu 4 hours ago | parent [-]

Parquet is probably an even better option. Columnar, compression, fast, succinct. All good things.

You can read them with DuckDB, but you don't end up with O(log n) writes -- which is, to speak plain English, batshit fucking insane for a system logger.

What those cursed writes buys you is O(log n) reads, but there's just no scenario that is necessary. If you have literally any time or subsystem constraints, parquet's predicate pushdowns means you get plenty fast access even with a full scan.

	▲	orf 3 hours ago \| parent [-]
		No, not at all. Parquet is great for building static content incrementally, but it’s not great for this: the aim is durable writes (it’s a log system after all), but with parquet you need large row group batches. Worst case (low log volumes and a time-based flush) you’d end up with loads of tiny row groups. You also need metadata in the file footer, so you can’t query it until the file is “done”. When is that?