Remix.run Logo
amluto an hour ago

The journald schema is surprisingly wide and has a bunch of boilerplate, and one of the goals is to keep the on-disk size under control (and an efficient format directly reduces write amplification). And you kind of want a format that allows a reader to just read the file without blocking concurrent writes. And the ability to use third-party tools to easily read the format is quite nice.

SQLite gets the last one but misses on the first two (although WAL and the improved read-only support in 3.20+ mostly gets #1). DuckDB might be decent except that you would need to connect through the daemon to read if the daemon is running. If a daemon that coordinates everything is okay, something like Clickhouse might work.

An LSM-style layer over Parquet gets all of this fairly naturally as long as readers using third party tools understand the LSM scheme. (In general there is a lack of consensus as to exactly how to correctly and efficiently use multiple Parquet files together.)