Remix.run Logo
otterley 2 hours ago

Is that true even with PRAGMA journal_mode = WAL?

https://www.sqlite.org/wal.html

amluto an hour ago | parent [-]

That design doc explicitly talks about what is, essentially, compression of duplicate values in the same column. Many column-oriented databases do this.

With SQLite you’re looking at third party extensions that compress the data, still in row-oriented format, and might rather inefficiently recover some benefit. But WAL probably does help with the write amplification above and beyond this.

journald-style logs really want a column store IMO. It would be highly entertaining to try something like ducklake with SQLite as the catalog — the whole stack is pretty lightweight and there’s support for inlining inserts in the catalog to avoid creating silly numbers of Parquet files.

otterley an hour ago | parent [-]

That would make for a fun experiment for a syslog service. I encourage anyone who wants to give it a shot!