| ▲ | pengaru 5 hours ago |
| I'm probably the main person responsible for making journald usable at all. But I never really made any effort to change the on-disk structure or how writes were performed. My focus was more on the read performance for journalctl and stability of the daemon. Back when I was paid to fix things in journald at CoreOS ages ago, it couldn't even avoid getting killed by its own service watchdog. My impression back then was the on-disk format dispersed the information too much within the same file, and those individual datums being written at discontiguous offsets were quite small, far smaller than an IO block size or even a disk sector size. Seemed like a write amplification problem due to the file format. If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written. The format had no consideration for these block-oriented storage details, then doing the IO via mmap rubs salt into the wound since the kernel has to try guess what to prefetch asynchronously... but I don't think that aspect amplifies the writes above what plain buffered IO would do - maybe I'm wrong. I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). I suppose there's probably compounding of the write amplification problem since the kernel will be dirtying pages at page size granularity vs. 512b sectors, and you have the same issue of small writes landing on page boundaries dirtying two pages. So that aspect of using mmap for the writes probably is exacerbating the problem. |
|
| ▲ | ValdikSS 5 hours ago | parent | next [-] |
| journald uses hash tables, I think it update it on every new log line, although I didn't debug it in depth yet. https://github.com/systemd/systemd/blob/199f75205b9c0625bf56... |
| |
|
| ▲ | cloudie78 5 hours ago | parent | prev | next [-] |
| Why not just have a SQLite file and call it a day? Also, why mmaped file? |
| |
| ▲ | pengaru 5 hours ago | parent | next [-] | | I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic. There was mailing list discussion at the time journald was conceived though, you can find it if you look. https://0pointer.de/blog/projects/the-journal.html might be a good entry-point. | | |
| ▲ | otterley 4 hours ago | parent | next [-] | | The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/ It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-Ja... | | |
| ▲ | pengaru 4 hours ago | parent [-] | | FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design. |
| |
| ▲ | marginalia_nu 4 hours ago | parent | prev [-] | | Well there was an ambition, apparently. > Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance > Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog. |
| |
| ▲ | quotemstr 4 hours ago | parent | prev [-] | | SQLite here is okay, but DuckDB or LevelDB would be better. Either way, no need to invent a new storage format. | | |
| ▲ | otterley 4 hours ago | parent | next [-] | | Neither DuckDB nor LevelDB existed when journald was created. Not to say it couldn't be done today, but just some historical context. | |
| ▲ | e2le 3 hours ago | parent | prev | next [-] | | Sqlite3 is present in the default installation of most Linux distributions. It has proven itself from years of battle testing in many different environments. To use DuckDB or LevelDB would probably require pulling in an additional dependency. | |
| ▲ | ElectricalUnion 4 hours ago | parent | prev [-] | | No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens. | | |
| ▲ | quotemstr 4 hours ago | parent [-] | | Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.) We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats. |
|
|
|
|
| ▲ | otterley 4 hours ago | parent | prev | next [-] |
| > I'm probably the main person responsible for making journald usable at all. Thank you for your service! |
|
| ▲ | quotemstr 4 hours ago | parent | prev [-] |
| Thank you for your work. ISTM the workload is naturally LSM-shaped. > If you write a few bytes into some arbitrary position within a file, the storage has to write back the whole block, despite your only changing a tiny fraction of it. If those few bytes happened to cross a block boundary, guess what? two blocks get written. Exactly. So either make the format append-only or make it append-mostly with occasional writebacks from the append-only log to the main data structure. Nice and simple. > I'd expect the mmap aspect to be causing more/mispredicted reads, and polluting the page cache with unrelated contents (you tend to end up with the entire journal cached IIRC, if you have enough memory). If you used an LSM or append-only approach, you could MADV_DONTNEED the pages behind your write cursor pretty easily. |
| |
| ▲ | amluto 4 hours ago | parent [-] | | Append-only -> Parquet -> bigger Parquet would do the trick. Sadly Parquet is useless for the append-only layer. Feather would work but is quite inefficient with a batch size of 1. | | |
| ▲ | quotemstr 4 hours ago | parent [-] | | Once you solve enough problems using raw Parquet or Feather or whatever and you end up with something that looks like a DB anyway, so you might as well use a DB. | | |
| ▲ | amluto an hour ago | parent | next [-] | | The journald schema is surprisingly wide and has a bunch of boilerplate, and one of the goals is to keep the on-disk size under control (and an efficient format directly reduces write amplification). And you kind of want a format that allows a reader to just read the file without blocking concurrent writes. And the ability to use third-party tools to easily read the format is quite nice. SQLite gets the last one but misses on the first two (although WAL and the improved read-only support in 3.20+ mostly gets #1). DuckDB might be decent except that you would need to connect through the daemon to read if the daemon is running. If a daemon that coordinates everything is okay, something like Clickhouse might work. An LSM-style layer over Parquet gets all of this fairly naturally as long as readers using third party tools understand the LSM scheme. (In general there is a lack of consensus as to exactly how to correctly and efficiently use multiple Parquet files together.) | |
| ▲ | hedora 2 hours ago | parent | prev [-] | | Or, you could write a plain text file. Yes, that means the FS will sometimes punch nulls towards the tail of the log. However, it is the lowest latency / write amplification way to get stuff on disk (other than a blocked compression format, which would be a small change to syslog), so if the text file gets holes punched in it, the journalctl file would be truncated before the hole anyway in practice. If you really care about nulls in logs for ideological reasons, you could write a few lines of code that finds the first stream of nulls in the text file, then truncates there. In practice, no one wants that. It is strictly worse than returning partial entries after the hole, and by the time you are hitting this corner case, you are debugging a kernel crash. |
|
|
|
