Remix.run Logo
malone 2 days ago

Why is the commit frame not sufficient to determine whether the transaction was fully written or not? Is there a scenario where the commit frame is fsynced to disk but the proceeding data isn't?

adambb 2 days ago | parent [-]

The disk controller may decide to write out blocks in a different order than the logical layout in the log file itself, and be interrupted before completing this work.

grumbelbart2 a day ago | parent | next [-]

Just wondering how SQLite would ever work if it had zero control over this. Surely there must be some "flush" operation that guarantees that everthing so far is written to disk? Otherwise, any "old" block that contains data might have not been written. SQLite says:

> Local devices also have a characteristic which is critical for enabling database management software to be designed to ensure ACID behavior: When all process writes to the device have completed, (when POSIX fsync() or Windows FlushFileBuffers() calls return), the filesystem then either has stored the "written" data or will do so before storing any subsequently written data.

mschuster91 a day ago | parent [-]

A "flush" command does indeed exist... but disk and controller vendors are like patients in Dr. House [1] - everybody lies. Especially if there are benchmarks to be "optimized". Other people here have written up that better than I ever could [2].

[1] https://house.fandom.com/wiki/Everybody_lies

[2] https://news.ycombinator.com/item?id=30371403

johncolanduoni 2 days ago | parent | prev | next [-]

It’s worth noting this is also dependent on filesystem behavior; most that do copy-on-write will not suffer from this issue regardless of drive behavior, even if they don’t do their own checksumming.

hinkley 2 days ago | parent | prev [-]

We still have the elevator algorithm on NVMe?

jrockway 2 days ago | parent | next [-]

NVMe drives do their own manipulation of the datastream. Wear leveling, GC, trying to avoid rewriting an entire block for your 1 bit change, etc. NVMe drives have CPUs and RAM for this purpose; they are full computers with a little bit of flash memory attached. And no, of course they're not open source even though they have full access to your system.

djfivyvusn a day ago | parent [-]

Skynet gotta start somewhere.

bob1029 2 days ago | parent | prev | next [-]

Anything that uses NAND storage technology is going to be optimized in some way like this. NVMe is just the messenger.

lxgr 2 days ago | parent | prev [-]

SQLite runs on anything from servers to Internet-connected lightbulbs.

jrockway 2 days ago | parent [-]

Which lightbulbs include SQLite? I kind of want one.

natebc 2 days ago | parent [-]

these guys have a Cree logo on their homepage so maybe Cree?

https://imaginovation.net/case-study/cree/

At least what I could turn up with a quick web search.