The ability to... correct 1-2 bit errors? Is that even a realistic failure mode on common hardware?

CRCs as used in SQLite are not intended to detect data corruption due to bit rot, and are certainly not ECCs.

▲ AlotOfReading 2 days ago | parent [-]

Yes, it's a pretty important error case with storage devices. It's so common that modern storage devices and filesystems include their own protections against it. Your system may or may not have these and bit flips may happen after that point, so WAL redundancy wouldn't be out of place.

Sure, the benefits to the incomplete write use case are limited, but there's basically no reason to ever use a fletcher these days.

It's also worth mentioning that the VFS checksums are explicitly documented as guarding against storage device bitrot and use the same fletcher algorithm.

▲ lxgr 2 days ago | parent [-]

It would be absolutely out of place if your lower layers already provided it, and not enough if they didn't (since you'd then also need checksums on the actual database file, which SQLite does not provide – all writes happen again there!)

▲ AlotOfReading 2 days ago | parent [-]

sqlite does actually provide database checksums, via the vfs extension I mentioned previously.

There's no harm to having redundant checksums and it's not truly redundant for small messages. It's pretty common for systems not to have lower level checksumming either. Lots of people are still running NTFS/EXT4 on hardware that doesn't do granular checksums or protect data in transit.

Of course this is all a moot point because sqlite does WAL checksums, it just does them with an obsolete algorithm.

▲ lxgr 2 days ago | parent [-]

> There's no harm to having redundant checksums

There sure is: Redundant checksums need extra storage and extra processing. SQLite often runs on embedded systems, where both can come at a premium.

> Of course this is all a moot point because sqlite does WAL checksums, it just does them with an obsolete algorithm.

That's not nearly the only thing missing for SQLite to provide full resistance to lower-level data corruption. At a very minimum, you'd also need checksums of the actual database file.

▲ AlotOfReading 2 days ago | parent [-]

There's no extra storage because it's already storing checksums. There's technically extra processing, but a good CRC is faster than reading memory on virtually all modern systems (even in embedded) so there's no reason to prefer a fletcher unless the few extra cycles of latency to get the CRC running are deeply concerning to you.

    That's not nearly the only thing missing for SQLite to provide full resistance to lower-level data corruption.

A CRC wouldn't provide full resistance anyway, so this clearly isn't what I'm arguing. What I'm saying is that a fletcher is strictly worse than a CRC here. Not an advanced algorithm, a bog-standard CRC. Fletcher actually takes more space in this case, has worse error detection, and doesn't allow recovery. A CRC would allow you to recover some of the time, from some kinds of errors, in some scenarios. It's simply better than nothing.

▲

lxgr 2 days ago | parent [-]

> There's no extra storage because it's already storing checksums.

At the database level (i.e. not just the WAL)? Are you sure?

> What I'm saying is that a fletcher is strictly worse than a CRC here.

I can't speak to the performance differences, but the only thing SQLite really needs the checksum to do is to expose partial writes, both due to reordered sector writes and partial intra-sector writes. (The former could also be solved by just using an epoch counter, but the latter would require some tricky write formats, and a checksum nicely addresses both).

In both cases, there's really nothing to recover: CRC won't catch an entire missing sector, and almost no partially written sectors (i.e. unless the failure somehow happens in the very last bytes of it, so that the ratio of "flipped" bits is low enough).

	▲	AlotOfReading 2 days ago \| parent [-]
		Just talking about the WAL. The database does have a page level checksum with the VFS extension, but it's optional.