It would be absolutely out of place if your lower layers already provided it, and not enough if they didn't (since you'd then also need checksums on the actual database file, which SQLite does not provide – all writes happen again there!)

▲ AlotOfReading 2 days ago | parent [-]

sqlite does actually provide database checksums, via the vfs extension I mentioned previously.

There's no harm to having redundant checksums and it's not truly redundant for small messages. It's pretty common for systems not to have lower level checksumming either. Lots of people are still running NTFS/EXT4 on hardware that doesn't do granular checksums or protect data in transit.

Of course this is all a moot point because sqlite does WAL checksums, it just does them with an obsolete algorithm.

▲ lxgr 2 days ago | parent [-]

> There's no harm to having redundant checksums

There sure is: Redundant checksums need extra storage and extra processing. SQLite often runs on embedded systems, where both can come at a premium.

> Of course this is all a moot point because sqlite does WAL checksums, it just does them with an obsolete algorithm.

That's not nearly the only thing missing for SQLite to provide full resistance to lower-level data corruption. At a very minimum, you'd also need checksums of the actual database file.

▲ AlotOfReading 2 days ago | parent [-]

There's no extra storage because it's already storing checksums. There's technically extra processing, but a good CRC is faster than reading memory on virtually all modern systems (even in embedded) so there's no reason to prefer a fletcher unless the few extra cycles of latency to get the CRC running are deeply concerning to you.

    That's not nearly the only thing missing for SQLite to provide full resistance to lower-level data corruption.

A CRC wouldn't provide full resistance anyway, so this clearly isn't what I'm arguing. What I'm saying is that a fletcher is strictly worse than a CRC here. Not an advanced algorithm, a bog-standard CRC. Fletcher actually takes more space in this case, has worse error detection, and doesn't allow recovery. A CRC would allow you to recover some of the time, from some kinds of errors, in some scenarios. It's simply better than nothing.

▲

lxgr 2 days ago | parent [-]

> There's no extra storage because it's already storing checksums.

At the database level (i.e. not just the WAL)? Are you sure?

> What I'm saying is that a fletcher is strictly worse than a CRC here.

I can't speak to the performance differences, but the only thing SQLite really needs the checksum to do is to expose partial writes, both due to reordered sector writes and partial intra-sector writes. (The former could also be solved by just using an epoch counter, but the latter would require some tricky write formats, and a checksum nicely addresses both).

In both cases, there's really nothing to recover: CRC won't catch an entire missing sector, and almost no partially written sectors (i.e. unless the failure somehow happens in the very last bytes of it, so that the ratio of "flipped" bits is low enough).

	▲	AlotOfReading 2 days ago \| parent [-]
		Just talking about the WAL. The database does have a page level checksum with the VFS extension, but it's optional.