dm-cache writeback mode is both amazing and terrifying. It reorders writes, so not only do you lose data if the cache fails, you probably just corrupted the entire backing disk.

Yeah, when I used it on a workstation many years ago, I layered it on top of an MD RAID-1 SSD array for the cache and an MD RAID-5 HDD array for the bulk store.

I used writeback mode, but expected to wipe the machine if the caching layer ever collapsed. In the end, the SSDs outlived my interest in the machine, though I think I did failover an HDD or two while the rest remained in normal operating mode.

Wow, meanwhile it'd be so easy to just take cache flush commands as "only" reordering barriers without breaking the single-system consistency (don't use it for a backing store of a Raft/PAXOS cluster, though!).