Remix.run Logo
loeg 3 days ago

We've observed FDP to make a surprisingly big difference in drive internal WA. If you can meaningfully tag different lifetime/stream data from your workloads, and you can expect hardware that supports it, it's very helpful. We some something like WAF reduction from ~1.60 to ~1.04 (on a synthetic but vaguely plausible workload).

jeffbee 3 days ago | parent [-]

With rocks, I assume?

loeg 3 days ago | parent [-]

Most of our writes (>99%) aren't rocks.

jeffbee 3 days ago | parent [-]

Hrmm. Still guessing about your workloads, but isn't it possible that workload A could cause a disproportionate amount of amplification, while still being much smaller in aggregate than workload B?

loeg 3 days ago | parent [-]

Seems possible, though we only incorporated FDP changes to the non-rocks writes and saw large WAF reduction. (We're approximately an object store as a service with rocksdb for metadata and direct disk writes for data; typical object size is hundreds of kB to single-digit MBs, and rocks updates are batched. Many of our users can accurately predict how long their objects will live, so we can segregate FDP streams by lifetime.)