I find that Ceph is pretty annoying to operate but if you do, it works fine. Keeps redundant copies of data on multiple disks, across multiple racks if your diversity is that wide. Supports erasure coding for effective redundancy less than 2x.

Don't go for the "object gateway" compatibility layer - just use raw Ceph if you're writing your own app.