Yeah, engineering high scale distributed data systems on top the cloud providers a very weird thing at times.

But the reality is that as large enterprise move to the cloud, but still need lots of different data systems, it is really hard to not play the cloud game. Buying bare metal and direct connect with AWS seems a reasonable solution... But it will add years to your timeline to sell to any large companies.

So instead, you work in the constraints the CSPs have, and in AWS, that means guaranteeing durability cross zone, and at scale, that means either huge cross az network costs or offloading it to s3.

You would think this massive cloud would remove constraints, and in some ways that is true, but in others you are even more constrained because you don't directly own any of it and are the whims of unit costs of 30 AWS teams.

But it is also kind of fun

If cross AZ bandwidth was more reasonably priced it would enable a lot of design options like running something like MinIO on nothing but directly connected NVMe Instance store volumes.