It is kinda of crazy how much work is done to mitigate the very high latency of S3 when we have NVMe SSDs with access latency of microseconds.

Yeah, engineering high scale distributed data systems on top the cloud providers a very weird thing at times.

But the reality is that as large enterprise move to the cloud, but still need lots of different data systems, it is really hard to not play the cloud game. Buying bare metal and direct connect with AWS seems a reasonable solution... But it will add years to your timeline to sell to any large companies.

So instead, you work in the constraints the CSPs have, and in AWS, that means guaranteeing durability cross zone, and at scale, that means either huge cross az network costs or offloading it to s3.

You would think this massive cloud would remove constraints, and in some ways that is true, but in others you are even more constrained because you don't directly own any of it and are the whims of unit costs of 30 AWS teams.

But it is also kind of fun