| ▲ | sensodine 2 hours ago | |
> every data system that doesn't need <100ms latency is moving to object storage I think the opportunity extends below 100ms too, particularly given the existence of faster object storage tiers like S3 express or more recently GCS rapid bucket (both only offering single-zone durability, so still need to do quorum writes to get region-level durability as with standard tiers). One of the tensions of course is how long to linger before flushing to object storage - you have to trade off directly between latency and cost of your API ops for PUTs. When building the serverless offering of s2.dev (which is in a similar space, full disclosure!), we designed around stateful backend processes capable of constantly flushing multi-tenant objects (i.e., containing records from many streams), allowing streams to offer low ack latencies (~50ms p99 from same region) without blowing up the unit economics. Congrats on the launch btw! | ||
| ▲ | shye 21 minutes ago | parent [-] | |
From my experience benchmarking S3 express, it is is faster, but not fast enough yet for many use cases. An obvious disclaimer is that the word "enough" here is carrying quite the weight: I expect it to get better, and each has their own requirements. Do benchmark yourself and don't make expensive decision based on an HN comment. | ||