Most startups can just scale your traditional separation of compute & storage here though. You’d be shocked how well duckdb against s3 scales for 99.9% of use cases

▲

zbentley an hour ago | parent [-]

It's one thing to be able to scale database compute/storage; it's another thing to be able to partition it. It's extremely common for bad queries/access patterns to cause noisy-neighbor effects on other simultaneous accesses to the database, to the extreme of knocking the whole database over with timeouts/OOMs/etc.

Scaling out DB compute can only help with that to a (expensive) point; eventually, you end up wanting to either prevent the bad queries from being added to the system (DBA culture) or ensure that the bad query runs on database infrastructure that doesn't affect other queries. That's why partitioning DB compute (and storage: noisy-neighbor effects from a bad query at the storage layer don't require storage to be running e.g. a BookKeeper or whatever on a server; they can manifest as hot S3 keys or cloud object/block store rate limiting) is a necessary capability if your plan for dealing with a culture of "anyone can add any access pattern they want" is to scale the DB.

	▲	jeremyjh 35 minutes ago \| parent [-]
		Iceberg & Delta Lake do partition the data. You can add as many DuckDB servers processing them as you want, each app can have as many as you need. You pay Amazon for all the I/O and let them worry about it scaling it. I'm not saying there is no point of contention - depending upon your write patterns you have practical limits, but for reading data that is already stored you pretty much just pay more to do more and the rate of that cost is close to linear. I'm sure there is a scale at which that pattern breaks, and I'm sure 99.9% of people reading this will never operate a system at that scale.