| ▲ | jeremyjh 2 hours ago | |
Iceberg & Delta Lake do partition the data. You can add as many DuckDB servers processing them as you want, each app can have as many as you need. You pay Amazon for all the I/O and let them worry about it scaling it. I'm not saying there is no point of contention - depending upon your write patterns you have practical limits, but for reading data that is already stored you pretty much just pay more to do more and the rate of that cost is close to linear. I'm sure there is a scale at which that pattern breaks, and I'm sure 99.9% of people reading this will never operate a system at that scale. | ||