Hi! Author here.

We engineered, then we measured and then we tried to make it work, probably for a too long of a time before we decided to completely scrap our current architecture and go back to the drawing board. Happy to answer more questions.

▲

a34729t 2 hours ago | parent | next [-]

Why didn't you just use Iceberg and Trino from the get go (or Paimon if you are fancy)? This has been the accepted answer for quite some time now. The fast OLAP stores just don't scale as well as going object store to a proper query engine like Trino.

	▲	roboben 2 hours ago \| parent [-]
		We tried Iceberg and it was too slow writing data is the short story. >The fast OLAP stores just don't scale as well as going object store to a proper query engine like Trino this exactly why we started with this project.

▲

benrutter 3 hours ago | parent | prev [-]

> Happy to answer more questions.

Ah yay, I appreciate it! I feel like its rare to get a look at how these decisions cone about.

What did you measure? How'd you realise there was a problem? And did you measure anything to give you confidence that Trino won't hit the same issues?

	▲	roboben 2 hours ago \| parent [-]
		With DuckDB and the queries we got, we had to provision such huge pods/nodes that we made it work but could never get to an economical viability. In the end, most of our customers had pods running 46 vCPUs / 360GB and asked for more. DuckDB is amazing, it really does everything right on a single node to make your query run successfully to the end almost always but it just doesn't work in our cloud env. Imagine when a query is running for 12h with this since and then another 10 from the same customer come in :) Trino on the other hand has a much more convenient distributed query concept, so you can much more dynamically assign resources.