I'm very curious about the convergence of the best in class fast OLAP products (StarRocks, ClickHouse) with Trino. It sounds like everybody is going for decoupled compute/storage, using S3 or similar as the storage layer, and thus forgoing colocated joins (ok I know ClickHouse joins suck)...

So what does this mean for ingestion (and indexing)? Iceberg V3? Paimon? Bespoke ingestion through the DB engine to do the indexing?

▲

jimmyl02 2 hours ago | parent | next [-]

It's also interesting how Clickhouse / Starrocks can now also act as a query planner and executor on top of non-native formats (ex. Iceberg).

I assume the native formats will always be faster / more optimized but the need for Trino as a separate executor while running either of these databases seems to be close to gone.

	▲	a34729t 2 hours ago \| parent [-]
		The benefit of StarRocks/ClickHouse over Trino is that you get secondary indices, but that means you have to do the indexing somehow. Native format is faster (especially for colocated joins), but it's way more expensive if you have to run a bunch of separate storage nodes vs just using S3, especially your query volume isn't that high. I liken it to the BigQuery cost model, where storage is effectively free.

▲

efromvt 2 hours ago | parent | prev | next [-]

I’ve historically read this as ‘open format compatible’ but ‘native preferred’ - where this opens up market space and dev velocity - but it’ll be interesting to see if native storage differentiation gets dumped entirely. It just seems like ‘fork and optimize for our engine’ would always be tempting enough that you’d want a native play for when you don’t need the decoupling.

▲

mrits 3 hours ago | parent | prev [-]

Clickhouse joins have been improving almost every month for the last couple of years. Maybe they still suck but a lot less