Yeah makes sense. The SQL thing has also been my differentiator (and yeah, straight up arrow if you aren't doing much transformation of the data), but now I'm curious whether polars sql might be just as good. I kind of like that duckdb allows me to work with a database file, like a sqlite db. But maybe persisting parquet (or arrow directly?) is just as good?

This is why I asked someone else who is also figuring this out!

Really comes down to query speed and management cognitive cost. I look forward to trying the new version for polars.