How does this compare with delta lake and iceberg?

Vortex is a file format, where as delta lake and iceberg are table formats. it should be compared to Parquet rather than delta lake and iceberg. This guest lecture by a maintainer of Vortex provides a good overview of the file format, motivations for its creation and its key features.

https://www.youtube.com/watch?v=zyn_T5uragA

▲

ks2048 6 hours ago | parent | next [-]

The website could use a comparison / motivation in comparison to Parquet (beyond just stating it's 100x better).

▲

3eb7988a1663 3 hours ago | parent [-]

Agreed, really need a tl;dr here, because Parquet is boring technology. Going to require quite the sales pitch to move. At minimum, I assume it will be years before I could expect native integration in pandas/polars/etc which would make it low effort enough to consider.

Parquet is ..fine, I guess. It is good enough. Why invoke churn? Sell me on the vision.

	▲	frisbm 2 hours ago \| parent [-]
		DuckDB just added support for vortex in their last release using the Vortex Python package so hopefully other tools wont be too far behind

▲

sys13 6 hours ago | parent | prev [-]

I think it would still make sense to compare with those table formats, or is the idea that you would only use this if you could not use a table format?

	▲	bz_bz_bz 5 hours ago \| parent [-]
		That’s like comparing words with characters. Vortex is, roughly, how you save data to files and Iceberg is the database-like manager of those files. You’ll soon be able to run Iceberg using Vortex because they are complementary, not competing, technologies.

▲

cpard 5 hours ago | parent | prev [-]

As others said, Vortex is complementary to the table Formats you mentioned.

There are other formats though that it can be compared to.

The Lance columnar format is one: https://github.com/lancedb/lancedb

And Nimble from Meta is another: https://github.com/facebookincubator/nimble

Parquet is so core to data infra and widespread, that removing it from its throne is a really really hard task.

The people behind these projects that are willing to try and do this, have my total respect.