I read that. But afaik, feather format is stable now. Hence my confusion. I use parquet at work a lot, where we store a lot of time series financial data. We like it. Creating the Parquet data is a pain since it's not append-able.

▲

yencabulator 5 hours ago | parent | next [-]

Generally Parquet files are combined in an LSM style, compacting smaller files into larger ones. Parquet isn't really meant for the "journal" of level-0 append-one-record style storage, it's meant for the levels that follow.

▲

aynyc 4 hours ago | parent [-]

So feather for journaling and parquet for long term processing?

▲

yencabulator 3 hours ago | parent [-]

You basically can't do row by row appends to any columnar format stored in a single file. You could kludge around it by allocating arenas inside the file but that's still a huge write amplification, instead of writing a row in a single block you'd have to write a block per column.

▲

amluto 2 hours ago | parent | next [-]

You can do row by row appends to a Feather (Arrow IPC — the naming is confusing). It works fine. The main problem is that the per-append overhead is kind of silly — it costs over 300 bytes (IIRC) per append.

I wish there was an industry standard format, schema-compatible with Parquet, that was actually optimized for this use case.

	▲	yencabulator an hour ago \| parent [-]
		Creating a new record batch for a single row is also a huge kludge leading to lot of write amplification. At that point, you're better off storing rows than pretending it's columnar. I actually wrote a row storage format reusing Arrow data types (not Feather), just laying them out row-wise not columnar. Validity bits of the different columns collected into a shared per-row bitmap, fixed offsets within a record allow extracting any field in a zerocopy fashion. I store those in RocksDB, for now. https://git.kantodb.com/kantodb/kantodb/src/branch/main/crat... https://git.kantodb.com/kantodb/kantodb/src/branch/main/crat... https://git.kantodb.com/kantodb/kantodb/src/branch/main/crat...

▲

gregw2 2 hours ago | parent | prev [-]

Agreed.

There is room still for an open source HTAP storage format to be designed and built. :-)

▲

dionian 4 hours ago | parent | prev [-]

Have you considered something like iceberg tables?

	▲	aynyc 9 minutes ago \| parent [-]
		Yes, but parquet hates small files.