For a dasaset that live in RAM, the best solution are DuckDB or clickhouse-local. Using SQLish data is easier than a bunch of bash script and really powerful.

Though ClickHouse is not limited to a single machine or local data processing. It's a full-featured distributed database.