Remix.run Logo
mrbluecoat an hour ago

A clever repurposing of technologies but realistically only worthwhile for static datasets with range payloads small enough to fit into a web response.

> your pipeline has to rebuild each customer’s file fast enough to meet the update cadence. ... data that updates on a coarse schedule rather than in realtime

simonw an hour ago | parent | next [-]

"static datasets with range payloads small enough to fit into a web response" fits a lot of workloads.

I expect that if your overall data is less than a GB this trick will work really well for you.

simlevesque an hour ago | parent | prev [-]

It doesn't have to all live in the same Parquet file. you can have a Parquet file for all your historical data, plus one for the current week which is updated often cheaply, and then when the week is over you merge that into your big parquet file.

You're making it seem like there's hard limits to what can be done but while there definitely is, you can do incredible stuff.