Remix.run Logo
srcreigh 2 hours ago

MapReduce is from a world with slow HDDs, expensive ram, expensive enterprise class servers, fast network.

In that case to get best performance, you’d have to shard your data across a cluster and use mapreduce.

Even in the authors 2014 SSDs multi-core consumer PC world, their aggregate pipeline would be around 2x faster if the work was split across two equivalent machines.

The limit of how much faster distributed computing is comes down to latency more than throughput. I’d not be surprised if this aggregate query could run in 10ms on pre sharded data in a distributed cluster.