Remix.run Logo
chrisweekly 2 hours ago

> "I can compute PageRank on a directed graph with one billion edges (graph500-26 from the Graphalytics dataset) using 5 GB of memory. Alternatively, I can identify all the weakly connected components in a graph with two billion edges (twitter_mpi from the same dataset collection) using 10 GB of memory. Neither NetworkX nor Igraph can do this; most existing graph algorithms require the graph to fit into memory. Previously, I thought you needed Apache Spark and GraphFrames for billion-scale graph analytics. Now, however, I think all you need is a laptop. I have completely changed my old opinion about using Apache DataFusion for graph analytics."

Impressive!

anon7725 an hour ago | parent [-]

> most existing graph algorithms require the graph to fit into memory.

You can get pretty far with sparse graphs, which are just arrays, in combination with memory mapping.