| ▲ | chrisweekly 2 hours ago | |
> "I can compute PageRank on a directed graph with one billion edges (graph500-26 from the Graphalytics dataset) using 5 GB of memory. Alternatively, I can identify all the weakly connected components in a graph with two billion edges (twitter_mpi from the same dataset collection) using 10 GB of memory. Neither NetworkX nor Igraph can do this; most existing graph algorithms require the graph to fit into memory. Previously, I thought you needed Apache Spark and GraphFrames for billion-scale graph analytics. Now, however, I think all you need is a laptop. I have completely changed my old opinion about using Apache DataFusion for graph analytics." Impressive! | ||
| ▲ | anon7725 an hour ago | parent [-] | |
> most existing graph algorithms require the graph to fit into memory. You can get pretty far with sparse graphs, which are just arrays, in combination with memory mapping. | ||