Cool stuff!

I still always want to know the numbers per edge. Because there'a a reaon why these graph DBs try to hold the stuff in memory, everything else is slow as heck. We have ~30bytes per edge in our succinct indexes, which makes the 1.5gb wiki dataset clock in at ~50gb.

If I understand correctly you shard the graph and then content adress each chunk:

- How do you decide on the sharding? I'd expect finding good connected subsets with nice memory locality to be extremely difficult computationally?

- How do you canonicalise your graph. Which has also been an extremely difficult problem in RDF land for example. Although that one is at least efficiently solvable.

Sorry, I didn't see this question when it came in (the comment was collapsed in the thread for some weird reason).

The wikidata image I was using (sourced from a huggingface dataset Arun Sharma kindly uploaded for LadybugDB) is about 10GB zipped I think, and when imported into Slater takes up about 20GB of disk space. It's about 133GB of raw Cypher.

So in bytes/edge, that's probably about 14.

In terms of how to shard (which I'm taking to mean allocation of nodes to ISAM blocks) during an initial build or during a consolidation step, it will try to pack neighbours into the same blocks along with metadata about neighbouring segments, with some additional heuristics to avoid superhubs causing over-filling issues. For on the fly writes it just uses a normal WAL that it builds tables out of until theres a segment merge, at which point it then uses the same logic as for the consolidation.

In terms of canonicalising, it asks for a pk property during import, and then uses a (label, key property, value) tuple for uniqueness. So in a sense it doesn't really do that itself, it relies on that being handled outside the import.

Hope that helps.

PS as a follow-up: one really important memory-saving trick I used was Elias-Fano encoding on the adjacency matrices. Each node keeps an array of nodes it's joined to, and Elias-Fano does an amazing job of compression on those, both on disk and in RAM.