Sorry, I didn't see this question when it came in (the comment was collapsed in the thread for some weird reason).

The wikidata image I was using (sourced from a huggingface dataset Arun Sharma kindly uploaded for LadybugDB) is about 10GB zipped I think, and when imported into Slater takes up about 20GB of disk space. It's about 133GB of raw Cypher.

So in bytes/edge, that's probably about 14.

In terms of how to shard (which I'm taking to mean allocation of nodes to ISAM blocks) during an initial build or during a consolidation step, it will try to pack neighbours into the same blocks along with metadata about neighbouring segments, with some additional heuristics to avoid superhubs causing over-filling issues. For on the fly writes it just uses a normal WAL that it builds tables out of until theres a segment merge, at which point it then uses the same logic as for the consolidation.

In terms of canonicalising, it asks for a pk property during import, and then uses a (label, key property, value) tuple for uniqueness. So in a sense it doesn't really do that itself, it relies on that being handled outside the import.

Hope that helps.

PS as a follow-up: one really important memory-saving trick I used was Elias-Fano encoding on the adjacency matrices. Each node keeps an array of nodes it's joined to, and Elias-Fano does an amazing job of compression on those, both on disk and in RAM.

Neat! Thank you for the in-depth!

So you're also using succinct datastructures with Elias-Fano, you might want to check out this paper https://aidanhogan.com/docs/ring-graph-wco.pdf on WCO joins over succinct indexes.