Remix.run Logo
local_surfer 4 hours ago

Interesting! Triblespace seems similar to TerminusDB and the solution presented here - would you mind stating the differences ?

j-pb 2 hours ago | parent [-]

Spot on!

In one word. Simplicity!

TerminusDB is a RDF database with build-in prolog, a heavy focus on succinct data structure indexes, and has a client server model.

Triblespace is purposefully not RDF, because RDF has horrible DX, and it does not have a good reconciliation and distributed consistency story.

It's virtually impossible to get RDF data into a canonical form [https://www.w3.org/TR/rdf-canon/#how-to-read] and trivial stuff like equality of two datasets is NP-Hard.

Triblespace, while also a data exchange standard like RDF, is closer to Datascript or Datomic. It's a Rust library, and great care has been taken to give it extremely nice DX.

In-memory datasets are cheaply clonable, and support efficient set operations. There's macros that integrate fully into the type system to perform data generation and queries.

    // The entity! macro returns a rooted fragment; merge its facts into
    // a TribleSet via `+=`.
    let herbert = ufoid();
    let dune = ufoid();
    let mut library = TribleSet::new();

    library += entity! { &herbert @
        literature::firstname: "Frank",
        literature::lastname: "Herbert",
    };

    library += entity! { &dune @
        literature::title: "Dune",
        literature::author: &herbert,
        literature::quote: ws.put(
            "I must not fear. Fear is the mind-killer."
        ),
    };

    ws.commit(library, "import dune");

    // `checkout(..)` returns a Checkout — a TribleSet paired with the
    // commits that produced it, usable for incremental delta queries.
    let catalog = ws.checkout(..)?;
    let title = "Dune";

    // Multi-entity join: find quotes by authors of a given title.
    // `_?author` is a pattern-local variable that joins without projecting.
    for (f, l, quote) in find!(
        (first: String, last: String, quote),
        pattern!(&catalog, [
            { _?author @
                literature::firstname: ?first,
                literature::lastname: ?last
            },
            { _?book @
                literature::title: title,
                literature::author: _?author,
                literature::quote: ?quote
            }
        ])
    ) {
        let quote: View<str> = ws.get(quote)?;
        let quote = quote.as_ref();
        println!("'{quote}'\n - from {title} by {f} {l}.");
    }
Data has a fully tracked history like in terminus, but we are overall more CRDT-like with multiple scopes of transactionality.

You can store stuff in either S3 or a single local file (for the local file you can union two databases by concatenating them with `cat`).

We also have just recently added sync through Iroh.

The core idea and main difference between RDF is that RDF is text based and weakly typed, we are binary and strongly typed.

We split everything into two basic structures: - the tribles (a pun on binary triple), 64byte units that are split into [16byte entity id | 16byte attribute id | 32byte Value] where the first two are basically high entropy identifiers like UUIDs, and the last is either a Blake3 hash, or an inlined <32byte value, with the type being disambiguated by metadata on the attribute ID (itself represented as more tribles) - blobs, content addressed, arbitrary length

It's pretty easy to see why canonical representations are pretty easy for us, we just take all of the tribles, sort them lexicographically, dedup them, store the resulting array in a blob. Done.

Everythign else is build up from that. Oh and we also have succinct datastructures, but because those are dense but slower, and immutable, we have a custom 256-ary radix trie to do all of the immutable set operations.

The query engine is also custom, we don't have a query planner which gives us 0.5-2.5microseconds of latency for queries depending on the number of joins, with a query engine that is fully extensible via traits in rust.