Remix clone Hacker News

The consistency model doesn't seem to make sense.

https://github.com/orbitinghail/graft/blob/main/docs/design....

> Graft clients commit locally and then asynchronously attempt to commit remotely. Because Graft enforces Strict Serializability globally, when two clients concurrently commit based on the same snapshot, one commit will succeed and the other will fail.

OK, but, the API provides only a single commit operation:

> commit(VolumeId, ClientId, Snapshot LSN, page_count, segments) Commit changes to a Volume if it is safe to do so. The provided Snapshot LSN is the snapshot the commit was based on. Returns the newly committed Snapshot on success.

So if a client commits something, and it succeeds, presumably locally, then how should that client discover that the "async" propagation of that commit has failed, and therefore everything it's done on top of that successful local commit needs to be rolled-back?

This model is kind of conflating multiple, very different, notions of "commit" with each other. Usually "commit" means the committed transaction/state/whatever is guaranteed to be valid. But here it seems like a "local commit" can be invalidated at some arbitrary point in the future, and is something totally different than an "async-validated commit"?

▲

carlsverre 2 days ago | parent | next [-]

You're right - it's a bit confusing! I took a crack at explaining it in the blog post under the Consistency section: https://sqlsync.dev/posts/stop-syncing-everything/#consisten...

The key idea is that if your system supports offline writes, then by definition the client making those writes can't have general purpose strict serializability. They have to exist under the assumption that when their transactions eventually sync, they are no longer valid. Graft attempts to provide a strong foundation (server side commits are strictly serialized), however let's the client choose how to handle local writes.

A client may choose any of these options:

1. If offline, reject all local writes - wait until we are online to commit

2. Rebase local writes on the latest snapshot when you come back online, resulting in the client experiencing "optimistic snapshot isolation"

3. Merge local changes with remote changes - this probably depends heavily on the datastructure you are storing in Graft. For example, storing a Conflict-Free Replicated Datatype (CRDT) would work nicely

4. Fork the Volume entirely and let the user figure out how to manually merge the branches back together

5. Throw away all local changes (probably not what you want, but it works!)

My goal is to build a building block on top of which edge native systems can be built. But I'm not too opinionated about what local write semantic you're application needs. :)

(edit: added newlines between list items)

▲

kiitos 2 days ago | parent [-]

What you've said here is totally different to what the repo docs claim.

The guarantees of Graft's "commit" operation are properties of the Graft system itself. If commit is e.g. strict-serializable when clients satisfy one set of requirements, and isn't strict-serializable if clients don't satisfy those requirements, then "commit" is not strict-serializable.

▲

carlsverre 2 days ago | parent | next [-]

Just to make sure I understand correctly, would you agree that if clients always synchronously commit (i.e. wait until the MetaStore commit returns ok) before acknowledging any commit locally, the client will experience Strict Serializability?

Assuming you agree with that, what would be a more clear way to explain the tradeoffs and resulting consistency models in the event that a client desires to asynchronously commit?

I think I see the issue, but I'd love to hear your take on how to update the docs.

▲

ratorx 2 days ago | parent | next [-]

Firstly, it might be worth separating the concepts of read and write consistency. For example, in your system reads are eventually consistent (because syncing is not enforced) and the situation with writes is more complicated.

I think the key point of your design is flexibility, rather than any individual consistency properties. It might be better to emphasise this and try to explain, at the top-level, the concrete ways an application can interact with the storage and the different tradeoffs.

So you might have strong write consistency with forced global serialisation or weaker properties with less enforced sync policies. From the perspective of the application, the internal details shouldn’t matter, but the external properties and how to achieve them (eg. CRDT style merging etc as a way to get strong consistency with less syncing, for certain domains).

▲

kiitos 2 days ago | parent | prev [-]

I think I'm probably operating with definitions of client and commit that are different than yours.

Specifically, I don't really see how a client can "commit locally" and "commit globally" as separate things. I understand a client to be something that interacts with your metastore API, which provides a single "commit" operation, that AFAICT will return success/failure based on local commit state, not global state.

Is that not correct?

Later on in the design.md you say

> The Client will be a Rust library ...

which might be the missing link in this discussion. Is the system model here assuming that clients are always gonna be Rust code in the same compilation unit as the Graft crate/library/etc.?

▲

mdavidn 2 days ago | parent [-]

Graft's definition sounds more like a Git "commit" than one found in a SQL standard. Perhaps that's the source of this confusion?

	▲	kiitos 6 hours ago \| parent \| next [-]
		That may be the case, but the Git notion of a "commit" is a domain concept that is unrelated to, and incompatible with, the distsys notion of a "commit". And terminology like "strict serialization" and "snapshot isolation" and so on -- which this system and its docs reference -- are distsys concepts, not Git concepts. So "commit" needs to abide distsys rules, not Git rules, as you can't really mix-and-match definitions across domains.
	▲	drewcoo a day ago \| parent \| prev [-]
		Git is a DVCS. The D stands for distributed, meaning (in old people language) masterless. Git doesn't have a "global."

▲

tyre 2 days ago | parent | prev [-]

The doc you linked and the author's response here do a good job of clarifying the conditions.

They're building an edge data store that has both local and global characteristics, with certain options meant for offline mode. It's reasonable to assume that the answer is more complicated when talking about the strict serializability of such a system.

▲

feverzsj 2 days ago | parent | prev [-]

It's basically single master asynchronous replication. And only works for sqlite's journal mode. The master saves all sqlite's journals as commit history and sends them to followers to replay them.