> The caveat is that Litestream replication is asynchronous. A restore can miss the newest local writes if the SQLite volume disappears before they are copied. That is fine for many AI and experimentation workflows

In short: SQLite is not all you need, unless you’re just experimenting don’t actually care about durability, in which case you also need litestream + object storage.

Right.

▲ gwking 7 hours ago | parent | next [-]

The suitability of Litestream for production disaster recovery is also an open question in my mind. I used 0.3.x for several years and when I tried to upgrade to the 0.5.x series there were runaway disk usage problems that would have caused downtime had they made it to prod. As far as I can tell these have not been entirely addressed, although recent bug reports suggest that they might be getting closer.

I want to love it, and I don't take open source projects like this for granted. But during my last production upgrade I chose to decommission Litestream in favor of a dumber, less granular solution using sqlite3_rsync and nightly backups because there is no point in using a backup system that is not rock solid.

▲ 0cf8612b2e1e 7 hours ago | parent | prev | next [-]

Postgres also does not synchronously replicate for free. You can setup both to get a confirmation write if you require that durability.

▲ orf 7 hours ago | parent | next [-]

> postgresql also does not synchronously replicate

By default. Generally your primary database is in a completely different failure category than a kubernetes node running an ephemeral workflow pod.

	▲	0cf8612b2e1e 6 hours ago \| parent [-]
		Either you have durable storage or you do not. SQLite and Postgres can both ensure local durability of commits. If you want distributed durability, you need to ship that data elsewhere. That is another Postgres node, object store, whatever that’s still an external dependency.

▲ paulddraper 7 hours ago | parent | prev [-]

Not for free, but without the needing additional software.

  synchronous_commit = on

	▲	0cf8612b2e1e 7 hours ago \| parent [-]
		That’s about the local transaction, not replication. SQLite WAL also gives you strict durability. `PRAGMA synchronous = full`

▲ bootsmann 7 hours ago | parent | prev | next [-]

S3 is strongly consistent, if you need it anyways you can just use s3 keys to deconflict and store the workflow state.

	▲	orf 7 hours ago \| parent [-]
		Yes, but directly using s3 as a key-value database is completely different from using SQLite + litestream.

▲ paulddraper 7 hours ago | parent | prev | next [-]

"Durable workflows without the durability"

That's distributed workflows :)

▲ dilyevsky 6 hours ago | parent | prev [-]

i mean it's durable as long as nothing crashes or litestream has a data corruption bug which only happens every other release...