Remix.run Logo
▲ Distributed, high-performance blob storage on SQLite(zerodha.tech)
65 points by ghostfoxgod 3 hours ago | 6 comments
▲bob1029 an hour ago | parent | next [-]

> JSON blobs

An alternative to faster blob storage is to represent the blob's schema in SQL directly.

If most blob updates are incremental you can eliminate a significant amount of storage IO and garbage collection this way.

One SQLite database per blob type with a metadata SQLite db that tracks each. We could leverage powerful AI reasoning models to produce SQLite schemas and virtual table adapters for each encountered JSON schema type.

▲feverzsj 2 hours ago | parent | prev | next [-]

Sounds like you should use a database-per-user design.

▲mkesper 2 hours ago | parent [-]

Right, then even sqlite should be enough?

▲CyrusGrissom an hour ago | parent | prev | next [-]

I couldn't find the word "replication" in the post or on the GitHub page. So, what about fault tolerance?

▲tomasol an hour ago | parent [-]

I'm curious too. The only backup I found on GitHub uses cp; they may use filesystem snapshots or Litestream in production. If it’s Litestream, litestream sync -wait can force a sync after a write and wait for remote replication to complete.

▲zodwick 40 minutes ago | parent | prev [-]

all roads lead to sqlite