Can you list some realistic workflows where people would be touching the same huge file but only changing much smaller parts of it?

And yes you can represent a whole repo as a giant tar file, but because the boundaries between hash segments won't line up with your file boundaries you get an efficiency hit with very little benefit. Unless you make it file-aware in which case it ends up even closer to what git already does.

Git knows how to store deltas between files. Making that mechanism more reliable is probably able to achieve more with less.

▲

bobmcnamara 5 days ago | parent | next [-]

Most Microsoft office documents.

One of our projects has a UI editor with a 60MB file for nearly everything except images, and people work on different UI flows at the same time.

▲

Dylan16807 4 days ago | parent [-]

So for office, you're looking at files that are archive formats already. Maybe you could improve that a bit, but because of the compression you wouldn't be able to diff text edits better, just save storage. So it would perform about the same as git already does. And you could make it smarter so the prolly tree works better, but you could also make git smarter in the same way, it's not a prolly tree specific optimization.

For your UI editor I'd need to understand the format more.

▲

jiggawatts 4 days ago | parent | next [-]

I picture a next-gen SCM having storage plugins, possibly implemented in something portable like WASM or a portable bytecode. This would enable archive formats to be unpacked for differencing and then repacked for consumption.

Alternatively, apps could detect and support SCM storage distinct from ordinary file storage.

As a trivial example: Office documents are just zip files, so compatible apps could just save them without compression when saving to SCM-managed folders.

Or better yet, compression, but restart it for each contained file to enable efficient binary diffs.

	▲	Dylan16807 4 days ago \| parent [-]
		Zip does restart for each file. But that still means a lot of wasted storage when you change one word for example.

▲

bobmcnamara 3 days ago | parent | prev [-]

The UI file is a single giant JSON file.

I was able to convince them to put in line endings though, so it diffs and compresses well, but GitHub will kick it out in another 40MB or so.

▲

jiggawatts 5 days ago | parent | prev [-]

Binary database files containing “master data”.

Merging would require support from the DB engine, however.