Hash the files and commit the hashes. Store the large files somewhere convenient, keyed by the hash. Voila, you’ve invented an LFS.

That's just how plain git works anyway. Commits are just a tree of hashes which refer to the blobs in the objects store. Diffing is done on demand. If you can implement diff for binary files, then you've got git.