| ▲ | zzsheng 7 days ago | |||||||||||||
Author here. This is not a general argument against fsync; the design depends on SSD-only deployment, preallocated files, O_DIRECT, single-key atomicity, and device write guarantees. | ||||||||||||||
| ▲ | atombender 4 days ago | parent | next [-] | |||||||||||||
I'm surprised none of the design decisions considered an indirection between the folder tree structure and the actual files. For example, if you map folders like /foo and /foo/bar to numeric IDs, then each file can simply refer their parent folder. Renaming a folder, or moving a folder to a new parent, does not need to update any files. You can take this a step further and have a three-level split: Tree, file-tree join table, and files. The tree describes the hierarchical structure of folders (which changes more rarely than files do), while the file-tree join table is essentially [folder_id, file_id]. When a file is moved, only the join table (which is much smaller than the files and super sortable and compressible) must be updated. I take the point that updating multiple discrete pieces of information puts more demand on the transactional layer, which has to ensure atomicity and consistency. But I'm surprised it wasn't even mentioned as an alternative that was evaluated and rejected. The article starts out with the premise that a flat key/value approach is the only choice on the table. | ||||||||||||||
| ▲ | 100ms 5 days ago | parent | prev [-] | |||||||||||||
Your approach looks interesting but I was curious when you talk about path-based splitting for ART, do you literally mean always on "/"? I know S3 directory buckets always use /, but the classical S3 model had no natural separator character and I was wondering if supporting those styles of prefix or custom delimiter queries suffered any impediment in your approach. Bookmarked your whole blog for later consumption, interesting stuff! | ||||||||||||||
| ||||||||||||||