| ▲ | MertsA 6 days ago | |||||||
>Tectonic some KV store, Tectonic is built on ZippyDB which is a distributed DB built on RocksDB. >What's important here with S3FS is that it supports (1) a fuse client - it just makes life so much easier Tectonic also has a FUSE client built for GenAI workloads on clusters backed by 100% NVMe storage. https://engineering.fb.com/2024/03/12/data-center-engineerin... Personally what stands out to me for 3FS isn't just that it has a FUSE client, but that they made it more of a hybrid of FUSE client and native IO path. You open the file just like normal but once you have a fd you use their native library to do the actual IO. You still need to adapt whatever AI training code to use 3FS natively if you want to avoid FUSE overhead, but now you use your FUSE client for all the metadata operations that the native client would have needed to implement. https://github.com/deepseek-ai/3FS/blob/ee9a5cee0a85c64f4797... | ||||||||
| ▲ | Scaevolus 5 days ago | parent [-] | |||||||
Being able to opt-in to the more complex and efficient user-mode IO path for critical use cases is a very good idea. | ||||||||
| ||||||||