CephFS implements a (fully?) POSIX filesystem while it seems that TernFS makes tradeoffs by losing permissions and mutability for further scale.

Their docs mention they have a custom kernel module, which I suppose is (today) shipped out of tree. Ceph is in-tree and also has a FUSE implementation.

The docs mention that TernFS also has its own S3 gateway, while RADOSGW is fully separate from CephFS.

My (limited) understanding is that cephfs, RGW (S3), RBD (block device) are all different things using the same underlying RADOS storage.

You can't mount and access RGW S3 objects as cephfs or anything, they are completely separate (not counting things like goofys, s3fs etc.), even if both are on the same rados cluster.

Not sure if TernFS differs there, would be kind of nice to have the option of both kinds of access to the same data.