I just bit the bullet last week and figured we are going to migrate our self hosted minio servers to ceph instead. So far 3 server ceph cluster has been setup with cephadm and last minio server is currently mirroring its ~120TB buckets to new cluster with a whopping 420MB/s - should finish any day now. The complexity of ceph and it's cluster nature of course if a bit scary at first compared to minio - a single Go binary with minimal configuration, but after learning the basics it should be smooth sailing. What's neat is that ceph allows expanding clusters, just throw more storage servers at it, in theory at least, not sure where the ceiling is for that yet. Shame minio went that way, it had a really neat console before they cut it out. I also contemplated le garage, but it seem elasticsearch is not happy with that S3 solution for snapshots, so ceph it is.

▲

lima 7 hours ago | parent [-]

It's complex, but Ceph's storage and consensus layer is battle-tested and a much more solid foundation for serious use. Just make sure that your nodes don't run full!

▲

antongribok 5 hours ago | parent [-]

Make sure you have solid Linux system monitoring in general. About 50% of running Ceph successfully at scale is just basic, solid system monitoring and alerting.

	▲	ecshafer 3 hours ago \| parent [-]
		This line of advice basically comes down to: have a competent infrastructure team. Sometimes you gotta move fast, but this is where having someone on infrastructure that knows what they are doing comes in and pays dividends. No competent infra guy is going to NOT set up linux monitoring. But you see some companies hit 100 people and get revenue then this type of thing blows up in their face.