Personally I would suggest that the "easiest S3" would be simply using NFS. You can get replication with RAID.

S3 is simple for the users, not the operators. For replicating something like S3 you need to manage a lot of parts and take a lot of decisions. The design space is huge:

Replication: RAID, distributed copies, distributed erasure codes...

Coordination: centralized, centralized with backup, decentralized, logic in client...

How to handle huge files: nope, client concats them, a coordinator node concats them...

How will be the network: local networking, wan, a mix. Slow or fast?

Nature of storage: 24/7 or sporadically connected.

How to handle network partitions, pick CAP sides...

Just for instance: network topology. In your own DC you may say each connection has the same cost. In AWS you may want connections to stay in the same AZ, use certain IPs for certain source-destination to leverage cheaper prices and so on...

▲

klodolph 4 hours ago | parent [-]

NFS in practice is too different from S3 to make this work.

I’ve been at a couple companies where somebody tried putting an S3 interface in front of an NFS cluster. In practice, the semantics of S3 and NFS are different enough that I’ve had to then deal with software failures. Software designed to work with S3 is designed to work with S3 semantics and S3 performance. Hook it up to an S3 API on what is otherwise an NFS server and you can get problems.

“You can get replication with RAID” is technically true, but it’s just not good enough in most NFS systems. S3 style replication keeps files available in spite of multiple node failures.

The problems I’m talking about arise because when you use an S3-compatible API on your NFS system, it’s often true that you’re rolling the dice with three different vendors—you have the storage appliance vendor, you have the vendor for the software talking to S3, and you have Amazon who wrote the S3 client libraries. It’s kind of a nightmare of compatibility problems in my experience. Amazon changes how the S3 client library works, the change wasn’t tested against the storage vendor’s implementation, and boom, things stop working. But your first call is to the application vendor, and they are completely unfamiliar with your storage appliance. :-(

	▲	themafia 3 hours ago \| parent [-]
		> but it’s just not good enough in most NFS systems. NFS is just an interface. At the end of the day it's on top of an FS. It's entirely possible and sometimes done in practice to replicate the underlying store served by NFS. As you would expect there are several means of doing this from the simple to the truly "high-availability."