Append-only is pretty much the only way to do robust replicated storage at scale, else you get into scenarios where, instead of a given logical block having two possible states (either existing somewhere or not existing anywhere), the block can exist with multiple values at different times, including an unbounded number of invalid ones, for instance in case a client died halfway through a block mutation. Immutability is just plain a very strong invariant.

It also does not at all preclude implementing a read-write layer on top of it, for instance with a log-structured FS design. That's however the solution to a problem these people are, it seems, not having.