I agree, and I had to think about it for a second, but now it seems obvious. It works for the exact same reason that averaging multiple independent measurements can give a more accurate result. The key fact is that the different random variables are all independent, so it's unlikely that the various deviations from the means will line up in the same direction.