I suspect there’s downweights the system can apply - and if five people voted it up, the -5 downweight was applied, and then four of the upvotes were removed, you’d end up at -4.