I independently came to that conclusion after noticing that my comments on some controversial subjects will slowly acquire upvotes, then nose dive 2 or 4 votes very quickly. At first I thought that some mods might have double-value votes (Hi Cowboy Neal!) but then I realized it's sometimes 4 successive downvotes, not only 2.

I've frequently encountered this same thing. Often the downvotes come a day later, but quickly.