If they all started at zero and the ones that are no longer in existence end at zero, then roughly speaking, wouldn’t that have to happen?

But in slightly more detail, not every empire has ended, yet, if you count Russia and the Chinese as empires. Also, some empires have had declines that reversed again for a while, such as Byzantine Empire.