Everybody should be allowed to vote, except for people who don't want everyone to vote.
Even people who openly aim to violently overthrow the government and abolish elections?
Yes, why not? If they are a minority, then there's no issue. If they are a majority (or close to it), then perhaps they have a point.
If they're an electoral majority then you already have a problem.
But the point is they're less likely to get there if they're part of the power structure.
A presumed but frequently not mentioned component of democracy is the peaceful transition of power once a decision is made.