I like this idea in theory. In practice, the problem is that someone gets to decide who is allowed to vote and on what grounds. If that institution is corrupted, it leads to worse outcome than allowing everyone to vote. And the bad actors would have all the incentives in the world to corrupt that institution.