You vote against them when they're up for re-election. If they're not up for re-election, you vote against anyone who allies themselves with said person.