Polls can be capricious, but Trump's recent numbers with some groups have seen big drops.