What are the downsides of taking what amounts to unsubstantiated gossip seriously?
Is there an existing phrase for doing precisely what the OP said people do as a response :D