Speaking up would have a higher failure rate. Speaking up overtly stimulates the annoying person's defensiveness and propensity to challenge back. Subtle manipulation always works better.