> just for the crime of delaying them by a few seconds

Often it's not an actual delay, but just a perception of a delay as the driver then arrives at the next queue a few seconds later which makes no difference to their total journey time.