Most around me seem to be in this mode. The traffic crawls past at pedestrian speed whilst a steadily growing group waiting to cross stand in vehicle fumes. There's a pretty clear principle here of flow conservation. If the light stopping traffic does not reduce overall flow then the light should change for pedestrians immediately. If the traffic flow is low, then a small reservoir is built up which clears immediately. If the flow is slow and saturated the gap that opens up can be refilled quickly. The flow range where the change actually impacts local vehicle flow is rather narrow (high, but not too high). Of course there are second order effects at associated junctions, but at that point maybe the first order interests of pedestrians should be prioritised.