Right, but the odds of this happening is small(ish) - I'm certain there is a sweet spot for witholding time. If they don't find a block within the time interval, then effectively all the work for that time is "wasted" by the other participants since it could not have been put on the chain anyway AND the witholder has a headstart of a couple of seconds searching for a new block.

Wasting time would mean not receiving the rewards if they find a block. But that's not the case here. If they find a block within the time interval, they get the rewards (and the withheld block is discarded).