I know that many people do use ZIP codes for grouping locations, and polygons may be useful for tieing a lat/long to a zipcode if it doesn't have a delivery location. But it's all kind of silly unless you're sorting mail. Also, I'd hate/love to see some of the polygons where delivery boundaries get weird.

Whatever you're grouping is likely better served by some grouping based on geography, not based on delivery routes. You probably wouldn't group by telephone company central office either, unless you're telco related.

We want to group by geography, zip codes are easy and vaguely geographic, lets use that is kind of a valid decision, but it's deeply frustrating :p

Completely agree on this. Something I have used for a few applications is the DMA (designated market area) boundaries. It's not ideal for everything, but it's very human-focused: If you live just a few miles across a state boundary from a major city, grouping you with the city makes more sense than with anything in your state.

In fact, these were invented by Nielsen, and they determined them just by measuring which city's TV stations are predominantly watched in each county, and assigning each county to one DMA based on that.