I think a big part of undergraduate math is theory/abstraction building, which is usually very different than the activity of programming. I've studied quite a bit of math, so I can appreciate both. But I can definitely see why the former drives people away.