Yup. Collider/mesh math is where the really esoteric, fun-killing bugs start to be introduced. 3D is cool, but on another level (hehe, literally).
Just use spheres