em is a generic "font-size" without regard to the axis. ch is specifically related to the font width, and ex is specifically related to the font height. So if you want things like padding and margins to be proportionate to the font size, then using em will be suboptimal for fonts that happen to be tall or short. And as a web developer, you generally don’t know which font will be used (content management systems, designers making different choices in the future, users overriding fonts for accessibility reasons, etc.)

For lengths that you specifically want to be axis-independent and related to the font size, e.g. border-radius, then you should still use ems.