Yup, using rem for as many things as possible has always been good to me.

This is bad advice. You should only use rem for text, e.g. font sizes and paragraph margins.

If the user is on a phone and has a larger default font size due to vision difficulties, making padding scale with the font size takes screen real estate away from the larger text the user needs.