Does that mean that if I have a struct with #pragma pack(push, 1) I can't use pointers to any members that don't happen to be aligned?

This is a non-standard extension, so your compiler may provide stronger guarantees.