> But why don't we also have laws criminalizing things like refusing to be friends with $SKIN_COLOR people? I think it's because it's only in those specific realms like employment and public accommodation where we have created rights.

Not just that, at least in my understanding of American political theory. It's because of the existing right to freedom of association. If it is criminal to refuse association, that association becomes compulsory, and thus not free.