Yeah, not sure where the phrase originated but it does sound bad when you put some thought into it. My sister is a redhead and people loved to make fun of her growing up, telling her there's no way two parents with brown hair could have a kid with red hair, so the mailman (who also had red hair) was obviously her dad.