It can go the other direction, too: exposure to moldy home environments gave me (now resolved) food sensitivities, dust allergies, pet-associated allergies, etc.

You can definitely undertrain, or overwhelm, the immune system if not cautious!