If anything, fitness makes you more susceptible to altitude sickness. It's not an inherent effect, but rather your habits driving you to do things you shouldn't. You are supposed to take things lazy and slow when acclimatizing to high altitude. But if you're fit, you may be used to moving too fast and pushing yourself too hard. You may not recover from exertion as quickly as you expect, and you may end up climbing higher every day than you should.

Altitude sickness typically starts after 12–24 hours. If you climb high and come back down in the same day, there is usually not enough time for the symptoms to start. And 11,500ft is not that high altitude. People routinely fly to Cuzco, La Paz, Lhasa, and Leh from sea level, and most of them suffer no serious ill effects.