I guess fitness makes a difference from what these other comments are saying. My ex-wife and I lived in Long Beach (which is obviously sea level) when we were in ROTC and pretty regularly took day trips out to San Gogornio and walked to the summit in about five hours, which is ~11,500ft. Not once did that have any noticeable effect, but we were both pretty serious runners back in the day trying to become Army officers. On the other hand, she tried to summit Aconcagua during a spring break and couldn't make it due to altitude sickness. I've never been higher than Mt. Whitney, personally.

Even if you don't feel it, the altitude still makes a difference, though. I recall doing two-a-day hell weeks at Big Bear at the end of summer cross-country training in high school and there was a 5k up there at the end of that week. We all got worse times than typical at sea level, and somewhat amusingly, I recall a high school senior from Rim of the World High School (who lived up there) getting 2nd place overall the first year I ever competed in that race, beating way more seasoned competitors just because he was used to the altitude.

It works in reverse, too. There was an officer in my Armor Basic Officer Course from Colorado who gave himself rhabdo during the two-mile test the first week we in-processed, apparently because he was so used to altitude that he hadn't quite acclimated to Fort Knox atmosphere.

It's pretty widely accepted in the climbing world that the primary effect of altitude in the short-term is a reduction in your cardiovascular fitness.

The better your heart is at getting oxygen into your muscles and organs, the better it can compensate for less oxygen.

Not a bulletproof solution to altitude sickness, but it's definitely one of a lot of variables that matters. It's also just true that some people are way more susceptible regardless, I've got friends who run competitive marathon times who get splitting headaches flying from sea level to denver.

If anything, fitness makes you more susceptible to altitude sickness. It's not an inherent effect, but rather your habits driving you to do things you shouldn't. You are supposed to take things lazy and slow when acclimatizing to high altitude. But if you're fit, you may be used to moving too fast and pushing yourself too hard. You may not recover from exertion as quickly as you expect, and you may end up climbing higher every day than you should.

Altitude sickness typically starts after 12–24 hours. If you climb high and come back down in the same day, there is usually not enough time for the symptoms to start. And 11,500ft is not that high altitude. People routinely fly to Cuzco, La Paz, Lhasa, and Leh from sea level, and most of them suffer no serious ill effects.

>I guess fitness makes a difference

Not really. Altitude sickness seems quite random in who it effects worst. I trekked to the top of Mera Peak (~21,000 ft) many years ago. 3 of the fittest people in our party got altitude sickness and didn't make it to the peak.