>I guess fitness makes a difference

Not really. Altitude sickness seems quite random in who it effects worst. I trekked to the top of Mera Peak (~21,000 ft) many years ago. 3 of the fittest people in our party got altitude sickness and didn't make it to the peak.