I mean Neil Armstrong literally smoked and did not "believe" in excercise so they were absolutely not the most physically fittest people. They were just freaks in terms of enduring a lot of stress tests. Physical endurance is just one aspect they train for. Other aspects were much more valued like them being military flight pilots/smart enough to understand the systems/mentally strong enough to not break down etc. You were not selecting for absolute raw fitness for the apollo missions.

They didn't select for pure physical fitness but they were already selected for fitness as a pilot and then again when they were selected from the pilots to train as an astonaut. Its not like they just picked arbitrarily from the potential pool of candidates and gambled on getting better than average.