That's just simply not true at all, I don't know where you're getting this idea. Literally every Olympic athlete was more fit that an any astronaut ever.

The military does not survey the population and then select the fittest. So, as a function, it cannot actually perform as you say.

It's the same with F1. "We have the best drivers in the world!" You have the best drivers from the self-selection mechanism you impose on the sport. There are zero reasons to think these categories have good overlap.

The exposure was brief, too. Wikipedia says mesothelioma has been known to develop from exposures of "only" 1 month. That's a scary short time if it's in your home or workplace, but comfortably longer than an Apollo mission. Could be an issue for a future base, though.

I mean Neil Armstrong literally smoked and did not "believe" in excercise so they were absolutely not the most physically fittest people. They were just freaks in terms of enduring a lot of stress tests. Physical endurance is just one aspect they train for. Other aspects were much more valued like them being military flight pilots/smart enough to understand the systems/mentally strong enough to not break down etc. You were not selecting for absolute raw fitness for the apollo missions.