The point of the saying is to get most people to understand that the biggest factor in the average person's weight loss is their diet, not their exercise levels.

Pointing out that elite endurance athletes and olympic athletes can have high calorie requirements isn't helpful or the point. Yes, energy must come from somewhere, but even among the generally fit and active portion of the population, only a vanishingly small number of them exercise at the intensity and time requirements to burn so many calories as to not easily have all of that thwarted with a single meal.

> The point of the saying is to get most people to understand that the biggest factor in the average person's weight loss is their diet, not their exercise levels.

Just say that then.