"People weren't their strongest even when most did manual labor."

Is there good historical data on some measure of strength across representative populations over time in the modern era? I'm doubtful.

We do know that the introduction of agriculture diminished strength:

"Bone mass was around 20% higher in the foragers - the equivalent to what an average person would lose after three months of weightlessness in space.

After ruling out diet differences and changes in body size as possible causes, researchers have concluded that reductions in physical activity are the root cause of degradation in human bone strength across millennia."

cam.ac.uk/research/news/hunter-gatherer-past-shows-our-fragile-bones-result-from-physical-inactivity-since-invention-of