I was in the best shape of my life after religiously going to the gym for ~3 years in my late twenties.

Outside of a couple of women asking me for help at the grocery store with some heavy produce, it barely made any difference to how people treated me.

Being fit is just 1 aspect. These days you also have to look like you're a top 20% in a bunch of other categories as well.