You have a lot more flexibility around those downswings at 40 than you do at 80. For example you could retire but continue doing work that brings you joy and also happens to make money. A lot of people in this situation start lifestyle businesses or do consulting work, for example.

At $4mm in a market account (not 401k), you also have the option to take out margin loans at shockingly low interest. This gives you untaxed cashflow without touching your principal. 160k untaxed is a lot more cashflow than the same number in salary or retirement distributions.