With exercise and cognition alike, the concern is motivational. If you let yourself get out of shape, it can be very difficult to rebuild the habits required to get back into shape. You get drawn into a bad local maximum (e.g. prompting everything, driving everywhere) and find you don't have the energy to get out.