But they are part of your net worth. If somebody who becomes richer every year, passively, has "no income", then the definition of income does not match people's intuition about what constitutes fair targets for taxation.

There is a good case to be made to tax wealth rather than income, which comes closer to people's intuition about what's fair.