There is not one discussion without this "argument".

But, let's say, they do. And they also have to stay out. Wouldn't that be.. kinda nice actually?

Your taxes would stay the same either way. Yes, of course, there would be less luxury spending, but the millions of normal people still exist. And they spend and they produce.

Why not let the ultra wealthy just leave and idk go to mars or something? Kick them out, seize their assets, run society without them if they do not want to be part of it anyway.