I'm not going to opine on philosophy. These "arbitrary rules" are the supreme law of the land, backed by the application of violence and our economy.

> Some other people may want to treat everyone fairly

"Fair" is a subjective term: are we obligated to provide an economic social safety net for the entire world regardless of contribution? (No.)

Thankfully, in a democracy, we decide collectively as a society what "fair" is.

> It follows that you didn't do anything to deserve more rights.

Ok? The legal fact of the matter is that I do indeed have more rights than a foreigner. Those same foreigners have more rights than I do in their home countries. This has been a constant theme since the dawn of civilization, and isn't going to change any time soon.