I don't have hard data yet but I'm pretty sure some cities have suburbs outside them, connected via road, that rich people use as tax havens so they can live near a city without being subject to the laws and taxation of the city

Right but if you go into a country then you're in the country, not in the outskirts. You still pay taxes (generally...), and, in many countries, don't get any social services.

If anything, many formally-colonial countries are leeching off their illegal immigrants, not the other way around.

Countries are generally big and with cities on both sides of a border so that doesn't seem like it would be a big worry for them.