People have varying distance to the server, and regardless, it would still feel really crappy for everyone.

People have varying equipment at home anyway, the only way to make it fair is to force people to come to the same room amd play on exact same equipment, a bit like keirin racers in Japan.