I honestly still see some benefit from working together, quick meetings over lunch and the psychological effects of leaving work. Needing to physically leave your workplace helps your brain leave work at work.

The downsides of it are well known, however. Higher costs of living near work, less mobility (both home and work need to be mostly in the same place, so you end up missing opportunities), unmet demands for housing (resources invested in office space), more time spent commuting in public transport (or worse, cars), and so on. Even for employers, the cost is limiting where you can hire talent. I live in Ireland and, while a lot of talent is concentrated in the Dublin area, there are other talent pools that are simply out of reach to us.

I too love to meet my colleagues in person, but the cost for society, and for myself, are non-trivial. I would love if we had a weekly office day, so we could have meetings, lunch, discuss work, projects, intersections on what we are doing, and so on.