I don't know why you think SV doesn't have those downsides. In fact the entire world suffers the price of SV driving away their own employees with institutional knowledge and massively diminishing the quality of their software as a result.

I worked at a series of startups and I think having lunch every day with the other employees was an excellent way to get to know them. These jobs didn't last long because the startups weren't all that successful, but I remember them well.

Or maybe it was because I was younger then?

Stayed at Google over a decade and it wasn't quite the same, particular when working with people in distant offices.