That's "just life" unfortunately. By the time most people reach their mid-30s, they accumulate enough commitments that they have to plan things in advance or they just don't happen.

If a friend turned up randomly and unannounced at my door and asked if I want to go to the pub, they have the following barriers to overcome:

I'm out for 10 hours a day during the work week and I'm asleep another ~8h. In the remaining 6h I want to go to the gym, I have to eat, I have to run errands, I have to spend time with my wife and in the remaining time I might just want some alone time. Odds are I'd have to turn my friend away, which would make me feel bad, even though I'd have gladly joined him in the pub if I could plan for it. And I have no kids! If I had kids, the odds are even worse.

We all wish we could be back in our 20s when we had all the freedom and none of the commitments, but the truth is that for 95% of people this isn't possible. When you are in uni (or fresh out), you have all the time and energy, but (generally) no money. So you can spend a lot of time with friends who are in a similar position. By mid-life (30-55), you have money and energy, but no time. And in your winter years, you have money and time, but no energy left.

In each of these phases, you can try to go against the flow and experience friction all the time, or you can try to make the most of it and adapt.

If you absolutely desire the freeform approach you describe, perhaps you need to step up and establish the clubs and spaces you'd like and select for members who have a similar desire.

Most spaces in cities have to cater to the lowest common denominator, and simply wouldn't be able to function without strict scheduling rules. It'd be patently unfair if the 20 year old uni student hogged the equipment when you turn up for your 2 hours of free time that you planned a week in advance, wouldn't it?