The only person who seems to claim that Dyson Spheres were a "joke" is Angela Collier and people quoting her. I've seen no other source for this claim.

But even if it's true, which I don't think it is, it doesn't matter. Why? Because the idea is actually sound and a lot of thought has gone into it by people who have backgrounds in physics and engineering.

It's worth adding that "Dyson Swarm" is the more common nomenclature now because of a mistaken belief that a Dyson sphere was a rigid spherical shell you build around a star. That was never the case and there's no even theorized material that could support this kind of rigid structure. It was always a cloud of small orbitals so the "Dyson Swarm" should alleviate any confusion (hopefully).

Our civilization is ultimately energy-limited. I've seen estimates that we use ~10^11 Watts of energy. The Sun's entire output is ~10^25 Watts. That's a truly staggering amount of energy. We can barely comprehend what might be possible with that much energy. But one of those things is interstellar travel.

Many discussions on interstellar travel gloss over just how large the energy budgets are. But propelling objects with solar sails, particularly if you concentrate energy, seems feasible. At least the physics holds up.

Spreading throughout the galaxy becomes an almost inevitable consequence once you have the capability to do so.