People don't want to rent those homes out because once you're doing so it's difficult to evict a long-term tenant. You just lose out a lot on flexibility - even if you try and manage that risk by leasing out housing e.g. on a yearly basis, landlord-tenant law often overrides that since there are strong ethical reasons for not evicting someone who has since come to treat that rental space as their home.

Short term rentals are better on that score: no one sensible forms a long-term expectation that they're going to live in an Airbnb that they've rented for a few days. (If you think short-term rentals are "bad" for the long-term market or have negative side-effects on the neighborhood, then tax them to manage that tradeoff. But banning them altogether is unconscionable and just leads to houses sitting empty and unused.)