good thing quantum computers that can factor numbers have never been built. No number was ever really factored without cheating, the actual shor's algorithm has never been implemented. And we're not really any closer to

That last sentence is not true; we have gotten much closer to building a quantum computer that can run Shor's algorithm. Organizations like Google and Cloudflare have declared a 2029 deadline to completely stop depending on the security of pre-quantum algorithms; hitting that deadline is going to cost a lot of engineering resources, but they're paying that cost because they think there's too great a chance that nation-state adversaries will have scalable quantum computers by then. See https://words.filippo.io/crqc-timeline/ and the various posts linked therein, including from the aforementioned companies.