For decades we were encrypting our communications with rsa, surely nothing is wrong with it?

There is nothing wrong with it, and RSA is still commonly used. In fact, RSA is better against quantum computers compared to ECC.