There are techniques to speed up the search for RSA keys quite significantly: they don't scale as with a pure brute force search, nor with a very useful rule of thumb (it's not even the case that doubling the RSA key length doubles its effective security, it's actually a fair bit less than that).