You also have to be an MP.

I don't think that's strictly true. You definitely don't have to be an MP in order to hold a cabinet office. I think that extends to the PM as well. It's never been tried obviously.

It's not strictly law, but at this point it might as well be. The last PM to be appointed without being an MP was Sir Alec Douglas-Home in the 1960s, and he immediately resigned his peerage and won a by-election to remedy that...