You can't always replace "let us" with "let's", but you can always replace "let's" with "let us".