"Let us go" does not only mean "you should let us go" but it is also the first person plural imperative implying that we go. Whether you shorten it to "let's go" or not does not change this.

Same as how "let us pray" is frequently used as well.