Words have more than one meaning.

Let also means "to cause to" as in "let me know", or can be "used in the imperative to introduce a request or proposal", as in "let us pray". (Or "let there be light.")

https://www.merriam-webster.com/dictionary/let

The definition you're referring to matches definition 2a, "to give opportunity to or fail to prevent", or definition 4: "to permit to enter, pass, or leave".

"Let's go" absolutely means "let us go". There's no way around it. It's just not the version of "let" that you may be used to, but that doesn't change anything.