I never assumed that my chosen password was the only input to the key derivation. Just that my chosen password is at least one of the inputs to the key derivation. LUKS has been around for quite a while.

You're asking for the encryption key to be derivable by an unrelated system that only has the encrypted file and your password.