I don't see how your argument makes sense. It's all just bits of entropy in the end, be it knowing a port to connect to or a character in your key.

Yeah absolutely. That was precisely my point — Requiring a secret (be it a password or the private part of an asymmetric key) isn't security through obscurity, and finding the sequence of knocks is equivalent to finding a password of equivalent complexity.