Yeah, I ran into this problem too. I tried a few different hacky solutions and then settled on using port knocking to sort inbound ssh connections into their intended destinations. Works great.

I have an architecture with a single IP hosting multiple LXC containers. I wanted users to be able to ssh into their containers as you would for any other environment. There's an option in sshd that allows you to run a script during a connection request so you can almost juggle connections according to the username -- if I remember right, it's been several years since I tried that -- but it's terribly fragile and tends to not pass TTYs properly and basically everything hates it.

But, set up knockd, and then generate a random knock sequence for each individual user and automatically update your knockd config with that, and each knock sequence then (temporarily) adds a nat rule that connects the user to their destination container.

When adding ssh users, I also provide them with a client config file that includes the ProxyCommand incantation that makes it work on their end.

Been using this for a few years and no problems so far.

Doesn't this require configuration at the end user, so you could just as easily ProxyJump or use a different port?

It's a nice solution but I've been looking for something more transparent (getting them to configure an SSH key is already difficult for them). A reverse proxy that selects backend based solely on the SSH key fingerprint would be ideal