I believe I got rid of most of the issues with some .ssh setting which simply uses the normal xterm I believe. I can't check right at this moment, but apparently, simply using xterm and not the xterm-xkitty or what it's called does not actually make much problems when ssh-ing into a machine - not in my case at least.

It was a setting in some ~/.ssh config file though I believe, not a kitty setting.

I currently just use this in my .zshrc:

alias ssh="kitty +kitten ssh"

sshpass() { TERM=xterm-256color command sshpass "$@"; }

Could also alias sshpass differently but I mostly use that for embedded devices where I just want a simple shell anyways.

The alternative you mean is in ~/.ssh/config:

Host *

    SetEnv TERM="xterm-256color"