Putting the following in my ~/.ssh/config fixed all ssh-related issues I had with kitty.

  Host *
    SetEnv TERM="xterm-256color"