I have tried every possible setting but SSH ends up breaking more often than not. As opposed to iTerm which just works.

Same here. I have a Ghostty often open but end up using iTerm2 for pretty much everything. It's not just having to hack the TERM (which I ended up doing), it's that sometimes something about the TERM internally breaks anyway and I'm back to seeing ESC codes when I press an arrow. Quit & restart fixes it but it's just enough friction to keep me away.