There is though - the ssh authentication and connection is already handled, and I'm already in a terminal. When I quit the app or session I'm back in the terminal.

If it worked it would greatly reduce the hassle.

Think about all the TUI apps that exist. They're useful because they're convenient when working in a terminal, not because they look like shit.

If I want to view an image file on a remote machine, and all I have is ssh... I just connect to that machine with filezilla and click on whatever files I want. I can even open files that aren't PNG! Even files that aren't even images at all. Mindblowing.

A terminal with in-band graphics primitives is called an RDP client.

We've had graphics terminals since RIP BBS's and even before that. If they were actually useful enough to be worth the bother, then we'd all have been using them all along and there wouldn't be posts like this.

It's not a case of there's this awesome idea that just for some reason no one knows about. No, it's just not that awesome of an idea. It's not harmful so it doesn't bother me that most xterms support tektronix graphics, it's just a gimmick of no real value. It's a solution to no problem.

Don't believe me? When was the last time you used passthrough printing? Or saw it being used even in some place where they do actually need to print? The terminals all still support it. It's just a thing that you don't need to do in-band in a tty, and today there is no reason to bother doing it that way even though you could. It's not better and does not solve a problem.

What you are looking for is forwarding an X session via SSH, and that has been supported since the dawn of time.