What you're describing is a graphical shell. If you want it over the network, we have a protocol for that, it's called X. Misusing a terminal for this is fundamentally pointless.

Nope. X is too slow to be useful except on local networks, and alternatives like FreeNX or xpra are difficulty and janky to set up. Also X is dying so you really mean something like waypipe or worse (which also jankily failed to run some external command when I last tried it).

But I don't want that anyway. I want something graphical that's actually integrated into the terminal.