I wonder if they could exploit terminal emulators... Could breach my VMs and get into my host that way.

Yes, there have been many CVEs for "terminal escape-sequence injection".

Damn, this is a good one. Sounds like we need an ANSI sanitizer, keep only basic formatting, remove all esoteric escapes, the fancy Sixel & co stuff.

For paranoia you could us a Chrome like multi-process architecture, the ANSI parser runs in it's own sandboxed process.