I recently built something just like this: https://github.com/sethdeckard/atria

It supports tmux, but you can use it without via embedded terminal. It also has native integration with a few select terminals that expose the right kind of APIs.

Installs as single binary (written in Go) with no external dependencies.