Thank you for your thoughts and support. I agree, doing this in X11 is definitely doable. I’ve seen others achieve similar results by running a daemon that listens for damage regions. I hadn’t considered using a custom backend video driver; that’s something I’ll be looking into.

My focus has mostly been on Wayland, but possibly prototyping in X11 could be a starting point. Some people I've met are interested in using computer vision techniques at the hardware level, or making a camera rig.

What excites me most is getting this into people’s hands, seeing what they create with it, and working together to improve it and explore new use cases. One use case that I learned about recently:

E-Ink News: The mystery of cuttlefish disguise, scientists use E Ink technology to explore its secrets

https://www.ereaderpro.co.uk/en/blogs/news/e-ink-news-the-my...

Heh, neat use re: tank + fish!