Nice. That latency is tough though. On that topic, I wrote an equivalent "magic mirror" type tool that lets you write to the mirror/oracle/VLM on the remarkable and get an answer back.

A few possible solutions I explored:

1) You can try and consume (and possibly write to) the frame buffer directly. https://github.com/ddvk/remarkable2-framebuffer was my starting point. This gets you instant updates about what's going on. I guess you could pair this with speculative decoding to get a much faster output.

2) You can use the streaming API on the device to stream the screen to a beefy server, possibly over Tailscale, letting you do everything off device.

3) You can write your own Qt app; ddvk's repos are a good starting point here.

I ultimately instructed claude to write me my own app. Which worked enough to scratch my itch, and I never use it. But this was five months ago, an eternity in vibe hobby projects, so perhaps modern tooling would let me get it in shape and be more usable. Basically, it worked, but VLMs weren't great at what I wanted -- decorating a blank page with a grimoire-style answer to a written question while leaving the original text alone -- and getting it started / stopped from the pen UI is difficult.

Where my mind goes for your project is that I think it'd be nicest to keep a sort of Jupyter notebook somewhere that's the canonical representation, that would have your handwritten blocks and an interpretation, and then the output. then a render layer to get it back onto the screen. At that point, I don't think I'd care very much if it's stored as a PDF on the device, which points back to having this be an app.

Either way it's fun to tinker! the RM line is very hackable, and I still wish it were even easier, the hardware just makes you think of so many possibilities.