Remix.run Logo
Show HN: Running a vision model on every screenshot on-device(github.com)
16 points by alexkarpathy 4 hours ago | 4 comments

hi author here, Screenmind is privacy first Microsoft recall alternative . It runs on gemma 4 which is one of the fewer models supporting vision audio and reasoning all 3, so your data never leaves you machine.

With screenmind you can keep a track of your timeline , how much time you spent on what..search any screenshot with any text on it.. and the coolest thing, you can chat with your screen history, like what did alex texted me on discord or did i received any mail from Microsoft, if it was on your screen , you can prompt it in the cha. and also you can make automations on top of it, like send me my whole day report on slack(it has integrations )..you can also write automation either though plain English for not so coders or use the python for devs who want to deep dive, and you can save voice memos(with a screenshot) with just a hotkey, and get you meeting transcribed and summarised(auto detects meeting)

the hardest part which i faced was keep running screenmind as a background service it would not have been not hard if chat feature didn't existed, as running local model requires compute ..and keep analyzing screenshots continuously will keep all the resouces hogged up for that i came up with a perceptual has cache .. the three tier cache system reduces inference upto 40% for an average user(which is me)..and to reduce the inference time more i came up with three modes..fast balanced and accurate..where the tradeoff is between time and accuracy

for now i use it daily on my 4gb gtx 1650 with fast mode, works pretty fine also it would be much faster on high end machine , it also has a mcp server so you can just ask claude desktop/cursor about the bug you saw in morning..

supports windows/mac/Linux

being upfront about rough edges , it is not extensively tested on mac and installation has some friction , for which i m working on one click installer thing

(reposting- i put up an earlier version a few days back, comments got flagged cuz of new account so couldn't reply to any )

repo:github.com/ayushh0110/ScreenMind

curious about anyone have idea for how to approach multi monitor support

wassimgr 2 hours ago | parent | next [-]

You’re basically building https://coreviz.io though

Oxodao 3 hours ago | parent | prev [-]

Could be quite interesting especially since it runs on hyprland!

Does it supports other languages than english by chance ?

alexkarpathy 2 hours ago | parent [-]

yea added the hyprland support few days back only And about languages i think gemma already supports more than 140 languages and handles multilingual audio as well, so i think everything should work fine except the daily summaries since the system prompts are in english .. making it fully multilingual is also straightforward tho just a small tweak in system prompt like”respond in same language” or can pick it up from the system local will work. Let me know how it works with your language

Oxodao 2 hours ago | parent [-]

Neat, i'll try this out!