I've been working on a utility that lets me "see through" app windows on macOS [1] (I was a dev on Apple's Xcode team and have a strong understanding of how to do this efficiently using private APIs).

I wondered how Claude Code would approach the problem. I fully expected it to do something most human engineers would do: brute-force with ScreenCaptureKit.

It almost instantly figured out that it didn't have to "see through" anything and (correctly) dismissed ScreenCaptureKit due to the performance overhead.

This obviously isn't a "frontier" type problem, but I was impressed that it came up with a novel solution.

[1]: https://imgur.com/a/gWTGGYa

▲

skc 3 hours ago | parent | next [-]

That's actually pretty cool. What made you think of doing this in the first place?

	▲	LatencyKills 3 hours ago \| parent [-]
		Thanks! I've been doing a lot of work on a laptop screen (I normally work on an ultrawide) and got tired of constantly switching between windows to find the information I need. I've also added the ability to create a picture-in-picture section of any application window, so you can move a window to the background while still seeing its important content. I'll probably do a Show HN at some point.

▲

saagarjha 2 hours ago | parent | prev | next [-]

Why is ScreenCaptureKit a bad choice for performance?

	▲	LatencyKills 2 hours ago \| parent [-]
		Because you can't control what the content server is doing. SCK doesn't care if you only need a small section of a window: it performs multiple full window memory copies that aren't a problem for normal screen recorders... but for a utility like mine, the user needs to see the updated content in milliseconds. Also, as I mentioned above, when using SCK, the user cannot minimize or maximize any "watched" window, which is, in most cases, a deal-breaker. My solution runs at under 2% cpu utilization because I don't have to first receive the full window content. SCK was not designed for this use case at all.

▲

stavros 3 hours ago | parent | prev [-]

What was the solution?

▲

LatencyKills 3 hours ago | parent [-]

Well, I'm not going to share either solution as this is actually a pretty useful utility that I plan on releasing, but the short answer is: 1) don't use ScreenCaptureKit, and 2) take advantage of what CGWindowListCreateImage() offers through the content server. This is a simple IPC mechanism that does not trigger all the SKC limitations (i.e., no multi-space or multi-desktop support). In fact, when using SKC, the user cannot even minimize the "watched" window.

Claude realized those issues right from the start.

One of the trickiest parts is tracking the window content while the window is moving - the content server doesn't, natively, provide that information.

▲

stavros 3 hours ago | parent [-]

Huh, Claude one-shotted it out of a single message from me. Man, LLMs have gotten good.

▲

LatencyKills 2 hours ago | parent [-]

No it didn't. Like I said... it may have gotten something that worked but there is no way Claude got it to work while supporting multi-spaces, multi-desktops, and using under 2% cpu utilization. My solution can display app window content even when those windows are minimized, which is not something the content server supports.

My point was that Claude realized all the SKC problems and came up with a solution that 99% of macOS devs wouldn't even know existed.

	▲	TeMPOraL 2 hours ago \| parent [-]
		> it may have gotten something that worked but there is no way Claude got it to work while supporting multi-spaces, multi-desktops, and using under 2% cpu utilization. Maybe, but that's the magic of LLMs - they can now one-shot or few-shot (N<10) you something good enough for a specific user. Like, not supporting multi-desktops is fine if one doesn't use them (and if that changes, few more prompts about this particular issue - now the user actually knows specifically what they need - should close the gap).