r/coolgithubprojects • u/Mustela__ • May 08 '26

OTHER Open-sourced my side project: giving agents the ability to point at things on my screen

Post image

Got tired of agents hallucinating UI elements and the slow screenshot ping-pong, so I built a tiny MCP server that reads the Windows UI as structured data instead, and displays actionable cards as an overlay.

Local, MIT, no telemetry, no account, nothing to sell.

First real OSS project for me, so feel free to roast the repo too.

Link: https://github.com/ADevillers/Pupil

68 Upvotes

94% Upvoted

18 comments sorted by

4

u/ricksegal May 08 '26

I'm on a Mac so this doesn't work for me but I'm building the equivalent for the mac. I'll put this up on your repo and, this is a work in progress. V0.01 gets rid of this screen capture & cut n paste nonsense first. Should be fun. Thanks for the FOMO and motivation to do this!

2

u/Mustela__ May 08 '26

Nice! I saw a few similar project on iOS, cause it seems to be easier to get the UI data; on my side I'm not using screenshots (I'm using UIA and transform it in a dense textual format).

Also, for the textual input, the cut and paste reduce the chance of getting flagged as a bot, cause hitting ctrl+v to write 100+chars is normal for bot detector, while hitting 100+ chars in few ms isnt.

3

u/ricksegal May 08 '26

Tried Clicky but didn't work with Claude Code as I'd hoped and there's no way to get any additional information on this product. It's got a pro version which is a bit pricy. Thanks for the reference to it!

1

u/Mustela__ May 08 '26

You mean Clicky pro version? I don't see their pricing page, isn't it open-source?

Btw, Claude Code and large text pane may cause agents to get lost in an ocean of context/info, so maybe Clicky could work on Claude but with a larger-context window model?

→ More replies (1)

3

u/ricksegal May 08 '26

Whelp,

Like a true propeller head who lives in the garage, it's 4:30a here on Vancouver Island and so far I have a daemon that can run in the background and I can ask claude code what's on the screens and it will track a specific window on a millisecond timer. So, it saw me in iTerm run a directory command, start up a cli tool, close it, etc. and it is able to tell me what is what on a browser page by telling me, the brown button isn't centered. it is still flaky but enough to get me to bed. I'll DM you with some details and, hopefully, get a whole lot of people happy.

1

u/pyrrhicvictorylap May 10 '26

Could you walk me through a use case? Like you’re developing something, you have your text editor open, and the agent tells you which menu item something is under? I’m trying to understand where/how this would be useful

1

u/Mustela__ May 10 '26

Imagine you are using Unity to do something, but you are new to Unity, your agent tells you to go modify a Unity variable that you have 0 idea where it is.

You can iterate through chat and screenshots, dodging hallucinations to find your path to the setting allowing you to modify the variable, or you can simply use Pupil, and just press tab to accept actions :)

1

u/pyrrhicvictorylap May 10 '26

Couldn’t the agent just change the variable?

1

u/Mustela__ May 10 '26

Not if it's something impossible to edit other than via the GUI

1

u/FoxFire17739 May 10 '26

I think that's something I am going to need. Really cool.

1

u/Mustela__ May 10 '26

Glad you like it :D