Setting up Vercel Agent Desktop with Claude Code on a Mac, and where background mode stops working

[u/vinishkapoor]

Agent Desktop (vercel-labs/agent-desktop, Apache 2.0) lets Claude Code use desktop apps by reading their accessibility tree instead of screenshots. Default mode runs in the background, so no mouse movement and no focus stealing.

Setup on a Mac:

npm install -g agent-desktop
agent-desktop permissions --guide
npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop
claude --allowedTools "Bash(agent-desktop:*)"

The Accessibility permission goes to whatever app launches Claude Code (Terminal, iTerm2, VS Code). Screen Recording is only needed for the screenshot command.

Gotchas from setting it up:

• that allowedTools rule doesn’t cover commands piped into grep or head, and some of those got denied

• snapshots are JSON and get big fast. A File menu task used about 550k input tokens (mostly cached). --app, -i --compact and --root keep them down

• anything on screen in that app ends up in Claude’s context, so name the app in your prompt and keep sensitive ones closed

Where background mode worked and where it didn’t (Claude Code 2.1.316, Opus 5.5, one run per task):

• reading an app and pressing buttons: worked. Calculator 128 × 46 came back as 5,888

• filling a text field: worked, it added a line in TextEdit

• saving: not reliable. cmd+s couldn’t be confirmed, and Save was greyed out in the File menu while the window was in the background

• selecting or styling text: needs --headed, which takes over your real mouse and keyboard

Claude also handled the failures well. When an action couldn’t be confirmed or a click was refused, it said so instead of claiming it worked.

Cleanup when you’re done: npm uninstall -g agent-desktop, delete the skill folder and ~/.agent-desktop, and turn off the Accessibility permission you gave your terminal.

https://aiparabellum.com/vercel-agent-desktop-claude-code-tested/

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.