agent-desktop: Open-source agent computer use

What changed

Vercel Labs has open-sourced agent-desktop, a command-line tool and C library that gives any AI agent native computer control without locking you into a single vendor’s platform. Version 1.0.2 shipped on October 30, 2026, under the Apache-2.0 license.

The tool runs on macOS 13 and later, Windows 10 (build 1809) and Server 2019, and Ubuntu 24.04 and later on both x64 and ARM64 architectures. It requires only a CPU and no GPU. The design is a Rust CLI with a C library that agents call as a tool, not the other way around. It embeds no large language model and has no graphical interface.

The technical approach differs sharply from screenshot-based computer use. Instead, agent-desktop reads each application through the operating system’s native accessibility tree. On macOS it uses the AX API, on Windows the UI Automation framework, and on Linux the AT-SPI standard. Each command prints one JSON output and exits, returning structured data with a ref identifier for every element the agent can act on. Refs belong to their snapshot; when the UI changes, an action returns STALE_REF or AMBIGUOUS_TARGET and the agent must take a new snapshot.

Why it matters

This architecture solves a real cost and reliability problem for teams building AI agents. Traditional computer use tools work from pixel coordinates extracted from screenshots. According to the documentation, each screenshot costs roughly 1,000 to 1,800 input tokens at model-side services, and the toolset definition itself adds about 4,500 input tokens per request. agent-desktop returns structure instead of images, cutting that overhead dramatically.

The builder reported reductions of 78% to 96% fewer tokens on dense applications when using the skeleton snapshot feature, which reads only three levels deep in the accessibility tree rather than dumping the entire tree. In one example, Apple Maps dropped from 3,811 tokens per look to 273. A simulated 52-step task fell from 198,000 tokens to 31,000 tokens. These are self-reported figures, not results from an independent benchmark.

agent-desktop is model-agnostic. You are not locked into Anthropic, OpenAI, or Microsoft. Any shell-capable coding agent can drive native applications without a vision model. Ref actions run headless by default, meaning they go through the accessibility API without moving your pointer or stealing keyboard focus. This lets agents work while you keep using the machine. The tool auto-waits up to 5,000 milliseconds for a target to become visible, enabled, and stable before acting.

For developers on Linux, Windows, and macOS, this removes friction. You install via npm and grant only the permissions your OS requires. macOS users run agent-desktop permissions --guide for a one-time permission grant. Windows users need no additional grant for apps at the terminal’s integrity level. Linux users need a graphical session with at-spi2-core installed. Homebrew and Scoop packages are being submitted but are not yet available in default catalogs.

What to test

Before deploying agent-desktop in production, verify these claims and limitations:

• Token savings in your own applications. The 78% to 96% token cuts came from the builder’s own testing. Your workflows may behave differently. Set up a controlled benchmark using your own target applications and compare snapshot sizes.

• Accessibility tree coverage. Applications without a properly implemented accessibility tree remain out of reach. Test whether your critical third-party tools expose the UI elements your agent needs to see and operate. Some legacy or custom applications may have gaps.

• Ref stability and error handling. The tool returns STALE_REF or AMBIGUOUS_TARGET when the UI changes and the ref no longer points to a valid element. Every error carries a disposition field; agents should retry only when it says safe. Test how often this occurs in your target workflows and whether your agent’s retry logic respects the disposition field rather than retrying blindly.

• Headless mode performance. The default headless behavior does not move your pointer or take focus. Verify that agents can accomplish your target tasks in this mode. The global --headed flag allows physical input when needed, but test whether this is necessary for your workflows.

• Permission model on each OS. macOS requires an accessibility grant that can be tedious to distribute at scale. Windows has fewer hurdles. Linux requires at-spi2-core. Confirm that your deployment environment allows these prerequisites.

• Dense application snapshots. The skeleton snapshot feature reads only three levels deep and reports a children_count for containers, with a drill ref to read one region into the same snapshot. Test whether your target dense applications work well with this approach or whether full-tree dumps are necessary.

Common questions

What does a desktop agent do?

An agent using agent-desktop can see desktop applications the way a person does by reading their UI structure, then click, type, scroll, and interact with them. Unlike humans, it does not use the mouse and keyboard directly. Instead, it works through the operating system’s accessibility layer, moving between observe, act, and verify cycles.

Can agents use a computer yet?

Yes. agent-desktop is now available as an open-source tool that any agent can use. Microsoft’s Copilot Studio also offers computer use for Windows agents. However, not all agents have this capability built in. Your agent needs to be explicitly designed to call computer use tools, and those tools need to be installed in your environment.

What is the difference between agent-desktop and pixel-based computer use?

agent-desktop reads UI structure through accessibility trees and returns JSON. Pixel-based tools take a screenshot, extract coordinates, and have the agent click at those coordinates. The accessibility approach cuts tokens dramatically, works without a vision model, and is more robust to UI changes that do not affect the underlying structure. Pixel-based tools are more universal but more expensive.

The conclusion

agent-desktop removes a significant barrier to building cost-effective AI agents that operate desktop software. By reading accessibility trees instead of screenshots, it cuts token costs by most measures and avoids vendor lock-in. The Apache-2.0 license means you can run it anywhere.

The main trade-offs are straightforward. Applications without good accessibility tree support are unreachable. Ref stability requires agents to handle stale references and error dispositions gracefully. Permission models differ by OS and can be awkward to scale on macOS. The token savings are real but come from the builder’s own testing, not an independent benchmark.

What to watch next: whether adoption by agent harnesses picks up, how well it handles complex workflows with rapid UI changes, and whether Homebrew and Scoop packages become available to ease installation. If you are building agents that need to operate native applications reliably, testing agent-desktop against your own workloads is worth a few hours of development time.

Related reading

• Sterling finance research agent outperforms Claude Fable 5.1
• AKV cache engineering cuts agent costs 27.7%
• Anthropic OSS Scanner free vulnerability detection for open source