The big idea

The universal API is a screen

Most of the world’s software has no API. Legacy portals, internal tools, airline checkouts, government forms — the only interface they offer is the one humans use: pixels, a mouse, a keyboard. A computer-use agent operates that interface directly, which makes it the most general — and most treacherous — kind of agent: every website is now its tool, and every website can surprise it.

Wrap a vision-language model in a see-think-act loop over a sandboxed machine: observe the screen, decide one action, perform it, observe again. The model supplies judgment; the system supplies eyes, hands, brakes, and a memory.

How to read this: Each step opens with a real design decision — make the call before seeing what ships. Watch the loop grow, hover any box, replay the flow. At the end, change the page mid-task and watch the recovery machinery earn its keep. Hit Begin.

Step 1 · The skeleton

One loop, one action at a time

The user says "book me on the 9am to Berlin." That’s dozens of clicks, fields and pages. What structure turns a goal into safe progress through an interface the agent has never seen?

Design decision: How should the agent work through a multi-page web task?

The call: A loop: observe the screen, decide ONE action, do it, observe again. — One action per iteration, each decided against the freshest screen. The loop absorbs surprise — which is the whole job when the world is a website.

The Agent Loop drives everything: capture the current screen, ask the model for the single next action, execute it, capture again, repeat. Progress is a chain of small, individually-verifiable moves — never a long blind script.

See → think → act: Computer use is the agent loop with pixels for observations and input events for actions. The tighter the loop, the faster mistakes are caught — one action per look is the discipline.

Step 2 · The eyes

What does the model actually see?

The agent needs to perceive the screen. You have three candidate senses: raw screenshots, the DOM/accessibility tree, or both. Screenshots show truth but cost vision tokens; trees are cheap and precise but only exist for well-built apps — and lie when the UI is canvas, Flutter, or a legacy desktop app.

Design decision: What should the agent perceive?

The call: Both: screenshots for ground truth, the a11y/DOM tree for cheap structure when available. — The hybrid is the production answer: trust pixels as the final arbiter, exploit structure wherever it exists, and degrade gracefully to pure vision when it doesn’t.

The VLM Pilot receives the task, recent history, and the current screenshot — augmented with the accessibility/DOM snapshot when one exists. Pixels are the ground truth; structure is an accelerant, not a dependency.

Pixels are the contract: Any agent that only reads the DOM is really an API client in disguise — and breaks the day the DOM lies. Vision is what makes computer use universal: if a human could see it, the agent can.

Step 3 · From intent to target

Grounding: where exactly is "Submit"?

The model says "click the Submit button." The mouse needs an (x, y). Bridging that gap — grounding — is where computer-use agents historically embarrassed themselves: right intent, wrong pixels, catastrophic click.

Design decision: How does "click Submit" become coordinates?

The call: Resolve against the a11y/DOM tree when possible; else label candidate elements on the screenshot (set-of-marks) and let the model pick a label. — Structured targets give exact rects for free. Where there’s no tree, drawing numbered boxes on detected elements turns "guess a coordinate" into "choose #7" — a far easier problem for the model.

The Grounding layer resolves intent to a target: exact element rects from the accessibility tree when available, set-of-marks labeling over the screenshot when not, raw model coordinates as the last resort. Every action is grounded against the current observation — never a remembered one.

Set-of-marks: Detect interactive elements, draw numbered boxes on the screenshot, ask the model "which number?" — converting a hard regression problem into an easy multiple-choice one. Grounding accuracy is the single best predictor of computer-use success rates.

Step 4 · The hands, in a padded room

Execute actions in a sandbox

The agent is about to click, type, scroll — with real side effects. On whose computer? A model driving your logged-in laptop can read your email, and a hijacked one (a webpage that says "ignore your instructions…") can act on what it reads. Where do the hands live?

Design decision: Where should the agent’s mouse and keyboard operate?

The call: In a disposable VM / isolated browser session with only the access the task needs. — The sandbox bounds the blast radius: scoped credentials go in, a task comes out, the VM is destroyed. Screenshots of it are streamed to the user for oversight.

The Action Executor performs grounded actions as synthetic input events inside a VM / browser sandbox — a disposable environment holding only task-scoped credentials. The user watches a live stream of the screen and can grab the wheel at any time.

Blast radius first: Everything a computer-use agent reads is untrusted input — a webpage can carry a prompt injection the same way an email carries a phish. You cannot filter what you can’t anticipate, so you bound what a hijacked agent can reach.

Step 5 · Close the loop

Verify, don’t assume

The click was sent. Did it work? The page might still be loading, a popup might have eaten the click, the button might have been disabled. Agents that assume success compound errors invisibly — by the time failure is obvious, they’re five wrong steps deep.

Design decision: The agent just clicked "Add to cart." What’s the next move?

The call: Capture a fresh observation and verify the expected effect before choosing the next action. — Screenshot again, confirm the cart badge incremented, THEN proceed. Every action is a hypothesis; the next observation is the test.

After every action the loop captures a fresh Observation — screenshot plus DOM state — and the model verifies the expected effect before deciding the next move. Loading states get bounded waits-for-evidence; failed effects trigger re-grounding and bounded retries, not blind repetition.

Act → verify → act: The observation after an action is not the next step’s input — it’s the previous step’s test result. This one discipline separates agents that finish 10-step tasks from agents that finish 3.

Step 6 · The brakes

Gate the irreversible

Somewhere in the task sits a click that can’t be un-clicked: Purchase. Send. Delete. Submit to the IRS. The model is right most of the time — and "most of the time" is a horrifying standard for spending money. Where do the brakes go?

Design decision: The agent reached the "Place order · $1,240" button. What happens?

The call: Pause, show the user the screen and the intended action, and wait for approval. — A Safety Gate classifies actions; the irreversible ones — payments, sends, deletes, credential entry — require explicit human sign-off with full context.

A Safety Gate sits between decision and execution: reversible actions flow freely; irreversible ones — purchases, sends, deletions, logins — pause with a screenshot and a plain-language "about to do X, approve?" The human approves the moments that matter, not every click.

Bounded autonomy: Gate by consequence, not by frequency: the agent moves fast through the forty reversible steps and stops dead at the two irreversible ones. That split is what makes delegation feel safe instead of reckless.

Step 7 · Going long

Memory and the paper trail

Real workflows run long: 60 steps, three sites, a confirmation code from page 2 needed on page 9. Screenshots are huge and the context window isn’t. And when the agent finishes — or fails — someone will ask exactly what it did. What persists?

Design decision: A 60-step task across 3 websites. How does the agent keep track?

The call: Structured task memory: running notes of progress + extracted facts; old screenshots dropped, recent ones kept. — "Done: flight selected · Learned: conf #XK42 · Next: passenger form" beats fifty stale screenshots. Notes compress; pixels expire.

Task Memory holds structured running notes — completed steps, extracted facts, remaining plan — while old screenshots age out of context. In parallel, the Trace records every frame and action: the audit trail for debugging, evals, and answering "what exactly did it do with my card?"

Notes compress, pixels expire: This is context engineering under a vision budget: screenshots are enormous and instantly stale, so the far past survives as written conclusions, not imagery. The trace keeps the pixels — outside the window, where they belong.

The payoff

You built a computer-use agent

From "the only API is a screen" to a system that works it safely: a see-think-act loop, hybrid vision + structure perception, set-of-marks grounding, sandboxed execution with scoped credentials, verification after every action, approval gates on the irreversible, task memory for the long haul, and a full replayable trace.

Now change the page mid-task and watch the machinery respond — re-observe, re-ground, bounded retry, and a clean escalation to the human when the world stops making sense.

• See-think-act — one verified action per fresh observation
• Hybrid perception — pixels as truth, a11y/DOM as accelerant
• Grounding — a11y rects → set-of-marks → raw coords, in that order
• Sandbox — disposable VM, scoped credentials, live stream
• Verify effects — every action is a hypothesis; the next frame tests it
• Safety gate — gate by consequence — approve the irreversible
• Task memory — notes compress; old screenshots expire
• Trace — every frame and action, replayable and auditable