The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
The universal API is a screen
Most of the world’s software has no API. Legacy portals, internal tools, airline checkouts, government forms — the only interface they offer is the one humans use: pixels, a mouse, a keyboard. A computer-use agent operates that interface directly, which makes it the most general — and most treacherous — kind of agent: every website is now its tool, and every website can surprise it.
Wrap a vision-language model in a see-think-act loop over a sandboxed machine: observe the screen, decide one action, perform it, observe again. The model supplies judgment; the system supplies eyes, hands, brakes, and a memory.
What the new pieces do
- Userclient
- Hands over a real computer task — "book the flight", "file this expense report" — and expects to approve anything scary.
Step 1 · The skeleton
One loop, one action at a time
The user says "book me on the 9am to Berlin." That’s dozens of clicks, fields and pages. What structure turns a goal into safe progress through an interface the agent has never seen?
How should the agent work through a multi-page web task?
The plan is stale by click three — pages load differently, popups appear, buttons move. Upfront scripts are how brittle RPA breaks.
One action per iteration, each decided against the freshest screen. The loop absorbs surprise — which is the whole job when the world is a website.
That’s not an agent, that’s a very slow mouse. Approval belongs at the few irreversible moments (step 6), not every step.
The Agent Loop drives everything: capture the current screen, ask the model for the single next action, execute it, capture again, repeat. Progress is a chain of small, individually-verifiable moves — never a long blind script.
Why this piece earns its place
The loop is the only component that can decide the task is over, and that is harder than it looks. Actions succeeding is not the same as progress: an agent can click, confirm the effect, and cycle between two pages forever, every iteration individually correct. Retry limits never catch this, because nothing failed. What catches it is a no-progress check on the loop itself — compare the observation and the notes against the last few iterations, and stop when neither has moved. The ceilings the retry story already assumes, a step count and a clock, only do their job if they arrive in the request rather than sitting in config, because a one-form expense claim and a three-site booking are not the same budget and one global number serves both badly. The cost model falls out of the same shape: every iteration is a model call over a fresh image, so both the bill and the elapsed time are linear in iterations and neither can be quoted up front. Make the loop resumable while you are at it — it owns live state in a running machine, and an orchestrator restart mid-task otherwise leaves a logged-in session with no owner.
What the new pieces do
- Agent Loopbackend
- The controller: capture an observation, ask the model for the next action, execute it, verify the effect, repeat until done.
Step 2 · The eyes
What does the model actually see?
The agent needs to perceive the screen. You have three candidate senses: raw screenshots, the DOM/accessibility tree, or both. Screenshots show truth but cost vision tokens; trees are cheap and precise but only exist for well-built apps — and lie when the UI is canvas, Flutter, or a legacy desktop app.
What should the agent perceive?
Great until the app is a canvas, a game engine, a PDF viewer, or just badly built — then the tree is empty or wrong. Trees describe what developers declared, not what’s on screen.
Pixels are truth but expensive and unaddressable: "the blue button" isn’t a coordinate. You’d burn vision tokens re-reading text the tree gives you for free.
The hybrid is the production answer: trust pixels as the final arbiter, exploit structure wherever it exists, and degrade gracefully to pure vision when it doesn’t.
The VLM Pilot receives the task, recent history, and the current screenshot — augmented with the accessibility/DOM snapshot when one exists. Pixels are the ground truth; structure is an accelerant, not a dependency.
Why this piece earns its place
Perception looks like one capture and is really two, of a moving target: grab the pixels, then the tree, and a page that re-renders in between hands the model a description of a screen that no longer exists. Take them as close together as the platform allows, timestamp both, and treat a disagreement as a reason to re-observe rather than something to reconcile. A sharper hazard belongs to the degraded path this step just defined. Where the tree exists, an element below the fold is still in the snapshot carrying a rect outside the viewport, so present-but-unseen and genuinely absent stay distinguishable. Drop to pure vision and they collapse into each other — a camera sees only the viewport — and the model will report the button missing with complete confidence. That path needs an explicit scroll-and-look behavior before absence is allowed to count as fact. Capture resolution decides more than it appears to: downscale the frame to save vision tokens and small labels turn to mush, so the right setting is per-app, not global. Fix it and record it, because every coordinate the next step produces is only meaningful against that exact frame.
- 1 screenshotper loop step
- a11y treewhen the app offers one
- pixelsfinal arbiter
What the new pieces do
- VLM Pilotservice
- A vision-language model that looks at the current screen (plus the task and history) and decides the next action.
Step 3 · From intent to target
Grounding: where exactly is "Submit"?
The model says "click the Submit button." The mouse needs an (x, y). Bridging that gap — grounding — is where computer-use agents historically embarrassed themselves: right intent, wrong pixels, catastrophic click.
How does "click Submit" become coordinates?
Modern VLMs can — but raw coordinate regression is the flakiest path, off by enough to hit the wrong control on dense UIs. Use it as the fallback, not the plan.
Structured targets give exact rects for free. Where there’s no tree, drawing numbered boxes on detected elements turns "guess a coordinate" into "choose #7" — a far easier problem for the model.
A joke — but stale-coordinate replay after a layout change is functionally this, which is why re-grounding on every fresh screenshot matters.
The Grounding layer resolves intent to a target: exact element rects from the accessibility tree when available, set-of-marks labeling over the screenshot when not, raw model coordinates as the last resort. Every action is grounded against the current observation — never a remembered one.
Why this piece earns its place
Grounding is where two coordinate systems quietly disagree. The browser reports rects in CSS pixels, the frame arrives in device pixels, and the page carries a scroll offset — get the conversion wrong and every target is off by the scale factor, an error that grows with distance from the origin, so a control near the top-left still lands and one near the far edge misses the window entirely while every log line reads as correct. Normalize once, at the boundary, and assert the grounded target actually falls inside the frame it was grounded against. Then make the layer emit what it chose, not just where: the target rect, which tier produced it, and the selector or label it matched. Without that record a failed step is unattributable — you cannot tell whether the model asked for the wrong thing or the resolver found the wrong thing, and those two have opposite fixes. It pays forward as well. A target that carries an identity is something the next observation can be checked against and the trace can replay faithfully, so the work done here is what lets verification mean more than the screen looks different now.
What the new pieces do
- Groundingservice
- Turns the model’s intent ("click Submit") into an exact target — via accessibility tree, DOM, or set-of-marks over the screenshot.
Step 4 · The hands, in a padded room
Execute actions in a sandbox
The agent is about to click, type, scroll — with real side effects. On whose computer? A model driving your logged-in laptop can read your email, and a hijacked one (a webpage that says "ignore your instructions…") can act on what it reads. Where do the hands live?
Where should the agent’s mouse and keyboard operate?
Maximum capability, maximum blast radius: every open tab, credential and file is exposed to both model mistakes and prompt injection from any page it reads. Reserve this for narrow, supervised flows.
The sandbox bounds the blast radius: scoped credentials go in, a task comes out, the VM is destroyed. Screenshots of it are streamed to the user for oversight.
Gates catch actions you predicted. Injection and misclicks exploit the ones you didn’t. Isolation is the defense that works when judgment fails.
The Action Executor performs grounded actions as synthetic input events inside a VM / browser sandbox — a disposable environment holding only task-scoped credentials. The user watches a live stream of the screen and can grab the wheel at any time.
Why this piece earns its place
Isolation is not the only control this step needs. Start with egress: a machine with no host access still has a network, and the cheapest thing a hijacked agent can do is put a secret in a URL and fetch it. Isolation bounds what the agent can reach on your side; a destination allowlist bounds what it can reach on the way out, and those two get conflated constantly. Then there is cost. A cold machine plus a browser plus a login is a long time before the first useful frame, so the first optimization anyone reaches for is a warm pool — and a pooled environment is not disposable. Cookies, cached sessions and a stray open tab survive into the next person's task, which is the exact property isolation was bought to provide. If you pool, pool the base image and reset the profile, treating a failed reset as a hard error rather than something to retry. The last one is that the site is watching back. A datacenter address, a profile with no history and input at machine speed is the fingerprint commercial bot defenses look for, and the block arrives as a challenge rather than an error — so the executor has to recognize one as a stop, not as a step worth retrying.
What the new pieces do
- Action Executorservice
- Performs the grounded action as synthetic input events — clicks, typing, scrolling, key chords — inside the sandbox.
- VM / Browserservice
- A disposable virtual machine or browser session the agent controls. Isolation is the blast-radius control.
Back of the envelope
- disposable VM
- fresh per task, destroyed after
- scoped credentials
- only the logins this task needs
- live screen stream
- user can watch and take over
- no host access
- the agent never touches your real machine
Step 5 · Close the loop
Verify, don’t assume
The click was sent. Did it work? The page might still be loading, a popup might have eaten the click, the button might have been disabled. Agents that assume success compound errors invisibly — by the time failure is obvious, they’re five wrong steps deep.
The agent just clicked "Add to cart." What’s the next move?
"Sent" is not "worked." Silent failures compound: the cart is empty at checkout and the agent starts improvising. Never chain actions on assumption.
Fixed waits are both too slow (most pages settle faster) and too fast (some don’t). Wait for evidence, not time.
Screenshot again, confirm the cart badge incremented, THEN proceed. Every action is a hypothesis; the next observation is the test.
After every action the loop captures a fresh Observation — screenshot plus DOM state — and the model verifies the expected effect before deciding the next move. Loading states get bounded waits-for-evidence; failed effects trigger re-grounding and bounded retries, not blind repetition.
Why this piece earns its place
None of this works unless the expectation is written down before the action. Act first, then show the model a new screen and ask whether it worked, and it will explain why it worked — post-hoc rationalization is the failure mode of self-verification, and it hides precisely because every step reports success. So make the decision emit its predicted effect as part of the action: this click should produce a cart count of one. Verification becomes a comparison, and a mismatch is a concrete event you can log and count rather than a judgment call. Counting them is worth more than it sounds. Mismatches bucketed by site and element are the earliest warning that a site shipped a redesign: the rate moves across every user at once, well before any single task fails outright, and it is the signal that separates the model got worse from the world changed. The retry path needs its own care, because most interesting actions are not idempotent. An ambiguous outcome — the click may have landed, the page may still be settling — is the worst case, and deciding whether the effect already happened has to come before deciding to act again, or the form gets submitted twice.
- every actionverified against pixels
- bounded waitsevidence, not timers
- re-groundon every fresh frame
What the new pieces do
- Observationstore
- The fresh screenshot (and accessibility/DOM snapshot) captured after every action — the only truth about what happened.
Step 6 · The brakes
Gate the irreversible
Somewhere in the task sits a click that can’t be un-clicked: Purchase. Send. Delete. Submit to the IRS. The model is right most of the time — and "most of the time" is a horrifying standard for spending money. Where do the brakes go?
The agent reached the "Place order · $1,240" button. What happens?
One misread field or hijacked instruction and the agent bought the wrong flight with real money. Autonomy without brakes is liability.
A Safety Gate classifies actions; the irreversible ones — payments, sends, deletes, credential entry — require explicit human sign-off with full context.
Then the agent can’t finish the tasks people most want delegated. The goal is bounded autonomy: free on reversible steps, gated on irreversible ones.
A Safety Gate sits between decision and execution: reversible actions flow freely; irreversible ones — purchases, sends, deletions, logins — pause with a screenshot and a plain-language "about to do X, approve?" The human approves the moments that matter, not every click.
Why this piece earns its place
An approval is asynchronous and the page is not. While someone reads the request on their phone, the session times out, the held seat is released, the price moves, a banner shifts the layout. What they approved was a screenshot; what executes runs against whatever is there when they say yes. So an approval is permission for an intent, not a license to replay a stale click — once the human agrees, re-observe, re-ground, and confirm the target still says what it said, or go back and ask again. The pause also has to end somehow. Nobody answering is a common case rather than a rare one, and a waiting task is a live sandbox holding a seat, a cart, a lock — so the request needs a deadline and a defined expiry: release what is held, tear the environment down, and report where the task stopped, rather than leaving a machine idling against a timeout nobody chose. The other risk sits on the human side. Gate too much and people approve without reading, which is worse than having no gate at all: you have manufactured consent and then logged it, so the audit trail now claims a person reviewed something nobody looked at.
What the new pieces do
- Safety Gateservice
- Pauses before purchases, sends, deletes, logins — anything hard to undo — and asks the human to approve.
Step 7 · Going long
Memory and the paper trail
Real workflows run long: 60 steps, three sites, a confirmation code from page 2 needed on page 9. Screenshots are huge and the context window isn’t. And when the agent finishes — or fails — someone will ask exactly what it did. What persists?
A 60-step task across 3 websites. How does the agent keep track?
Dozens of images overflow any window and bury the signal. Old pixels are the least useful tokens you can carry.
"Done: flight selected · Learned: conf #XK42 · Next: passenger form" beats fifty stale screenshots. Notes compress; pixels expire.
Re-doing a half-completed purchase flow is how you buy two tickets. Long tasks need memory, not amnesia and a rerun.
Task Memory holds structured running notes — completed steps, extracted facts, remaining plan — while old screenshots age out of context. In parallel, the Trace records every frame and action: the audit trail for debugging, evals, and answering "what exactly did it do with my card?"
Why this piece earns its place
The trace is the most useful artifact in this design and the most dangerous one. Every frame it keeps is a picture of whatever happened to be on screen — a card number mid-entry, somebody's inbox, a password field between keystrokes — so it holds the highest concentration of secrets anywhere in the system, and it holds them as images, where no string scanner will ever find them. Redact at capture rather than at review: mask credential fields and known-sensitive regions before the frame is written, and give the store a retention clock of its own, separate from the task's lifetime. It is also the largest thing you keep by a wide margin, since a full frame per step across every task dwarfs the notes sitting beside it — so store keyframes plus the actions between them instead of everything at full fidelity. What buys all of that back is evals. A recorded failure replays as a fixed test, which is the only honest way to answer whether a new model got better at your tasks or merely better at the ones you remembered to try.
What the new pieces do
- Task Memorystore
- Running notes for long workflows: what’s done, what was learned (order number, form values), what remains.
- Trace / Replaybus
- Full recording of observations and actions — for debugging, evals, audits, and replaying exactly what the agent did.
The payoff
You built a computer-use agent
From "the only API is a screen" to a system that works it safely: a see-think-act loop, hybrid vision + structure perception, set-of-marks grounding, sandboxed execution with scoped credentials, verification after every action, approval gates on the irreversible, task memory for the long haul, and a full replayable trace.
Now change the page mid-task and watch the machinery respond — re-observe, re-ground, bounded retry, and a clean escalation to the human when the world stops making sense.
Everything you assembled, in order
- See-think-act — one verified action per fresh observation
- Hybrid perception — pixels as truth, a11y/DOM as accelerant
- Grounding — a11y rects → set-of-marks → raw coords, in that order
- Sandbox — disposable VM, scoped credentials, live stream
- Verify effects — every action is a hypothesis; the next frame tests it
- Safety gate — gate by consequence — approve the irreversible
- Task memory — notes compress; old screenshots expire
- Trace — every frame and action, replayable and auditable