Agents that
use the screen.

Most agent tools are clean APIs — call a function, get a result. But a huge amount of the world's software has no API, only a screen. Browser and computer-use agents close that gap: instead of a programmatic interface, they operate the actual UI the way a person does — read the screen, move to an element, click or type, watch what changes. That unlocks the long tail of tools no one exposed programmatically, from legacy web apps to desktop software.

The capability is broad but the substrate is unforgiving. A UI wasn't designed to be driven by a model; it was designed for a human who effortlessly parses layout, ignores clutter, and knows a button when they see one. The agent has to reconstruct all of that from a DOM tree or a raw screenshot, decide what to do, and act with precision — then cope with the page changing underneath it. Everything hard about these agents traces back to operating an interface built for eyes and hands, not for code.

A UI agent runs a specialized version of the agent loop: observe → act → verify. Observe: capture the current screen state — the DOM elements (for web) or a screenshot (for general computer use), often annotated with the interactable elements. Act: choose an action — usually click or type on a specific target element — and execute it. Verify: observe again to confirm the action did what was intended before continuing.

The verify step is what separates a toy from a system. UI actions fail silently all the time — a click lands on the wrong element, a page hasn't finished loading, a modal intercepts the input — and without verification the agent charges ahead on a false belief about the screen state, and the error cascades. So a good UI agent doesn't fire-and-forget; it checks the effect of each action before taking the next, and recovers when reality doesn't match its plan. But verification only helps if the agent can act accurately in the first place, and that's the hard middle step: grounding.

Grounding is the act of mapping an intention — "click the submit button" — to the exact on-screen target: the specific element, or the specific pixel coordinate. It sounds trivial and it's the dominant source of failure. Pages are cluttered with dozens of similar elements; labels are ambiguous or missing; the same visual button might be several nested DOM nodes; a screenshot gives no clean handles at all, just pixels. Pick the wrong target and the action does the wrong thing — or nothing.

This is the same challenge that vision-language-action models face in robotics: turning "pick up the cup" into the right motor command is grounding in physical space; turning "click checkout" into the right element is grounding in interface space. Much of the research on UI agents is really research on grounding — better element representations, set-of-marks prompting that labels clickable regions, models trained specifically to locate UI targets. Because grounding accuracy isn't just one factor in reliability; over a multi-step task, it's the factor that compounds.

A single action succeeds only if the agent grounds to the right element. A multi-step task succeeds only if every step grounds correctly — one wrong click can derail the rest. So with per-step grounding accuracy p over k steps, the task success rate is:

Even excellent per-step accuracy decays fast: at p=0.9, a 3-step task succeeds 73% of the time but a 10-step task only ~35% (0.9¹⁰).

This exponential decay is the reason long-horizon UI automation is so brittle — and it points straight at the levers:

Push per-step grounding toward 1, or break long tasks into short verified segments — both attack the exponent. The runnable version below grounds actions to page elements and shows task success collapsing as steps grow.

Because the exponent is brutal, robust UI agents attack both p and k:

The single biggest win is usually representation: an agent grounding over the structured DOM or accessibility tree, with interactable elements explicitly labelled, is far more accurate than one squinting at a screenshot. Verification converts the inevitable misses into recoverable ones, and decomposition keeps any single task's step-count low so the exponent stays kind. And as always, wrap it in a bounded harness — a UI agent that can click anything needs a hard stop and, for consequential actions (purchases, deletions), a human confirm.

The reliability trap is underestimating the exponent: a 90%-accurate agent feels great in a 3-step demo and falls apart on a 15-step real task, and teams are surprised because they never multiplied it out. Design for the compounding: keep tasks short, verify relentlessly, and measure end-to-end success on realistic step counts, not per-action accuracy. The security trap is bigger. A UI agent reading arbitrary web pages is a prompt-injection magnet — a malicious page can instruct the agent to click something harmful — so treat page content as untrusted, sandbox the environment (a throwaway browser profile, no saved credentials), and gate consequential actions.

Two more. Non-determinism and flakiness: pages load at different speeds, popups appear, layouts shift, so an agent that assumes a fixed screen breaks — always re-observe, never act on a stale view. And consequential actions without confirmation: a wrong click can buy the wrong thing, send the wrong message, or delete real data, so route anything irreversible past a human or a strong guardrail. Browser and computer-use agents are a genuinely powerful frontier — the path to software that operates other software — but they demand humility about grounding, discipline about verification, and real care about safety. Respect the exponent and the injection surface, and they become dependable rather than merely impressive.