Plain-language explainer
Computer-use agents, explained
How can AI click around apps, and when is that safe?
A computer-use agent operates a screen the way a person would: it takes a screenshot, plans a step, clicks or types, looks at the result, and verifies before moving on. That loop, look, plan, act, observe, verify, is what lets a model use software that has no API. The catch is that interfaces are brittle and some actions cannot be undone. So verification and human approval on risky steps are not extras; they are what separates a useful agent from one that confidently clicks the wrong button.
Last reviewed
Reading is the slow way. Start with a free lesson you can operate right now.
Start free: The harness: the loop, made real →Free, no code, no signup.
Then go deeper: Computer-use agents: look, plan, click, verify Locked
What people get wrong
- It understands the screen perfectly. It reads a screenshot and can misidentify elements, so it must verify.
- It can safely do anything a user can. Irreversible actions need a human approval gate.
- It is just a macro recorder. It plans and adapts from what it sees, rather than replaying fixed steps.
Where you see it in real products
- Browser agents fill forms and gather information across sites.
- QA and automation tools drive apps that expose no API.
- Assistants take real actions, gated behind your confirmation for risky steps.
Common questions
- How does an agent use a computer?
- It takes a screenshot, reasons about what is on screen, and emits a click or a keystroke at specific coordinates. Then it takes another screenshot to see what happened. It is the same act-and-observe loop, with pixels as input and mouse and keyboard as the tools.
- Why is it slower and less reliable than an API?
- Every step costs a full screenshot to process and one action to take, and the model has to recover from anything unexpected: a popup, a slow load, a moved button. An API call skips the interface entirely. Computer use is the fallback for software that has no API.
- Is it safe to let an agent use my computer?
- Treat it like handing the keyboard to a capable stranger who reads everything on screen and can be talked into things by the text it encounters. Run it in a sandbox or a separate profile, keep credentials out of reach, and require confirmation before anything that spends money or sends messages.
Related explainers
More in Agents and tools
One idea at a time, in your inbox
New lessons and explainers, written the way these pages are. Now and then, not daily, and never a sales sequence.
We email you a confirmation link first. Unsubscribe in one click, any time. Privacy.
Part of See How AI Works, a free interactive course, where you learn how modern AI works by operating it, not watching videos.