Computer-use agents: look, plan, click, verify
An agent that clicks buttons and fills forms sounds magical, until it clicks the wrong thing on a screen it only half understands, and the action can't be undone.
The idea inside
A computer-use agent loops: screenshot, plan, act, observe, verify. UIs are brittle and some actions are irreversible, so verification and human approval gates are not optional.
After this lesson
You can explain how computer-use agents operate interfaces (screenshot, plan, act, observe, verify) and why brittle UIs and irreversible actions demand verification and human approval.
Where it leads
Inside this lesson
That's the real lesson stage, paused. Claim your pass to operate it.
See how AI actually works, end to end.
This lesson is one stop on the full arc. Unlock all of it, and keep it for life.
What you get
- The 34-lesson main path, a finishable route from a word to agents
- Goal tracks for using AI at work and building AI features
- Boss labs that make you apply a whole act, not just recognize it
- Spaced recall that brings each idea back before you forget
- Course memory: every term defined, with links to where it first appears
- A shareable capability card when you finish the main path
- Lifetime access on every device, every future lesson included
Not videos to watch. You predict, operate the machine, then prove it. That is why it stays.
99 interactive lessons and challenges. No videos, no code.
Free launch pass: lifetime access, no card needed
New here? The first lessons are free to try. Start with lesson 0.1
What this lesson shows
A computer-use agent loops: screenshot, plan, act, observe, verify. UIs are brittle and some actions are irreversible, so verification and human approval gates are not optional.
The question it opens with
An agent that clicks buttons and fills forms sounds magical, until it clicks the wrong thing on a screen it only half understands, and the action can't be undone.
The walkthrough, in the lesson's own words
- Drive the agent one step at a time. Watch it look, plan, act, then verify.
- The screen changed since the agent's last screenshot. Predict what its old plan does now.
- A computer-use agent plans from a screenshot. Brittle UIs and risky actions need a human gate.
- Discount applied. New total $37.80. The payment is confirmed.
- The agent wants to confirm a payment. That action can't be undone, so it pauses and asks you.
- You approved it, the payment confirmed, and the agent verified the new screen. Reversible steps ran on their own, the irreversible one waited for you.
- You held the action, so nothing was charged. A human stayed in the loop. Run the step again when you're ready to decide.
- The agent looked, planned, acted, then verified the result. Run the next step.
- Press Run next step to start the agent's loop.
- A computer-use agent takes a screenshot (look), decides a click or keystroke from what it sees (plan), performs it (act), then takes a new screenshot to confirm the result (verify). Then it loops. The verify step is what keeps it from charging blindly ahead.
- This screen is a simulation. The loop and the approval gate are the real pattern.
- The agent planned its click from the OLD screenshot. If it acts without re-checking the screen, what happens?
- A modal popped up and the button moved. The agent's saved coordinates now land on the wrong element, so it clicks the wrong thing. Screens are brittle: a layout shift breaks a stale plan.
- The fix is to observe a fresh screenshot every step and verify the result, instead of trusting a plan made from an old view.
- Run it in a sandbox (a throwaway browser or VM) so a misclick can't touch your real files, and give it scoped tool permissions, the narrow set of actions it's allowed, so it can read a page but not, say, send money without your sign-off. Same idea as scoping any tool's access (see lesson 6.5).
- Today's browser and OS agents (Claude's computer use, ChatGPT's agent mode, and friends) avoid raw pixels when they can. They prefer an API, the page's structure (the DOM), or the accessibility tree, the labeled map of buttons and fields that screen readers use, because 'the Confirm button' survives a layout shift that saved pixel coordinates don't. Screenshots and clicks are the fallback for apps with no better handle, and that fallback is exactly the brittle case you just watched. These agents are past demo-only, but they still fail a meaningful share of long multi-step tasks, which is why the verify loop and the approval gate stay.
- A computer-use agent reliably sees and clicks like a person.
- It plans from a screenshot, UIs are brittle, and irreversible actions need verification and a human approval gate.
- Before you let an agent run your browser, email, or bank, what is the real question to ask?
- Not 'can it click', but 'what happens when it clicks wrong, and which actions need my sign-off'. Reversible actions can run free; irreversible ones need verification and a human gate.
- You can read a computer-use agent's loop, look, plan, act, verify, and decide which actions it may run on its own and which need to pause for your approval.
Key takeaway
You ran the look-plan-act-verify loop, caught a misclick by verifying, and gated an irreversible action behind your approval.
What you can do after this lesson
You can explain how computer-use agents operate interfaces (screenshot, plan, act, observe, verify) and why brittle UIs and irreversible actions demand verification and human approval.
Prefer reading first? This lesson has a companion explainer: Computer-use agents
This is the written summary. The lesson itself is interactive: you predict, drag and operate the mechanism above, and the reveal answers you.