Computer Use: When the Agent Needs the Screen

Some work has no API: a legacy desktop app, a settings dialog, a website with no export button. Computer use closes that gap: an idapt agent can see a screen, move the cursor, click, and type, on a computer you explicitly allow.

How it works

Give an agent a task that needs a GUI and a computer with computer use enabled. The agent takes screenshots, reasons about what it sees, and acts: clicks, keystrokes, scrolling, window management. Screenshots land in the chat so you watch the same screen the agent does.

Built defensive, on purpose

Screen control is the most consequential tool an agent can hold, so the design leads with control:

• Per-computer consent: computer use is off until you enable "Allow computer use" on that specific machine; headless setups confirm via CLI.
• One controller at a time: an exclusive lease means one chat controls a computer at a time, released automatically when the run ends.
• A visible border on the controlled screen while control is active, with an on-screen emergency Stop that cuts the run instantly.
• Mis-click detection: the daemon reads back where the cursor actually landed and flags divergence instead of clicking blind.
• The autonomy dial applies: ask mode confirms actions; budgets and step limits bound every run; the full trace is auditable after.

Good to know

• Works on your own machines and on cloud computers with a desktop.
• Available from the verified free tier; anonymous sessions are excluded.
• A Computer Use tab on each machine shows its control history.

Get started

Enable "Allow computer use" on a computer you control, then hand an agent a small GUI task and watch the border. The feature page covers capabilities and limits, and the agents overview explains the permission model it inherits.