Melaya Blog

Computer Use: How AI Agents Operate a Screen

Computer use lets an AI model see a screen and operate the mouse and keyboard. Learn how it works, where it fits, the risks, and how to set it up.

Computer use is a capability that lets an AI model look at a screen and operate it the way a person does: it reads screenshots, moves the pointer, clicks, types, and scrolls. It matters because it reaches software that has no API. Anthropic, OpenAI, and Google each ship a version of it, and you can run the same pattern across providers if you put a controlled execution layer between the model and the machine.

This guide covers how the loop works, how it compares with API and DOM automation, what to build first, and how to contain the security risks.

TL;DR

• Computer use is a screenshot-in, action-out loop: the model sees the screen, picks a click or keystroke, and repeats until the task is done.
• It is the fallback for software with no API or stable selectors. Where an API exists, use the API.
• Anthropic, OpenAI, and Google each document a computer use tool, so the model is a swappable part, not a commitment.
• The main risks are prompt injection and over-broad permissions. Contain them with allowlists, isolated environments, and human approval on consequential actions.
• Melaya is bring-your-own-model: Claude, GPT, Gemini, DeepSeek, Mistral, Qwen, Ollama, and LM Studio all work, and we do not sell a model.

What computer use means for AI agents

Ordinary computer use is you, a keyboard, and a screen. In the AI sense, the term describes a model that does that work in your place. OpenAI describes it as a model operating a browser or desktop interface to fill out forms, test user flows, or complete tasks in an application. Anthropic's documentation describes a tool that provides screenshot capabilities and mouse and keyboard control for autonomous interaction with computer environments.

The key idea is that the interface is the integration. A model that can read pixels and issue input events can use any application a human can, including legacy desktop tools, vendor portals, and internal admin screens that never got an API.

You will also see the terms "computer use agent" and "computer use model." The model is the vision-capable component that decides what to click. The agent is the whole system around it: the model, the screen it controls, the permissions it holds, the memory it keeps, and the log of what it did.

How an agent perceives a screen and acts

Every computer use implementation runs the same loop:

1. Capture. The harness takes a screenshot of the target screen or window.
2. Reason. The model receives the screenshot plus the task and the history of past steps. It decides on the next action.
3. Act. The model returns a structured action: click at coordinates, type a string, press a key combination, scroll, or wait.
4. Execute. Your harness, not the model, carries out the action on the real machine.
5. Observe. A new screenshot is taken and returned to the model.
6. Stop or repeat. The loop ends when the model reports completion or a step limit, timeout, or rejected approval ends it.

Step 4 is where control lives. The model only proposes. Whatever sits in your harness decides what is permitted, what is logged, and what pauses for a person.

Vision-based versus text-based control

There are two ways to give a model its view of the interface:

• Vision-based: the model sees pixels and clicks coordinates. It works on anything visible, including remote desktops, canvas apps, and native software.
• Text or DOM-based: the model reads the page structure (the DOM or accessibility tree) and targets elements by role or label. It is faster and cheaper per step, but only works where that structure is exposed.

Many production setups combine them: structured access when available, screenshots when not.

Anthropic described the underlying idea when it introduced its computer use model in October 2024: the aim was to lower the barrier for AI systems to apply existing skills to tools made for people, not to invent a new kind of intelligence.

Anthropic computer use and the other provider tools

Anthropic computer use is a tool you declare in an API request. Claude then returns actions for screenshots, mouse movement, clicks, and typing, and your code executes them in a sandboxed environment and sends results back. Anthropic's docs are the reference for current model support and tool versions, since both change.

It is no longer the only option:

The practical consequence: the loop is the same, the action schemas differ slightly, and the best model for a task changes over months. If you wire your agent directly to one vendor's schema, switching means rewriting. If you put a provider-independent layer between the model and the machine, switching means changing a setting.

Computer use versus API, browser, and DOM automation

Computer use is not always the right tool. Pick by how much structure the target exposes.

A workable rule:

1. If an API exists and covers the task, call the API.
2. If it is a website, use a browser agent with element-level access first.
3. If it is a native desktop app, a legacy system, or a workflow that crosses several apps, use computer use.
4. On Android apps with no API, use a device agent.

Red Hat frames computer use as a way to automate tedious manual work and enterprise workflows where custom integrations are not available. That is the right mental model: it is the integration of last resort, and it is often the only one.

What you can automate with it

Tasks that fit share three traits: they are repetitive, they run through a screen, and the outcome is checkable.

• Data entry across systems. Copy values from a legacy invoice viewer into a finance tool. For accounting teams, these accounting workflow automation examples show where the repetitive steps usually sit.
• Reconciliation and reporting. Pull exports from several portals, compare them, and assemble a report for review. See our operations crew use case.
• QA and user-flow testing. Walk a signup flow in a browser, then repeat it on an Android device. See web and mobile QA agents.
• Form filling on vendor portals. Submit routine forms where no API exists, with a person approving the final submission.
• Data extraction from apps. Read figures out of mobile apps that expose no export. See extracting data from mobile apps.
• Sales and operations chores. Update CRM records, log call outcomes, and prepare follow-ups. See sales and BizDev agents.

Tasks that do not fit: anything where a wrong click is irreversible and nobody reviews it, and anything an API already does in 200 milliseconds.

What you need to build a cross-model computer use agent

The minimum viable stack has six parts:

1. A model with vision. Any cloud or local model that can interpret screenshots and return structured actions.
2. An execution environment. A virtual machine, container, or dedicated desktop that the agent controls, separate from your daily machine.
3. An action executor. Code that turns "click (412, 288)" into a real input event and returns the next screenshot.
4. Permissions. Which apps, domains, and actions are allowed.
5. Approvals. A pause point before consequential steps.
6. Traces. A record of every screenshot, action, and decision, so a failed run can be replayed and debugged.

Building parts 3 to 6 yourself takes real engineering time, and you rebuild them for each provider's schema. This is the work Melaya handles.

Perspective: the model is the brain, Melaya is the controlled pair of hands

We built Melaya on one premise: your AI is the brain, Melaya is the controlled pair of hands.

Computer use gets more useful when the model is replaceable. Melaya is bring-your-own-model. You connect Claude, GPT, Gemini, DeepSeek, Mistral, Qwen, Ollama, or LM Studio, and the same tools, memory, schedules, and approvals apply whichever you pick. We do not sell a model, and agents never require one from us. When a better model ships, you change a setting instead of rebuilding workflows.

Control matters as much as capability. Melaya uses granular permissions, execution traces, and replayable runs. Every consequential action, whether publishing, sending, paying, or writing to a connected app, runs behind human approval. Agents prepare the work, and you approve the step that has consequences.

The surfaces are split by what they control. Browser Control operates websites. Mobile Agents operate Android applications. Computer Control operates desktop software through the Melaya desktop app. Our security posture is backed by independent security audits.

Melaya Team

Security risks and how to contain them

A model that can click anything can be steered into clicking the wrong thing. Treat that as a design constraint, not an edge case.

Prompt injection

The agent reads whatever is on screen: web pages, emails, documents. Text in that content can contain instructions aimed at the model. Anthropic's own announcement flagged this, noting that screenshots from internet-connected machines may expose Claude to injected instructions. The same applies to every provider's model.

Defenses that work at the system level, not the prompt level:

• Restrict the agent to an allowlist of domains and applications.
• Keep credentials out of the model's context. Let the executor inject them.
• Enforce authorization outside the model, so a persuasive page cannot grant itself new permissions.
• Require approval before any action that sends, publishes, pays, or writes.

Over-broad access

Run the agent in an isolated environment with only the accounts it needs. A dedicated VM with a scoped service account limits the damage from a bad run to that account.

Errors and drift

Vision models misclick, misread small text, and loop. Set a hard step limit, a timeout, and a stop condition. Review traces for the first runs of any new workflow.

Governance

For regulated teams, controls need to be documented and auditable. A guide to AI compliance automation covers capabilities risk and compliance teams use to map controls to frameworks. Combined with execution traces and approval records, you can show who approved what and when.

How Melaya runs computer use

Melaya splits control into three surfaces, each with its own permission scope:

You assemble these in the visual Agent Builder, combining a model, tools, memory, schedules, triggers, approvals, and multi-agent pipelines. Each run is traceable and replayable: you see the steps the agent took and can replay a failed one.

If you already work in ChatGPT, Claude, Cursor, or Codex, the Melaya MCP Server brings these tools into those clients. For larger organizations, the enterprise agent platform covers private, provider-independent deployment, including local infrastructure.

Start with Melaya

Start on the free Sandbox plan at $0 per month, then add what you need. Published prices:

See current details on the pricing page.

Pick one task, such as a weekly report assembled from two portals. Build it in the Agent Builder with your preferred model, keep approval on the final step, and read the first three traces before you schedule it. You can start from the Melaya home page.

A first-week plan

1. Day 1: choose one workflow with a clear success check and no API.
2. Day 2: set up an isolated environment and the narrowest permissions that work.
3. Day 3: build the agent, choose a model, and add approval before any send, publish, pay, or write step.
4. Day 4: run it manually five times and read every trace.
5. Day 5: try a second model on the same workflow and compare success rate and cost per run.
6. Week 2: add a schedule only after the manual runs are consistently clean.

The model comparison on day 5 is cheap when the tool layer does not depend on a provider.

FAQ

What is computer use in AI?

It is a capability where a model views screenshots of a screen and issues mouse and keyboard actions to complete tasks in software. A harness executes those actions and returns a new screenshot. The loop repeats until the task finishes or a limit stops it.

What is Anthropic computer use?

It is Anthropic's tool for Claude, introduced as a public beta capability in October 2024. Claude receives screenshots and returns actions such as clicks and typing, which your code runs in a sandboxed environment. Anthropic's documentation covers current setup.

Is computer use safe?

It is safe to the degree you constrain it. Use isolated environments, allowlists, scoped accounts, step limits, and human approval for any consequential action. In Melaya, publishing, sending, paying, and writing to connected apps run behind approval.

Can I use computer use with a local model?

Yes, if the model can interpret screenshots and return structured actions. Melaya works with local models through Ollama and LM Studio as well as cloud models, so the same tools apply either way.

Sources

• Computer use: How AI agents can automate almost anything, Red Hat Developer
• AI Compliance Automation: Six Capabilities for Risk & Compliance Teams
• Accounting workflow automation examples for 2026

Recommended

• AI Computer Use Agent for Windows, Mac & Linux
• Android AI Agent: AI That Controls Your Phone
• AI Browser Agent: Automate Any Site, You Approve
• AI Phone Control for Android: Agents Use Your Apps