Visual Prompt Injection Against Computer-Use Agents: Screenshots, Pop-Ups, and GUI Attacks

Visual prompt injection (VPI) targets the one input a computer-use agent cannot avoid trusting at some level: the screenshot it takes of its own screen before deciding what to click, type, or run next. A pop-up dialog, a banner inside a PDF viewer, or text an attacker renders in low contrast against a webpage's background can carry an instruction the agent's vision-language model reads as part of the environment it is operating in, not as content to treat with suspicion. Unlike a browser-scoped attack that ends at a stolen cookie, a general-purpose computer-use agent's screen can belong to any application: a terminal, a file manager, an email client, or a document viewer, so the same injection path can reach destructive, cross-application actions a browser-only threat model does not cover.

What it is

The defining object is the rendered visual surface a computer-use agent captures as a screenshot and feeds into its model's perception step, on every cycle of its observe-act loop, across whatever application currently has focus. This is broader than a single webpage: a full-desktop agent screenshots PDF viewers, terminals, native dialogs, and file managers the same way it screenshots a browser tab, because its action space (click, type, scroll, run) is defined against the operating system, not against HTML. The defining cause is adversarial content placed somewhere that render will put it in front of the agent's vision model: a pop-up timed to appear during a task, hidden or low-contrast text layered into an otherwise ordinary page or document, or an image carrying text a human would skim past but an OCR-capable model reads in full. The defining consequence is that the agent's next action follows the injected instruction instead of, or alongside, the user's actual task, because nothing in the perception step marks pixels on the screen as data to describe rather than instructions to obey.

This differs from ordinary indirect prompt injection in channel, not mechanism. A text-based tool result can be filtered or diffed before it reaches the model; a screenshot cannot be filtered the same way without defeating the point of a vision-driven agent, since the agent needs to see the screen to do its job, and the injected content arrives embedded in the very thing it needs to see.

What it is not

Agentic browser session hijacking, scoped to an already-authenticated browser session, cookies, logins, and stored payment data, with account takeover as the documented consequence. This entry is not browser-scoped; its consequence is unauthorized action across any application the agent's desktop session can reach, authenticated or not.
MCP tool poisoning, where untrusted content arrives in a tool's text description or structured result, not as rendered pixels read through a vision or OCR path.
Agent clipboard hijacking, where the contested object is the shared OS clipboard specifically. A visual injection can lead an agent toward a clipboard action, but the entry path here is what the agent sees on screen, not what it copies or pastes.
Agent command-approval bypass, where a guardrail inspects a command string and a shell rewrites it before execution. This entry's failure happens one step earlier, at perception, before any command string exists to inspect.
Markdown image and link exfiltration, the output-side counterpart covering a client rendering the agent's own generated Markdown. This entry covers the input side: the agent reading the screen's content before it generates anything.
Reproducible injection payloads or a working proof of concept tuned to a specific vendor's current classifier.

In scope

Pop-ups, overlays, and modal dialogs timed to appear mid-task, containing text written to read as a system or user instruction.
Hidden, low-contrast, or off-screen text layered into a webpage, PDF, or image that only a vision or OCR path, not a human skimming the same screen, is likely to read.
Content delivered through any application a full-desktop computer-use agent can screenshot, not webpages alone: PDF viewers, terminals, file managers, and native dialogs.
The documented, limited effectiveness of existing defenses, including vendor-trained classifiers and system-prompt-level warnings, against this channel.

Target state and trust boundaries

The target state is the agent's own perception of its environment: the screenshot that functions as ground truth for what the desktop currently looks like and what action makes sense next. The trust boundary that fails is the assumption that content appearing on screen is passive information to describe, when a vision-language model with no separate channel for "trusted system state" versus "untrusted rendered content" reads both through the same pipeline. A computer-use agent's action space, being defined over the whole operating system rather than over one application's document object model, means this boundary sits underneath every application the agent can see, not only a browser.

What the research establishes

VPI-Bench, accepted to ICLR 2026, built an end-to-end threat model tracing visual prompt injection from a visually embedded malicious prompt through to a harmful, executed outcome, evaluated with 306 test cases across five realistic, interactive platform clones (Amazon, Booking.com, BBC News, Messenger, and Email). The paper reports that current computer-use agents (CUAs) and browser-use agents (BUAs) "can be deceived at rates of up to 51% and 100%, respectively, on certain platforms," and that existing defense methods, including system-prompt-level warnings, "offer only limited improvements." Against Anthropic's Claude computer-use framework (Sonnet 3.5 and 3.7), success stayed below 60% on every platform; against several browser-use agent and model combinations with no equivalent defense layer, success reached as high as 100% on individual platforms (arXiv 2506.02456, VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents).

Anthropic's own documentation for the computer-use tool independently confirms the channel: "instructions on webpages or contained in images might override your instructions or cause Claude to make mistakes," and when the tool is active, "classifiers will automatically scan what the tools return, such as screenshots, to flag potential prompt injections" and steer the model to check with the user before acting. The same page recommends a dedicated, minimally privileged virtual machine, a domain allowlist, and human confirmation before any consequential action, as defenses independent of the classifier (Anthropic, Computer use tool documentation).

A documented case shows the consequence reaching past any single application. On 2024-10-24, HiddenLayer researcher Jason Martin disclosed that a PDF opened during a Claude Computer Use session, carrying an instruction obfuscated with base64 and ROT13 encoding (a plain, unobfuscated rm -rf command triggered Claude's own refusal first), caused the agent to decode and execute a filesystem-deleting shell command (HiddenLayer, Indirect Prompt Injection of Claude Computer Use). The case involved no browser, login, or authenticated session; the chain ran through a document viewer and a terminal, the cross-application reach this entry's scope is built around.

Detection: observable signs

A screenshot or document the agent processed contains imperative-language text, in a pop-up, overlay, or low-contrast layer, that does not describe the visible task.
The agent's next action (a command run, a file opened, a destination typed) does not match what the user's own visible instruction requested.
A vendor's built-in prompt-injection classifier flags a screenshot, and the agent proceeds without a logged user confirmation.
An agent's action log shows an obfuscated or encoded string (base64, ROT13, or similar) decoded and executed shortly after a document or image was read.
The same visual bait (a specific pop-up or overlay pattern) succeeds against one agent configuration and fails against another running with stricter sandboxing or confirmation gates.

Defenses, containment, and recovery

The table below is SSA editorial guidance for defenders. It is neither a standard requirement nor a measured effectiveness ranking.

Control	Transition it interrupts	Applicability	Residual risk and verification
Minimally privileged, dedicated VM or container per session	A compromised action reaching durable host state or unrelated applications	Any computer-use agent with file or shell access	Limits blast radius, not the initial injection; verify the VM cannot reach host-level credentials or persistent storage the user cares about.
Allowlisted domains and applications	Untrusted, attacker-controlled content reaching the agent's screen at all	Agents that browse or open externally sourced documents	An allowlist does not stop injection from a compromised or legitimate-but-hijacked site already on the list; pair with human confirmation for consequential actions.
Human confirmation before consequential actions	A single injected instruction completing an irreversible action unreviewed	Any deployment, regardless of classifier coverage	VPI-Bench's own results show this is where current defenses still fall short; confirmation fatigue can erode this control under high request volume.
Vendor screenshot classifiers	An injected instruction being acted on without the model pausing to check intent	Platforms that provide this as a built-in feature	VPI-Bench measured only limited improvement from comparable defenses; do not treat a classifier as sufficient on its own.

Prevention should treat every screenshot as untrusted content by default, the posture already applied to tool results elsewhere on this site. Detection should log the full chain from ingested visual content to the action it preceded. Containment should pause the session and revoke any access the agent held the moment an unreviewed, consequential action is suspected. Recovery should assume any action taken after the suspect screenshot may be compromised and re-verify it, rather than trusting the agent's own summary of what it did.

Boundaries with similar concepts

This entry sits one layer upstream of agentic browser session hijacking: both share a vision- or OCR-read entry path, but that entry's consequence is specific to an already-authenticated browser session, while this entry's scope is the full desktop a general-purpose computer-use agent can act on, authenticated or not. It also sits upstream of agent clipboard hijacking and agent command-approval bypass: a visual injection can be the precondition that puts a command on the clipboard or in front of an approval check, but each targets a distinct mechanism further along the same chain.

Mapping to standards and research

Source	Source-owned term or item	Relationship to this article
VPI-Bench, ICLR 2026 (arXiv 2506.02456)	Visual Prompt Injection (VPI)	Primary research term and benchmark; direct evidence for success rates and limited defense effectiveness.
Anthropic	Computer use tool documentation and screenshot classifiers	Vendor-acknowledged risk and first-party mitigation for the same channel.
HiddenLayer (2024-10-24)	Indirect Prompt Injection of Claude Computer Use	Documented case showing cross-application, destructive consequence with no browser or authenticated session involved.
MITRE ATLAS v2026.06	`AML.T0051.001 LLM Prompt Injection: Indirect`; `AML.T0053 AI Agent Tool Invocation`	Attacker-behavior classification for the injection delivery and resulting action.
OWASP Top 10 for Agentic Applications 2026	`ASI02 Tool Misuse and Exploitation`; `ASI05 Unexpected Code Execution (RCE)`	Broader risk categories this pattern's consequences fall under.
Agent State Attack	Visual prompt injection against computer-use agents	SSA framing of this entry under the site's state-surface taxonomy, not attributed to the sources above.

Related terms

Agent state attack: the umbrella definition.
Agentic browser session hijacking: the adjacent, browser- and authenticated-session-scoped variant of screen-content injection.
Agent clipboard hijacking: the adjacent surface for the shared clipboard channel a visual injection can lead toward.
Agent command-approval bypass: the adjacent surface for what happens after an injected instruction becomes a command string a guardrail must evaluate.

FAQ

Does this require the agent to use a browser?

No. The defining scope is a full-desktop, computer-use agent that screenshots whatever application has focus. VPI-Bench's test platforms were web-based, but HiddenLayer's documented case ran entirely through a PDF viewer and a terminal, with no browser or login involved.

Do vendor-trained classifiers solve this?

Not on their own. VPI-Bench found Anthropic's computer-use framework, which includes screenshot classifiers, kept success rates meaningfully lower than browser-use agents without an equivalent layer, but still measurably above zero. Anthropic's own documentation frames the classifier as one layer among several, not a standalone fix, and recommends sandboxing, domain allowlisting, and human confirmation alongside it.

Sources

Verification and change history

Standards and versions referenced: MITRE ATLAS, data release v2026.06; OWASP Top 10 for Agentic Applications, Version 2026 (December 2025); arXiv 2506.02456 (v2).

Last verified: 2026-11-03. The VPI-Bench abstract, per-platform success rates, and defense findings were read from the paper's own HTML rendering on this date. Anthropic's computer-use documentation text was read from its current page on this date. The HiddenLayer case's date, author, and attack description were read from the source's own page on this date.

Change history: 2026-11-03, first version.

Working-term label: Mixed provenance. "Visual Prompt Injection" and "VPI-Bench" are Research Terms owned by arXiv 2506.02456. AML.T0051.001 and AML.T0053 are Standard Terms owned by MITRE. ASI02 and ASI05 are Standard Terms owned by OWASP. This entry's framing of visual prompt injection as a full-desktop, cross-application state surface distinct from browser-scoped session hijacking is SSA editorial synthesis, not attributed to the sources above.