Cua Driver Optional addition Perception Visual extension

Article

Cua Driver Give Agents “eyes”: even pure-pixel interfaces can recognize buttons and text

Beating AI news flash: open-source Computer Use tool Cua adds Cua Driver an optional Perception visual extension. Driver originally prioritized using webpage structure and the system accessibility tree to find buttons. When it encounters Canvas, games, remote desktops, or custom-drawn applications, these structures may be empty.Perception can directly read screenshots, breaking the display into regions with text, types, and positions for the Agent to continue operating.

The first release uses OmniParser to recognize icons and controls, and PP-OCR to read text. The entire parsing process is completed locally on CPU the device, without requiring GPU or sending screenshots over the network. Official testing found that a single screenshot takes approximately 2 to 2.5 seconds on macOS, 3.5 to 4 seconds on Linux, and 8 to 9 seconds on Windows. It mainly serves as a fallback when webpage structure and the accessibility tree fail; visual parsing is not run by default at every step.

Cua also restricts the reuse of visual operations on old screenshots. Each visual recognition is bound to the `capture_id` of the current screenshot, and subsequent clicks must come from the same screenshot. If a screenshot is more than 60 seconds old, or an operation has already been performed, Driver refuses to continue using it, preventing erroneous clicks based on stale coordinates after the interface changes.

Cua previously already supported visual localization through `cua-som` and OmniParser. This capability is now integrated directly into Cua Driver, allowing upper-layer Agents to obtain recognition results through a unified interface. Even if the model itself cannot see images, it can determine the next operation based on this text and region information.Perception is not a default component. Its OmniParser detector uses AGPL-3.0；Cua Driver and still maintains the MIT protocol.

Original link https://m.theblockbeats.info/flash/369547

Key points

Cua Driver adds an optional Perception visual extension.
Perception Directly reads screenshots and breaks the display into regions with text, types, and positions.
The first release uses OmniParser to recognize icons and controls, and PP-OCR to read text.
The parsing process is completed locally on CPU the device, without requiring GPU or sending screenshots over the network.
Official testing found that a single screenshot takes approximately 2 to 2.5 seconds on macOS, and 3.5 to 4 seconds on Linux, and 8 to 9 seconds on Windows.
Visual recognition is bound to the current screenshot’s capture_id; if the screenshot is more than 60 seconds old or an operation has already been performed, Driver refuses to continue using it.
Perception is not a default component; its OmniParser detector uses AGPL-3.0，Cua Driver and still maintains the MIT protocol.

AI-assisted interpretation

The following is analysis, separate from reported facts. Verify important claims independently.

Webpage structure and the system accessibility tree sometimes cannot provide button information, such as in Canvas, games, remote desktops, or custom-drawn applications. In such cases,Perception can inspect the current screenshot and return text, control types, and positions, allowing the Agent to continue operating.

Why it matters to readers

It adds a visual recognition entry point for pixel interfaces that cannot provide webpage structure or accessibility information, while also restricting the repeated use of old screenshots.

This can be understood as giving a computer-operating Agent an optional “see-the-screen” capability, but recognition takes time and is not automatically launched at every step.

Risks and unknowns

The supplied materials do not include an official Cua announcement or code repository evidence; the feature, performance, and licensing information still require verification against project materials.
The materials do not explain OmniParser uses AGPL-3.0 or how this affects specific integration or usage scenarios.

Cua Driver Optional addition Perception Visual extension

Article

AI-assisted interpretation

Why it matters to readers

Risks and unknowns

Related Developments

Related concepts

Keep following

Security Center