ESP32 iPhone Agent is open source, enabling a low-cost development board to operate a real iPhone

Article

ESP32 iPhone Agent is open source, allowing 40 multiple ESP32 to become an iPhone version of Doubao's phone assistant

Beating AI news flash: developer Henry has open-sourced ESP32 iPhone Agent. With only a development board costing approximately 40–50 yuan and a mobile app, it can make XIAO ESP32S3 control a real iPhone, open other apps, click buttons, and swipe the screen. It does not require jailbreaking, nor does it rely on Apple's AI testing permissions.

The implementation is ingenious.XCTest establishes its own Wi-Fi hotspot, serving as an ESP32 screen-casting receiver. The iPhone casts its screen to the development board. The board does not decode it; instead, it sends the compressed video back to the Agent app on the same phone via Bluetooth.

The app calls a vision model to recognize the interface and decide the next step, then sends instructions via Bluetooth, allowing AirPlay to simulate ESP32 mouse and keyboard operations on the phone. To avoid click-position offsets, the system also automatically calibrates the mouse pointer and, when necessary, corrects its position based on the image.

This solution combines screen reception and input simulation using a standard USB development board, without requiring dedicated video-capture hardware such as NanoKVM Go. The Agent runs directly on the iPhone, while model inference depends on an external ESP32-S3 or a self-hosted server. Because the phone's Wi-Fi is occupied by the development board's hotspot, access to the model by default requires a cellular network. Flashing the firmware for the first time and compiling and installing the iPhone app still require a Mac and Xcode.

Due to Bluetooth bandwidth limitations, the returned image has a resolution of only API and a maximum of 480×1040 frames per second; each operation typically takes several seconds. Touch coordinates currently support only portrait interfaces, keyboard input supports only 5 characters, and screen casting cannot continue after the screen is locked.

Original article:ASCII https://m.theblockbeats.info/flash/372185

Key points

Developer Henry has open-sourced ESP32 iPhone Agent。
The solution uses a development board costing approximately 40–50 yuan and a mobile app.XIAO ESP32S3
The system can open other apps, click buttons, and swipe the screen. It does not require jailbreaking or Apple's XCTest testing permissions.
ESP32 establishes a Wi-Fi hotspot and serves as an AirPlay screen-casting receiver; the compressed video is sent back via Bluetooth to the Agent app on the same phone.
The app uses a vision model to recognize the interface and decide the next step, then sends instructions via Bluetooth, allowing ESP32 to simulate USB mouse and keyboard operations on the phone.
The image resolution is 480×1040 with a maximum of 5 frames per second; each operation typically takes several seconds.
Touch coordinates currently support only portrait interfaces, and keyboard input supports only ASCII characters; screen casting cannot continue after the screen is locked.
Model inference depends on an external API or a self-hosted server; flashing the firmware for the first time and compiling and installing the iPhone app still require a Mac and Xcode.

AI-assisted interpretation

The following is analysis, separate from reported facts. Verify important claims independently.

It connects a small ESP32-S3 development board with an iPhone app: the development board receives the phone's screen, the app uses a vision model to interpret the interface, and the development board then simulates mouse and keyboard input to complete the operation.

Why it matters to readers

This demonstrates a solution for operating an iPhone using a standard ESP32-S3、AirPlay, Bluetooth, and a vision model, with hardware costing approximately 40–50 yuan.

You can think of it as an experimental iPhone assistant in which a “camera views the screen, a model makes decisions, and a mouse and keyboard execute operations,” but it requires a Mac, Xcode, and an external model service.

Risks and unknowns

The evidence does not provide specific addresses for the project repository, firmware, or iPhone app; whether the code and solution can be independently reproduced has not been confirmed.
Image frame rate, portrait-orientation limitations,ASCII input limitations, and screen-lock limitations may affect the practical scope of use.
The evidence does not state the solution's compatibility with different iOS versions.

ESP32 iPhone Agent is open source, enabling a low-cost development board to operate a real iPhone

Article

AI-assisted interpretation

Why it matters to readers

Risks and unknowns

Related Developments

Related concepts

Keep following

Security Center