Who AutoCua is for#
QA engineers automating cross-application regression tests
AutoCua can run a single test that opens a desktop app, reads data from it, pastes into a browser form, and verifies the result without needing separate automation tools for each surface. The shell-use mode lets the agent write test helper scripts in a real terminal mid-run, keeping the test logic in one place.
Skip if:
Your test suite lives entirely within a single web application. Dedicated browser automation tools like Playwright handle that case with less configuration overhead and a larger ecosystem of plugins and reporters.
AI researchers comparing model performance on computer-use tasks
Switching among 48 models across 9 providers is a single parameter change in run_agent(). With no project telemetry, benchmark runs do not send task details to a vendor. The multi-agent compression keeps long evaluation tasks from failing at token limits, so the same complex task can run to completion across multiple models.
Skip if:
You need a hosted benchmark platform with team access controls, reproducible sandboxed environments, and audit logs. AutoCua has no built-in multi-user management or environment isolation.
Developers building automation pipelines across desktop and browser
A single run_agent() call can move data from a PDF viewer through a spreadsheet and into a browser form in sequence. The accessibility-tree view works across native desktop apps, Chrome, and the shell without requiring separate API integrations for each surface.
Skip if:
Your pipeline involves only web pages with a stable DOM structure. A headless browser library handles that case with less setup than AutoCua requires.
Privacy-sensitive teams running local model inference
Because AutoCua sends only the API call to the model provider you configure, teams running a local endpoint keep all data on-device: no screenshots leave the machine, and no usage data goes to the AutoCua project. This covers Ollama, LM Studio, or any private API that speaks an OpenAI-compatible protocol.
Skip if:
You need enterprise-grade audit trails, role-based access controls, or SLA-backed uptime. AutoCua has no managed service or enterprise support tier.
The problem it solves#
Automating a real desktop workflow is harder than it looks. Browser automation libraries drive a browser through a debugging port, which browsers flag and anti-bot systems detect. Screen-coordinate automation breaks on high-DPI displays and whenever a window changes size or position. None of these approaches cross application boundaries: a workflow that moves data from a PDF reader into a spreadsheet and then files a form in a browser requires three separate automation tools that were never designed to share state.
Windows UAC elevation prompts stop user-mode automation cold: the secure desktop that UAC shows cannot receive synthetic input from user-level code. Linux desktop automation has historically required knowledge of AT-SPI bindings that most frameworks leave undocumented. Privacy-sensitive teams face a further problem: most managed computer-use services require sending screenshots of your screen to a cloud endpoint with no option to run locally.
How it solves it#
Cross-Platform Desktop Automation with Real Input Events
Drives any desktop application on macOS, Windows 10/11, and Linux GNOME using the operating system's own input layer: kCGHIDEventTap on macOS, pywinauto's SendInput on Windows, and AT-SPI on Linux. Applications see the same mouse and keyboard events they receive from a physical device, so accessibility APIs, native UI controls, and anti-automation detection all respond normally. Supports clicks, multi-line text paste via clipboard, drag gestures, hotkeys, and AppleScript on macOS.
Browser Agent via Chrome Extension
Drives your own Chrome profile through the AutoCua extension, which Chrome loads at launch without a store install. Chrome shows its own debugging bar and a Cancel button that returns control to you immediately. A headless path opens a separate Chrome instance on a debugging port when needed for unattended runs. The page scanner never calls Runtime.enable and never injects numbered labels into the DOM; interaction points are painted into the screenshot only.
Accessibility Tree and OCR Screen Reading
Each step fuses the OS accessibility tree with OCR to produce a numbered element list. On macOS, Apple Vision runs at the accurate level while the accessibility walk is still in progress; anything under 1 percent visible in the real window stack is discarded. On Windows, WinRT OCR, taskbar walk, Win32, and UIA scans run in parallel and merge at 5 px. The model picks a numbered element, never a pixel coordinate; a number not on the list is rejected before any input fires.
iOS Simulator and iPhone Automation
Drives an iOS Simulator by default or a USB-paired iPhone or iPad from a macOS host. Agent mode is set to 'mobile use, ios' in the run_agent() call. WebDriverAgent is cloned automatically on the first iOS run; full Xcode and git are the only prerequisites, with no manual setup required beyond that. iOS automation requires a macOS host; Android support is not yet available.
Multi-Agent Context Compression
A separate compression agent monitors the transcript during a run and splices a handoff summary into it when the context approaches the limit. This lets long multi-step workflows finish rather than fail at the token boundary. The desktop app's memory ring shows the compression event count in real time. Without this, complex tasks spanning many application interactions would require manual restarts and state recovery between steps.
48 Models Across 9 Providers
Model selection is a single parameter in the run_agent() call. Pass any model name from model_list.txt in the repository along with the provider key in your .env file. Providers include Anthropic, OpenRouter, and others; the full list is in model_list.txt. A model name not on the list is forwarded verbatim to the provider API and returns the provider's own error, making it easy to test new model releases as they ship.
Strengths and trade-offs#
Strengths
- Real OS-Level Input, Not Synthetic AutomationAutoCua posts events at the operating system's own input layer rather than through a debugging port, automation flag, or JavaScript injection. On macOS, events go to kCGHIDEventTap, the same tap a physical mouse feeds. On Windows, real SendInput calls are used; a kernel-mode driver handles UAC-elevated windows. Applications, browsers, and anti-bot systems all see normal user input events, because they are exactly that.
- No Telemetry, Local Data StorageAutoCua sends nothing to the project: the only outbound traffic is the API call to the model provider you configure. Screenshots are processed locally. Session data, conversations, provider keys, and browser profiles all write to an AutoCua_data directory on your own machine. Teams running a local model endpoint such as Ollama or LM Studio can operate entirely offline, with no data leaving the device.
- MIT License with No Commercial RestrictionsThe MIT license permits commercial use, modification, redistribution, and embedding in proprietary products without restriction. The only dependency with different terms is the optional Interception kernel driver for Windows UAC handling, which is obtained separately from its author under LGPL-3.0 non-commercial terms; it is never required for AutoCua to function, and every other mode works without it.
- Prebuilt Binaries on All Major PlatformsThe pip wheel ships prebuilt Rust binaries for the browser agent on Windows x64, Apple Silicon macOS, and Linux x86_64. No Rust toolchain or C compiler is needed on supported platforms. WebDriverAgent for iOS clones itself on first use. The optional Electron desktop app downloads on first launch when you confirm it; without it the same app opens in pywebview with no further setup.
Trade-offs
- -Linux Requires One-Time System Package InstallThe Linux wheel requires a GNOME desktop (tested on Ubuntu with Wayland; X11 is not tested) and a one-time apt install of AT-SPI, GTK 3, XDG desktop portal, and related bindings. Ubuntu and Debian also require a virtualenv built on the distro's own Python 3 version to load system PyGObject correctly; a venv on a different minor version fails with a gi import error. This is a documented one-time setup, but it is more involved than the macOS or Windows setup scripts.
- -iOS Automation Requires macOS; Android Not Yet AvailableMobile automation only runs on a macOS host. iOS support covers simulators and USB-paired iPhones and iPads; Android automation is listed in the roadmap but not yet available. Teams that need cross-platform mobile automation or that run automation infrastructure on Linux or Windows cannot use the mobile surface at all.
- -Windows UAC Requires a Kernel Driver Not Installable via pipAnswering Windows UAC elevation prompts requires the Interception kernel-mode driver, which pip cannot install because it must be loaded at system boot. Without it, every other mode still works, but any task that hits an elevation prompt stops there. Installing the driver requires running setup_windows.bat as administrator and rebooting once; it is never bundled due to its dual-license terms.
- -Project Is Under Two Weeks OldAutoCua was created on 2026-10-08 and had 191 stars and 1 open issue at research time. Active development is evident from daily commits, but API stability, long-term maintenance trajectory, and community size are not yet established. Teams evaluating this for production workflows should track the repository's release cadence and issue volume over the coming months before committing to it.
AutoCua vs alternatives#
AutoCua vs OpenAI Operator
Both automate computer tasks through AI, but they run in different environments. OpenAI Operator is a managed cloud service that operates inside a sandboxed browser; AutoCua is an open source tool you run on your own machine, able to drive any desktop app, browser, terminal, or iOS device.
| Feature | AutoCua | OpenAI Operator |
|---|---|---|
| License | MIT | Proprietary |
| Self-hosting | Yes | No |
| Desktop automation | Yes | No |
| iOS automation | Yes (macOS host) | No |
| Model choice | 48 models, 9 providers | OpenAI models only |
| Data handling | Local only | Cloud-processed |
| Cost | Free (pay for model API) | ChatGPT Plus or API |
AutoCua is the better choice when you need native desktop or iOS automation, when screenshots must stay on your machine, or when you want to run the same workflow across different model providers for cost comparison. OpenAI Operator is the right fit when you need only browser automation and prefer a fully managed service with no local setup; its chat interface is also more accessible to non-developers who prefer not to write Python.
AutoCua vs Simular Sai
Simular Sai is a commercial computer-use product for professional desktop automation. Both tools drive desktop apps with OS-level input, but differ in license, hosting model, and platform breadth.
| Feature | AutoCua | Simular Sai |
|---|---|---|
| License | MIT | Proprietary |
| Self-hosting | Yes | No |
| Platforms | Windows, macOS, Linux, iOS | Windows, macOS |
| Model choice | 48 models, 9 providers | Limited |
| Cost | Free (model API cost only) | Commercial |
AutoCua fits teams embedding computer-use into pipelines, running in CI, or keeping data entirely on-premise. Simular Sai may suit teams that want a commercial product with vendor support and no configuration overhead, or non-technical users who need a polished managed interface rather than a pip-based setup.
Quick start#
Install via pip on Python 3.10 or later; Linux users need AT-SPI system packages before running this step.
```bash
pip install AutoCua
```What it's built on#
- Languages
- JavaScriptPythonRust
- Frameworks
- Flask
FAQ#
Does AutoCua work on Windows and Linux, not just macOS?
AutoCua supports Windows 10 and 11, Linux (GNOME desktop, tested on Ubuntu with Wayland), and macOS on Apple Silicon or Intel. On Linux, a one-time apt install of AT-SPI and GTK bindings is required before use. iOS mobile automation requires a macOS host; there is no iOS path on Windows or Linux.
Which AI models does AutoCua support?
AutoCua supports 48 models across 9 providers. The full list is in model_list.txt in the repository. You pass a provider name and model name as parameters to run_agent(); the names are case-sensitive. A model name not in the list is forwarded verbatim to the provider API and returns the provider's own error response.
Does AutoCua send screenshots or usage data to any external service?
No. AutoCua sends nothing except the API call to your chosen model provider. Screenshots are processed locally and never transmitted by the AutoCua runtime. Session data, conversations, and provider keys write to an AutoCua_data directory on your machine. If you use a local model endpoint, all inference also stays on-premise.
Is AutoCua free for commercial use?
AutoCua is MIT licensed, so commercial use, modification, and redistribution are permitted with no fees to the project. You pay only for the model API calls your runs make. The one dependency with different terms is the optional Interception driver for Windows UAC handling, which is dual-licensed LGPL-3.0 non-commercial for free use; commercial use of that driver requires a paid license from its author.
How is AutoCua different from Playwright or Selenium?
Playwright and Selenium drive a browser through a debugging port and cannot automate native desktop apps. AutoCua drives Chrome through a browser extension and posts input events at the OS level, so it can automate any desktop app, shell terminal, and iOS device in the same run. For web-only testing with a stable DOM, Playwright is simpler and more mature; AutoCua is the right choice when your workflow crosses multiple application surfaces or includes native apps.
Similar open-source tools#
Atomic Agent
3,397 GitHub starsLocal-first AI agent that runs on your machine, drives real tools.
Alternative to
Grok Bot
rakazo
4,904 GitHub starsAI teammates you own: your keys, your model, your machine.
Alternative toGrok Bot
Voice Assistant
2 GitHub starsVoice interface for Claude Code with local Whisper and spoken answers
Alternative to
Superwhisper
OCRmyPDF
35,105 GitHub starsMake scanned PDFs searchable and copy-pasteable in one command.
Alternative to
ABBYY FineReader PDF
JellyBoxPlayer
750 GitHub starsNative Jellyfin and Emby music player for every platform
Alternative to
Symfonium
FckSignups
4,495 GitHub starsOpen-source tools that work instantly, no signup required