Skip to content

Native Windows backend (full computer-use, not stub) - #13

Merged
Aurora Capital (auroracapital) merged 2 commits into
mainfrom
feat/windows-native-backend
Jul 30, 2026
Merged

Native Windows backend (full computer-use, not stub)#13
Aurora Capital (auroracapital) merged 2 commits into
mainfrom
feat/windows-native-backend

Conversation

@auroracapital

@auroracapital Aurora Capital (auroracapital) commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Prior multi-agent work left WinBackend as a limited stub. This ships a real Windows backend with the same MCP tool surface as Linux/macOS.

  • WinBackend: screenshot (Pillow ImageGrab), click/type/keypress/scroll (pyautogui or ctypes SendInput), list_windows (EnumWindows), launch_app (startfile/shell)
  • Leases: single interactive desktop win-main (macOS model) — exclusive owner + heartbeat/reap of lease metadata
  • Deps: pyautogui only on win32 (marker); SendInput fallback if missing
  • Docs: README Windows section + architecture notes corrected (was “X11 everywhere else”)
  • Tests: tests/test_win_backend.py — method surface, off-Windows guards, lease roundtrip on Linux CI

Capability matrix

Linux macOS Windows
Primitives (shot/click/type/key/scroll) yes yes yes
Multi-seat Xvnc pool yes no no
Exclusive leases yes yes yes
noVNC live view yes optional no

Test plan

  • python3 -m py_compile mcp-server/server.py
  • pytest tests/ (leases + win backend) on Linux
  • Manual smoke on a Windows interactive session: ensure_desktopscreenshotclicktype_textlist_windowsrelease_desktop

Note

Medium Risk
Large new code path drives real mouse/keyboard and app launch on the logged-in Windows session; incorrect input or lease handling could affect concurrent agents sharing one desktop, though Linux pool behavior is unchanged.

Overview
Replaces the limited WinBackend stub with a native implementation that matches Linux/macOS MCP tools: screenshot (Pillow ImageGrab), click/type/keypress/scroll (pyautogui or ctypes SendInput + clipboard paste fallback), list_windows (EnumWindows), and launch_app (os.startfile / shell).

Desktop pooling on Windows stays a single interactive session win-main (macOS-style): exclusive leases, heartbeat, release, and reap clear metadata only—no Xvnc multi-seat pool. Default display becomes win:main when on Windows; MCP instructions and architecture notes now describe three backends.

Dependencies & tests: pyautogui is added with a win32 marker; tests/test_win_backend.py validates the full method surface, off-Windows guards for GUI ops, and lease roundtrips on Linux CI.

Reviewed by Cursor Bugbot for commit 938ffba. Bugbot is set up for automated code reviews on this repo. Configure here.

Replace the Windows stub with real primitives (ImageGrab, pyautogui/SendInput,
EnumWindows, launch_app) and the same exclusive-lease model as macOS. Document
Windows install/limits, pin pyautogui for win32, and add unit tests that run
on Linux CI.
@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_c52c93dc-6945-4e08-8bd1-c520b434c83c)

@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@auroracapital, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 56 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e4febeaa-baad-46ca-84ef-52fd1c2cab16

📥 Commits

Reviewing files that changed from the base of the PR and between 86fe8ac and 938ffba.

📒 Files selected for processing (4)
  • README.md
  • mcp-server/server.py
  • requirements.txt
  • tests/test_win_backend.py
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Fix failing CI checks
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/windows-native-backend

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_de051245-e300-48df-ab68-cb69f97d639f)

Comment thread mcp-server/server.py
x, y, w, h = parts
bbox = (x, y, x + w, y + h)
img = ImageGrab.grab(bbox=bbox, all_screens=True)
path, meta = _encode_screenshot(img, fmt=fmt, max_width=max_width, use_cache=use_cache)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The call to _encode_screenshot in WinBackend is missing required arguments (display, grab_ms) and incorrectly unpacks the dictionary return value, causing a TypeError.
Severity: CRITICAL

Suggested Fix

Update the call to _encode_screenshot in WinBackend._take_screenshot to provide all required arguments (img, display, grab_ms, fmt, max_width, use_cache). The returned dictionary should be assigned to a single variable, and its keys should be used to access the values, similar to how it is handled in the MacBackend.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: mcp-server/server.py#L1709

Potential issue: The `_take_screenshot` method in the `WinBackend` class calls the
`_encode_screenshot` function with incorrect arguments. The call at line 1709 is missing
the required positional arguments `display` and `grab_ms`. Furthermore, it incorrectly
attempts to unpack the dictionary returned by `_encode_screenshot` into a tuple `(path,
meta)`. This will raise a `TypeError` whenever a screenshot is taken on a Windows
system, causing the feature to fail completely on that platform.

Did we get this right? 👍 / 👎 to inform future reviews.

@auroracapital
Aurora Capital (auroracapital) merged commit 9278d88 into main Jul 30, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant