Add element_name parameter to Click-Tool and Move-Tool#80
Open
ikoskela wants to merge 1 commit intoCursorTouch:mainfrom
Open
Add element_name parameter to Click-Tool and Move-Tool#80ikoskela wants to merge 1 commit intoCursorTouch:mainfrom
ikoskela wants to merge 1 commit intoCursorTouch:mainfrom
Conversation
Enable clicking and moving to UI elements by accessible name using Windows UI Automation, without requiring coordinate estimation. When element_name is provided, the tool uses UIA to locate the element, gets its bounding rectangle center, and clicks/moves there. The optional window_title parameter scopes the search to a specific window. This is fully backward-compatible: existing loc-based calls work unchanged. element_name is an optional alternative to loc. Uses the existing windows_mcp.uia module — no new dependencies. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
element_nameandwindow_titleparameters to Click-Tool and Move-Toolelement_nameis provided, uses Windows UI Automation to find the element by its accessible name and click/move to its center coordinates — no coordinate estimation neededwindow_titleoptionally scopes the search to a specific window (substring match)loc-based calls work unchangedMotivation
Currently, clicking a UI element requires the caller to estimate coordinates — either from State-Tool output, screenshots, or manual calculation. This is error-prone, especially on high-DPI displays where coordinate systems can be mismatched.
UI Automation can locate elements precisely by name, returning exact bounding rectangles. This PR exposes that capability directly in Click-Tool and Move-Tool, so callers can say
click(element_name="Save")instead of guessing pixel coordinates.Implementation
_resolve_element_location()helper that uses the existingwindows_mcp.uiamodule (no new dependencies)uia.Control(Name=...)for element search withExists(maxSearchSeconds=3)timeoutuia.WindowControl(SubName=...)for optional window scopingBoundingRectangle— ready forpyautoguiExamples
Test plan
click_tool(loc=[x, y])still works as beforeclick_tool(element_name="Start")finds and clicks the Start buttonclick_tool(element_name="Nonexistent")returns a clear errorwindow_titlescoping works correctlymove_toolwithelement_namemoves cursor to the element