Skip to content

feat: click-xy 和 eval-val,提升 UI 自动化健壮性 (#24)#25

Merged
4ier merged 1 commit into
mainfrom
feat/click-xy-press
Apr 28, 2026
Merged

feat: click-xy 和 eval-val,提升 UI 自动化健壮性 (#24)#25
4ier merged 1 commit into
mainfrom
feat/click-xy-press

Conversation

@4ier
Copy link
Copy Markdown
Owner

@4ier 4ier commented Apr 28, 2026

背景

解决 #24 — 从 browser-harness 实践里总结出来的 Neo UI 自动化痛点:a11y tree 抽象遇到 React/Material 自定义组件就崩。

变更

P0: neo click-xy <x> <y>

绕开 a11y tree,直接走 CDP Input.dispatchMouseEvent 在视口坐标点击。解决 #24 的 Bug 1/3:

  • 自定义 <div role="combobox"> 的 dropdown 选项,resolveRef 拿不到稳定 objectId
  • Portal/popover 里的选项节点存在但无 box model
  • Shadow DOM、跨域 iframe 里的元素

P1: neo eval-val <selector> <value>

native setter + 事件派发 模式写入 input/textarea/select:

var setter = Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, 'value');
setter.set.call(el, value);
el.dispatchEvent(new Event('input', { bubbles: true }));
el.dispatchEvent(new Event('change', { bubbles: true }));

解决 React/Vue/Angular 受控输入忽略 el.value = x 的问题——框架监听的是原生 setter 被调用后的事件链,直接赋值不会进入它们的 state。

为什么不加自动 fallback(#24 的 P3)

保持行为可预测。让用户/agent 显式选 click vs click-xyfill vs eval-val——静默 fallback 会让排错变难。

未做:keyboard 命令

neo press 已经存在并支持 Enter/Tab/Escape/Arrow/Ctrl+a 等(我来之前就有了),无需重做。

未做:ref 稳定性 / selector fallback

#24 P4/P5 需要 snapshot 输出格式改造,留给后续独立 PR。当下 click-xy + eval-val 已能兜住绝大多数 snapshot 失败场景。

实现要点

  • 两个纯 helper 便于测试:buildClickXyEvents(x, y)buildEvalValExpression(selector, value)
  • 完全不改动 commands.click / commands.fill 既有行为,只新增
  • Help 文本同步更新(文件头注释 + 运行时 usage 两处)

验证

单元测试

173 passed, 0 failed

新增 6 条:

  • buildClickXyEvents produces mousePressed + mouseReleased at given coords
  • buildClickXyEvents accepts numeric strings and floats
  • buildClickXyEvents rejects non-finite coordinates
  • buildEvalValExpression JSON-escapes selector and value safely
  • buildEvalValExpression escapes quotes and backslashes in value
  • buildEvalValExpression coerces null/undefined value to empty string

CLI 冒烟

$ neo click-xy            → Usage: neo click-xy <x> <y>        (exit 1)
$ neo click-xy foo bar    → Usage: ... (finite numbers required) (exit 1)
$ neo eval-val            → Usage: neo eval-val <selector> <value> (exit 1)

真实 Chrome 端到端

测试页:<input id='e' value='old'><button onclick="e.value='clicked'">

$ neo click-xy 460 320 → Clicked (460, 320)
$ neo eval "document.getElementById('e').value" → clicked   ✅
$ neo eval-val "#e" "hello from native setter" → Set value on #e
$ neo eval "#e.value" → hello from native setter              ✅

影响范围

  • 纯增量,不改动既有命令
  • 依赖 CDP 已有能力(Input.dispatchMouseEvent, Runtime.evaluate)
  • Help 文本同步,向后兼容

closes #24 (P0 + P1 部分)

Addresses issue #24 lessons from browser-harness.

- neo click-xy <x> <y>: bypass a11y tree, click at absolute viewport
  coordinates via CDP Input.dispatchMouseEvent. Fixes combobox options,
  shadow DOM, portals, cross-origin iframes where resolveRef fails.

- neo eval-val <selector> <value>: set input/textarea/select value using
  the framework-safe native setter pattern. Works with React/Vue/Angular
  controlled inputs that ignore naive '.value = x'. Dispatches input +
  change events with bubbles:true.

- Extract buildClickXyEvents() and buildEvalValExpression() as pure
  helpers + unit tests (+6 tests, 173 passed).

- Update help text (header comment + runtime usage).
@4ier 4ier merged commit 7c52c4b into main Apr 28, 2026
1 check passed
@4ier 4ier deleted the feat/click-xy-press branch April 28, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UI automation: lessons from browser-harness — coordinate clicks, reactive inputs, combobox fixes

1 participant