Skip to content

fix(page-controller): index new (*-prefixed) elements in getElementTextMap - #633

Open
JOhnsonKC201 wants to merge 1 commit into
alibaba:mainfrom
JOhnsonKC201:fix/element-text-map-new-elements
Open

fix(page-controller): index new (*-prefixed) elements in getElementTextMap#633
JOhnsonKC201 wants to merge 1 commit into
alibaba:mainfrom
JOhnsonKC201:fix/element-text-map-new-elements

Conversation

@JOhnsonKC201

Copy link
Copy Markdown

Problem

getElementTextMap (packages/page-controller/src/dom/index.ts) builds the index → line map that PageController uses to echo a descriptive result for clickElement / inputText. It matched interactive lines with:

/^\[(\d+)\]<[^>]+>([^<]*)/

But flatTreeToString renders new elements with a leading *:

const highlightIndicator = node.isNew ? `*[${node.highlightIndex}]` : `[${node.highlightIndex}]`

which the system prompt documents ("Elements tagged with *[ are the new clickable elements"). The ^\[ anchor never matches a *[...] line, so:

  • new elements (dropdown options, freshly-rendered modals, etc.) get no entry, and
  • on the first updateTree() every element is new, so the map is empty entirely.

PageController then falls back to the bare index — Clicked element (5). instead of Clicked element (*[5]<a role=button>Get started />). — losing traceability for exactly the elements the agent most often acts on, and for the whole first extraction.

Fix

Allow an optional leading * in the regex (/^\*?\[(\d+)\]<[^>]+>([^<]*)/). Existing (non-*) lines are unchanged, and plain text still can't match because the [i]<…> structure is still required.

Tests

Adds packages/page-controller/src/dom/index.test.ts covering the existing-element, new-element (*-prefixed), and indented cases. npm test passes across all packages; typecheck and eslint are clean.

…xtMap

getElementTextMap matched interactive lines with /^\[(\d+)\]<.../, but
flatTreeToString renders new elements with a leading '*' (`*[i]<...`), which
the system prompt documents as the "new clickable elements". The anchored
regex never matched those lines, so new elements got no entry in
elementTextMap — and on the first updateTree(), when every element is new,
the map was empty entirely.

As a result clickElement/inputText fell back to the bare index in their
result messages (e.g. "Clicked element (5).") instead of the descriptive
line, losing the traceability the tool feedback is meant to provide.

Allow an optional leading '*' in the regex so new elements are indexed the
same as existing ones. Existing lines are unaffected. Adds unit tests for
the new-element and indented cases.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants