mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
filip131311
aa90873b59
Android `describe` rendered an `android.webkit.WebView` as a single opaque leaf labelled `[web-view]` and threw the subtree away. Chromium publishes the web DOM to the accessibility tree on both capture paths (the android-devtools helper and the `uiautomator dump` fallback), so the sentinel hid content the agent can act on. This keeps the WebView as a landmark and lets the normal trim run over the DOM beneath it. Supersedes #955, which reached the same diagnosis. The parser rules for the doubled in-app node, the web list clip and the text-run role are carried over from it; the rendered-output cap, the `truncated` flag plumbing and the flow-tree parity rework are not, so this stays confined to the Android parser and adapter (about 120 source lines) with no renderer, contract, skill or tool-description changes. ### What changes inside a WebView, and only there - **Doubled node merged.** An in-app WebView dumps as a nested pair: the app's view (resource-id, gesture flags), then Chromium's root web area under the same class carrying the page `<title>` as text. The inner half folds into the outer node, so the tree reads `WebView "<title>"`. A Chrome tab arrives as one node and is untouched. - **Web list clip.** A `<ul>` arrives as `android.widget.ListView` with `scrollable="false"`. Inside a WebView only the framework flag decides what clips, so a list never drops the items laid out around it, and the role is `List` rather than `ScrollView`. - **Text runs.** A labelled leaf `android.view.View` with no gesture flag is reported as `StaticText`. Outside a WebView a bare `View` keeps its role (Compose semantics nodes). - **Headings survive.** The WebView branch returns before the label dedupe, so an `h1` that repeats the page title is kept. ### The cold-read race is fixed, not documented Chromium builds a WebView's accessibility tree on the first request for it. Measured on API 35 / WebView 124: a read taken 6 s after page load still shows the WebView with no children; the page appears only after a read has asked for it — 10–400 ms later for a small in-app page, ~600 ms later for a Chrome tab over a long article. `describeAndroid` re-reads in 250 ms steps within a 1.5 s budget while the tree still has the unread shape, on both paths. A screen without a WebView never pays for it (one call, pinned by test). A WebView that stays empty after the budget is labelled `(no web content exposed)`. The unread shape is an `android.webkit.WebView` with no children, or — for a browser tab, which has no WebView view of its own before the page is published — Chrome's childless content view labelled "Web View" (English UI only; other locales see the tab the way they do today). ### Measured on a Pixel 9 / API 35 emulator with the branch tool-server | Screen | Before | After | |---|---|---| | In-app checkout form, cold launch | `WebView "[web-view] (no label)"` | full form in one call, 1.2 KB, 0.6 s | | 120-paragraph page | sentinel | the ten visible paragraphs, 1.8 KB (existing scroll clip bounds it) | | Wikipedia article in Chrome, helper attached after the page loaded | `FrameLayout "Web View"` | 4.7 KB, heading, links, lists, 1.4 s | | Settings home (native) | — | byte-identical to the previous parser | ### Testing - New `android-webview-describe.test.ts` on three live captures checked in as fixtures (in-app, Chrome, cold): merged landmark, `List` role, heading kept, re-read fires on the cold in-app shape and on Chrome's content-view placeholder, never on a native screen, and gives up within the budget. - `uiautomator-parser-v2-trim.test.ts`: the opaque-leaf case is replaced by the merge, the empty-WebView leaf, and a native `View` that must not be remapped. - Full tool-server suite, `tsc`, eslint, prettier, docs build. ### Docs No docs, skill or tool-description change. The page already names the accessibility tree as the source on Android; WebView content is now part of that tree and the agent needs no new instruction, since the cold read no longer produces an empty WebView. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01GZbkzeVC9faSxTFi4DbMuT <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Android WebView content is now captured and presented with meaningful controls, text, lists, and links. * Web content is consolidated into a clearer, unified accessibility view. * **Bug Fixes** * Improved Android screen descriptions when WebView content is initially unavailable by retrying the accessibility hierarchy. * Added safeguards for WebViews that never expose content, including a clear fallback message. * **Tests** * Expanded coverage for in-app WebViews, Chrome tabs, cold captures, and native Android UI filtering. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Hubert Gancarczyk <hubert.gancarczyk@swmansion.com>