mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
filip131311
aa90873b59
Android `describe` rendered an `android.webkit.WebView` as a single opaque leaf labelled `[web-view]` and threw the subtree away. Chromium publishes the web DOM to the accessibility tree on both capture paths (the android-devtools helper and the `uiautomator dump` fallback), so the sentinel hid content the agent can act on. This keeps the WebView as a landmark and lets the normal trim run over the DOM beneath it. Supersedes #955, which reached the same diagnosis. The parser rules for the doubled in-app node, the web list clip and the text-run role are carried over from it; the rendered-output cap, the `truncated` flag plumbing and the flow-tree parity rework are not, so this stays confined to the Android parser and adapter (about 120 source lines) with no renderer, contract, skill or tool-description changes. ### What changes inside a WebView, and only there - **Doubled node merged.** An in-app WebView dumps as a nested pair: the app's view (resource-id, gesture flags), then Chromium's root web area under the same class carrying the page `<title>` as text. The inner half folds into the outer node, so the tree reads `WebView "<title>"`. A Chrome tab arrives as one node and is untouched. - **Web list clip.** A `<ul>` arrives as `android.widget.ListView` with `scrollable="false"`. Inside a WebView only the framework flag decides what clips, so a list never drops the items laid out around it, and the role is `List` rather than `ScrollView`. - **Text runs.** A labelled leaf `android.view.View` with no gesture flag is reported as `StaticText`. Outside a WebView a bare `View` keeps its role (Compose semantics nodes). - **Headings survive.** The WebView branch returns before the label dedupe, so an `h1` that repeats the page title is kept. ### The cold-read race is fixed, not documented Chromium builds a WebView's accessibility tree on the first request for it. Measured on API 35 / WebView 124: a read taken 6 s after page load still shows the WebView with no children; the page appears only after a read has asked for it — 10–400 ms later for a small in-app page, ~600 ms later for a Chrome tab over a long article. `describeAndroid` re-reads in 250 ms steps within a 1.5 s budget while the tree still has the unread shape, on both paths. A screen without a WebView never pays for it (one call, pinned by test). A WebView that stays empty after the budget is labelled `(no web content exposed)`. The unread shape is an `android.webkit.WebView` with no children, or — for a browser tab, which has no WebView view of its own before the page is published — Chrome's childless content view labelled "Web View" (English UI only; other locales see the tab the way they do today). ### Measured on a Pixel 9 / API 35 emulator with the branch tool-server | Screen | Before | After | |---|---|---| | In-app checkout form, cold launch | `WebView "[web-view] (no label)"` | full form in one call, 1.2 KB, 0.6 s | | 120-paragraph page | sentinel | the ten visible paragraphs, 1.8 KB (existing scroll clip bounds it) | | Wikipedia article in Chrome, helper attached after the page loaded | `FrameLayout "Web View"` | 4.7 KB, heading, links, lists, 1.4 s | | Settings home (native) | — | byte-identical to the previous parser | ### Testing - New `android-webview-describe.test.ts` on three live captures checked in as fixtures (in-app, Chrome, cold): merged landmark, `List` role, heading kept, re-read fires on the cold in-app shape and on Chrome's content-view placeholder, never on a native screen, and gives up within the budget. - `uiautomator-parser-v2-trim.test.ts`: the opaque-leaf case is replaced by the merge, the empty-WebView leaf, and a native `View` that must not be remapped. - Full tool-server suite, `tsc`, eslint, prettier, docs build. ### Docs No docs, skill or tool-description change. The page already names the accessibility tree as the source on Android; WebView content is now part of that tree and the agent needs no new instruction, since the cold read no longer produces an empty WebView. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01GZbkzeVC9faSxTFi4DbMuT <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Android WebView content is now captured and presented with meaningful controls, text, lists, and links. * Web content is consolidated into a clearer, unified accessibility view. * **Bug Fixes** * Improved Android screen descriptions when WebView content is initially unavailable by retrying the accessibility hierarchy. * Added safeguards for WebViews that never expose content, including a clear fallback message. * **Tests** * Expanded coverage for in-app WebViews, Chrome tabs, cold captures, and native Android UI filtering. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Hubert Gancarczyk <hubert.gancarczyk@swmansion.com>
1 line
15 KiB
XML
1 line
15 KiB
XML
<?xml version='1.0' encoding='UTF-8' standalone='yes' ?><hierarchy rotation="0"><node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,142]"><node index="0" text="" resource-id="com.android.systemui:id/status_bar_launch_animation_container" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,142]" /><node index="1" text="" resource-id="com.android.systemui:id/status_bar_container" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,142]"><node index="0" text="" resource-id="com.android.systemui:id/status_bar" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,142]"><node index="0" text="" resource-id="com.android.systemui:id/status_bar_contents" class="android.widget.LinearLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1027,142]"><node index="0" text="" resource-id="com.android.systemui:id/status_bar_start_side_container" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[11,0][458,142]"><node index="0" text="" resource-id="com.android.systemui:id/status_bar_start_side_content" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[11,0][252,142]"><node index="0" text="" resource-id="com.android.systemui:id/status_bar_start_side_except_heads_up" class="android.widget.LinearLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[11,0][252,142]"><node index="0" text="12:07" resource-id="com.android.systemui:id/clock" class="android.widget.TextView" package="com.android.systemui" content-desc="12:07 PM" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[11,49][136,92]" /><node index="1" text="" resource-id="com.android.systemui:id/notification_icon_area" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[136,0][252,142]"><node index="0" text="" resource-id="com.android.systemui:id/notificationIcons" class="android.view.ViewGroup" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[136,0][252,142]"><node index="0" text="" resource-id="" class="android.widget.ImageView" package="com.android.systemui" content-desc="Security & privacy notification: Set a screen lock" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[136,0][194,142]" /><node index="1" text="" resource-id="" class="android.widget.ImageView" package="com.android.systemui" content-desc="Android System notification: Serial console enabled" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[194,0][252,142]" /></node></node></node></node></node><node index="1" text="" resource-id="com.android.systemui:id/cutout_space_view" class="android.view.View" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[458,0][568,142]" /><node index="2" text="" resource-id="com.android.systemui:id/status_bar_end_side_container" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[568,0][1016,142]"><node index="0" text="" resource-id="com.android.systemui:id/status_bar_end_side_content" class="android.widget.LinearLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[876,0][1016,142]"><node index="0" text="" resource-id="com.android.systemui:id/system_icons" class="android.widget.LinearLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[876,42][1016,100]"><node index="0" text="" resource-id="com.android.systemui:id/statusIcons" class="android.widget.LinearLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[884,42][985,100]"><node index="0" text="" resource-id="com.android.systemui:id/wifi_combo" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[884,42][930,100]"><node index="0" text="" resource-id="com.android.systemui:id/wifi_group" class="android.widget.LinearLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[891,42][930,100]"><node index="0" text="" resource-id="com.android.systemui:id/wifi_combo" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[891,51][930,90]"><node index="0" text="" resource-id="com.android.systemui:id/wifi_signal" class="android.widget.ImageView" package="com.android.systemui" content-desc="Wifi signal full." checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[891,51][930,90]" /></node></node></node><node index="1" text="" resource-id="" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[884,71][884,71]" /><node index="2" text="" resource-id="com.android.systemui:id/mobile_combo" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="Phone three bars." checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[930,42][969,100]"><node index="0" text="" resource-id="com.android.systemui:id/mobile_group" class="android.widget.LinearLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[930,42][969,100]"><node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[930,51][969,90]"><node index="0" text="" resource-id="com.android.systemui:id/mobile_signal" class="android.widget.ImageView" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[930,51][969,90]" /></node></node></node></node><node index="1" text="" resource-id="com.android.systemui:id/battery" class="android.widget.LinearLayout" package="com.android.systemui" content-desc="Battery 100 percent." checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[985,54][1005,88]"><node index="0" text="" resource-id="" class="android.widget.ImageView" package="com.android.systemui" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[985,54][1005,88]" /></node></node></node></node></node></node></node></node><node index="1" text="" resource-id="" class="android.widget.FrameLayout" package="com.android.htmlviewer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2424]"><node index="0" text="" resource-id="android:id/decor_content_parent" class="android.view.ViewGroup" package="com.android.htmlviewer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2424]"><node index="0" text="" resource-id="android:id/content" class="android.widget.FrameLayout" package="com.android.htmlviewer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2424]"><node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.android.htmlviewer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,289][1080,2361]"><node index="0" text="" resource-id="com.android.htmlviewer:id/webview" class="android.webkit.WebView" package="com.android.htmlviewer" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,289][1080,2361]" /></node></node><node index="1" text="" resource-id="android:id/action_bar_container" class="android.widget.FrameLayout" package="com.android.htmlviewer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,289]"><node index="0" text="" resource-id="android:id/action_bar" class="android.view.ViewGroup" package="com.android.htmlviewer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,142][1080,289]"><node index="0" text="" resource-id="" class="android.widget.ImageButton" package="com.android.htmlviewer" content-desc="Navigate up" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,142][147,289]" /><node index="1" text="WV Fixture" resource-id="" class="android.widget.TextView" package="com.android.htmlviewer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[189,180][443,251]" /><node index="2" text="" resource-id="" class="android.widget.LinearLayout" package="com.android.htmlviewer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[1080,142][1080,289]" /></node></node></node><node index="1" text="" resource-id="android:id/statusBarBackground" class="android.view.View" package="com.android.htmlviewer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,142]" /><node index="2" text="" resource-id="android:id/navigationBarBackground" class="android.view.View" package="com.android.htmlviewer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,2361][1080,2424]" /></node></hierarchy> |