mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
aa90873b59
Android `describe` rendered an `android.webkit.WebView` as a single opaque leaf labelled `[web-view]` and threw the subtree away. Chromium publishes the web DOM to the accessibility tree on both capture paths (the android-devtools helper and the `uiautomator dump` fallback), so the sentinel hid content the agent can act on. This keeps the WebView as a landmark and lets the normal trim run over the DOM beneath it. Supersedes #955, which reached the same diagnosis. The parser rules for the doubled in-app node, the web list clip and the text-run role are carried over from it; the rendered-output cap, the `truncated` flag plumbing and the flow-tree parity rework are not, so this stays confined to the Android parser and adapter (about 120 source lines) with no renderer, contract, skill or tool-description changes. ### What changes inside a WebView, and only there - **Doubled node merged.** An in-app WebView dumps as a nested pair: the app's view (resource-id, gesture flags), then Chromium's root web area under the same class carrying the page `<title>` as text. The inner half folds into the outer node, so the tree reads `WebView "<title>"`. A Chrome tab arrives as one node and is untouched. - **Web list clip.** A `<ul>` arrives as `android.widget.ListView` with `scrollable="false"`. Inside a WebView only the framework flag decides what clips, so a list never drops the items laid out around it, and the role is `List` rather than `ScrollView`. - **Text runs.** A labelled leaf `android.view.View` with no gesture flag is reported as `StaticText`. Outside a WebView a bare `View` keeps its role (Compose semantics nodes). - **Headings survive.** The WebView branch returns before the label dedupe, so an `h1` that repeats the page title is kept. ### The cold-read race is fixed, not documented Chromium builds a WebView's accessibility tree on the first request for it. Measured on API 35 / WebView 124: a read taken 6 s after page load still shows the WebView with no children; the page appears only after a read has asked for it — 10–400 ms later for a small in-app page, ~600 ms later for a Chrome tab over a long article. `describeAndroid` re-reads in 250 ms steps within a 1.5 s budget while the tree still has the unread shape, on both paths. A screen without a WebView never pays for it (one call, pinned by test). A WebView that stays empty after the budget is labelled `(no web content exposed)`. The unread shape is an `android.webkit.WebView` with no children, or — for a browser tab, which has no WebView view of its own before the page is published — Chrome's childless content view labelled "Web View" (English UI only; other locales see the tab the way they do today). ### Measured on a Pixel 9 / API 35 emulator with the branch tool-server | Screen | Before | After | |---|---|---| | In-app checkout form, cold launch | `WebView "[web-view] (no label)"` | full form in one call, 1.2 KB, 0.6 s | | 120-paragraph page | sentinel | the ten visible paragraphs, 1.8 KB (existing scroll clip bounds it) | | Wikipedia article in Chrome, helper attached after the page loaded | `FrameLayout "Web View"` | 4.7 KB, heading, links, lists, 1.4 s | | Settings home (native) | — | byte-identical to the previous parser | ### Testing - New `android-webview-describe.test.ts` on three live captures checked in as fixtures (in-app, Chrome, cold): merged landmark, `List` role, heading kept, re-read fires on the cold in-app shape and on Chrome's content-view placeholder, never on a native screen, and gives up within the budget. - `uiautomator-parser-v2-trim.test.ts`: the opaque-leaf case is replaced by the merge, the empty-WebView leaf, and a native `View` that must not be remapped. - Full tool-server suite, `tsc`, eslint, prettier, docs build. ### Docs No docs, skill or tool-description change. The page already names the accessibility tree as the source on Android; WebView content is now part of that tree and the agent needs no new instruction, since the cold read no longer produces an empty WebView. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01GZbkzeVC9faSxTFi4DbMuT <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Android WebView content is now captured and presented with meaningful controls, text, lists, and links. * Web content is consolidated into a clearer, unified accessibility view. * **Bug Fixes** * Improved Android screen descriptions when WebView content is initially unavailable by retrying the accessibility hierarchy. * Added safeguards for WebViews that never expose content, including a clear fallback message. * **Tests** * Expanded coverage for in-app WebViews, Chrome tabs, cold captures, and native Android UI filtering. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Hubert Gancarczyk <hubert.gancarczyk@swmansion.com>
121 lines
5.9 KiB
TypeScript
121 lines
5.9 KiB
TypeScript
// Android WebView coverage on real captures (API 35 / WebView 124, 1080x2424):
|
|
// an in-app WebView (HtmlViewer), a Chrome tab, and a cold capture taken before
|
|
// Chromium published the page. The android-devtools helper and the
|
|
// `uiautomator dump` fallback emit the same XML shape, so one parser pass
|
|
// covers both; `describeAndroid` adds the one re-read the cold shape needs.
|
|
import { readFileSync } from "node:fs";
|
|
import { join } from "node:path";
|
|
import { describe, it, expect, vi } from "vitest";
|
|
import type { Registry } from "@argent/registry";
|
|
import { parseUiAutomatorDump } from "../src/tools/describe/platforms/android/uiautomator-parser";
|
|
import { describeAndroid, hasUnreadWebView } from "../src/tools/describe/platforms/android";
|
|
import { formatDescribeTree } from "../src/tools/describe/format-tree";
|
|
import type { DescribeNode } from "../src/tools/describe/contract";
|
|
|
|
const fixture = (name: string): string =>
|
|
readFileSync(join(__dirname, "fixtures", `android-webview-${name}.xml`), "utf-8");
|
|
|
|
function flatten(tree: DescribeNode): DescribeNode[] {
|
|
const out: DescribeNode[] = [];
|
|
const stack: DescribeNode[] = [tree];
|
|
while (stack.length > 0) {
|
|
const n = stack.pop()!;
|
|
out.push(n);
|
|
for (let i = n.children.length - 1; i >= 0; i--) stack.push(n.children[i]!);
|
|
}
|
|
return out;
|
|
}
|
|
|
|
describe("parseUiAutomatorDump — WebView captures", () => {
|
|
it("lists the form inside an in-app WebView as one merged landmark", () => {
|
|
const tree = parseUiAutomatorDump(fixture("inapp"), 1080, 2424);
|
|
const webviews = flatten(tree).filter((n) => n.role === "WebView");
|
|
expect(webviews).toHaveLength(1);
|
|
const labels = flatten(webviews[0]!).map((n) => `${n.role} ${n.label ?? ""}`.trim());
|
|
expect(labels).toContain("Button Pay now");
|
|
expect(labels).toContain("TextField");
|
|
expect(labels.some((l) => l.startsWith("StaticText"))).toBe(true);
|
|
// A web `<ul>` is a List, never a scroll clip.
|
|
expect(labels).toContain("List");
|
|
expect(labels.some((l) => l.startsWith("ScrollView"))).toBe(false);
|
|
});
|
|
|
|
it("keeps a heading that repeats the page title in a Chrome tab", () => {
|
|
const tree = parseUiAutomatorDump(fixture("chrome"), 1080, 2424);
|
|
const webview = flatten(tree).find((n) => n.role === "WebView");
|
|
expect(webview?.label).toMatch(/Wikipedia/);
|
|
const rendered = formatDescribeTree(tree, { source: "android-devtools" });
|
|
expect(rendered).toContain('StaticText "Android version history"');
|
|
expect(rendered).toContain("[clickable]");
|
|
});
|
|
|
|
it("renders the cold capture as a leaf the adapter re-reads on", () => {
|
|
const tree = parseUiAutomatorDump(fixture("cold"), 1080, 2424);
|
|
expect(hasUnreadWebView(tree)).toBe(true);
|
|
expect(hasUnreadWebView(parseUiAutomatorDump(fixture("inapp"), 1080, 2424))).toBe(false);
|
|
});
|
|
|
|
it("does not surface the native chrome text through the WebView branch", () => {
|
|
// Status-bar nodes stay filtered exactly as before.
|
|
const rendered = formatDescribeTree(parseUiAutomatorDump(fixture("inapp"), 1080, 2424), {
|
|
source: "android-devtools",
|
|
});
|
|
expect(rendered).not.toContain("Battery");
|
|
});
|
|
});
|
|
|
|
describe("describeAndroid — re-read on an unread WebView", () => {
|
|
function fakeRegistry(hierarchies: string[]) {
|
|
// Serve the captures in order and repeat the last one.
|
|
const getHierarchy = vi.fn(async () => ({
|
|
xml: hierarchies.length > 1 ? hierarchies.shift()! : hierarchies[0]!,
|
|
}));
|
|
const resolveService = vi.fn(async () => ({
|
|
isReady: () => true,
|
|
getHierarchy,
|
|
getScreenSize: async () => ({ width: 1080, height: 2424, rotation: 0 }),
|
|
}));
|
|
return { registry: { resolveService } as unknown as Registry, getHierarchy };
|
|
}
|
|
|
|
it("reads once more when the first tree holds an empty WebView", async () => {
|
|
const { registry, getHierarchy } = fakeRegistry([fixture("cold"), fixture("inapp")]);
|
|
const { tree, source } = await describeAndroid(registry, "emulator-5554", undefined, false);
|
|
expect(source).toBe("android-devtools");
|
|
expect(getHierarchy).toHaveBeenCalledTimes(2);
|
|
expect(flatten(tree).some((n) => n.label === "Pay now")).toBe(true);
|
|
});
|
|
|
|
it("reads once more on Chrome's unpublished content view", async () => {
|
|
// A browser tab before the page is published: no `android.webkit.WebView`
|
|
// node yet, just Chrome's childless content view.
|
|
const cold = `<?xml version='1.0'?><hierarchy rotation="0">
|
|
<node class="android.widget.FrameLayout" content-desc="Web View" bounds="[0,142][1080,2361]"/>
|
|
<node class="android.widget.EditText" resource-id="com.android.chrome:id/url_bar" text="example.org" clickable="true" bounds="[210,150][690,280]"/>
|
|
</hierarchy>`;
|
|
const { registry, getHierarchy } = fakeRegistry([cold, fixture("chrome")]);
|
|
const { tree } = await describeAndroid(registry, "emulator-5554", undefined, false);
|
|
expect(getHierarchy).toHaveBeenCalledTimes(2);
|
|
expect(flatten(tree).find((n) => n.role === "WebView")?.label).toMatch(/Wikipedia/);
|
|
});
|
|
|
|
it("reads exactly once on a screen without a WebView", async () => {
|
|
const native = `<?xml version='1.0'?><hierarchy rotation="0">
|
|
<node class="android.widget.Button" text="Settings" clickable="true" bounds="[0,0][200,100]"/>
|
|
</hierarchy>`;
|
|
const { registry, getHierarchy } = fakeRegistry([native]);
|
|
await describeAndroid(registry, "emulator-5554", undefined, false);
|
|
expect(getHierarchy).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("gives up within the budget when the WebView never publishes", async () => {
|
|
const { registry, getHierarchy } = fakeRegistry([fixture("cold")]);
|
|
const started = Date.now();
|
|
const { tree } = await describeAndroid(registry, "emulator-5554", undefined, false);
|
|
// 1.5 s budget in 250 ms steps: the first read plus six re-reads.
|
|
expect(getHierarchy).toHaveBeenCalledTimes(7);
|
|
expect(Date.now() - started).toBeLessThan(3_000);
|
|
expect(flatten(tree).find((n) => n.role === "WebView")?.label).toBe("(no web content exposed)");
|
|
});
|
|
});
|