Files
software-mansion__argent/packages/tool-server/test/android-webview-describe.test.ts
filip131311 aa90873b59 feat(android): read the web DOM inside a WebView in describe (#1052)
Android `describe` rendered an `android.webkit.WebView` as a single
opaque leaf labelled `[web-view]` and threw the subtree away. Chromium
publishes the web DOM to the accessibility tree on both capture paths
(the android-devtools helper and the `uiautomator dump` fallback), so
the sentinel hid content the agent can act on. This keeps the WebView as
a landmark and lets the normal trim run over the DOM beneath it.

Supersedes #955, which reached the same diagnosis. The parser rules for
the doubled in-app node, the web list clip and the text-run role are
carried over from it; the rendered-output cap, the `truncated` flag
plumbing and the flow-tree parity rework are not, so this stays confined
to the Android parser and adapter (about 120 source lines) with no
renderer, contract, skill or tool-description changes.

### What changes inside a WebView, and only there

- **Doubled node merged.** An in-app WebView dumps as a nested pair: the
app's view (resource-id, gesture flags), then Chromium's root web area
under the same class carrying the page `<title>` as text. The inner half
folds into the outer node, so the tree reads `WebView "<title>"`. A
Chrome tab arrives as one node and is untouched.
- **Web list clip.** A `<ul>` arrives as `android.widget.ListView` with
`scrollable="false"`. Inside a WebView only the framework flag decides
what clips, so a list never drops the items laid out around it, and the
role is `List` rather than `ScrollView`.
- **Text runs.** A labelled leaf `android.view.View` with no gesture
flag is reported as `StaticText`. Outside a WebView a bare `View` keeps
its role (Compose semantics nodes).
- **Headings survive.** The WebView branch returns before the label
dedupe, so an `h1` that repeats the page title is kept.

### The cold-read race is fixed, not documented

Chromium builds a WebView's accessibility tree on the first request for
it. Measured on API 35 / WebView 124: a read taken 6 s after page load
still shows the WebView with no children; the page appears only after a
read has asked for it — 10–400 ms later for a small in-app page, ~600 ms
later for a Chrome tab over a long article. `describeAndroid` re-reads
in 250 ms steps within a 1.5 s budget while the tree still has the
unread shape, on both paths. A screen without a WebView never pays for
it (one call, pinned by test). A WebView that stays empty after the
budget is labelled `(no web content exposed)`.

The unread shape is an `android.webkit.WebView` with no children, or —
for a browser tab, which has no WebView view of its own before the page
is published — Chrome's childless content view labelled "Web View"
(English UI only; other locales see the tab the way they do today).

### Measured on a Pixel 9 / API 35 emulator with the branch tool-server

| Screen | Before | After |
|---|---|---|
| In-app checkout form, cold launch | `WebView "[web-view] (no label)"`
| full form in one call, 1.2 KB, 0.6 s |
| 120-paragraph page | sentinel | the ten visible paragraphs, 1.8 KB
(existing scroll clip bounds it) |
| Wikipedia article in Chrome, helper attached after the page loaded |
`FrameLayout "Web View"` | 4.7 KB, heading, links, lists, 1.4 s |
| Settings home (native) | — | byte-identical to the previous parser |

### Testing

- New `android-webview-describe.test.ts` on three live captures checked
in as fixtures (in-app, Chrome, cold): merged landmark, `List` role,
heading kept, re-read fires on the cold in-app shape and on Chrome's
content-view placeholder, never on a native screen, and gives up within
the budget.
- `uiautomator-parser-v2-trim.test.ts`: the opaque-leaf case is replaced
by the merge, the empty-WebView leaf, and a native `View` that must not
be remapped.
- Full tool-server suite, `tsc`, eslint, prettier, docs build.

### Docs

No docs, skill or tool-description change. The page already names the
accessibility tree as the source on Android; WebView content is now part
of that tree and the agent needs no new instruction, since the cold read
no longer produces an empty WebView.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01GZbkzeVC9faSxTFi4DbMuT


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Android WebView content is now captured and presented with meaningful
controls, text, lists, and links.
* Web content is consolidated into a clearer, unified accessibility
view.
* **Bug Fixes**
* Improved Android screen descriptions when WebView content is initially
unavailable by retrying the accessibility hierarchy.
* Added safeguards for WebViews that never expose content, including a
clear fallback message.
* **Tests**
* Expanded coverage for in-app WebViews, Chrome tabs, cold captures, and
native Android UI filtering.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Hubert Gancarczyk <hubert.gancarczyk@swmansion.com>
2026-09-07 16:35:42 +02:00

121 lines
5.9 KiB
TypeScript

// Android WebView coverage on real captures (API 35 / WebView 124, 1080x2424):
// an in-app WebView (HtmlViewer), a Chrome tab, and a cold capture taken before
// Chromium published the page. The android-devtools helper and the
// `uiautomator dump` fallback emit the same XML shape, so one parser pass
// covers both; `describeAndroid` adds the one re-read the cold shape needs.
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { describe, it, expect, vi } from "vitest";
import type { Registry } from "@argent/registry";
import { parseUiAutomatorDump } from "../src/tools/describe/platforms/android/uiautomator-parser";
import { describeAndroid, hasUnreadWebView } from "../src/tools/describe/platforms/android";
import { formatDescribeTree } from "../src/tools/describe/format-tree";
import type { DescribeNode } from "../src/tools/describe/contract";
const fixture = (name: string): string =>
readFileSync(join(__dirname, "fixtures", `android-webview-${name}.xml`), "utf-8");
function flatten(tree: DescribeNode): DescribeNode[] {
const out: DescribeNode[] = [];
const stack: DescribeNode[] = [tree];
while (stack.length > 0) {
const n = stack.pop()!;
out.push(n);
for (let i = n.children.length - 1; i >= 0; i--) stack.push(n.children[i]!);
}
return out;
}
describe("parseUiAutomatorDump — WebView captures", () => {
it("lists the form inside an in-app WebView as one merged landmark", () => {
const tree = parseUiAutomatorDump(fixture("inapp"), 1080, 2424);
const webviews = flatten(tree).filter((n) => n.role === "WebView");
expect(webviews).toHaveLength(1);
const labels = flatten(webviews[0]!).map((n) => `${n.role} ${n.label ?? ""}`.trim());
expect(labels).toContain("Button Pay now");
expect(labels).toContain("TextField");
expect(labels.some((l) => l.startsWith("StaticText"))).toBe(true);
// A web `<ul>` is a List, never a scroll clip.
expect(labels).toContain("List");
expect(labels.some((l) => l.startsWith("ScrollView"))).toBe(false);
});
it("keeps a heading that repeats the page title in a Chrome tab", () => {
const tree = parseUiAutomatorDump(fixture("chrome"), 1080, 2424);
const webview = flatten(tree).find((n) => n.role === "WebView");
expect(webview?.label).toMatch(/Wikipedia/);
const rendered = formatDescribeTree(tree, { source: "android-devtools" });
expect(rendered).toContain('StaticText "Android version history"');
expect(rendered).toContain("[clickable]");
});
it("renders the cold capture as a leaf the adapter re-reads on", () => {
const tree = parseUiAutomatorDump(fixture("cold"), 1080, 2424);
expect(hasUnreadWebView(tree)).toBe(true);
expect(hasUnreadWebView(parseUiAutomatorDump(fixture("inapp"), 1080, 2424))).toBe(false);
});
it("does not surface the native chrome text through the WebView branch", () => {
// Status-bar nodes stay filtered exactly as before.
const rendered = formatDescribeTree(parseUiAutomatorDump(fixture("inapp"), 1080, 2424), {
source: "android-devtools",
});
expect(rendered).not.toContain("Battery");
});
});
describe("describeAndroid — re-read on an unread WebView", () => {
function fakeRegistry(hierarchies: string[]) {
// Serve the captures in order and repeat the last one.
const getHierarchy = vi.fn(async () => ({
xml: hierarchies.length > 1 ? hierarchies.shift()! : hierarchies[0]!,
}));
const resolveService = vi.fn(async () => ({
isReady: () => true,
getHierarchy,
getScreenSize: async () => ({ width: 1080, height: 2424, rotation: 0 }),
}));
return { registry: { resolveService } as unknown as Registry, getHierarchy };
}
it("reads once more when the first tree holds an empty WebView", async () => {
const { registry, getHierarchy } = fakeRegistry([fixture("cold"), fixture("inapp")]);
const { tree, source } = await describeAndroid(registry, "emulator-5554", undefined, false);
expect(source).toBe("android-devtools");
expect(getHierarchy).toHaveBeenCalledTimes(2);
expect(flatten(tree).some((n) => n.label === "Pay now")).toBe(true);
});
it("reads once more on Chrome's unpublished content view", async () => {
// A browser tab before the page is published: no `android.webkit.WebView`
// node yet, just Chrome's childless content view.
const cold = `<?xml version='1.0'?><hierarchy rotation="0">
<node class="android.widget.FrameLayout" content-desc="Web View" bounds="[0,142][1080,2361]"/>
<node class="android.widget.EditText" resource-id="com.android.chrome:id/url_bar" text="example.org" clickable="true" bounds="[210,150][690,280]"/>
</hierarchy>`;
const { registry, getHierarchy } = fakeRegistry([cold, fixture("chrome")]);
const { tree } = await describeAndroid(registry, "emulator-5554", undefined, false);
expect(getHierarchy).toHaveBeenCalledTimes(2);
expect(flatten(tree).find((n) => n.role === "WebView")?.label).toMatch(/Wikipedia/);
});
it("reads exactly once on a screen without a WebView", async () => {
const native = `<?xml version='1.0'?><hierarchy rotation="0">
<node class="android.widget.Button" text="Settings" clickable="true" bounds="[0,0][200,100]"/>
</hierarchy>`;
const { registry, getHierarchy } = fakeRegistry([native]);
await describeAndroid(registry, "emulator-5554", undefined, false);
expect(getHierarchy).toHaveBeenCalledTimes(1);
});
it("gives up within the budget when the WebView never publishes", async () => {
const { registry, getHierarchy } = fakeRegistry([fixture("cold")]);
const started = Date.now();
const { tree } = await describeAndroid(registry, "emulator-5554", undefined, false);
// 1.5 s budget in 250 ms steps: the first read plus six re-reads.
expect(getHierarchy).toHaveBeenCalledTimes(7);
expect(Date.now() - started).toBeLessThan(3_000);
expect(flatten(tree).find((n) => n.role === "WebView")?.label).toBe("(no web content exposed)");
});
});