mirror of
https://github.com/software-mansion/argent.git
synced 2026-09-14 19:27:14 +08:00
172e4d3a64
Part of #609. ## What was wrong On a rotated Android device `screenshot` returned a portrait-framed image with the content lying sideways. `describe` frames and gesture coordinates were **already upright** — uiautomator measures against the rotated display — so the capture was the one surface out of step. An agent reading that image saw geometry that disagreed with every coordinate it was given. Measured on a landscape Pixel_9 (API 36, `mCurrentOrientation=1`): | | space | |---|---| | `describe` | upright — `"Search Settings"` centre `(0.530, 0.234)` | | taps | upright — tapping describe's `"Display & touch"` centre opens it | | `screenshot` (default) | **portrait-native, sideways** | ## The fix simulator-server already applies `rotation` correctly; nothing was telling it which one. The rotation is now queried from the device per capture, so `screenshot`, `screenshot-diff`'s live captures and the auto-attached "Screen after action" screenshot all land in `describe`/tap space. **Nothing is tracked.** A cached orientation could only be staler than the ~40 ms probe, and serving a stale one is exactly the failure being fixed. When the rotation cannot be read the request is left byte-identical to before rather than guessing — a wrong orientation is a confidently wrong image, which is worse than a sideways one. **iOS is deliberately untouched.** There the whole surface is consistently portrait-native, so rotating only the capture would *break* an agreement rather than restore one. The iOS half of #609 needs a different change and is not in this PR. ### Also fixed: the legacy uiautomator describe path Same root cause, worse symptom. `wm size` reports the **unrotated** size — on the landscape Pixel_9 it still answered `Physical size: 1080x2424`. Dividing rotated bounds by that does not merely squash frames: `isVisibleRect` prunes nodes whose left edge is past the screen width, so **the right-hand half of a landscape screen vanished from the tree**. The dump carries `<hierarchy rotation="N">`, so the divisor is oriented from bytes already in hand — no extra round-trip on the path that runs precisely when the helper could not be reached. ## On the rotation-name table Pinned by measurement, never by what the names mean — the reporter of #609 found the same names are 180° inverted on iOS. `adb exec-out screencap` is rotation-aware, so it is ground truth; each candidate was scored as mean absolute difference against it both as-is and rotated 180°, so an inverted mapping could not pass: ``` rotation 1 LandscapeLeft 2.19 (vs 6.54 for its 180° twin) rotation 3 LandscapeRight 2.19 (vs 6.34) rotation 2 PortraitUpsideDown 2.67 (vs 13.13) rotation 0 Portrait identity ``` This table is the **inverse** of simulator-server's own `90° → LandscapeRight` convention, because it compensates for a rotation that layer applies while decoding the scrcpy stream. That makes it a property of a particular simulator-server build rather than of Android — which is why a table test alone is not enough, and the delivered PNG's aspect is checked against the rotation requested, falling back to an unrotated capture if they disagree. ## Verified live Branch tool-server against a real emulator: - landscape, default capture → **2424×1080 upright**, MAD 2.19 vs ground truth (6.54 vs its 180° twin) - cropping `describe`'s frame for the toolbar out of that default capture shows exactly the "Display & touch" toolbar — the two spaces agree - rotation 0 → 1080×2424, MAD 2.82, unchanged - rotated **iPad** → still 2064×2752 sideways, i.e. iOS provably untouched Plus 27 new unit tests, including one that asserts the legacy path *drops* the right-hand node before the fix and keeps it after. Full suite 3125 passing; lint, prettier, `typecheck:tests` and the tool-description gate all clean. ## Not in scope - The iOS half of #609. - `rotate` on Android silently no-ops when auto-rotate is enabled (it returned `{orientation:"LandscapeLeft"}` while `dumpsys` still read `cur=1080x2424`). Filing separately. - `screen-recording` still records sideways — same root cause, different pipeline. Worth noting a session's screenshots are now upright while its video is not. - Existing visual baselines captured on a rotated Android device will stop matching, since the baseline key embeds capture dimensions and those now swap. This fails loudly as a missing baseline rather than silently comparing the wrong thing — and those baselines were sideways captures of an upright UI. --- > **#673 and #658 are both stacked on this branch** — each conflicts with this one (`screenshot/index.ts` and `screenshot-diff/index.ts` respectively) but not with each other, so both were rebased onto this branch as siblings rather than chained. Merge this first; GitHub then retargets both to `main`. --------- Co-authored-by: Filip131311 <f.kaminski2000@gmail.com>
123 lines
6.1 KiB
TypeScript
123 lines
6.1 KiB
TypeScript
import { describe, it, expect } from "vitest";
|
|
import { orientScreenSize, parseDumpRotation } from "../src/utils/android-screen";
|
|
import { parseUiAutomatorDump } from "../src/tools/describe/platforms/android/uiautomator-parser";
|
|
|
|
/**
|
|
* Issue #609, legacy describe path. `wm size` reports the UNROTATED size —
|
|
* measured on a landscape Pixel_9 (API 36) whose display was really 2424x1080,
|
|
* `wm size` still answered "Physical size: 1080x2424" with no Override line.
|
|
*
|
|
* uiautomator, by contrast, reports bounds against the rotated display. Dividing
|
|
* one by the other did not merely squash the frames: nodes whose left edge lies
|
|
* past the (too small) screen width are treated as off-screen and pruned, so the
|
|
* right-hand half of a landscape screen vanished from the tree entirely.
|
|
*
|
|
* The dump states its own rotation, so the divisor can be oriented with no extra
|
|
* round-trip — which matters because this path runs exactly when the
|
|
* android-devtools helper could not be reached.
|
|
*/
|
|
|
|
/** Shaped like a real landscape dump: 2424x1080, with content past x=1080. */
|
|
const LANDSCAPE_DUMP = `<?xml version='1.0' encoding='UTF-8' standalone='yes' ?>
|
|
<hierarchy rotation="1">
|
|
<node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.example.app" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][2424,1080]">
|
|
<node index="0" text="Left edge" resource-id="id/left" class="android.widget.TextView" package="com.example.app" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[24,100][500,200]" />
|
|
<node index="1" text="Right edge" resource-id="id/right" class="android.widget.TextView" package="com.example.app" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[1900,100][2380,200]" />
|
|
</node>
|
|
</hierarchy>`;
|
|
|
|
const PORTRAIT_DUMP = LANDSCAPE_DUMP.replace('rotation="1"', 'rotation="0"');
|
|
|
|
/** Every label anywhere in the tree. uiautomator `text` surfaces as `label`. */
|
|
function labels(node: unknown): string[] {
|
|
const out: string[] = [];
|
|
const walk = (n: Record<string, unknown>) => {
|
|
if (typeof n.label === "string" && n.label) out.push(n.label);
|
|
for (const c of (n.children as Record<string, unknown>[] | undefined) ?? []) walk(c);
|
|
};
|
|
walk(node as Record<string, unknown>);
|
|
return out;
|
|
}
|
|
|
|
function findFrame(node: unknown, text: string): { x: number; width: number } | undefined {
|
|
let found: { x: number; width: number } | undefined;
|
|
const walk = (n: Record<string, unknown>) => {
|
|
if (n.label === text && n.frame) found = n.frame as { x: number; width: number };
|
|
for (const c of (n.children as Record<string, unknown>[] | undefined) ?? []) walk(c);
|
|
};
|
|
walk(node as Record<string, unknown>);
|
|
return found;
|
|
}
|
|
|
|
describe("parseDumpRotation", () => {
|
|
it("reads the rotation the dump was taken at", () => {
|
|
expect(parseDumpRotation(LANDSCAPE_DUMP)).toBe(1);
|
|
expect(parseDumpRotation(PORTRAIT_DUMP)).toBe(0);
|
|
});
|
|
|
|
it("reads every rotation the platform can report", () => {
|
|
for (const r of [0, 1, 2, 3]) {
|
|
expect(parseDumpRotation(`<hierarchy rotation="${r}"><node /></hierarchy>`)).toBe(r);
|
|
}
|
|
});
|
|
|
|
it("returns null when the attribute is absent, so older dumps still work", () => {
|
|
expect(parseDumpRotation("<hierarchy><node /></hierarchy>")).toBeNull();
|
|
});
|
|
|
|
it("returns null for an unparseable value rather than guessing", () => {
|
|
expect(parseDumpRotation('<hierarchy rotation="x"><node /></hierarchy>')).toBeNull();
|
|
});
|
|
});
|
|
|
|
describe("orientScreenSize", () => {
|
|
const portrait = { width: 1080, height: 2424 };
|
|
|
|
it("swaps the axes when the device is on its side", () => {
|
|
expect(orientScreenSize(portrait, 1)).toEqual({ width: 2424, height: 1080 });
|
|
expect(orientScreenSize(portrait, 3)).toEqual({ width: 2424, height: 1080 });
|
|
});
|
|
|
|
it("leaves an upright or upside-down device alone", () => {
|
|
expect(orientScreenSize(portrait, 0)).toEqual(portrait);
|
|
expect(orientScreenSize(portrait, 2)).toEqual(portrait);
|
|
});
|
|
|
|
it("leaves the size alone when the rotation is unknown", () => {
|
|
expect(orientScreenSize(portrait, null)).toEqual(portrait);
|
|
});
|
|
});
|
|
|
|
describe("a rotated dump normalized against the oriented size", () => {
|
|
const wmSize = { width: 1080, height: 2424 }; // what `wm size` really answers
|
|
|
|
it("drops the right-hand half of the screen when the size is not oriented", () => {
|
|
// This is the bug, stated as a test: with the unrotated divisor the node at
|
|
// x=1900 is past screenW=1080 and is pruned as off-screen, so it is missing
|
|
// from the tree entirely — not merely clamped.
|
|
const tree = parseUiAutomatorDump(LANDSCAPE_DUMP, wmSize.width, wmSize.height);
|
|
expect(labels(tree)).toContain("Left edge");
|
|
expect(labels(tree)).not.toContain("Right edge");
|
|
});
|
|
|
|
it("keeps both edges once the size is oriented by the dump's own rotation", () => {
|
|
const oriented = orientScreenSize(wmSize, parseDumpRotation(LANDSCAPE_DUMP));
|
|
const tree = parseUiAutomatorDump(LANDSCAPE_DUMP, oriented.width, oriented.height);
|
|
expect(labels(tree)).toEqual(expect.arrayContaining(["Left edge", "Right edge"]));
|
|
});
|
|
|
|
it("puts the right-hand node where it actually is on screen", () => {
|
|
const oriented = orientScreenSize(wmSize, parseDumpRotation(LANDSCAPE_DUMP));
|
|
const tree = parseUiAutomatorDump(LANDSCAPE_DUMP, oriented.width, oriented.height);
|
|
const frame = findFrame(tree, "Right edge")!;
|
|
// 1900/2424 ≈ 0.784 — on the right-hand side, which is the whole point.
|
|
expect(frame.x).toBeCloseTo(1900 / 2424, 2);
|
|
expect(frame.width).toBeCloseTo(480 / 2424, 2);
|
|
});
|
|
|
|
it("leaves an unrotated dump exactly as it was", () => {
|
|
const oriented = orientScreenSize(wmSize, parseDumpRotation(PORTRAIT_DUMP));
|
|
expect(oriented).toEqual(wmSize);
|
|
});
|
|
});
|