Files
software-mansion__argent/packages/tool-server/test/android-rotated-uiautomator.test.ts
filip131311 172e4d3a64 fix(android): capture the screen the way it is actually oriented (#693)
Part of #609.

## What was wrong

On a rotated Android device `screenshot` returned a portrait-framed
image with the content lying sideways. `describe` frames and gesture
coordinates were **already upright** — uiautomator measures against the
rotated display — so the capture was the one surface out of step. An
agent reading that image saw geometry that disagreed with every
coordinate it was given.

Measured on a landscape Pixel_9 (API 36, `mCurrentOrientation=1`):

| | space |
|---|---|
| `describe` | upright — `"Search Settings"` centre `(0.530, 0.234)` |
| taps | upright — tapping describe's `"Display & touch"` centre opens
it |
| `screenshot` (default) | **portrait-native, sideways** |

## The fix

simulator-server already applies `rotation` correctly; nothing was
telling it which one. The rotation is now queried from the device per
capture, so `screenshot`, `screenshot-diff`'s live captures and the
auto-attached "Screen after action" screenshot all land in
`describe`/tap space.

**Nothing is tracked.** A cached orientation could only be staler than
the ~40 ms probe, and serving a stale one is exactly the failure being
fixed. When the rotation cannot be read the request is left
byte-identical to before rather than guessing — a wrong orientation is a
confidently wrong image, which is worse than a sideways one.

**iOS is deliberately untouched.** There the whole surface is
consistently portrait-native, so rotating only the capture would *break*
an agreement rather than restore one. The iOS half of #609 needs a
different change and is not in this PR.

### Also fixed: the legacy uiautomator describe path

Same root cause, worse symptom. `wm size` reports the **unrotated** size
— on the landscape Pixel_9 it still answered `Physical size: 1080x2424`.
Dividing rotated bounds by that does not merely squash frames:
`isVisibleRect` prunes nodes whose left edge is past the screen width,
so **the right-hand half of a landscape screen vanished from the tree**.
The dump carries `<hierarchy rotation="N">`, so the divisor is oriented
from bytes already in hand — no extra round-trip on the path that runs
precisely when the helper could not be reached.

## On the rotation-name table

Pinned by measurement, never by what the names mean — the reporter of
#609 found the same names are 180° inverted on iOS. `adb exec-out
screencap` is rotation-aware, so it is ground truth; each candidate was
scored as mean absolute difference against it both as-is and rotated
180°, so an inverted mapping could not pass:

```
rotation 1  LandscapeLeft       2.19  (vs 6.54 for its 180° twin)
rotation 3  LandscapeRight      2.19  (vs 6.34)
rotation 2  PortraitUpsideDown  2.67  (vs 13.13)
rotation 0  Portrait            identity
```

This table is the **inverse** of simulator-server's own `90° →
LandscapeRight` convention, because it compensates for a rotation that
layer applies while decoding the scrcpy stream. That makes it a property
of a particular simulator-server build rather than of Android — which is
why a table test alone is not enough, and the delivered PNG's aspect is
checked against the rotation requested, falling back to an unrotated
capture if they disagree.

## Verified live

Branch tool-server against a real emulator:

- landscape, default capture → **2424×1080 upright**, MAD 2.19 vs ground
truth (6.54 vs its 180° twin)
- cropping `describe`'s frame for the toolbar out of that default
capture shows exactly the "Display & touch" toolbar — the two spaces
agree
- rotation 0 → 1080×2424, MAD 2.82, unchanged
- rotated **iPad** → still 2064×2752 sideways, i.e. iOS provably
untouched

Plus 27 new unit tests, including one that asserts the legacy path
*drops* the right-hand node before the fix and keeps it after. Full
suite 3125 passing; lint, prettier, `typecheck:tests` and the
tool-description gate all clean.

## Not in scope

- The iOS half of #609.
- `rotate` on Android silently no-ops when auto-rotate is enabled (it
returned `{orientation:"LandscapeLeft"}` while `dumpsys` still read
`cur=1080x2424`). Filing separately.
- `screen-recording` still records sideways — same root cause, different
pipeline. Worth noting a session's screenshots are now upright while its
video is not.
- Existing visual baselines captured on a rotated Android device will
stop matching, since the baseline key embeds capture dimensions and
those now swap. This fails loudly as a missing baseline rather than
silently comparing the wrong thing — and those baselines were sideways
captures of an upright UI.

---

> **#673 and #658 are both stacked on this branch** — each conflicts
with this one (`screenshot/index.ts` and `screenshot-diff/index.ts`
respectively) but not with each other, so both were rebased onto this
branch as siblings rather than chained. Merge this first; GitHub then
retargets both to `main`.

---------

Co-authored-by: Filip131311 <f.kaminski2000@gmail.com>
2026-08-21 17:34:59 +02:00

123 lines
6.1 KiB
TypeScript

import { describe, it, expect } from "vitest";
import { orientScreenSize, parseDumpRotation } from "../src/utils/android-screen";
import { parseUiAutomatorDump } from "../src/tools/describe/platforms/android/uiautomator-parser";
/**
* Issue #609, legacy describe path. `wm size` reports the UNROTATED size —
* measured on a landscape Pixel_9 (API 36) whose display was really 2424x1080,
* `wm size` still answered "Physical size: 1080x2424" with no Override line.
*
* uiautomator, by contrast, reports bounds against the rotated display. Dividing
* one by the other did not merely squash the frames: nodes whose left edge lies
* past the (too small) screen width are treated as off-screen and pruned, so the
* right-hand half of a landscape screen vanished from the tree entirely.
*
* The dump states its own rotation, so the divisor can be oriented with no extra
* round-trip — which matters because this path runs exactly when the
* android-devtools helper could not be reached.
*/
/** Shaped like a real landscape dump: 2424x1080, with content past x=1080. */
const LANDSCAPE_DUMP = `<?xml version='1.0' encoding='UTF-8' standalone='yes' ?>
<hierarchy rotation="1">
<node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.example.app" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][2424,1080]">
<node index="0" text="Left edge" resource-id="id/left" class="android.widget.TextView" package="com.example.app" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[24,100][500,200]" />
<node index="1" text="Right edge" resource-id="id/right" class="android.widget.TextView" package="com.example.app" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[1900,100][2380,200]" />
</node>
</hierarchy>`;
const PORTRAIT_DUMP = LANDSCAPE_DUMP.replace('rotation="1"', 'rotation="0"');
/** Every label anywhere in the tree. uiautomator `text` surfaces as `label`. */
function labels(node: unknown): string[] {
const out: string[] = [];
const walk = (n: Record<string, unknown>) => {
if (typeof n.label === "string" && n.label) out.push(n.label);
for (const c of (n.children as Record<string, unknown>[] | undefined) ?? []) walk(c);
};
walk(node as Record<string, unknown>);
return out;
}
function findFrame(node: unknown, text: string): { x: number; width: number } | undefined {
let found: { x: number; width: number } | undefined;
const walk = (n: Record<string, unknown>) => {
if (n.label === text && n.frame) found = n.frame as { x: number; width: number };
for (const c of (n.children as Record<string, unknown>[] | undefined) ?? []) walk(c);
};
walk(node as Record<string, unknown>);
return found;
}
describe("parseDumpRotation", () => {
it("reads the rotation the dump was taken at", () => {
expect(parseDumpRotation(LANDSCAPE_DUMP)).toBe(1);
expect(parseDumpRotation(PORTRAIT_DUMP)).toBe(0);
});
it("reads every rotation the platform can report", () => {
for (const r of [0, 1, 2, 3]) {
expect(parseDumpRotation(`<hierarchy rotation="${r}"><node /></hierarchy>`)).toBe(r);
}
});
it("returns null when the attribute is absent, so older dumps still work", () => {
expect(parseDumpRotation("<hierarchy><node /></hierarchy>")).toBeNull();
});
it("returns null for an unparseable value rather than guessing", () => {
expect(parseDumpRotation('<hierarchy rotation="x"><node /></hierarchy>')).toBeNull();
});
});
describe("orientScreenSize", () => {
const portrait = { width: 1080, height: 2424 };
it("swaps the axes when the device is on its side", () => {
expect(orientScreenSize(portrait, 1)).toEqual({ width: 2424, height: 1080 });
expect(orientScreenSize(portrait, 3)).toEqual({ width: 2424, height: 1080 });
});
it("leaves an upright or upside-down device alone", () => {
expect(orientScreenSize(portrait, 0)).toEqual(portrait);
expect(orientScreenSize(portrait, 2)).toEqual(portrait);
});
it("leaves the size alone when the rotation is unknown", () => {
expect(orientScreenSize(portrait, null)).toEqual(portrait);
});
});
describe("a rotated dump normalized against the oriented size", () => {
const wmSize = { width: 1080, height: 2424 }; // what `wm size` really answers
it("drops the right-hand half of the screen when the size is not oriented", () => {
// This is the bug, stated as a test: with the unrotated divisor the node at
// x=1900 is past screenW=1080 and is pruned as off-screen, so it is missing
// from the tree entirely — not merely clamped.
const tree = parseUiAutomatorDump(LANDSCAPE_DUMP, wmSize.width, wmSize.height);
expect(labels(tree)).toContain("Left edge");
expect(labels(tree)).not.toContain("Right edge");
});
it("keeps both edges once the size is oriented by the dump's own rotation", () => {
const oriented = orientScreenSize(wmSize, parseDumpRotation(LANDSCAPE_DUMP));
const tree = parseUiAutomatorDump(LANDSCAPE_DUMP, oriented.width, oriented.height);
expect(labels(tree)).toEqual(expect.arrayContaining(["Left edge", "Right edge"]));
});
it("puts the right-hand node where it actually is on screen", () => {
const oriented = orientScreenSize(wmSize, parseDumpRotation(LANDSCAPE_DUMP));
const tree = parseUiAutomatorDump(LANDSCAPE_DUMP, oriented.width, oriented.height);
const frame = findFrame(tree, "Right edge")!;
// 1900/2424 ≈ 0.784 — on the right-hand side, which is the whole point.
expect(frame.x).toBeCloseTo(1900 / 2424, 2);
expect(frame.width).toBeCloseTo(480 / 2424, 2);
});
it("leaves an unrotated dump exactly as it was", () => {
const oriented = orientScreenSize(wmSize, parseDumpRotation(PORTRAIT_DUMP));
expect(oriented).toEqual(wmSize);
});
});