Files
software-mansion__argent/packages/tool-server/test/device-orientation.test.ts
filip131311 172e4d3a64 fix(android): capture the screen the way it is actually oriented (#693)
Part of #609.

## What was wrong

On a rotated Android device `screenshot` returned a portrait-framed
image with the content lying sideways. `describe` frames and gesture
coordinates were **already upright** — uiautomator measures against the
rotated display — so the capture was the one surface out of step. An
agent reading that image saw geometry that disagreed with every
coordinate it was given.

Measured on a landscape Pixel_9 (API 36, `mCurrentOrientation=1`):

| | space |
|---|---|
| `describe` | upright — `"Search Settings"` centre `(0.530, 0.234)` |
| taps | upright — tapping describe's `"Display & touch"` centre opens
it |
| `screenshot` (default) | **portrait-native, sideways** |

## The fix

simulator-server already applies `rotation` correctly; nothing was
telling it which one. The rotation is now queried from the device per
capture, so `screenshot`, `screenshot-diff`'s live captures and the
auto-attached "Screen after action" screenshot all land in
`describe`/tap space.

**Nothing is tracked.** A cached orientation could only be staler than
the ~40 ms probe, and serving a stale one is exactly the failure being
fixed. When the rotation cannot be read the request is left
byte-identical to before rather than guessing — a wrong orientation is a
confidently wrong image, which is worse than a sideways one.

**iOS is deliberately untouched.** There the whole surface is
consistently portrait-native, so rotating only the capture would *break*
an agreement rather than restore one. The iOS half of #609 needs a
different change and is not in this PR.

### Also fixed: the legacy uiautomator describe path

Same root cause, worse symptom. `wm size` reports the **unrotated** size
— on the landscape Pixel_9 it still answered `Physical size: 1080x2424`.
Dividing rotated bounds by that does not merely squash frames:
`isVisibleRect` prunes nodes whose left edge is past the screen width,
so **the right-hand half of a landscape screen vanished from the tree**.
The dump carries `<hierarchy rotation="N">`, so the divisor is oriented
from bytes already in hand — no extra round-trip on the path that runs
precisely when the helper could not be reached.

## On the rotation-name table

Pinned by measurement, never by what the names mean — the reporter of
#609 found the same names are 180° inverted on iOS. `adb exec-out
screencap` is rotation-aware, so it is ground truth; each candidate was
scored as mean absolute difference against it both as-is and rotated
180°, so an inverted mapping could not pass:

```
rotation 1  LandscapeLeft       2.19  (vs 6.54 for its 180° twin)
rotation 3  LandscapeRight      2.19  (vs 6.34)
rotation 2  PortraitUpsideDown  2.67  (vs 13.13)
rotation 0  Portrait            identity
```

This table is the **inverse** of simulator-server's own `90° →
LandscapeRight` convention, because it compensates for a rotation that
layer applies while decoding the scrcpy stream. That makes it a property
of a particular simulator-server build rather than of Android — which is
why a table test alone is not enough, and the delivered PNG's aspect is
checked against the rotation requested, falling back to an unrotated
capture if they disagree.

## Verified live

Branch tool-server against a real emulator:

- landscape, default capture → **2424×1080 upright**, MAD 2.19 vs ground
truth (6.54 vs its 180° twin)
- cropping `describe`'s frame for the toolbar out of that default
capture shows exactly the "Display & touch" toolbar — the two spaces
agree
- rotation 0 → 1080×2424, MAD 2.82, unchanged
- rotated **iPad** → still 2064×2752 sideways, i.e. iOS provably
untouched

Plus 27 new unit tests, including one that asserts the legacy path
*drops* the right-hand node before the fix and keeps it after. Full
suite 3125 passing; lint, prettier, `typecheck:tests` and the
tool-description gate all clean.

## Not in scope

- The iOS half of #609.
- `rotate` on Android silently no-ops when auto-rotate is enabled (it
returned `{orientation:"LandscapeLeft"}` while `dumpsys` still read
`cur=1080x2424`). Filing separately.
- `screen-recording` still records sideways — same root cause, different
pipeline. Worth noting a session's screenshots are now upright while its
video is not.
- Existing visual baselines captured on a rotated Android device will
stop matching, since the baseline key embeds capture dimensions and
those now swap. This fails loudly as a missing baseline rather than
silently comparing the wrong thing — and those baselines were sideways
captures of an upright UI.

---

> **#673 and #658 are both stacked on this branch** — each conflicts
with this one (`screenshot/index.ts` and `screenshot-diff/index.ts`
respectively) but not with each other, so both were rebased onto this
branch as siblings rather than chained. Merge this first; GitHub then
retargets both to `main`.

---------

Co-authored-by: Filip131311 <f.kaminski2000@gmail.com>
2026-08-21 17:34:59 +02:00

264 lines
11 KiB
TypeScript

import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
import { promises as fs } from "node:fs";
import * as os from "node:os";
import * as path from "node:path";
/**
* Issue #609: on a rotated Android device the screenshot came back
* portrait-framed with the content lying sideways, while `describe` frames and
* gesture coordinates were already upright. The capture was the one surface out
* of step.
*
* The rotation is queried per capture rather than tracked, so these tests are
* about reading it honestly and — crucially — about refusing to guess when it
* cannot be read. A wrong orientation yields a confidently wrong image, which is
* worse than the sideways one being fixed.
*/
const adbShell = vi.hoisted(() => vi.fn(async (_serial: string, _cmd: string) => ""));
vi.mock("../src/utils/adb", () => ({ adbShell }));
// The adb-server socket path is tried before the spawned client. Default it to
// "server not reachable" so the existing probe tests keep exercising the client.
const adbServerShell = vi.hoisted(() =>
vi.fn(async (_serial: string, _cmd: string): Promise<string> => {
throw new Error("ECONNREFUSED");
})
);
vi.mock("../src/utils/adb-server", () => ({ adbServerShell }));
import {
SURFACE_ROTATION_TO_NAME,
captureLooksUpright,
captureRotationForSurface,
readAndroidSurfaceRotation,
readPngSize,
} from "../src/utils/device-orientation";
/** Verbatim shape of the two probes, captured from a Pixel_9 (API 36). */
const DUMPSYS_DISPLAY = (r: number) => ` mCurrentOrientation=${r}\n`;
const DUMPSYS_WINDOW = (r: number) =>
`Display: mDisplayId=0\n init=1080x2424 420dpi\n mRotation=${r}\n mCurrentRotation=ROTATION_90\n`;
beforeEach(() => {
vi.clearAllMocks();
adbShell.mockResolvedValue("");
adbServerShell.mockRejectedValue(new Error("ECONNREFUSED"));
});
describe("the rotation is read from the cheapest source that answers", () => {
// Measured on a Pixel_9 emulator: helper peek ~1 ms, adb server socket ~8 ms,
// spawned adb client ~19 ms. All three report the same platform value, so
// the order only changes latency — these tests pin that order and that each
// tier's failure hands over to the next rather than becoming a guess.
it("prefers a peek that answers, and then touches no adb at all", async () => {
const peek = vi.fn(async () => 3);
expect(await readAndroidSurfaceRotation("emulator-5554", peek)).toBe(3);
expect(adbServerShell).not.toHaveBeenCalled();
expect(adbShell).not.toHaveBeenCalled();
});
it("falls through when the peek has no answer", async () => {
adbServerShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(1));
expect(await readAndroidSurfaceRotation("emulator-5554", async () => null)).toBe(1);
expect(adbServerShell).toHaveBeenCalledTimes(1);
});
it("falls through when the peek throws or answers nonsense", async () => {
adbServerShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(2));
expect(
await readAndroidSurfaceRotation("emulator-5554", async () => {
throw new Error("helper gone");
})
).toBe(2);
adbServerShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(2));
expect(await readAndroidSurfaceRotation("emulator-5554", async () => 7)).toBe(2);
});
it("uses the adb server socket before spawning a client", async () => {
adbServerShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(1));
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(1);
expect(adbServerShell).toHaveBeenCalledWith(
"emulator-5554",
expect.stringContaining("mCurrentOrientation"),
expect.anything()
);
expect(adbShell).not.toHaveBeenCalled();
});
it("tries the second probe over the socket before giving up on it", async () => {
adbServerShell.mockResolvedValueOnce("").mockResolvedValueOnce(DUMPSYS_WINDOW(3));
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(3);
expect(adbServerShell).toHaveBeenCalledTimes(2);
expect(adbShell).not.toHaveBeenCalled();
});
it("spawns the client when the server socket cannot be used", async () => {
// e.g. the adb server is not running: the client will start it.
adbShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(1));
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(1);
expect(adbServerShell).toHaveBeenCalledTimes(2);
expect(adbShell).toHaveBeenCalledTimes(1);
});
it("still refuses to guess when every source is silent", async () => {
expect(await readAndroidSurfaceRotation("emulator-5554", async () => undefined)).toBeNull();
});
});
describe("the surface-rotation → rotation-name table", () => {
// Pinned by measurement against `adb exec-out screencap`, which IS
// rotation-aware: each candidate name was scored as mean absolute difference
// against the screencap reference and against that reference rotated 180°, so
// an inverted mapping could not pass. The correct name won by ~3x in every row
// (e.g. rotation 1: LandscapeLeft 2.19 vs LandscapeRight 6.54).
//
// This table is the INVERSE of simulator-server's own 90°→LandscapeRight
// convention because we are compensating for a rotation it applies while
// decoding. That is exactly why this test cannot be the only guard — see
// `captureLooksUpright`, which checks the real capture instead.
it("maps each rotation to the name that yields an upright capture", () => {
expect(SURFACE_ROTATION_TO_NAME).toEqual({
0: "Portrait",
1: "LandscapeLeft",
2: "PortraitUpsideDown",
3: "LandscapeRight",
});
});
it("sends no rotation at all for an unrotated device", () => {
// Not "Portrait": keeping it undefined leaves the request body byte-identical
// to what it was before any of this existed, so the common case is provably
// unchanged.
expect(captureRotationForSurface(0)).toBeUndefined();
});
it("sends no rotation when the rotation could not be read", () => {
expect(captureRotationForSurface(null)).toBeUndefined();
});
it("requests a rotation for each side the device can be on", () => {
expect(captureRotationForSurface(1)).toBe("LandscapeLeft");
expect(captureRotationForSurface(2)).toBe("PortraitUpsideDown");
expect(captureRotationForSurface(3)).toBe("LandscapeRight");
});
});
describe("reading the rotation off the device", () => {
it("reads the primary probe", async () => {
adbShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(1));
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(1);
expect(adbShell).toHaveBeenCalledTimes(1);
});
it("reads every rotation the platform can report", async () => {
for (const r of [0, 1, 2, 3]) {
adbShell.mockResolvedValueOnce(DUMPSYS_DISPLAY(r));
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(r);
}
});
it("falls back to the second probe when the first says nothing", async () => {
// `dumpsys` output is not a stable API, so a single regex is a single point
// of failure across vendors and versions.
adbShell.mockResolvedValueOnce("").mockResolvedValueOnce(DUMPSYS_WINDOW(3));
expect(await readAndroidSurfaceRotation("emulator-5554")).toBe(3);
expect(adbShell).toHaveBeenCalledTimes(2);
});
it("keeps a grep miss from throwing", async () => {
// The probe appends `|| true` precisely so that grep exiting 1 — the normal
// outcome when the line is absent — is not an error.
expect(adbShell.mock.calls).toHaveLength(0);
await readAndroidSurfaceRotation("emulator-5554");
for (const [, cmd] of adbShell.mock.calls) expect(cmd).toContain("|| true");
});
it("returns null rather than guessing when both probes fail", async () => {
adbShell.mockRejectedValue(new Error("device offline"));
expect(await readAndroidSurfaceRotation("emulator-5554")).toBeNull();
});
it("returns null on unparseable output", async () => {
adbShell.mockResolvedValue("mCurrentOrientation=banana");
expect(await readAndroidSurfaceRotation("emulator-5554")).toBeNull();
});
it("ignores an out-of-range rotation", async () => {
// Never coerce something unrecognised into a rotation — a wrong one is worse
// than none.
adbShell.mockResolvedValue("mCurrentOrientation=7");
expect(await readAndroidSurfaceRotation("emulator-5554")).toBeNull();
});
});
describe("the aspect guard on the delivered capture", () => {
// The mapping compensates for what simulator-server does at decode time. If
// that changes, the compensation silently becomes a 180° error or a no-op and
// no test of our own constant could see it. This checks the actual image.
it("accepts a landscape image for a landscape rotation", () => {
expect(captureLooksUpright("LandscapeLeft", { width: 2424, height: 1080 })).toBe(true);
expect(captureLooksUpright("LandscapeRight", { width: 2424, height: 1080 })).toBe(true);
});
it("rejects a portrait image delivered for a landscape rotation", () => {
expect(captureLooksUpright("LandscapeLeft", { width: 1080, height: 2424 })).toBe(false);
});
it("accepts a portrait image for an upside-down portrait rotation", () => {
expect(captureLooksUpright("PortraitUpsideDown", { width: 1080, height: 2424 })).toBe(true);
});
it("does not treat an unreadable image as evidence of a problem", () => {
expect(captureLooksUpright("LandscapeLeft", null)).toBe(true);
});
it("passes a square image, which carries no aspect information", () => {
expect(captureLooksUpright("LandscapeLeft", { width: 512, height: 512 })).toBe(true);
});
});
describe("readPngSize", () => {
let dir: string;
beforeEach(async () => {
dir = await fs.mkdtemp(path.join(os.tmpdir(), "argent-png-size-"));
});
afterEach(async () => {
await fs.rm(dir, { recursive: true, force: true });
});
/** A minimal valid PNG header — signature, then IHDR length/type/w/h. */
function pngHeader(width: number, height: number): Buffer {
const buf = Buffer.alloc(24);
buf.writeUInt32BE(0x89504e47, 0);
buf.writeUInt32BE(0x0d0a1a0a, 4);
buf.writeUInt32BE(13, 8);
buf.write("IHDR", 12, "ascii");
buf.writeUInt32BE(width, 16);
buf.writeUInt32BE(height, 20);
return buf;
}
it("reads the dimensions out of the IHDR chunk", async () => {
const file = path.join(dir, "a.png");
await fs.writeFile(file, pngHeader(2424, 1080));
expect(await readPngSize(file)).toEqual({ width: 2424, height: 1080 });
});
it("returns null for a file that is not a PNG", async () => {
const file = path.join(dir, "b.png");
await fs.writeFile(file, Buffer.alloc(24));
expect(await readPngSize(file)).toBeNull();
});
it("returns null for a truncated file rather than throwing", async () => {
const file = path.join(dir, "c.png");
await fs.writeFile(file, Buffer.alloc(8));
expect(await readPngSize(file)).toBeNull();
});
it("returns null for a missing file rather than throwing", async () => {
expect(await readPngSize(path.join(dir, "nope.png"))).toBeNull();
});
});